The Problem and the Solution to 21st Century Organizational Innovation. Trever Pearson PA 740 Professor Hyde 12.10.12 DEFINITION.  Big Data is a phenomenon defined by the rapid acceleration of the expanding volume of high velocity, complex, and diverse types of data which require advanced technologies and methods to enable their collection, storage, dissemination, management, and analysis. TechAmerica Foundation (2012) WHY IS IT PROBLEMATIC?     Increased velocity of available data is faster than most organizations can keep pace with. Data synthesis requires advanced technologies and appropriate staff and expertise on an ongoing basis. Implementation requires structural and organizational culture change. … And failure to respond will leave a lagging organization seriously behind. Data Storage in Exabytes Origins & Trends. 350 300 250 Data Storage. 200 150 Global Data Storage has increased from 0 to over 300 exabytes between1986 and 2007.1 100 50 0 1986 1993 2000 2007 Detail: % Exabytes 100% The type of global data stored has changed from 99% Analog in 1986 to 96% Digital in 2007  90% 80% 70% 60% Digital 50% Analog 40% 30% 20% 10% 0% 1986 1 1993 2000 2007 (5 Exabytes = 10^18 gigs: Enough to contain every word ever spoken by all humans on Earth. MGI (2011) Computation Capacity (Million Instructions per Second Origins & Trends. 350 (cont…) 300 Computational Capacity. 250  Computation capacity has grown from 0 to over 300 exabytes of traffic from 1986 to 2007. 150 200 100 50 0 1986  Information-producing devices such as, mobile phones, tablets, sensors etc… have doubled since 100% 2000. Coupled with 90% personal computing, traffic 80% 70% in these areas increased 60% from under 40 to nearly 50% 90% of all data created form 40% 1986 to 2007. 30% 1993 2000 Detail: % Million Instructions per Second Personal Computers Video Game Consoles Mobile Phones/PDA Servers and Minframes Supercomputers Pocket Calculators 20% 10% MGI (2011); Economist (2012) 2007 0% 1986 1993 2000 2007  The storage required for all of this data doubled between 1999 and 2002, a 25% compound annual growth rate.  1.8 zetabytes of data (the amount of 200 billion 2-hour HD movies) were created globally in 2011; an amount projected to double every year.  800 exabytes were created in 2009, projected to increase 44 times by 2020. It’s just like the universe, increasingly and exponentially expanding. MGI (2011) DATA TYPES.   15% Structured (database or spreadsheet data) 85% Unstructured (email, video, blogs, call center conversations, Facebook posts, Tweets, etc…) Economist (2012) DATA SOURCES.     Customer transactions with personal information and consumer behavior like Visa, Amazon, etc…) Multimedia content such as High-Res health procedure videos, YouTube, etc… Social Media such as Facebook and Twitter Sensors and devices used in industries such as, retail, healthcare & automotive  The effective response to Big Data is crucial for leading organizations to outperform their peers.  Companies are projected to increase operating margins by more than 60% with the effective response to BIG DATA.  Management decision making will be built upon evidence and information.  Data driven decisions are just plain better decisions. “You don’t manage what you don’t measure”. McAffee & Brynjolfsson (2012) HOW WILL BIG DATA HELP?          By… Replacing human decisionmaking with automated formulas where appropriate Reducing inefficiencies Creating transparency Discovering variability Reducing security threats and crime Increasing ability to predict mission outcomes Reducing or eliminating waste …just being innovative. MGI (2011) WHO WILL BIG DATA HELP? The five sectors to gain the most from the use of Big Data:  Health Care  Public Sector Administration  Manufacturing  Retail  Business/Organization using Personal Local data  WHO IS AFFECTED? HOW ARE THEY AFFECTED?  The Public   Policy Makers   Contractors   Employees  Government transparency, Bureaucratic efficiency… …Privacy Informed decision-making, evidence based legislation Monitoring contract deliverables, reporting Facilitation in workplace tasks, enhanced communication, etc… Increased transparency over organizational activity OPPORTUNITIES.     Data-driven organizations perform better on measures of financial and operational results than those who do not Data facilitate efficient processes, saving time and money Data lead to innovation Data will ultimately lead to funding. McAffee & Brynjolfsson (2012) CHALLENGES.      Data-driven decision making and collection processes require organizational cultural change Strong Leadership is necessary to set clear goals and to ask the right questions Skillful and talented Data/IT Specialists must be on staff. Lack of statistical and technical skills in the labor force Potential cost of implementation Step 1. Source Data: Speed, Type and Amount. What kind and how much data are we working with? Assessing how hard it is to access  Determining how it needs to be transformed  Identifying the technologies to facilitate the process  Step 2. Data Preparation: Cleansing and Verification. What do the data need for operational requirements?  Define methods required for data prep such as:  Standardization, verification, filtering, etc… Step 3. Data Transformation. What is required to leverage the data? Unstructured data may be broken down and presented in a structured format  Data sources can be aggregated to determine not-so-obvious relationships between data types  TechAmerica Foundation (2012) Step 4. Business Intelligence/Decision Support. Tools, methods, techniques to leverage data    Data Mining Visualization/Simulations Keyword Searches & Syntax Analysis Step 5. Analysts/Visualization. How should the data be used?   Present data visually so it can be explored Use data as is to support/enhance/improve existing organizational processes TechAmerica Foundation (2012) TechAmerica Foundation (2012)  Staffing  Data Analysts/IT Specialists, etc…  Infrastructure  Funding  Data storage, Software, Hardware, Connectivity, etc…  Technological investment  Performance objectives related to desired mission outcomes  Standards/Metrics to compare operational efficacy with mission outcomes  A Data-Driven organizational culture  Data Prioritization as driving force of organizational direction and the culture to support it.  Openness to organizational change  Data prioritization will require change! 1. Identify and Define mission objectives that need Big Data solutions 2. Assess current organizational capability, data sources, and technical requirements 3. Identify success criteria, implementation timeline, potential subsequent phases, required staffing levels, and “entry point” 1. Streams as entry point for high-velocity data needs 2. Un-bounded database/warehouse infrastructure for high-volume data needs 3. “Hadoop”1 or similar type technologies for high-variety data needs 4. Execute the plan as required 5. Review on an ongoing basis 1 The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. TechAmerica Foundation (2012), Apache Hadoop     Assessment of mission outcome achievement with improvement measures including increased savings, improved efficiency, etc… Identification of gaps in the links of the process chain, if any (see slide 13) Assessment of decisions being made (are the right data available to facilitate the process?) What are your Data/IT staff telling you? • Expand and invest in the talent pool by creating a formal track for IT/Data managers with training and certification in BIG DATA Analysis and technologies. •Establish and broaden coalitions between industry academic and associations to develop professional standards and shared best practices for the field. •Expand “college-to-government service” internship programs focused on technical aspects of BIG DATA. •Strengthen and expand Office of Science and Technology Policy to facilitate further research into new techniques and their applications to important problems across program and policy sectors. •Align incentives to promote data sharing for the common good. •Provide further guidance with industry and stakeholders on privacy and data protection practices. •Develop intellectual property policies to promote innovation. •Support necessary underlying IT/Communications infrastructure MGI (2012), TechAmerica Foundation (2012) Political resistance to BIG DATA may be minimal, resulting from a history of activity including:  Government (Library of Congress, Bureau of Information Resource Management)  Finance (Banks, Credit Card companies)  Internet search engines (Google) …HOWEVER… Bottom-up Resistance is likely  The Public  Privacy concerns and the notion of “Big Brother”  Employees  Data errors and the documentation of mistakes  Contractors  Less room for error, increased competition and accountability FAQs. 1. How do you know if you have a big BIG DATA problem? 2. How do you obtain insight from your data? 3. Which technology is right for my organization? 4. How long should it take to implement? 5. What skills/expertise are required on staff? 6. What about Privacy? City A.M. (2012) 1. When available data is beyond your ability to manage or when tapping into the insight it provides is problematic. 2. Start by placing mission objectives at the heart of every decision. While this might require change, even the more traditional change management practices may be of service. Let your Data staff tell you what they need. 3. It depends on your mission objectives and the type/amount/speed of data you need to inform your decisions. To start, build upon what you already have. 4. Start with small, manageable steps and allow for constant evaluation and revision. If the first phase takes longer than 6 months, you’re too slow. 5. Data Analysis and Communication, Technical skills, Database Management, and good ol’ fashioned Critical Thinking. 6. As with any data collection/sharing advancement, policies must be adjusted to address issues of privacy as they affect the organization within the context of the standards set in place (statutory or otherwise). Congress is working on it. As far as the public is concerned: Welcome to the 21st Century. Identifying what big data means to you. (2012, Feb 24). City A.M. London. McAffee, A., Brynjolfsson, E. (2012). Big Data: The Management Revolution. Harvard Business Review. Pp. 59-68. Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C. Byers, A.H. (2011). Big Data: The Next Frontier for Innovation, Competition and Productivity. McKinsey Global Institute (MGI). TechAmerica Foundation (2012). Demystifying Big Data: A Practical Guide to Transforming the Business of Government. Washington, D.C. Geography matters as much as ever despite digital revolution, says Patrick Lane. The Economist. (2012). Trever Pearson is a third-year Master’s student in Public Administration at San Francisco State University. With an emphasis in Policy Analysis and Finance, his interests lie mostly in evidence-based improvement in the policy arena in sectors such as health, education, finance, and income security. Trever comes from a solid background in health care policy implementation and evaluation in the San Francisco public health network. He is currently working as a Data Analyst for Curry Senior Center, a community clinic serving the elderly in San Francisco’s Tenderloin neighborhood. His achievements there include the development of agencywide data collection and reporting processes for service quality improvements and contract reporting. With coursework in Urban Administration, Financial Management and Applied Statistics, Trever aspires to use BIG DATA and research solutions for the improvement of state and federal policies and agency operations.