We'll respond shortly.
This blog originally appeared at http://blog.gopivotal.com/p-o-v/making-big-data-real-time-and-ubiquitous-is-key-to-enterprise-adoption
Last week, Pivotal’s COO Bill Cook spent some time talking with the Wall Street Journal’s Clint Boulton on why big data is not producing value for most businesses. This conversation was spurred by a recent report from Bain & Co. that cited that only 16 out of 400 businesses are seeing business value from big data today. With those 16 companies representing just 4% of the total surveyed, drawing doubt to predictions that big data is the biggest growth area in IT today.
While the number of companies exacting benefits today may be small, advancements in big data technologies—from speed to scale to cost—are about to swing those numbers higher and live up to the forecasts that big data technology and services market is growing at an incredible 31.7%—or about 7x faster than all of IT.
In the interview, Cook gives us the first clue on what has to change, zeroing in on how companies need to start thinking of big data differently to gain strategic advantages:
“By definition, most enterprises are not there,” Mr. Cook said in an interview at CIO Journal’s offices Wednesday. The traditional software and processes most businesses operate today are not built to accrue value from analytics, he explained. Instead IT staffs must extract data from business applications and analyze it in a separate analytics application. This is a time-consuming process in which data sets become outdated as new data is created.
Mr. Cook said companies must organize all of their business data, figure out what information to retain based on the business challenges they wish to solve, and build software that marries data and analytics. Essentially, companies must make analytics a feature — not a separate function — of the business software, Mr. Cook said. And it should be self-service, meaning business analysts can access the analytics data without asking IT to build them a report. Then businesses can begin thinking about how to use the data to gain competitive advantage. “Analytics is the algorithm by how you interact with your customers,” he said.
Cook suggests companies need to start building analytics into the DNA of business processes, and he is absolutely right.
Still for many companies, big data projects are too complex or too costly to justify. I submit this is just because companies first need to think differently about big data analytics to justify new IT projects.
To start, let’s look closer at Cook’s comment about how data sets become outdated quickly. If data is allowed to get stale, or you need to wait for another group to run your report, the entire process of using the data becomes halted and loses the window for timely action. This is what happened to your business already. Data is not incredibly accessible or relevant at the time, so it hardly gets used.
By placing big data within reach in real-time for the entire organization, your business processes will have amazing possibilities to change. To illustrate, think about old school product development processes. Back when I was doing product management work at Siebel Systems, we went through formal Marketing Requirement Document (MRD) processes once every 9-12 months. This process lasted months, and was planned for well in advance. We would outline our marketing research requirements and take months to get the analysis compiled, many times hired out to market research firms, so we could rationalize customer requirements against engineering budgets. How users used features required focus groups and extensive UE testing. The process was long, and required a lot of work to do properly.
Today, 3 months of planning what you are going to do product-wise will sink you. It defies all the tenets of agile development, where processes demand you deliver results in less than half that time, not a plan to have results.
To keep progress in today’s market, we simply need to compress decision time and focus on action. Big data analytics works to unlock insights hidden previously by data too costly to process. With big data on your side, organizations are able to identify correlations and causal relationships, classify and predict events, identify patterns and anomalies, and infer probabilities, interest, and sentiment in order to establish priorities more quickly. Having data available in a reasonable timeframe compresses the time and improves the accuracy of market research, promoting an investigative approach to data where follow on questions can be answered just as quickly as the initial questions, and decisions can be made faster. Imagine these scenarios:
There are more examples, but a common theme here is that each of these scenarios requires big data to be built into real-time processes. It can not be a separate silo in the business, and needs to be accessible to all.
Previously, such as the days when Facebook built out their application, only serious engineering outfits had the skills or the funds to work on this. Most data was analyzed by ETL or batch processes. Near real-time was usually the best anyone could do.
Today, we are in a different world. Software has evolved to deal with the scale of big data, and simultaneously the cloud era has paved a way to commoditize compute power, dramatically reducing costs. Large software companies (like Pivotal) have emerged that are further packaging data solutions to make it easier for companies to deploy big data solutions, and re-architect their data infrastructures to be real-time.
This has helped engineering and data science teams to deploy big data solutions easier, but the next hurdle is getting everyone in the enterprise to use it as a strategic part of their real-time processes. Part of this means harnessing this data to be used within automation programs, such as the customer churn example. Another part is not so scripted though, and needs to be a flexible enough sandbox to find new data insights that can be actioned on immediately or nominated for automation projects.
To do this, forward thinking organizations like the New York Stock Exchange have created a portal that pools data sources, provides access control and security logging, and opens up ad-hoc data investigation for all users. For NYSE, this means that they have the ability to do market surveillance to an open-ended number of data investigations. Using their platform, now available as Data Dispatch, NYSE can identify cyber threats, research customer spending patterns, and service regulatory needs without making IT or a separate data warehousing team the bottleneck. Data consumers are empowered to do their own data investigation and take action within their own domains.
In short, while today big data may only be deployed as part of the process in 4% of Bain & Co.’s 400 enterprise user study, this is because big data has only recently become within reach of the enterprise. And now that its real-time and affordable to open up to the entire organization, we are very much on the precipice of seeing big data adoption accelerate.