Earlier this month I attended the Big Data Innovation Summit in Santa Clara that brought together over 600 data scientists, enterprise and data architects, business analysts, BI executives, customer experience professionals and C-level executives. It seemed like the perfect opportunity to poll the attendees to gauge their perceptions of real-time Big Data Analytics, and in particular, streaming Big Data Analytics. The three key questions we posed to the attendees were as follows:
I recently ran across an article in TechTarget that talks extensively about Hadoop’s limitations around real-time analytics applications. The article, authored by Ed Burns, emphasizes that while Hadoop was designed to process large sets of structured, unstructured and semi-structured data, it was built as a batch processing system which imposes significant limitations around real-time analysis. In the article Burns features excerpts from an interview with Forrester analyst Mike Gualtieri who mentions that there are plenty of vendors and end users asking, “Why can’t we execute real-time data analytics and ad hoc queries using Hadoop?” It’s a valid question, and Mike cites a key obstacle Hadoop faces with respect to real-time analytics.
Only an Operational Intelligence (OI) platform can unify continuous, real-time analytics on Big Data in a single, actionable view. With an OI platform, users can compare, combine, and monitor a wide variety of data sources, from Hadoop to complex events, in a simple, live dashboard. Additionally, users can easily perform predictive analysis by correlating past, present, and future events to forecast next-best actions.
Big Data is beyond the scope of traditional database software tools to capture, store, manage and analyze the enormous volume of operational data being generated from transactional interactions.
By: Brian Bohan, Vice President of Worldwide Sales Consulting
Operational Intelligence (OI) can be leveraged to overcome the “store just-in-case” mindset to Big Data. I see too many businesses capturing terabytes, and in some cases petabytes, of data “just-in-case” it might come in handy for some future initiative. In these situations, the data in question has not been mapped into any known business decision and action process, but instead stored just-in-case it might aid in some decision in the future. Why is this the case? I believe it is because dealing with the Volume and Variety of data is no longer the long pole in the tent, thanks to the advances in Big Data frameworks, but mainstreaming this data into core business transactions, processes and actions is far more difficult. The lack of qualified data analysts is well-known. But there is an emerging belief that perhaps it is the amount of data itself that is the issue; or at least the focus on collecting and storing this data without always having a clear idea on how best to put it to use while it still matters.
In addition to “continuous analytics” supported through a native complex-event processing (CEP) engine, Operational Intelligence (OI) also supports online and offline analytics through third-party tools, such as a Mondrian OLAP Server:
On Wednesday, June 29, Hortonworks (named after the Dr. Seuss elephant) was created as an independent, privately-held, VC-funded company to lead the Hadoop community and market the open-source product for the future. Its parent, Yahoo!, is now one of its customers.