From the Founder and CEO, Cloudscale Inc.

Bill McColl

Subscribe to Bill McColl: eMailAlertsEmail Alerts
Get Bill McColl: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: Cloud Computing, Cloudscale, Cloud Computing Newswire, Cloudcel, Big Data on Ulitzer, OpenStack Journal

Cloud Computing: Article

The Consumerization of Big Data Analytics

Dataclouds

With Dropbox, Jive, Yammer, Chatter and a number of other new services, the modern enterprise is rapidly becoming "consumerized". And it's not just business, the same is happening in major web companies, on Wall Street, in government agencies, and in science labs.  Thirty years of bad "enterprise software" experiences is making this transition happen much more quickly than anyone would have expected. The shift to cloud computing is also accelerating the trend, as is the goal of developing a much more "social" approach to business.

The other major change that's going on today throughout business, web, finance, government and science, is that every organisation now realizes that it needs to be data-driven. Big data and analytics have the potential to unleash creativity and innovation everywhere - generating new ideas and new insights continuously. To achieve and maintain competitive advantage today, it is becoming essential for everyone in an organisation to have instant access to all the information they need, at all times.

Making big data analytics available to everyone in an organisation means that it has to be much simpler than traditional data analytics solutions such as databases, data warehouses and Hadoop clusters. It needs to be consumerized! We need a new generation of data analytics solutions that are not just powerful and scalable, but also very easy-to-use.

Dataclouds
At Cloudscale we've been working on the hard problem of delivering this extreme simplicity, extreme power and extreme scale. Our "datacloud" solution combines a number of advanced technologies in a unique way to achieve these challenging goals. The patented in-memory architecture is massively parallel, cloud-based, and fault tolerant. It runs on standard commodity hardware, either in the cloud (e.g. Amazon) or as an in-house (OpenStack) appliance.

Cloudscale lets anyone easily store, share, explore and analyze the exponentially growing volumes of data in their work and in their life. It's like a "Dropbox for Big Data Analytics". The Cloudscale data store and app store allow users to easily create, share and collaborate on all kinds of data and apps.

It’s designed for everyone - business users, data scientists, app developers, individuals - anyone, or any organization, that needs a simpler way to handle today’s explosively growing data volumes. And it’s viral - sharing data and apps creates powerful network effects within organizations, unleashing data-driven creativity and innovation everywhere.

With this new technology, anyone can now become a "big data rocket scientist". Through simple, easy-to-use interfaces, users can:

  • Work with all types of data - structured and unstructured - from any source
  • Work with live data streams and massive stored data sets
  • Quickly discover important patterns, correlations, statistics, trends, predictions,...
  • Quickly develop, deploy and scale big data apps - mapreduce, realtime analytics, statistics, pattern matching, machine learning, graph algorithms, time series,...
  • Evaluate millions of scenarios and potential opportunities and threats every second
  • Go from data to decision to action instantly

It's super-fast and super-scalable! For example, Cloudscale can be used to analyze a live stream in realtime at more than 150MB/sec on just three 8-core AWS cluster instances. That corresponds to processing a SINGLE STREAM in parallel at a rate of TWO MILLION ROWS PER SECOND, or well over ONE TRILLION EVENTS per week. To give some idea of how fast this is, the nationwide call log systems of even the biggest US telcos only generate about 50,000 rows/sec, even at peak. For processing even more data, the solution scales linearly.

The performance of the Cloudscale datacloud is more than 125x faster than Yahoo's S4 (Realtime MapReduce) system, on the same hardware - about the difference in speed between walking from San Francisco to New York (4mph) versus taking a plane (500mph).

These are just the first steps in the consumerization of the $30Billion+ analytics industry. As powerful analytics gets democratised in this way, we can expect that it will spread virally into every corner of every organisation.

More Stories By Bill McColl

Bill McColl left Oxford University to found Cloudscale. At Oxford he was Professor of Computer Science, Head of the Parallel Computing Research Center, and Chairman of the Computer Science Faculty. Along with Les Valiant of Harvard, he developed the BSP approach to parallel programming. He has led research, product, and business teams, in a number of areas: massively parallel algorithms and architectures, parallel programming languages and tools, datacenter virtualization, realtime stream processing, big data analytics, and cloud computing. He lives in Palo Alto, CA.