Top Digital Transformation and DevOps Influencer

Jason Bloomberg

Subscribe to Jason Bloomberg: eMailAlertsEmail Alerts
Get Jason Bloomberg via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Agile Digital Transformation, cloudbigdataajaybudhraja

Blog Feed Post

Reimagining the Data Pipeline Paradigm as a Continuous Data Insights Platform

Since the dawn of the modern computer, data has been a core architectural component. Every system needed an interface to collect and present data, a way to process that data and some place to store it.

As simple as that paradigm may be – or perhaps because of its simplicity – it has persisted throughout the evolution of digital systems from the mainframe to the cloud. We have now, however, entered the era of big data and a time in which competitive advantage goes to those organizations who can most rapidly leverage their data and find insight from it. And that means that the underlying model of how organizations store and manage data must change.

The Data Pipeline Evolution

In the beginning, it was simple: each system stored data in one place for its own use. As systems grew in complexity, however, multiple systems needed that same data. IT organizations, therefore, had to find ways to move data and share it between systems – what is sometimes called a data pipeline.

These original data pipelines were simple and based on the concept of extracting the data from the original system, transforming the data into a format usable by the next system and loading the data into the new system – or extract, transform, and load (ETL).

ETL processes were slow, cumbersome and processed in batch mode, typically at the end of some pre-set period of time. As these data pipelines grew longer and more complex, however, and as the need for the data became more urgent, the traditional ETL approach broke down. This form of data flow is simply too unwieldly for many of today’s needs.

Moving Beyond Storage: iguazio and the Modern Data Pipeline

Organizations built early data pipelines based on the idea of moving data between storage locations where various systems could access that data. The primary purpose of the data was for use in transactional systems of record.

As data, however, has become the chief enabler of both competitive advantage and the customer experience, organizations must now analyze and act upon that data in real-time. This need was the driver behind the creation of iguazio.

iguazio’s goal is to simplify the data pipeline and help organizations move from processing data in batch to being able to uncover real-time insights. It does this by re-envisioning the entire data pipeline paradigm, but it is important to recognize that this is not merely a new approach to storage or data management.

iguazio’s solution is a continuous data insights platform that uses commodity hardware to enable organizations to rapidly ingest, process, and consume data. Architecturally, the company builds a unified real-time database which uses a combination of distributed, virtual, non-volatile memory and dense tiered storage to deliver a single data platform that can dynamically manage both streaming and persistent data structures.

This approach enables organizations to use these continuous insights to guide real-time actions in both transactional and analytical use cases.

Creating Continuous Analytics with a Unified Data Model

The secret to iguazio’s approach is what it calls its Unified Data Model. This proprietary model enables the platform to store and index data once, but then present or query that data via standard APIs as a stream, as file objects, as database records or in almost any way necessary to allow subscribing systems to consume it.

This unified model is protected with ‘fine-grained’ content-aware security. This includes full auditing, end to end authentication and authorization, which enables secure data sharing across business departments or external users. At the same time, this intricate level of control satisfies the most rigorous data compliance needs and helps organizations break through organizational barriers.

Instead of forcing organizations to use a new set of APIs, the platform exposes data through Amazon-compatible streaming and database APIs and also integrates with Hadoop, Spark, and Kubernetes. In this way, applications can consume and manage iguazio’s platform services with self-service and multi-tenant portals deployed either on-premises or in the cloud.

The result is an ability to ingest data once – whether it is streaming or persistent – and then dynamically enrich, analyze, use machine learning, and serve that data, or its derivatives, in real-time to identify and act upon the insights gleaned from it.

The company calls this capability, continuous data insights. Much more than merely a new approach to storage or data management, iguazio enables organizations to realize value from their data more quickly and to get to insights from that data faster to drive competitive advantage for the organization.

The Intellyx Take

If there is one hallmark of the digital era, it is this: everything is changing.

The traditional conceptualizations of storage and data pipelines are no exception. It may be tempting to look at iguazio’s approach as simply a new way of storing data, but that would be a mistake.

The fundamental essence of their approach is to abstract data from its underlying storage and to create a unified data model that enables organizations to stop worrying about where or how they are storing their data and instead focus only on the insights and the competitive value they can harvest from it.

Copyright © Intellyx LLC. iguazio is an Intellyx client. Intellyx retains full editorial control over the content of this paper.

Read the original blog entry...

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.