Top Digital Transformation and DevOps Influencer

Jason Bloomberg

Subscribe to Jason Bloomberg: eMailAlertsEmail Alerts
Get Jason Bloomberg via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Cloud Computing, Change Leadership Journal, SOA Best Practices Digest, Microservices Journal

Agile Development: Article

Big Data-Driven Innovation: Disruption vs. Optimization

Optimize what you can optimize and disrupt what you must disrupt

Big Data-Driven Innovation: Disruption vs. Optimization

The recipe for successful innovation: begin with a good measure of disruption. Add a heaping helping of talent, and don’t forget to mix in plenty of creativity. Finally, a pinch of intuition. Stir and bake.

Recipe for innovation? Perhaps. But successful innovation? That’s another story. After all, whether an innovation will actually succeed – that is, meet or exceed its business goals – seems to require some unknown, missing ingredient to the mix. The recipe above may include necessary elements, but taken together, they are still not sufficient to guarantee success.

How’s a Digital Transformation professional to determine that secret element, that je ne sais quoi that leads to successful innovations, every time – or at least, most of the time? Simple. Let’s get out our Big Data analytics tools and crunch some data. Surely, boiling down Big Data to a syrup of innovation to that rare sprinkling of insight will provide that missing ingredient. Magnifique!

Driving Agility with Big Data

At Intellyx we consider innovativeness to be the most important, strategic aspect of business agility. Also along for the ride: responsiveness and resilience. Put these core business drivers together and you have a multifaceted picture of how and why businesses deal with change.

Analyzing the information at our disposal, of course, can help any organization deal better with change through a straightforward optimization process. Gather data on anything you’re doing, crunch the numbers, and make recommendations on what to adjust to make the process better. After all, such analysis has been the primary purpose of business information since early managers crawled out from their cave and held the first punch card up to the wan light of morning.

Data analytics in support of human decision making, however, has one flaw – the human. This weak link in the data-driven agility chain becomes apparent as we move to Big Data: as the data grow so too do the results of the analyses, and yet people have a limited attention span and with it, the ability to process information. It doesn’t matter how wonderful the reports your newfangled Big Data tool generate if no one has the time or predilection to read them – or even worse, understand them!

The answer to each human’s exasperating lack of attention, of course, is to say goodbye to the weakest link, and establish fully automated feedback loops. Here’s an example: any business with equipment to maintain, from airlines to factories to data centers, knows that various widgets will eventually fail. Gather enough data to determine each part’s mean time between failures (MBTF), a statistical measure that predicts when a particular doodad will give up the ghost. Consider this statistical analysis Level #1 on our road to Big Data nirvana.

Based upon the MBTF, then, we schedule a replacement before each whatzit is expected to break. However, no one really knows when it’s gonna go, so we now have to balance the prospect of replacing it too early (thus wasting money on a part that still has some useful life to it), and waiting too long, thus increasing the odds it’ll break in production – an even more expensive headache.

The traditional approach to this kind of problem is to generate a bunch of reports, feed the data from those reports into various mathematical formulas, and adjust our maintenance schedule accordingly. Such mathematical machinations would constitute analysis at Level #2.

We’ll get much better results, however, if we automate this feedback loop, thus accelerating our ability to respond to changes in the environment, as well as taking error-prone, lazy humans out of the equation. Now we’ve reached Level #3: feeding back our analysis to improve results automatically over time. In other words, we’re now more responsive to change in the business environment – an essential aspect to business agility.

As you might expect, such automated feedback has been a staple in the equipment maintenance world for a while now. However, there are many other business areas that have yet to take a page out of the broken whatzit playbook. Does your organization, say, optimize its salaries based upon business outcomes – in real-time? Do your security policies balance the cost of security with the corresponding risk of loss, again in real-time? In general, how many of the processes in your organization are subject to continual optimization based upon automated, data-driven improvements?

Driving Innovation with Big Data

Such business-centric automated feedback is perhaps one of the largest potential Big Data benefits in the enterprise today – although adoption of this approach is spotty at best. Let’s say, however, that you’re one of the progressive early adopters in the automated Big Data-driven feedback world. Does this streamlined ability to optimize all aspects of your business improve your ability to innovate?

Perhaps, but not in a straightforward fashion. As I explained in an earlier Cortex newsletter, optimization activities in the absence of disruption will lead to local optima, but only disruption-driven innovation will lead to the most strategic outcomes. Does that mean innovation is always a crap shoot? Or is there some way we can improve our odds of successful innovation, perhaps by leveraging feedback-driven Big Data optimization techniques?

Time for a hypothetical example. Let’s say a leading consumer electronics manufacturer, perhaps named after a fruit, decides to create their first ever wrist-mounted Internet-of-Things device: the Banana Watch. Such devices are a new market for the Banana Company, and in fact, their device is so innovative that it’s essentially creating an entirely new market segment on its own (much as the Banana Pad did a few years ago).

So, is Banana Inc. taking a complete flyer on the Banana Watch? Or do they have data to suggest the Watch will be successful? And furthermore, how do those data help them to optimize their innovation process itself?

One the one hand, entering a new market, especially when you’re creating that market as you go, is always risky. But on the other hand, they do have extensive data about customer buying behavior, price points, design metrics, related markets, and a plethora of other information that feeds an internal analytics algorithm that is indubitably second-to-none.

The reason Banana is so successful in the broader consumer electronics market – and in fact, has been so successful over and over again – is they optimize what they can optimize and they disrupt what they must disrupt. They never make the mistake of trying to optimize what they should disrupt, as that approach would stifle innovation. But they also avoid the mistake of disrupting what they should optimize.

The Intellyx Take

The lack of useful optimization, either because of lack of data, insufficient analytics, or a simple dearth of management will is actually a risk factor for innovation, just as excessive or misapplied optimization is. In other words, it’s a gigantic mistake to believe that you can innovate your way out of a lack of insight – insight into your market, your customers, or your business. Sure, people try it all the time – but most of the time they fall flat on their face. The few people that manage to succeed at such innovation-in-the-dark are simply the lucky ones.

For companies that understand how to balance optimization and disruption, luck is still a factor in how successful any particular innovation will be – just as luck is a factor in whether a particular widget will outlive its MTBF. Taking Big Data-driven optimization to the highest level is how companies make the most of their breakage-prone widgets, just as it is the secret to any organization who wishes to establish innovation as a core competency. Know when to optimize, and know when to disrupt – and above all, know how to tell the difference.

Thanks to Jim Scott at Intellyx subscriber MapR and Raj Dalal at BigInsights for helpful ideas for this article. Please click here to take the BigInsights 2014 Big Data Study survey and enter to win a Samsung Galaxy S5. Image credit: Jason Bloomberg.


More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.