While driving Big Data Delivery my team and I have gained invaluable insights that may be very familiar to the ones that you have experienced in your own organizations. Just close your eyes and imagine the world where Google built an online Data warehousing company, where it brought data from various smaller set of selected sources and spent most of its focus on cleansing, integrating and standardizing data based on industry standards and exposing them through a standard set of interfaces instead of focusing most of its effort in creating search algorithms that answer user questions. If that were the case, Google would not have become the company that has changed the world. Google is an ANSWER company, not a data movement, data integration, cleansing, standardization or certification company. Google is exponentially more valuable, not for the quality and consistency of the information it provides, but the hugely rich and highly contextual answers it provides as and when the answer is needed.

Here are three simple thoughts that may make you think.

  • Philosophical shift vs. Technology shift – Has your team philosophically understood the Big Data paradigm? In the Big Data paradigm, we focus less on defining business rules and automating them. Instead, we focus on getting our hands on a vast amount of richer data (facts and opinions), creating models that study this extremely large and rich set of dataset to identify patterns that resulted in a positive or negative outcome and then apply the same patterns on new data to predict outcomes. We build systems, that for the first time take advantage of the feedback loop and learn and evolve at an extremely fast pace. We become more of a coach than the player. We recruit the right data, coach systems (predictive models) to play with the data and win the game for us and continue to do this over. Make sure that your team and your leadership understands this philosophical paradigm shift. This is the hardest change you will have to make, much harder than teaching your technical team how to use Big Data tools.
  • Airline vs. the Space Program – Imagine when the Space program began. Now imagine, as the Space Program struggled to take off the ground it was forced to compete with airline companies. To survive the Space Program tries to act and behave like an airline company that can also move people around, from Chicago to Florida, from Beijing to Moscow and so on. Users complain that the space ships are not as comfortable and there are no stewardesses who serve them drinks and peanuts. The Space program complies by spending more of its focus on catering to these needs and making their ships look and feel more like the regular planes. The Space Program gets so deeply involved in taking care of day to day air travel challenges that it gets off the vision of Space exploration. Does your Big Data Program vision match that of a better airline company or that of a Space program? Aggressively define and refine the Big Data vision, crisply differentiate the vision of space travel (Big Data) from air travel (pre-Big Data technologies) and educate the organization on its dramatic value proposition.
  • NASA or SpaceX Program – There was a reason why the NASA program became outdated. It needed billions of dollars and years to deliver results. It worked well when our setup was dependent on federally funded budgets. In today’s world, that model is neither financially sustainable nor aligned with customer’s expectation around Speed to value. Are you able to explore and find micro use-cases that can be delivered within a very short amount of time while you continue to look for the larger investment that will place the organization on a firm Big Data path?

I am sure there are innumerable thoughts and insights that you have gained that are beyond mastering the technical intricacies and that will help us make Big Data pervasive, part of our day to day lives outside the box that is labeled SPECIAL, where every business or personal decision is for the first time FACT based and evolves seamlessly as new facts emerge and new patterns are identified.


This guest article is written by Sufian Abu


By Techwacky

Editor-in-Chef of TechWacky.com