Mind the gap

Data is good. And the more there is, wider the possibilities of analytics to be accurate, effective and useful. Sounds about right? Nope.

Often, organisations pride themselves in gathering as much data as they possibility can, gathering it in massive lakes of data (let’s stay far away from the term ‘Big Data’), and churn the lake with analytics expecting miraculous insights to pop-up. The problem with this approach is that organisations end up working hard for data analytics, when in fact it should be the other way around (data analytics should work hard for you). The largest contributing factor to this is the Crap Gap.

What is the ‘Crap Gap’?

The Crap Gap is the difference between the total available data and the actual usable data.

Consider this scenario. One of the most popular CRM platforms out there is being used by a multinational consultancy firm with 600+ Partners and Principles across the world. Everyone is using the same system, which costs hundreds of thousands of Euros to implement. But, everyone is using it in their own way. The Head of Marketing of our multinational consultancy firm wants to run campaigns across different industry verticals and so aggregates all the contacts in his CRM. However, it is discovered that most of the data is either incomplete, out-dated or simply inaccurate. The Crap Gap was too wide for the campaign idea to come to fruition.

Organisations are finding it increasingly easier to acquire data and hold it; And in some cases, it’s a matter of pride. But managing and maintaining growing volumes and variety of data comes at a cost and an increased risk of the Crap Gap. There is also the associated possibility of ghost data or orphaned data where the data is entirely valid but no one knows why it is collected or who owns it.

Mind the Gap

So then, what are the few basic steps to deal with the Crap Gap?

Start with the (business) objective in mind
Instead of trying to collect as much data as possible, start with the (business) problem you are trying to solve. Clarity and focus is essential if you want to avoid a large Crap Gap and drown in the data lake you end up with. Examples of good starting points include, “How do I reduce my customer churn?” or “How do I reduce waste and scrap from this production process?”

Working backwards from the objective is useful because it allows you to:

  • Discover where the data-gaps exist (build a data-map) and identify ways to bridge these gaps.
  • Identify the appropriate technical platform(s) you might need for a full implementation, along with costs.
  • Test the hypothesis (against defined KPIs) in a sandbox without significantly reducing your bank account.
  • Identify key-stakeholders and any process changes that may be needed in your organisation.
    Build a success-based business case for budget allocation.

Note that the sandboxed Proof of Concept (PoC) and the larger scale implementation can be supported by totally different technologies/platforms which provides for a significant cost advantage for the PoC. Once the Crap Gap is addressed at the PoC stage, production-level implementation of the solution is a matter of reproducing the PoC in the appropriate technologies supporting production scale. It is like test-driving a car before you decide to commit to the entire cost-of-ownership and buy it.

Think Small to Think Big; Fail Fast to Discover More

Don’t start by trawling the ocean. Take small bites (bytes?) of data. Break it down into the simplest components. This keeps things focussed on the objective and makes understanding the data easier. Relationships, influences and causality are easier to note. Additionally, processes of cleansing and harmonising the data are documentable. Bringing it all together to paint the bigger picture is, therefore, more manageable.

For example, if your objective is “How do I reduce my customer churn?”, start by looking at a sample of your CRM data, then move to transaction data and finally to financial data to build a customer behaviour map for that sample.

Keep an open mind to look for different aspects of your data. Staying with our example, it may be worth having another look at the way you segment your customers. Perhaps another approach to segmentation works better to identify churn? If that doesn’t work, move on to the next hypothesis quickly. Small data sets allow you to efficiently connect the dots in different ways to keep what works and discard what doesn’t without a disproportionate amount of effort. Fail fast to discover more.

Embrace the Gap by diversifying your team

Not all low-quality data needs to be discarded. Sometimes sharper insights are concluded through fuzzy approaches. Data scientists, while being experts in analytics, are seldom also experts in business. Bringing in subject matter experts (SMEs), designers, operators and business analysts who connect data analytics with business decisions can ‘fill the crap gap’ with experience and intuition. This can often lead to emergent insights. For instance, a sales person might help identify the impact of a disruptive change in the customer environment early on. Quality analytics relies not just on data, but also on human experience.

Make data work for you

Working hard and spending big to capture copious quantities of data does not guarantee effective analytics. It is the relevance of the captured data that leads to impact through analytics. Data is indeed good, as long as you can Mind the (crap)Gap.

Stay In Touch.

Inflection Point

Let's Bring AI To Life