Dear fellow data friends, I am back after a break period from writing. My returning gift - a question for you all:
“If this year you can improve only one aspect of your data platform, what would that be?”
It is easy to have a long list of things you’d like to improve. Yet in reality, resource constraints always force us to prioritise and place our best bets. To prioritise is quite a difficult thing, especially in an emerging, dynamic and complex domain like data platform operation. In this article, I will offer a framework for you to systematically answer that question. It consists of 3 steps:
Understand different performance attributes of a data platform
Understand the key drivers in your situations
Make decision and move on
Let’s dive in!
Step 1: Understand different performance attributes of a Data Platform
Regardless of technologies, operation models or maturity levels, the performance of every data platform can be assessed in three main attributes:
Input
Processing
Consumption
You can see in this model, a data platform is similar to any common factory, which takes a certain type of input, processes it, then offers it to the users to consume. From a data-product manufacturing perspective, a data platform is a factory - a specific kind of factory whose input, processing and consumption can be measured in very specific ways:
Measurements of input:
Number of data sources integrated: covering a wide range of source systems such as internal applications, 3rd-parties sources, public sources.
Number of data types processed: structured, unstructured.
Measurements of processing:
Speed and timeliness of processing: data pipelines are run at sufficient speed and frequency, enabled by either batch or stream processing.
Quality of processing:
Errors are well-prevented, detected, and managed, enabled by capabilities such as observability, CI/CD, testing, version controls.
Environment is secured, enabled by RBAC and other network security tactics.
Processing is cost-efficient, enabled by configuration of storage and compute technologies.
Integrity of processed inventory: data are well integrated, showing consistent values for a record, enabled by master data management and reference data library
Measurement of consumption:
Quality of output:
Quality of data products are transparent, enabled by data quality metrics, or reference to data validation results, data owners.
Ease of consumption:
Data products are discoverable and understandable to business users, enabled by data catalogue, data marketplace, data dictionary.
Ease of use of BI, Analytics, ML/ AI tools, mostly enabled by the choice of vendors for these.
As you can see, when the measurements of the data platform’s attributes are clear, we can improve them, and therefore improve the data platform, by changing the specific components enabling those measures. For example, your choices can be:
Increase the number of data sources integrated in the platform
Add a data observability tool to the platform
Embed a communication channel with data product owners in the consumption interfaces
Etc … .
Knowing what choices are there, now we can move on to the next step.
Step 2: Understand the key drivers in your situations
There are often three kind of drivers for any organisational transformation initiative, no matter big or small:
Business needs
Existing capabilities
Future trends and foreseeable changes
For business needs, you can ask yourself what are the most common use cases for data in your organisation. Is this analytics for cost efficiencies, is this real-time detection of fraud? Identify the most painful challenges that your data consumers are facing, and put a big star on it.
For existing capabilities, the goal is to identify if you have enough fuel to go to the end of the change initiative. Depending on which attribute is the potential target of change (Input, Processing, Consumption), you will have to survey different capabilities (data domain expertise, engineering, analytics, product management, etc….). Look both in your team and across the organisation. Don’t limit yourself only to the resources you can directly control.
Finally, be aware of future trends and foreseeable changes. I’d specifically consider those foreseeable changes within a 6-month horizon. Look for factors that may disrupt the current data landscape, both from business needs and existing capability perspectives. You can ask questions such as:
Will a specific need go away due to organisational restructure? For example, your company is selling off the real-estate division, focusing on private equity only. That will change your business needs, impacting your plan.
Will a capability be significantly enhanced soon due adoption of new technology? If a department is looking to migrate an on-prem database to cloud, that would definitely impact your integration agenda.
Step 3: Make decision and move on
After considering the business needs, existing capabilities, future trends and foreseeable changes, a few candidates of priorities will stand out, which:
Support critical and significant business needs.
Can be supported by existing capabilities, or require a reasonable amount of investment with tangible ROI.
Will still stay relevant given the changing landscape, or will even become more relevant.
To make the final cut and pick your one and only candidate, simply narrow down to the one with the highest score across those 3 boxes. Also look around for buy-ins of your ideas and choose the path of least-resistant. Then, make the decision and stick with your plan.
Once the change has been implemented, remember to measure the impact, provide tangible numbers for your achievements. You can refer to the measurements of the data platform mentioned in step 1 for ideas of how to measure your success. After that, broadcast your results to your CDO and business stakeholders. You can use that as a platform to move on to the next items on your priority list.
If you enjoy or get puzzled by this content, please leave a comment so we can continue the discussion. Throw in a like as or share as well if you know someone who may enjoy this newsletter. Thanks!
The “input, processing, consumption” model is a useful way to think about data-platform priorities, especially as AI increases the pressure on all three.
More sources and faster processing don't necessarily create more value if the resulting data isn't trustworthy, discoverable, and usable by the systems consuming it.
As agents become another major consumer of enterprise data, I think “ease of consumption” will increasingly include whether machines can understand the semantics and quality of the data, not just whether a human can find it in a catalog.