Matheus Dellagnelo is the CEO and Co-founder of Indicium Tech, a company that uses specialized technologies and data science initiatives to assist businesses in advancing their analytical maturity. Indicium has expanded to over 20 countries and completed over 100 projects, accounting for more than 200 million in ROI for their clients.
As humans, we rose to global rule through a series of defining characteristics like brain power, innovation, and teamwork. Success, for us, has always been a collective effort of forming groups and removing blockades in the pursuit of shared goals.
Today, this idea has gone mainstream in the context of data-driven businesses.
Building a successful data-driven company requires a holistic strategy and a strong force of change ushered in by the data teams. For most companies, maximizing the use of data for strategic success is no longer just about rooting out the technical problems but also about resolving the underlying problems that these data teams are facing.
Smart businesses understand that data teams are their core organizational constructs. Like concertmasters in a chamber orchestra, I imagine data teams to be the key players in creating, improvising, adapting, and innovating data assets for the greater good. They merge your business teams with insights, ensuring that everyone can shine together and take advantage of data the way they want it.
Ten years ago, the nature of the challenges that data teams faced were solely limited to managing storage and compute resources for the entire enterprise. These teams would tinker with hardware in cold server rooms, ensuring that storage, communication servers, and other equipment were properly stacked.
As cloud-based applications and inexpensive warehousing technologies have become more popular, these problems have waned. Yet, new problems have emerged that permeate beyond the technology: communication barriers, overwhelming data and an increasing burden of data maintenance, lack of documented practices and standardization, and transitory data talent.
By solving problems that data “evangelizing” teams face, businesses gain momentum in harnessing data in every aspect of their products and operations. Despite what you may believe, data teams do not have broad shoulders and cannot solve every problem for your business. They rely on the support of other teams and leadership to anchor their own efforts. As a result, it behooves us to understand and solve their challenges as they become the linchpins of a winning data-driven culture.
Data Teams’ Four Biggest Challenges—And How to Solve Them
Data Teams Challenge 1: Finding a Common Language
Language is a unifying instrument that binds people together. When people speak one language, they become one.
– S. I. Hayakawa
Yuval Noah Harari, historian and bestselling author of “Sapiens,” writes in his book that our Hominid ancestors didn’t become modern humans by creating tools. It was our shared language and storytelling that evolved us from wandering African apes into today’s Homo sapiens.
Step into the 21st century, and his idea is still relevant today.
In most companies, solving problems using data and making business decisions have always fallen under separate functions. Data expertise is heavily centralized on the data teams, while decisions about business data are left to managers and executives. As a result, there is not just a communication gap, but a wide chasm: business experts place unrealistic expectations due to a lack of technical know-how, while data scientists and analysts struggle to understand business objectives linked to data. Consequently, misunderstandings happen, assumptions get made, data models lose accuracy, productivity drops, and disconnect grows.
The Data Language Barrier Study by Sigma
“Successful evolution” in business comes from a language that encompasses not just spoken words but also shared understandings and interests. When both business and data teams are working for the best interests of the company and end users, it’s essential that they communicate in a lingua franca with clear, informed, and agreed-upon objectives.
Data teams must be able to accurately interpret requests made to them and understand the nuances of the business problem to deliver the correct solutions in a timely manner.
Solution: Breaking Down the Data Language Barrier
Part of this disconnect can be bridged through training.
As data and analytics become pervasive, the ability to communicate in this language, to becoming data-literate, is the new organizational readiness factor.
– Carlie J. Idoine, Analyst at Gartner
I like to think of learning to “speak data” as trying to learn a new language. Your business teams are fluent in English, but they need to learn data fundamentals in conjunction with programming languages like SQL so they can ask data questions the right way and with confidence.
More than three-quarters (79%) of data experts and well over half (64%) of business experts desire to work together more closely. For them to collaborate more effectively, organizations need to provide a common platform that allows both groups to make the best use of their expertise and participate in the data conversation.
The Data Language Barrier Study by Sigma
Giving your business teams access to cloud analytics and BI tools in the modern data stack can help them access and analyze data autonomously and parlay their existing understanding of data. This can make them more informed and participatory while working on data projects with data teams.
Business leaders must learn to treat data as a second language and bring more people into the conversation so that it becomes a norm. Having “translators” like business analysts or analytics engineers who are fluent in both the technical and the business side of the project can provide a clearer path forward. These people are also adept at performing data transformations or visualizations for the business teams if ad-hoc requests pop up, taking some load off the data teams.
A New Way Ahead With Data Leadership
In my extensive experience of solving data problems, training or tools alone can’t completely eliminate your language barriers. While we’ve seen companies make some progress, they still struggle to translate a new dialect or technology into tangible outcomes.
For data teams to be successful, they require data leadership with enterprise-wide reach and direct influence on all corners. If yours is a young company without data leadership, Indicium specialists can help you build an effective data strategy encompassing both data and non-data teams.
Data leadership acts as an overlay that connects business strategy and operations with technology. They are the torchbearers of data teams who coordinate functions with an excellent understanding of the data teams’ processes so that people can build more trust in data, contribute, and communicate better.
Data leaders can firm up nebulous communications, resolve misaligned expectations from the start, and help data teams gather broader points of view on what is ideal and what is sought after. A good example of this is when your marketing teams need to identify the right set of audiences. Data leadership can guide data teams in determining whether the data model should identify customers with a high propensity to transact, or those who will transact if they are contacted via campaign. This will likely determine the path that the data team takes, including the training data, modeling approach, and level of effort, as well as how the business will be affected.
If your organization has a fractured data hierarchy, your data teams might find themselves reporting to various other functional teams. Data leadership can ensure that a precise schedule is established to avoid wasting valuable time. As the saying goes, the better the data leadership, the better the team can ride the challenges of difficult times.
Data Teams Challenge 2: Data Deluge and the Growing Burden of Data Maintenance
While businesses are generating more data than they know, data teams are scuffling to provide value and sustain momentum in operations.
– Matheus Dellagnelo, CEO at Indicium
Data is getting more pervasive than ever before, at an unprecedented rate. As the volume and complexity of data grow, so do the struggles of data teams to rapidly operationalize and leverage data for strategic purposes.
In the wake of the explosive growth in data sources used by business teams, data teams are deprived of time and energy they could spend on strategy. Having more software systems means having more varieties of data, which leads to more time being spent on data maintenance tasks and making sure that data is 100% consistent.
On average, data professionals report that 45% of their time is spent in getting data ready (loading and cleansing) before they can use it to develop models and visualizations.
Anaconda’s 2020 State of Data Science Report
Anaconda’s 2020 State of Data Science Report
For most businesses, the responsibility of managing data is usually assigned to data teams (55%) and business units (45%) (MarketPulse survey by IDG Research), and yet, in reality, I find data teams to be mostly untangling the messiness created in business systems, like resolving duplicate records and misstated data, purging and normalizing diverse datasets, integrating discrete data sources, etc. If your data teams are solving someone else’s problems, when would they solve theirs?
Matallion’s survey of data teams corroborates this by identifying three main tasks that consume a disproportionate amount of their time and effort. These unsung tasks rob data teams of valuable time while stymieing business value for other dependent teams.
The Data Divide Survey by Matallion
Solution: Take Advantage of Less-Maintenance, More Composable Tech Stack
There is no slowing down for the increasing volumes of data: as data volumes grow, so will the data teams’ time spent on data migration and maintenance tasks. When data is gathered from many sources, inconsistencies in the data come as complimentary additions.
Getting future-ready means employing solutions that can withstand the challenges posed by an always-online consumer-driven world and an increasing number of business applications. Such solutions must accelerate time to insight and remove barriers between data and action. They must integrate data with the numerous tools used by business teams in a way that requires little to no maintenance while in use.
A modern solution is to introduce a flexible technology that can integrate and manage data at scale and act as a data operating system for the entire business. No-code data integration platforms like Hevo Data help you perform this function by extracting data from 150+ disparate sources and automating data replication and transformation. This allows your data teams to expedite data collection to your warehouse and make the data ready for BI and analysis so other teams can leverage the data quickly and independently in making data-driven decisions.
Hevo Pipelines bring your data teams the promise of speed and scalability at zero maintenance. They make it easy to add more sources and pipelines to your data infrastructure, and you don’t have to pay a premium to employees possessing highly technical skills who can do complex and rapid hand coding. Using Hevo, even non-data professionals can configure and set up their own pipelines in minutes without having to worry about pipeline maintenance, schema mapping, or replication accuracy.
Data Teams Challenge 3: Lack of Data Standardization, Ownership, and Best Practices
Aside from bridging the “talk divide” and resolving the janitorial tasks of constantly maintaining disparate sources, another pressing challenge that data teams face is a lack of best practices, data standardization, and ownership into every step of their data strategy.
Supporting and getting the most out of your data teams requires a documented set of guidelines or best practices that share knowledge of:
- how to design end-to-end data frameworks and optimize them for business requirements,
- how to uncover and resolve issues such as data access, cleanliness, and timeliness,
- how to improve monitoring and tracking of data quality,
- how to lessen risk and save time using automation,
- how to develop tooling/processes and help other teams meet data governance policies and standards more easily.
The lack of data standardization and best practices can prevent data teams from deriving the full benefit of data and creating trusted data sources that other people can use. It can also obscure other teams from freely accessing and interacting with data.
To help you understand better, here’s a simple analogy: Imagine a situation where your data teams are in charge of maintaining a library with books organized on shelves according to various criteria. One shelf arranges books by the author’s name, another by the publishing company, and the third by ISBN number. In such a case, business teams will have a difficult time locating and accessing the book (dataset) that they require.
Data standardization, in essence, is the process of reviewing and documenting the names, meanings, and characteristics of all data assets so that users of the data can have a common, shared understanding of it. Over and above integrating data from discrete systems and maintaining it, data teams need standardization to bring in data portability (the ability to transfer data without affecting its content) and interoperability (the ability to integrate two or more datasets).
Lack of data standardization was found to constitute an important driver of costs, especially for small and medium businesses but also for large firms, potentially jeopardizing competition as well as innovation.
Data Standardization (Gal, Rubinfeld, 2018)
Data without standardization can cause chaos and simply become a way to store information without providing the opportunity needed to be able to use it. This effect can compound and create disorder by isolating data systems from each other, ultimately creating blind spots or data silos in your business.
Nearly 40% of data teams surveyed admit they don’t fully understand how data is being used in their organizations, which underscores the growing problem of information gaps across the enterprise.
The Data Divide Survey by Matallian
In addition to standardization, data teams need data ownership- knowing the “who, what, when, where, and why” of every change. Lack of such measures can result in a perfect circle of finger pointing, in which no one truly knows who uses the company’s datasets and how.
Solution: Start Incrementally and Set Standardization and Ownership Criteria Based on Your Needs
Standardizing data makes interpreting and analyzing data faster and more accurate, which ultimately facilitates better business decisions. For data teams, implementing data standardization practices in the data value chain can help them:
- Reduce metadata uncertainties by introducing data semantics,
- Minimize the number of barriers to data transformation,
- Share data easily,
- Manage data governance better,
- Enhance data flows and setting a good base for ML and AI initiatives.
Since every business has its own data needs, objectives, and data sources, it’s tough to give blanket advice. After all, the volume of data you collect and generate will only increase as you scale. Hence, it is in the best interest of your data teams to start small and keep things as simple as possible.
One of the best ways to do this is to examine the current flow of data- from sources to destination, and understand how information is disseminated in your organization. Your data teams can check and see if the data they collect is helping you achieve your current business goals.
I would be wrong if I said that you should be standardizing data from every single source, because that is ideal and certainly not 100% achievable: some data sources might not match your standardization criteria, some stakeholders might disagree with enrolling their business applications in the process, etc. In such cases, focusing on the right and essential data and standardizing only those sets is more sensible. A key role here will also be played by your data leadership.
Start with current data and build it incrementally.
This brings two twofold benefits to your data teams:
- One, it is easier to work backward and modify old data according to new criteria rather than doing it the opposite way. In this way, your data teams will have a much better sense of upcoming data sources and data types, and they can suitably devise standardization criteria according to your changing needs.
- Two, your data teams can realize smaller wins and better validate and gather support from higher management on a larger change.
Another requirement that data teams must fulfill is to put auditing policies in place. Auditing helps data teams gain visibility into any risks of misuse or breaches and ensures that other teams are using accurate and trustworthy data throughout the organization.
Data teams can choose to appoint a data controller, in consultation with the business teams, who can take charge of providing oversight and keeping track of audit trails. Such individuals can prepare reports and discuss them routinely with the data teams and leadership to identify areas that need improvement in data quality and data security and take corrective action when required.
Data Teams Challenge 4: Retaining the Right Talent
Most companies I’ve spoken with acknowledge that their data teams are like high-performance engines within their organization. These teams can know inside and out (with support from data leadership and other business teams) how to turn their organization into a fully-functional working organism and turn business data into fuel for growth.
If someone from the data team leaves, it is more than just an inconvenience: the work gets hampered since all team members are used to the output of the engine, there are transactional losses such as training costs, loss of expertise, and it’s difficult for other team members to evolve and adjust their workflows.
In such a competitive market, data specialists are looking for more than just jobs.
– Matt Hennessey, Chief Intelligence and Analytics Officer at Greater Manchester Health and Social Care Partnership
As technology commoditizes quickly, it is talent that will enable companies to develop competitive differentiation. For companies, it means creating an environment that data teams can enjoy and thrive in, and it goes beyond offering better monetary incentives. In fact, if you look at the most important job factors for data professionals, office environment, flexibility of schedule, and opportunities for professional development surface as high priorities.
Out of 49,349 people who answered the Stack Overflow survey, 35.5% (17,519 people) identify themselves as data professionals (database administrator, data scientist, data analyst, etc.). Here, I’ve taken the liberty of standardizing the overall survey results for data professionals as well.
2020 Stack Overflow Survey
Solution: Develop a Deep Understanding of What Data Talents Want
Retention requires finding opportunities for data professionals to grow and prosper. It might require giving your data teams new tools to experiment with and providing them with the freedom to learn and explore. It also helps if you cultivate a positive sense of career movement by rewarding strong performers frequently—not only through raises but also through recognition.
Having a career trajectory is hugely important to retention, as the lack of one is often cited as a reason why people leave jobs. If your data professionals can see a path beyond their current role, whether it is about becoming a senior analyst, a senior manager, or a specialist in C-suite, they would be in a much better position to stay.
– Paul Chapman, Global Director of Performance Management, BI and Innovation, JLL
Another important factor to consider is increasing ownership. You must have your data teams spend time with business or product teams in order to better understand the real issues they are dealing with. This will make them feel like a valuable member of the team while also improving their analyses and understanding of the company. As a result, they can contribute to the development of better products and the provision of better services for your customers.
Taken together, these two measures will increase the way your data teams feel engaged in their work environment and likely provide them with an environment that is conducive to both professional and personal growth. Making your data professionals happy goes a long way toward both retaining them and attracting new ones.
Wrapping Up…
Paving a path to becoming a successful data-driven company is a long journey that calls for big contributions from the data teams. It starts with removing barriers for them, so they can evangelize data-oriented thinking and catalyze substantial shifts across business functions.
This may require the presence of data leadership to achieve balance between business and technology, automated platforms like Hevo Data to streamline data replication and transformation from different data sources and avoid the hassles of infrastructure maintenance, a set of documented practices to administer ownership and standardization, or a culture that cultivates a genuine passion for working together and retaining the best talent.
At Indicium, we’ve helped hundreds of businesses develop an effective data strategy, which goes beyond just offering prescriptions. Our dedicated team of data specialists undertakes the tough work of resolving inefficiencies in your data infrastructure, whether it’s data acquisition, data transformation, storage, or analytics. We closely work with your data teams in bringing cogent solutions by not only meeting their technical needs but also creating cohesion with other business teams, so that everyone wins.
Get in touch with us by talking to our data specialists if you need help in resolving your data team challenges or transforming data into smart decisions.