In the data landscape, the people are represented by two separate yet equally important groups. The data engineers who design the Lego blocks and the data scientists who build something extraordinary out of them. These are their stories.
DUN DUN!
And we’re back! Last time, we went over the toolkit needed to get your foot in the door as a data engineer. You’ve gotten over the first hurdle, but I hope you haven’t fallen prey to the Dunning-Kruger Effect. It’s a cognitive bias where people with limited competence overestimate their competence.
The Dunning Kruger Effect
You still have a long way to go, friend. Last time, we ended on a slight bummer note, explaining why it’s not all rosy on the other side. If the grass isn’t greener on the other side, it’s not completely bleak either. If it’s not the best of times, it’s not the worst of times either. It’s time to peer into the crystal ball and see what the future holds. Spoiler alert: This is what top data engineers could end up earning:
Salary Statistics for Data Engineers in USA
With enough skills, you could easily bump this up to 200-250K, maybe even more.
Swimmin’ in Gold like Scrooge McDuck!
Now that I have your attention, let’s dive into what the future of data engineering looks like.
Upgrades, People, Upgrades
Previously, we covered how commercial software can perform a large chunk of a data engineer’s responsibilities. Now, your data scientists and analysts can build a pipeline without any data engineering experience!
Well, color me blue, because that doesn’t sound very hopeful now. On the surface, it appears that the way forward is to “sell your soul,” “do drab sh*t because the money’s good,” and “wait until your golden 60s,” because off-the-shelf tools are allegedly eating up your livelihood. And then it begins…
The Rise of Skynet
We aren’t there yet. So take a deep breath and keep reading.
Is it time to pack it in? There are two facets to this. Let’s cover them one by one.
First, if you hire data engineers and ask them to build data pipelines, you are boxing their responsibilities. If they’re focused on just building data pipelines, commercial tools would look like direct enemies. But the aim of the tools was to free up time for more innovative work. Think Baymax in Big Hero 6.
Alright, alright, alrighty then. If the boring parts of data engineering are being automated, what else can the data engineer work on?
Nope, no more SQL-based data transformations, and building ingestion pipelines. Here’s what a data engineer could focus on, according to Tristan Handy:
- Building and maintaining custom ingestion pipelines
- Building non-SQL transformation pipelines
- Optimizing and managing core data infrastructure
- Supporting data team resources with performance and design optimization…
A data engineer can also build products to “scratch their itch.” For example, the data engineers at Airbnb didn’t have a way to construct and schedule DAGs. So, they built Airflow.
Besides, the role of the data engineer is splintering into more specialized roles. Like software engineering, the responsibilities and roles of data engineering are changing, specifically for more mature organizations. The traditional data engineer is fading as data warehousing moves to the cloud. Now, data engineers are more responsible for managing data reliability and performance.
Now, data engineers are no longer responsible for managing storage and compute. Their role is changing from infrastructure development to more performance-based data stack elements. This includes the rise of more specialized roles.
The Splintering of the Data Engineer Role
Data Reliability Engineer
With companies pushing for higher and higher ROI use cases, the importance of data has been dialed up to 11. This has caused a surge in the demand for reliability and quality.
As a relatively new field, data reliability engineering focuses on improving data quality. It keeps the data moving on time, and ensures that machine learning and analytics products have a healthy set of inputs. A large portion of data reliability engineering involves late-night backfills and spot-checks while hand-rolling some SQL into Grafana monitoring.
Top Skills Needed
A data reliability engineer needs a very particular set of skills, earned over a very long career. Skills such as the ability to optimize and check data pipelines are invaluable. As a data reliability engineer, you should be aware of the following tools and concepts:
- Data lake technologies: PrestoDB, Azure Synapse, Databricks, and Hive.
- Databases: PostgreSQL, SQL Server, MySQL, and others.
- Data Warehouses: BigQuery, Snowflake, Redshift, Firebolt.
- Discovery and Governance Tools: Collibra, Immuta, Stemma, Select Star, Castor, and Alation.
- Testing and Observability for pinpointing and solving issues: Data observability tools like Observe.AI, infrastructure observability tools like Datadog, data testing tools like dbt tests and Great Expectations, along with the definition and tracking of data SLOs, SLAs, and SLIs.
- Data pipeline tools: Orchestration tools like Prefect, Airflow, and Dagster. Transformation tools like DBT, and Reverse-ETL tools like Hightouch and Census.
- Working knowledge of networking: FTP, HTTPS, DNS, VPNs, firewalls, and load balancing.
What does the Bedrock of Data Reliability Engineering look like?
Data Reliability Engineering takes a page from Google’s SRE handbook to create a strong foundation for data teams to work from:
- Monitor Everything: You can’t mitigate or control problems if you can’t pinpoint them first. Alerting and monitoring provide teams with the visibility they need to understand when something goes off the rails and how to fix it.
- Embrace Risk: Teams need to control, detect, and mitigate failures that do occur, as opposed to hoping that they achieve perfection someday.
- Reduce Toil: To improve efficiency, try to reduce the amount of effort spent to decrease the overhead. For instance, tools like Hevo Data can reduce the toil needed for replicating data, while Looker training sessions can decrease the amount of effort that goes into responding to BI requests.
- Set Standards: Is the data high quality or not? That’s a subjective question that should be quantified, defined, and unanimously agreed upon for teams to make progress on it. If the definition of what’s good and what isn’t is blurry, it would be hard to do anything about it.
- Control Releases: Making changes might leave your system vulnerable to breakages as well. This is a lesson that data teams can borrow directly from DevOps and SRE, CI/CD Pipelines, and code review. At the end of the day, pipeline code is still code.
- Use Automation: Data platform complexity has grown multifold while the management to take care of it has grown linearly. It doesn’t take an expert to see that this is slowly burning a hole in your pocket while being unsustainable. Automating manual processes helps data teams scale reliability efforts. It frees up the necessary bandwidth to go after more complicated issues.
- Maintain Simplicity: Complexity is the antonym of simplicity. Now you can’t remove complexity — but you can make an effort to reduce it. Minimizing and isolating the complexity in any pipeline job helps keep it on top of the reliability charts.
Analytics Engineer
The analytics engineer is a role that stands at the intersection of data analytics and data engineering. It applies an analytical, business-oriented approach to dealing with data. The analytics engineer ensures that the data lives with business analysis and intelligence.
They are responsible for providing clean data sets to end users. They also model data in a way that allows end users to answer their questions. The role of an analytics engineer used to overlap with that of a data analyst and a data engineer to some extent. But with time, the demarcation has become clearer.
According to DBT’s guide, here are a few questions that run through the mind of an analytics engineer:
- What do analysts or other business users need to understand about this table to be able to quickly use it?
- What is the clearest possible naming convention for tables in your data warehouse?
- How can one improve the quality of data as it’s produced, instead of cleaning it downstream?
- Is it possible to construct a single table that lets them answer this set of business questions?
- What if one could be notified of a problem in the data before a business user finds a broken chart in the business intelligence software?
Top Skills Needed
- Python: Although analytics engineers don’t need to be master coders like software or data engineers, it is still pivotal that they know how to code in Python. A vast majority of data pipelines leverage Python and use it while writing their pipelines.
- SQL: Similar to a data engineer, an analytics engineer owes its bread and butter to SQL. On one side, you have BI and data analysts that might prefer low-code drag-and-drop platforms to create SQL. On the other side, you have analytics engineers that start with SQL for more complex queries and greater control.
- Considerable DBT experience: DBT has been doing the rounds for a while now, and is widely known in the data space as the leading data transformation tool in the industry. You’ll mostly be using it to create your data models. DBT is fairly simple to use once you have the fundamentals of SQL down pat. That’s because DBT is built on top of SQL.
- Experience with modern data stack tools: An analytics engineer also needs to be familiar with the most popular tools in the data stack such as Snowflake, Google BigQuery, Hevo Data, Fivetran, Looker, and Power BI, to name a few. This infers a familiarity with transformation, ingestion, deployment, and warehouse tools. You don’t need to have a grasp over every tool on the market, just a thorough understanding of the underlying concepts would suffice. With that said, if you are aware of one tool in each component of the data stack, you might find it easier to get accustomed to others in the same category.
- Collaboration: Having the ability to collaborate is an integral part of becoming a good analytics engineer. Since, as an analytics engineer, you bridge the gap between data analysts and data engineers, you need to master communication to allow you to gel well with different data/business teams. The onus of keeping everyone up-to-date on the status of the data, when data quality is compromised, or when a pipeline is down, and above all, clearly understanding what the business needs.
When is this role a good pick for you?
Becoming an analytics engineer might be the right path for you if:
- You enjoy working with new tools and learning about them on the trot. As an analytics engineer, you’ll always find areas where you can make a positive impact with a new tool. Spoilers, a large part of understanding new tools is researching and reading through new tools. It might be one of the most tiresome aspects of this job. But, it’ll be worth it when you see the new tool you painstakingly looked into making your team more productive.
- Simply leaning toward the technical side of things won’t help here, you also need to have a sense of how the business operates. This requires you to have a good understanding of terms like retention, ROI, KPI, customer acquisition costs (CAC), etc. As a data engineer, you can go about your day without worrying about how the business grows and makes money. Since you’re required to come up with new ways to better solve the problems your business faces, it would be a simpler transition if you love it.
- You like traveling in uncharted territory to sketch out a path for yourself. You aren’t satisfied with just executing the tasks assigned to you, you like to go above and beyond. This involves keeping a keen eye on data quality and monitoring metrics that might slip unnoticed otherwise. In layman’s terms, you need to sniff out the problems before they turn into full-blown, urgent issues.
Splintering is a good thing since it prevents the data engineers from trying to do everything, a precursor to burnout. Instead, this allows them to focus on stuff that truly matters: making sure that the data is accessible, trustworthy, and secure at every point in its lifecycle.
The Opportunity to Treat Data as a Product
With the rise of off-the-shelf products, engineers have an amazing opportunity to own the movement toward treating data like a product. From afar, this might look like a move towards agile project management. But, at a deeper level, this might indicate the move to data tools that allow cross-organizational collaboration, monitoring, and version control. There are four pivotal steps that allow you to build your data platform like a product:
- Ensure Your Product’s Goals Match the Business Goals: When scaling or building your data platform, the first question to ask is: how does the data align with your company’s goals? This requires looking at this situation from a data-platform product manager’s perspective. This needs a deep understanding of the bigger picture since data will feed into the requirements of every other team.
- Play the Long Game: Data platforms usually don’t owe their success to being the first players on the field. No sir. The bigwigs got there by building and iterating their platforms to keep up with the changing times and customer needs. So, you should choose solutions that align with your organization’s deadlines and expectations. This means, you might face some short-term pain, and this might tempt you to jump to a symptomatic solution. But hold your horses, woman! Even though quick wins can get internal product support as part of a larger product development plan. Don’t lose sight of the horizon.
- Set the Tone for Your Data: The greatness of the data platform you built would go to waste if the incoming data is unreliable. But, data quality might have different connotations for different stakeholders. So, your data platform won’t succeed if you and your stakeholders have different definitions of data quality. It is crucial that you set baseline expectations for your data reliability. Setting clear Service-Level Indicators (SLIs) and Service-Level Objectives (SLOs) for software application reliability should become the norm for data teams. This helps simplify things down the road when working with data pipelines.
- Iterative Feedback is Pivotal… with the Right Peeps: Receiving iterative feedback and support throughout the product development process are key aspects of the data platform journey. What people gloss over is the importance of getting the right iterative feedback. Having an idea of the right people to approach to give you the feedback you need also helps save time in the long run. Otherwise, you might waste it trying to convince the wrong person.
Simplifying the Communication Gap
After you’ve collected and cleaned the data, you might hit a roadblock when defining the data before analyzing it. The problem doesn’t stem from a lack of knowledge or skills to get to the final answer. It stems from siloed data that isn’t collaborative. This is the data debt problem. It manifests as undocumented, unused, inconsistent, and incomplete data. This creates inefficiencies that compound over time, making it difficult to tackle down the road.
Stuck in a Loop?
This could create a communication problem for teams trying to get on the same page about how to define key metrics and what to measure. Also, a data professional spends almost 20—30% of the week answering trivial ad-hoc questions!
So, there is a need for a central repository that stores the information running in a data team’s head. It should also answer the questions of data consumers in a self-service manner. There’s also a need for better communication and collaboration to bridge the gap between the data consumers and the producers.
As data becomes more mature, the data engineer is in a unique position to have a much greater impact on your organization. That starts by developing a data culture alongside their data products. Here are a few ways Emily Riederer feels you can foster a data practice or community:
- Engage with users to understand the business questions that brought them to your data’s doorstep and how they interact with it. You can do this through huge design sessions or one-on-one interviews.
- Query the usage logs and publish them (keep privacy permits in mind). This helps users connect and get answers to common questions in a self-service manner.
- Share your work. As much as you can. For instance, share scripts that are a part of your pipeline. This’ll help advanced users understand and reuse them as they see fit.
- Give users the resources and training to let them interact with data in more advanced ways. Don’t brush off supposedly “basic skills,” because they might be transformative for other teams. For instance, teaching Airflow to data scientists or teaching marketing teams SQL can help these teams make their workflows more efficient.
- Create centralized tools that help users in consuming data and invite community participation.
The Switch from Feature Teams to Foundation Teams
Data engineers used to be members of feature teams. But, as data volumes exploded, this led to data silos and a lack of global consistency. This is why companies started to adapt by developing cross-functional teams. This allowed the data engineers to impact the organization on various points.
As we saw with the splintering of this role, the future data teams won’t work on a particular data product. Their aim would shift to making the product teams more productive. To do that, they’ll be responsible for providing the right set of tools. This is where the “data mesh” paradigm comes into play. It’s a distributed ownership model with a foundation team that provides all the necessary tools to build data products.
Holistically Speaking, What’s Up? Days of Future Past
So, automation appears to be a genuine threat… to mediocrity. Data engineers who’ll go above and beyond to get to the next level would reap the benefits of going the extra mile. Let’s call the data engineers that fall into this category the kickass bunch. What’s the ace up their sleeves, and can you arm yourself with these tools to join their ranks? It’s time to find out.
Back to Square One
Rock-solid understanding is the foundation for success. In a landscape where the form and function of technology change in the blink of an eye, it can be difficult to keep up. On top of your Hadoops, Kafkas, and Sparks, if you keep going down that rabbit hole, the list of programming tools and languages becomes a never-ending pile of terms.
Baas Geerdink suggested a way out.
Rather than learning the tools (frameworks, languages, products, and engines), you should focus on the concepts (best practices, common practices, and techniques). If, after studying, you come across a new gimmick, doing some quick research should be enough to give you an idea of where to place it in the landscape.
Learning the fundamentals is the key to fast-tracking your understanding of new-fangled ideas. This is aptly captured by the Lindy Effect (see Nassim Taleb’s Incerto Series), which states that the future life expectancy of a non-perishable entity, such as a technological idea, is directly proportional to how long it has been around.
The Lindy Effect
So, since the fundamentals have stood the test of time, they are likely to last much, much longer down the road too.
No Country for Safe Players
The score was all tied at 0, as David Luiz watched the clones running towards the goal after beating the goalkeeper, about to upstage the “real” players. Just then a voice rang inside his head, “There’s no greater danger than playing it safe,” and he got up with a smirk as Miss Alissa by Eagles of Death Metal played in the background. And well, the good guys win in the end. That was the brilliant ‘Risk Everything-The Last Game’ by Nike.
The takeaway from the video was clear. There’s no greater danger than playing it safe. The players in the video are great because they find it within themselves to take bold risks. This may appear reckless to an outsider, but it is the driving force behind what makes sports so interesting.
Now, that isn’t to say you throw everything on the line recklessly in the hopes of getting ahead. The kickass bunch isn’t afraid of taking risks. Calculated risks. To the untrained eye, this may appear to be a gamble, but to the badass crowd, these are steps with little downside but significant upside. Naval Ravikant espouses the need to take on accountability to spearhead your success:
Embrace accountability and take business risks under your name. Society will reward you with responsibility, equity, and leverage.
Without accountability, you can’t possibly build your credibility. But, keep in mind that accountability is a double-edged sword. You get to take the credit when things go well, but you’ll also have to face the music when things go sideways.
The Rise of the Pre-Cog
A key feature that the kickass bunch shares is the ability to predict problems and prepare for them in advance. Kind of like the pre-cogs in Minority Report (2002), immortalized in science fiction lore for their ability to predict future crimes.
The Pre-crime Division, Washington DC, 2054
We’ve all seen people who seem to know what’s going to happen before everyone else. A more grounded example would be Michael Burry, who saw the housing crisis of 2007 long before everyone else. Do these people have a crystal ball that allows them to peer into the future? Well, not quite. But “there are always markers.” Go deep enough, and you’ll see trends forming that eventually lead to the big event everyone sees.
Having the ability to see latent trends before they have materialized puts you in the creme de la creme of data engineers.
How can you go about developing this ability? You need to know the ins and outs of the space, and you need to understand data like the back of your hand. Once you become Neo in the third act of The Matrix (1999), you can better identify the markers that would serve as early indicators for an event and cash in on them.
But how do you master the ins and outs of this space when what you learn today might become obsolete tomorrow?
You become a lifelong learner. Pedro Marcelino answers that in his take on the importance of fundamental knowledge:
In a world flooded with an immense amount of knowledge, we need to be able to make sense of information, distinguish what is important from what is unimportant, and, above all, combine what we know to fuel our learning and keep up the pace.
Soo, what does the road look like for a lifelong learner?
Never Stop Learning
To be a lifelong learner in anything, you need to be genuinely interested in it. As Naval Ravikant said:
That doesn’t mean you quit your job this instant if you aren’t genuinely curious about data engineering. It just means that someone who fits that category has better odds of making it to the top than you. Tough love all the way.
There’s a Japanese term, “shokunin,” that best captures the essence of lifelong learning. It translates to “the craftsman’s spirit” and is a way of life where the person believes they are always a work-in-progress. There’s a simple 4-step cycle to imbibe Shokunin within yourself:
- Set Goals: First, you need to have a clear idea of the problem you want to solve or the skill you want to master. Only when you park yourself in that seat to identify and analyze what you plan on achieving, do your nebulous thoughts give way to some clarity.
- Experiment: Once you’ve set your eyes on a goal, it can be comforting to perfect the plan, instead of rolling up your sleeves and start taking action to achieve it. The fear of failure is strong with this one. But you need to remember that you’re simply working on a hypothesis. More on mechanisms to fail forward in the DISSSect failure section.
- Evaluate Outcomes: Once you’ve achieved your goals, scrutinize and analyze the outcomes. It will help you realize if:
- your hypothesis about the problem hit the nail on the head or not?
- Did you implement the right solution? Is there anything that you can do better?
However, if you skip this step, you risk getting back to square one. Play the devil’s advocate and track your progress, Jane.
- Refine and Repeat: Next, you can correct the mistakes you made, go back to the drawing board, refine them and come back with a stronger plan. This iterative cycle allows you to make informed decisions about what’s working and what’s not. Here’s the cherry on top of the sundae bar: since everything is backed by outcomes, you can learn from your mistakes and improve.
Aand, repeat. That’s it.
Writing software, according to Mitch Seymour is an art form. Like a haiku, writing software requires creativity, a lot of hard work, and good decision-making. If you’re uninterested in the medium or the project, it’ll be reflected in the final product.
But, if you leave room to be creatively challenged, you provide yourself an opportunity to make something impactful for your business or customers while enjoying the process.
Lead the Troops
Every role ends up involving people management as you climb the corporate ladder. As a result, the 1 percenter isn’t just good at what he or she does but is also a prolific leader or teacher.
Bono said this in “The 2009 Time 100: The World’s Most Influential People”,
It has been said that after meeting with the great British Prime Minister William Ewart Gladstone, you left feeling he was the smartest person in the world, but after meeting with his rival Benjamin Disraeli, you left thinking you were the smartest person.
Liz Wiseman, in her book Multipliers, described a horde of leaders who could get the best out of the teams they led. The people under the “Multipliers” didn’t just feel smarter; they became smarter. They could solve harder problems, adapt more quickly, and take more intelligent action.
Here are a few tips covered in “Multipliers” that take you from being just another “genius” to a “genius-maker”:
Start With the Assumptions
Since behaviors follow assumptions, you can bring out a wide range of behaviors by adopting the Multiplier mindset. For instance, if you were to assign a task to a member of your team, these are the assumptions that can help you extract the maximum value from them:
- If I can find someone’s innate talent/genius, I can leverage it for the benefit of the team.
- People get smarter by being challenged. For the nitty-gritty of how stepping out of the comfort zone allows someone to grow, you can jump to the ‘Get Uncomfortable’ section.
- People are smart and will figure things out.
- With enough minds, anything can be figured out.
- People’s best thinking has to be given. It can’t be taken by exerting pressure.
Lars Meisingseth, Data Platform Lead with Norwegian Public Roads, suggested that setting the stage for a strong culture of accountability and discovery will set a precedent for new analysts and data engineers, allowing them to grow with the company once they get started. Additionally, this also helps cultivate a culture of valuing data.
It’s simple. If your data engineers work in an environment where their contributions are appreciated, they are more likely to be excited and satisfied while growing alongside the rest of the team.
Similarly, Chad Sanderson, Head of Product, Data Platform at Convoy, noted that:
If your data engineers are just running pipelines, that’s not very interesting work, but if they’re building the systems and technologies that power the data platform, that’s much more compelling.
To encourage autonomy, you need to start by building a culture of trust in your team’s ability to own and execute the projects assigned to them. Also, ask for their input on what they wish to work on before they even join your team.
This is completely opposite to some of the common beliefs of a manager in the workplace:
- People need to report to me to get them to do anything.
- I need to have and find all the answers by myself.
- People will never be able to figure it out without me.
- There are only a few people worth listening to.
- Pressure is necessary for optimal performance.
These characteristics are aptly described as “diminishing” in the workplace.
Work the Extremes
To excel as a multiplier, study your leadership practices and work the two extremes:
- Top off a strength: This one’s self-explanatory. Invest your energy to go from good to kickass by topping off one of your strengths.
- Neutralize a weakness: One of the oldest and overplayed quotes is “turn your weaknesses into strengths”. But, you might not be able to turn every weakness into your biggest strength. So, it might be worth your while to set a realistic target, i.e., making sure you aren’t bad at it. This means you would need to neutralize the weakness. This frees up the capacity to do more important development work: turning your modest strengths into outstanding strengths.
Experiment
At the heart of any transformation is experimenting. Effective and enduring learning consists of small, successive experiments with new approaches—testing new behaviors, analyzing feedback, modifying, and repeating. When these small experiments produce successful outcomes, the resultant energy fuels the next, slightly bigger experiment. With time, these experiments form new patterns of behavior that establish a new baseline.
Be Ready for Setbacks
Even though the Multiplier disciplines might be pretty easy to grasp, the same can’t be said for the effort it takes to put them into practice. Very rarely will you find a multiplier effect stemming simply from knowledge. Often, replacing diminishing habits with multiplier behaviors can only come through resilience and persistence. So, it is vital to expect and create tools to withstand possible setbacks along the way.
While on this path, allow yourself to stumble as you cultivate new Multiplier behaviors while changing your old habits. Understand that it’ll be hard; developing the new mindset and skills isn’t going to be a cakewalk. Don’t beat yourself up if you don’t get it right the first time; commit to your plan and keep at it.
Making the Leap from Good to Kickass
Alright, now that we’ve established what a member of the kickass tribe looks like, how do you make the leap from good to kickass? Here are five ways to advance:
Consistency is Key
Every day it gets a little easier, but you gotta do it everyday.
There’s no silver bullet or magic pill that’s going to turn you into a master of the space. What you need to make that breakthrough is the compound effect. It’s the principle of reaping huge rewards from a series of small, smart choices. What makes compounding so amazing is the fact that even seemingly trivial changes can lead to amazing results.
Small Actions -> Big Results in Future
While we’re on the topic of what it takes to unlock greatness, a popular rule espoused by Malcolm Gladwell is the 10,000-hour rule. It states that the key to achieving true expertise is simply a matter of practicing in the right way for 10,000 hours.
At first glance, that number can seem quite daunting. Putting this into perspective, 8 hours per day will get you there in about 3.5 years. At 4 hours a day, you get there in 7 years. Using this range and based on this rule, you can expect to attain “mastery” anytime between 3.5-7 years. Now, it is true that skill is only developed through hours and hours of beating on your craft, but 10,000 hours isn’t a number set in stone. Since there are a lot of variables that come into play, you can maximize the impact and expedite your journey with Pareto’s principle.
Sara Baxter Orr (VP, Strategic Growth, Anaplan Inc.,) in her conversation with Correlation One, talked about how pivotal it is to just show up:
Show up when you’re invited to things and get to work. Show up when you’re terrified of something or have been asked to do some things you maybe didn’t have to. I might have a bunch of things on my mind, but let that all go and be ready to learn when you’re ready to learn. I think you’re going to get the most out of your career that way.
Next, on “Have You Got What It Takes To Be A Kickass Data Engineer?” we’ll be covering how you can make the most of your time, fail early, fail fast, learn, and keep movin’.
DISSSect Failure
We have been taught from a very early age that good things take time. Rome wasn’t built in a day, and other overplayed quotes to that effect. But does that mean you have to toil for 10-15 years to reach the top? Is it possible to fast-track this journey? Keep in mind that this isn’t meant to be an excuse to slack off. Even though it might be possible to knock a few years off that number, you can’t become an expert in, say, 15 minutes. Gotta keep it real. With that in mind, we’ll take a look at the DISSS method, which was popularized by Tim Ferriss to speed up learning.
- Deconstruction: Each skill has parts. For effective learning, you need to break it down into its basic elements. This takes something that might seem overwhelming and divides it into manageable chunks. Try to break down the activity/skill you want to pick up and break it down into 8-10 activities.
- Selection: Starting with the basics is a good practice to apply if you’ve got ample time on your hands. Now, I’m assuming you don’t. If you do, you can skip to the next section. The key here is to forget what’s fundamental and question what’s important to get to competency. David Epstein, the author of Sports Gene, said, “The hallmark of expertise is figuring out what information is important.” This is where the Pareto principle will come into play. It states that 80% of the results come from 20% of your actions, and it applies to almost everything. Based on this rule, if you were to learn a new language, simply focusing on the most commonly used expressions (the top 20%) should give you a leg up on everybody else trying to start with the basics first. You must determine what is important from the 8-10 activities obtained in the first step.
- Sequencing: Surprisingly enough, most people get this wrong. While learning something, we usually don’t focus on the important stuff. Even more surprising is the fact that we don’t get it out of the way right off the bat either. Tim believes that putting things in the right progression is the key to getting hang of something new remarkably fast. We talked about the Pareto principle above. Once you’ve figured out the 20% of activities that will yield 80% of the results, do it again. This will give you the 4% of activities that will provide you with 64% of the results. Keep going till you arrive at the most important activity and set a sequence in decreasing order of impact.
- Stakes: There need to be consequences attached to the activity to keep you accountable and make sure you don’t fall off the wagon. So, you need an incentive to practice. Scratch that, you need a penalty for not practicing. Tim calls this a “commitment device”. Say, for instance, give your friend $200. If you get a task done within the time frame you chose, you get the money back. If you don’t put in the hours, you lose that money. The most important thing to note here is the default position. You cannot say “I’ll give them $200 if I fail.” No, you gotta fork over the money first. And, boom, you’re motivated. If that still isn’t enough for you, you can always jack up the number to a figure you’re uneasy about losing.
Another way to fast-track your learning is to get a mentor. Or, collaborate with people in this industry. Hanging out with people who might be further ahead in the learning curve will decrease the time it takes you to get there yourself. It’s really simple: if what you’re about to do has been done before, talking with people who came before you gives you a sneak peek into their journey, what to avoid, what to double down on, and overall how to get to the destination faster.
To really go for the jugular here, adopt the agile approach and fail fast. Iterating fast on failures achieves a result faster than perfecting the solution. A high school ceramics teacher wanted to experiment. So, at the beginning of the school year, he split his class into two groups:
- Group A: This group had a single objective. They had to make the perfect pot. They only had to make one pot the entire year, but only a perfect pot would yield an A.
- Group B: This group had a slightly different objective. They were given the liberty to build as many pots as they wanted to. They were graded on weight, which meant, the more pots they made, the higher the chance to get an A.
At the end of the school year, the ceramics teacher had startling results. The best pots came from Group B. In hindsight, the reason behind this observation is iteration. Group B made pots over and over again, learning from their mistakes. They got considerably better with each new pot. So fail fast and keep iterating to keep polishing your craft. You can be sure you’ll be far ahead of the people who aren’t willing to take the first step, even if you don’t end up mastering your craft. Couple this with the DISSS method to boost your progress even more.
Get Uncomfortable
John C. Maxwell talked about how growth comes from
the tension between where we are and where we want to be
in what is now known as “The Law of the Rubber Band.” It states that to unlock your true potential, it is important to keep stepping out of your comfort zone. With that said, if you venture too far off of your comfort zone, similar to a rubber band, you might snap.
Now, this might seem counterintuitive to the six words that seem to have been etched into the lore: “Break Out of Your Comfort Zone.” But think about it. Growth happens when something is just out of your grasp.
For instance, video game levels are designed to be slightly harder than your character’s level. This spurs you to push harder to finish the level because you know that eventually, you will. But say you were at Level 1 and were dropped to fight a Level 65 boss; would you still be interested in playing the game? Of course not. Game’s over within seconds.
And cue the frustration. However, if you slowly make your way to Level 65, beating the boss wouldn’t be as difficult.
The road to the top isn’t a linear path. It’s filled with zillion bends along the way. Now, you can either develop a toolbox of varying skills to navigate the challenges involved, or you can pack it in and go home. Here’s what Sarah Baxter Orr had to say about pivoting her way to the top:
I often did the jobs nobody else wanted to do. Another time I was asked to go do a financial reporting role and that was the last thing I wanted to do. I remember the CFO at the time looked at me and said ‘does it sound like I’m asking you?’ So it’s been important to be really flexible where you know these leaders actually know more than you do, and you have to just trust them, I would encourage people to be open-minded and take things that maybe don’t sound all that sexy.
The trick is to slowly expand your comfort zone. Here’s a strategy for the same espoused by fear expert Rhonda Britten: Stretch, Risk, and Die. To understand what those terms mean, she says you first need to picture a dartboard. The bullseye is your comfort zone. The next ring is your ‘Stretch’ zone; the one next to that is your ‘Risk’ zone; and everything that exists outside of those would be dubbed the ‘Die’ zone. Every time you move into a new zone, you have to go through a little fear because you have to think differently about yourself and what you’re capable of.
- Stretch: A stretch is something that we tell ourselves we ought to do, but haven’t brought ourselves to do it just yet. This consists of all the self-improvement moves we know we’re capable of making, if only we weren’t so afraid/lazy/misguided. But, whenever we try to do it, all of a sudden it feels hard. Since it’s out of our comfort zone, we have no idea how to act here.
- Risk: A risk is something that you aren’t sure you’ll be successful at. It’s a part of wishful thinking (for now), but never believed was possible for you.
- Die: A die turns the dial to 11 for a task. It feels like a crazy choice. As in, if I try doing that, I might die. For instance, for various people, public speaking could be daunting. Or bungee jumping, or salary negotiation.
To summarize, a slight amount of discomfort is essential to growth, but too much of it would simply inundate you and be detrimental to your progress.
Get comfortable with being uncomfortable, because that’s where the real growth will take place.
Understand Your Circles of Competence
Ted Williams. That’s a name that still manages to loom as the greatest hitter in baseball history. To provide a little perspective, he recorded a hit in 40% of his at-bats during the 1941 season—one of baseball’s most mind-boggling feats. Even more baffling is the fact that despite the rapid advancements in nutrition, sports science, equipment technology, and training methods, he is still the only person to have achieved this feat. Apart from talent, what set him apart was his thinking. He perfected his approach with a data-driven approach that was unheard of back then.
After every game, he wrote down every pitch of every at-bat. Which produced a strike? A hit? And so on. Armed with this knowledge, he divided his striking zone into 77 units. He then labeled each one with a hitting percentage, as depicted in the heat map for swing choice below. He called it his “happy zone.”
Ted Williams’ Happy Zone
Sometimes, life’s most powerful words are “I don’t know.” And knowing your strike zone—what you know and don’t know—is one of the most effective ways to gain an advantage in a field where big success is dependent on the ability to make big decisions.
And awareness of what you don’t know is more common than you think. Take a look at the Dunning-Kruger effect we began this article with. Here’s what Warren Buffett had to say about the circle of competence: Although he meant it for investing, it can easily be applied in any walk of life:
The trick to investing is just to sit there and watch pitch after pitch go by, waiting for the one right in your sweet spot. And if people are yelling, “Swing you bum!” ignore them. Over the years, you develop a lot of filters. I do know what I call my “circle of competence.” So I stay within that circle, and I don’t worry about things that are outside that circle. Defining what your game is—where you’re going to have the edge—is enormously important.
Sticking to your guns could cost you in some scenarios. The kickass bunch has mastered the art of “quitting while they are ahead.” So, this requires a keen understanding of when it is time to quit, apart from knowing their circles of competence.
Story time. Stewart Butterfield was trying to build a multi-player, massive, online roleplaying game called Game Neverending. Driven by his passion for it, he launches it, with positive responses from critics and venture capitalists (roping in Andreessen Horowitz to invest in it). Post-launch, the game managed to garner 5,000 die-hard users. Still, a sizable number, as these ardent users were the ones who could be monetized.
The only issue was that around 95% of the users had a game time of close to 7 minutes, after which they left, never to return! This is a worrisome signal because, in order to make the money, the number of people you’d have to get this in front of was simply humongous.
After consulting with team members and investors, they decide to launch this massive marketing campaign to compensate for the low conversion rate. And the marketing push was also working wonders, bringing in 6–7% new users week after week. Come November 2012, they report their best week yet. However, despite such promising results, over the weekend, Stuart sent out an email to all the co-founders and investors, letting them know that he felt certain that Glitch was over. You might think that’s crazy, because why would anyone think it’s not going to work when you’ve just had your best week? The co-founders and investors felt the same and asked him to show why he felt it was the end of it.
According to Stewart’s estimation, even if they were to sustain the growth they had week-on-week, they could just hope to break even in about 31 weeks. All in all, he loved the game, but there just weren’t enough people willing to play it, so he felt it was high time they shut it down.
Two days later, he started work on a new product. To make it easier for employees to talk to each other internally, they’d developed a communication tool within Glitch. It combined all the best parts of email, instant messaging, texting, etc. After he took the decision to shut down Glitch, he decided to work on this communication system instead. He called it the “Searchable Log of All Company Knowledge.”
Or, Slack.
Slack wouldn’t have surfaced as a product if Stewart Butterfield hadn’t made the wise choice of quitting back when he did. Because, even though Slack as a product existed for various years within Glitch, they’d turned a blind eye to it, and it only caught their eye once they gave up Glitch. This is a beautiful example of the art of quitting.
Be a Linchpin
Today, simply following instructions isn’t enough to make you valuable. In his book Linchpin, Seth Godin said the key to overarching success in the workplace—or anywhere else for that matter—is to make yourself indispensable.
The primary reason why just following instructions isn’t a prudent move to get ahead is that we’re already outsourcing most of this, and robots are being fine-tuned to take over the rest before long.
What makes a “Linchpin” stand out?
The Art of Unconditional Gifting
The economy started off with the barter system, a system where you got something in return for something else you had. The medium might have changed, but the thinking still remains. Everyone expects something in return for everything.
However, giving away genuine gifts with good intentions and zero expectations is picking up steam as a winning tactic. Genuine gifts here refer to giving away your best work for free.
Gary Vaynerchuk is a great example of this. He always gives his best advice away for free and talks about it openly. Instead of “not getting paid,” it actually makes people appreciate his unique skills so much that he receives plenty of paid work, and companies pay whatever he wants, just to keep him around. Quite contrary to popular opinion.
Therefore, if you want to stand out, start giving like a true artist, as described by Seth, and don’t expect anything in return. Give it time, and you’ll see the universe paying you back ten times over.
“Maximum Effort”
You’ve probably heard of this before. A restaurant shuts down after the chef leaves, or a company struggles because the top salesperson moves on. In these instances, the people leaving are the so-called linchpins. They are indispensable for the business because they are 100 times more valuable than the average employee. They are the people who make a business great by giving it everything.
If you give your work all you’ve got, are present in every moment, put emotion into each task, and pour your heart and soul into it, you’ll build a reputation. Linchpins don’t need instructions, and they never just do what is asked of them. They approach their work creatively, solve problems when they see them, and consistently over-deliver.
That’s why linchpins will always find work, be treated fairly, and be the last people that a company might let go of.
“I’m Vengeance”
Now if all it took to become a linchpin was to invest a little more than your co-workers, everyone and their grandmother should have become one by now. What’s stopping us? In most cases, fear bogs us down. We’ve been conditioned from a tender age to follow the rules and merge with the crowd to stay comfortable and safe.
We’ve all been conditioned by fear to play it safe. Linchpins are no different. They are also afraid but decide to do it anyway. Sound like anyone you know?
Well, there’s the Dark Knight.
Deshi Basara!
Batman is a fictional character who is driven by fear. He is defined by his fear rather than confined by it. He’s afraid of bats, and he harnesses that fear to strike fear in the hearts of the criminals of his city, Gotham.
Don’t let fear sap your determination to do something productive. Accept that it’s there and consciously decide that you’ll act anyway.
Final Thoughts
That’s it for this one. We’ve gone over what the future might look like for a data engineer, a little respite from the slightly jarring note we ended the last article on. We then proceeded to cover what sets a kickass data engineer apart and how you can join their ranks if that’s something you want. Above everything else:
Let Go Of The Past
Zig Ziglar demonstrated this through a flea-training experiment. Although tiny, a flea has the capability of jumping 8 inches, which is 150 times its own height. However, if you place the fleas inside a jar and keep the lid closed for hours, you can see the flea trying to escape, repeatedly hitting their head against the lid.
With time, they adjust and no longer hit the top. Now, even if you remove the lid, the fleas will make no attempt to escape until doomsday. They jump, but don’t jump out. They’ve conditioned themselves to jump only so high.
Sound familiar?
We, too, make adjustments to meet the standards we set for ourselves. It’s time to reconsider the limits you’ve set for yourself. Unless you take the leap, you can’t know how high you can really go.
“When will I know I’m ready?”
“You won’t. It’s a leap of faith. That’s all it is, Miles.”
“A leap of faith.”
A Leap of Faith- Into the Spider-Verse
References