Impact Engineering
So where I really wanted to spend time for this session is on this notion of impact engineering. And in particular actually focusing a little bit more on some general principles that I think really matter. I think they matter to the data engineering ecosystem. I think they matter to the software engineering ecosystem. I've been increasingly chatty on LinkedIn over the last few weeks and, for those of you who follow me there, you have probably seen an increasingly large number of posts from me really talking about the concepts of engineering impact and engineering productivity.
Cause I think it's incredibly important. For us as an industry to be investing in this. And, I think when we spend time talking about productivity and this notion of 10x-ing I do think it's first important to really define what do we mean by 10 X.
There's been really long historical conversations around what is a 10 x engineer or what is a 10 x team. Which one is more important? And I think it, for us, we are big believers that everybody's kind of gotta find their own way. You gotta figure out what 10 X means to you as a team, as a company and what you're really trying to achieve.
And so, one thing that I always like to preface here is there's many ways to find success. We at Ascend are really working through our ways of how we have found success and how we have created a really high performance engineering team, and how we can how we've been able to really produce some pretty significant impacts for our team. So hopefully in sharing some of these with folks y'all will get some nice takeaways and really think through how this could also apply to you in your environment. Why I think it's important for us to really be talking about this now of all times. I think it's always important to reflect as teams and figure out how can we do more and more importantly, how can we do more with less? Because I do think that helps push us into some pretty unique different challenges and rethink how we build software, how we build data products.
And, you know, oftentimes when we look at how teams build software, if you assume that a team generally cannot grow and you just keep a headcount flat, for example, the productivity of that team would generally approach zero over the course of time. As you build more software that creates more maintenance burden, and without culling that software and reducing the overall footprint over time, your productivity does ultimately go to zero. And so I think that this is an important backdrop against which we really start to think about why it's an important to invest in impact engineering and productivity. I think there's a couple more things, however, that make this pain particularly acute today. For those of you who saw my keynote earlier on today, one thing that you are already familiar with is what we described as a new scaling challenge for data teams. And I think for data teams, in particular, it is particularly relevant in how do we get data teams higher leverage, higher impact because of the current landscape, because the vast majority of data teams are at or over capacity because the demand for what they do is dramatically higher than their ability to grow their overall headcount.
We've seen this, that 96% of those teams, the overwhelming majority or at or over capacity, and so the overwhelming majority of those have increased overall demand for what they do. Then three out of the four of those teams still indicates that the demand for what they do is growing faster than their ability to hire. I think that for data teams has created a pretty pointed challenge already. Now, when we take that and apply an overlay on top of that and a really important overlay, the current macroeconomic situation makes it pretty challenging.
Really, really, I think daunting stats for all of us to look at is a recent survey by resume builder highlighted that 70% of companies are likely to implement the hiring freeze in 2023. 61% are planning to do layoffs or consider it very likely for them to do layoffs and those who are still fortunate enough to be able to hire and expand their team, they still have to deal with this quarter million person shortage of data experts over the next decade. And so when we think about this as a backdrop, it compels each and every single one of us to invest heavily in impact for the people we have, for these incredibly talented resources.
How do we help maximize their overall impact for us and the team? And so when we think about this as a guiding principle, at Ascend one of the things that we really try and oriented towards is: how do we maximize the impact per hour invested for each and every single individual inside the company? And when we really apply a critical lens here, we try and create some structure around this, we think is incredibly impactful and important for us and our culture as to how we connect create this overall impact and outcome for our customers. And we do a couple of things that I think are, are pretty unique here and this is one of our overarching philosophies that probably seems a little counterintuitive to many folks at first blush. We are believers, individual impact goes up when the number of people you depend upon you deliver your results goes down. I think this is really important because oftentimes we are told and taught, rightly so, that to achieve greater Im, outcomes and to have greater impact, we need larger teams and bodies of people to go do the work.
And that is a hundred percent true. The reason why we orient towards this at Ascend is we all too often end up with dependencies on others to do even some more basic tasks that end up slowing us down. Dependencies from a frontend engineering team on a backend engineering team, from a backend engineering team onto an infrastructure team. And each one of these teams has oftentimes, in other companies, a different management structure, a different priority queue, a different sprint cycle. And all of these end up introducing low or high latency low productivity hops dependencies across teams. They also introduce a lot more dependency around coordination in collaboration.
And while to achieve really large outcomes, of course, we need far more people to swarm on certain problems. We still find that for most activities today, and a lot of the projects and initiatives that we drive on, we are able to produce a significantly greater outcome for our customers and deliver significantly greater product when we focus heavily on reducing the overall dependencies across the organization. Now, that's tricky at times because you have many different teams. You have marketing teams, you have sales teams, you have engineering teams, you have product teams. And how you apply this can vary a lot depending on the domain.
And it can depend a lot based on the technologies you use and the products you use today. So one of the things that I'm gonna do is I'm gonna walk you through some of the decision making process we've made and some of the frameworks we've applied at Ascend, that I think can be really helpful for other teams as you seek to try and increase the overall impact of those teams. So one of the frameworks we use here, at Ascend for identifying these, these opportunities is identifying these opportunities for step change advancements. And to do this, what we really look for are two things. First, is we look for the high ROI waves. And for that, what we're really looking for are what are the things that are huge resource consumers of our team.
And more of what we're looking for, what are the things that if we deliver on these, they will have an incredibly large impact compared to the amount of work required? Sometimes this is introducing brand new technology to achieve an outcome. Oftentimes, this is replacing existing technology with newer waves of that technology that just make it easier for you to support, maintain, and iterate all. I'll walk you through a couple of examples. For those who saw my keynote earlier on today, one of the things we are incredibly proud of is the fact that we released three times as many data connectors as we did as we had at the start of the year.
It is incredibly impactful for us. And one of the reasons why we were able to do this is we had a material overhaul of our connector framework. A couple of years ago, we add a new connector to the Ascend code base, required touching three to four different systems writing code in three to four different languages.
Java, Scala, Python, and oftentimes React in a Java script as well. To add a different connector, oftentimes would require thousands of lines of code. And to do so across multiple teams and multiple parts of the code base would take tremendous effort, tremendous coordination. And so as a result, one of the things we found was, we just didn't do it very often. The amount of work required was really high, and as a result, the ROI equation for adding new connectors was pretty low. Yet we found that a lot of our customers really needed significantly higher connectivity across the products.
So one of the things that we did is we took a really big step back and we said, Hey, what can we actually do inside of Ascend to make this much easier? What if we set some extremely ridiculous goal? What if we want it such that a single engineer could add an entirely new connector in less than a hundred lines of code and can do it all by themselves? What do we have to go do to achieve that? That was actually the seed of our entire new connector framework and strategy that has allowed us to deliver on such such large amounts of new connectors. We took a huge step back and said, well, the vast majority of our connector ecosystem either runs on Spark or has Python libraries. So we know we need to write that in Python. What are all the common elements? What is everything that a connector does? When we boiled all of this down to a single architecture of what a connector itself does and wrote all of the frameworks around it such that now when we add a new connector, we don't have to write Scala code or go code or JavaScript code. We just have to write small amounts of code specific to the connector.
And what we found was, not only did require less code, Not only were we able to deliver it much faster, and not only could an individual engineer do it. We found we actually ended up with far more powerful connectors that can deploy far greater functionality because all of the code associated this was very isolated. Now it all was at that exact end point of attack, if you will, where it interacted with other systems. And so when we look at these air ROIs approaches like this, this is a really great example for us. A second example that we've really been investing a lot in here is our PLG stack.
One of the things that we are really excited about at Ascend is getting more datand analytics around how our customers use our product, and how do we help them encourage to use more of our product. One of the reasons why we really like technologies like Intercom and Heap is not only are they no code solutions, There are solutions that a product person and a marketing person can use all by themselves. They directly hook into the dom, they can navigate the dom. You can tag events. You can build funnels, all without having to write code. And so when we apply our same principles from the engineering side to our product and our marketing teams, we find we can be incredibly productive and have people be incredibly high impact, without having to have cross team dependencies, that allows them to run as free and fast as they are able to do, without having to ask other teams for help.
We're really excited by technologies like this because they adhere to the same principlesthat, that we have. So one of the things I want to do is I wanna walk folks through a little bit of a thought experiment. How do you uncover these opportunities for 10 x improvements. I think these are really a fun and exciting framework to go through. And something that we certainly like to run here at Ascend.
So when you're running these thought experiments and you start to look for how do I find these opportunities to create a 10 x team and start to inject into our team this 10 x culture, First thing is I think is always important is to set ground rules and allow time for the ideas to grow. The general approach to this is you really do want to encourage an iterative yes and approach to brainstorming. For those of you who have done improv in the past, "Yes, and" is this really incredible way of building on each other's ideas and directly avoiding a person that might cut down a certain path of exploration or ideation. It can be really hard at first, especially as engineers, where we we're trying to whittle away at the problem space and stay focused on, on certain domains. Yes, and helps expand the overall level of creativity and exploration. I strongly recommend embracing that mindset.
The third thing that I always like to preface with folks from running these thought experiments: be patient. Rejection is a natural first reaction. Resistance is the second. I think this is really, really important because when we try and create some really hard thought processes like this, It's going to be natural for people to, at first, just not believe it's possible.
And so be patient. Sometimes you'll get a team together and say, you know what? Nobody came up with any ideas, but people are gonna go home, they're gonna be working on another project, they're gonna be talking to other teammates. They might even be talking to a teammate and be like, oh my gosh, Sean brought us into this other crazy mediand he gave us this crazy challenge, and he is off his rocker. There is no way that is even possible.
But it might trigger an idea for somebody else and they might end up following a "yes, and" approach offline. And the next time you bring the team together, they might actually have a really brilliant idea. They might not be the final idea. But if you truly embrace the "yes, and" approach, you'll actually start to see some really incredible ideas emerge. And so against this backdrop, we wanna give folks a fun challenge, and this is how we like to frame it.
And you can customize it to your sort of general structure here. But this is one approach that we oftentimes do, which is what if I had to double our output as a team, we just had to produce twice as much next month, next quarter as we did the last month or quarter, without growing our team. The third part here is, and you can't work more hours.
I think this is a really important one. Otherwise, you're tired. So I do strongly recommend including the, "you cannot work more hours". Hours don't scale.
There's only so many hours in the day. Your hands can only type so fast. It is very hard to change those laws of physics. And so when we think about a framework as to where do you invest your time, how can you increase your overall output and how do you do it outgrowing your team and not working more hours.
You'll start to find some really interesting ideas. Oftentimes, as we explore with our teams, we ask, well, what are the things you're doing that are incredibly repeatable? What do you find yourself doing day in and day out? Who do you find yourself depending on day in and day out? Can we help automate those away? Can we give you product, a SaaS product, a tool, a framework, something that can actually help alleviate your dependencies so that you can self-serve more? The more we can have people do that, the more likely they are to be high impact and highly productive. Let me give you a couple of examples of how we further have applied this at Ascend. Two major changes we've made over the last couple of years have been: one, we've made a pretty strong shift to Python.
One of the reasons why we did this is we decided the limiting factor for us is engineering hours, not system performance, or throughput. Most of the data processing that happens in and around ascend isn't through Python code, just as it's not going to be through our Scala or our Java or our Go Code. It actually is in the underlying engines, and as a result, those are already incredibly optimized. And things that we do actually do end touch that processes data and that are performance constrained, generally have fallbacks into c C++ libraries that are incredibly performant. And so we said, well, we're not trying to find languages that are incredibly performant from a throughput or a processing perspective.
What are we trying to optimize for? And one of the things we decided on was we want to optimize for engineering productivity. What are the languages that allow us to write code the fastest, allows us to write code very compact, That allows us to leverage more frameworks, more libraries that are out there. And that's one of the reasons why we have increasingly been investing our time and our energy in Python.
We think it's a incredibly powerful language. It continues to progress incredibly well, and we as an engineering team are able to produce more with this language than any other language out there. Very similarly, we also have been making greater investments in Dash as a UI rendering framework. For those of you who have used our observe service, which I referenced earlier on in my keynote this morning, that, that, service is actually built in dash, Dash allows us to create a really powerful UI framework, a React based UI, all in Python. We're able to leverage these really powerful bridges in component libraries that give us a modern, elegant framework, but still write all of the function handling and all of the behavioral changes in Python. And again, why this matters so much: it allows us to enable a larger number of people to have fewer dependencies and create really compelling experiences, without having to rely on, say, a front end engineering team or a design team or a graphics team.
So this is again, a really powerful example for us at Ascend of how we've been able to be so productive and how we can continue to turn out a lot of really great product because we can leverage these creative applications that make our lives easier. So if we take a bit of a step back too. Beyond the exercises and a couple of examples. One of the areas where I'd love to spend some time is talking a bit about engineering culture itself.
What are the aspects of an engineering culture that can enable higher impact teams? And how do we how do we achieve some of this at Ascend? Well, the first thing that we focus on is this notion of activities versus outcomes. And we're really clear internally what our general take on activities: activity is a cost. When you think of a balance sheet, if everyone puts their finance hat on. Activity is the thing that takes time exam energy, and you pay people to do. There's not a monetizable thing.
Outcomes are monetizable. Outcomes drive revenue, outcomes drive impact. So when we think about how we run our teams and how we orient our efforts, it's incredibly important that we focus on outcomes, not activity. We used to have this old saying back in the day, which was an A for effort ended in elementary school. Because outcomes are around grades, around the performance that you can actually create. And so for us, one of the really important parts is how do we continue to drive outcomes.
Now, when we think about outcomes, outcomes are really important. Outcomes for us are successful customers. And for us, it is incredibly important that we build a customer oriented mindset.
For those of you who are already existing, Ascend customers, you'll see we have granted huge amounts of access and we put tons of engineers directly on our customer accounts as we build new features. And we really encourage a strong level of engagement for this reason. I'm a deep believer that context and understanding of what our customers are doing, how they're using their product, helps us produce better overall results. So for us as an engineering team we believe strongly in having deep connection with our customers and how they're applying our technologies to make sure that we are building the right features, the right capabilities that deliver the actual outcomes for our customers. And you'll find this woven deeply throughout our engineering team. The other thing that we focus a lot on too is building a culture of polymaths.
Polymath, we believe, is incredibly important to a strong engineering culture. Core reason why: over time we see patterns emerge over and over and over again, whether it is big data patterns compared to operating system patterns compared to software engineering patterns. We watch, the movement for DataOps is very, very similar to the movement for DevOps. For students of DataOps, we can better understand the world of DataOps. Very similarly, the patterns we've employed around our Scala part of the code base, we can leverage for our Python part of the code base. When we become students of how Snowflake works incredibly well, we will better understand how BigQuery and how Databricks work as well.
We really encourage ,across our team, to build deep knowledge around how the technologies and the frameworks and the services around the company work. It helps us actually be better overall engineers. The other thing that we have found that is really powerful is the more polymathic in nature we become, the more we tend to avoid language and frameworks electric. And this is something that I think is incredibly important and can really permeate a lot of engineering cultures. One of the reasons why polymathism is so powerful is when you have this incredibly large tool set and you equally understand that the ecosystem that surrounds you, you then are able to pull from this larger toolkit or the best tool for the job.
As opposed to just saying, all we can ever do is Python, or all we can ever use is go. Being able to tap into a larger framework of tools and capabilities helps you generally find the best overall tool to deliver the outcome for your customers. And last section here, I wanna talk a little bit about too. When we think about building the core. It's not just around an engineering culture. There's also incredible importance in a company-wide culture that reinforces those core behaviors you want in any organization, whether it's marketing, sales, product, engineering, you name it.
So one of the things that we've done to be really thoughtful here at Ascend is we spent a lot of time on our core values and we focus pretty heavily on the core values. That don't just define a company we want to work for, but a company that will, beyond that, actually help drive a pattern of engagement and collaboration across the team that drives success. And so we think about our core values as things that should actually be tiebreakers, things that help us decide between some of the hardest decisions and trade-offs in a company, and instead drive a really focused culture for outcomes and impact. So, to walk folks through that, our first core value that you all if you've worked with Ascend, have probably heard or seen somebody wearing the t-shirts on is "I got it." We are deep believers in this as part of our culture. When somebody says, I got it the team can trust that person will deliver the agreed upon outcome.
And this is so important. We value people who deliver those overall outcomes and reinforce this on a daily and weekly basis across the company. The I got it notion allows teams to divide and conquer. You'll remember when I mentioned earlier at the start of this talk by reducing the number of dependencies to deliver an outcome, we fundamentally increase an individual's overall impact. And "I got it" as a core part of reinforcing that for us. When teams can properly divide and conquer, and a person can take their portion of the outcome and say, I'm going to deliver this, and others can trust that they will they will achieve that.
Everybody else can put their time and energy and attention on delivering other outcomes that are equally as important and impactful for the business. So we really focus on this one a lot and celebrate this one across our culture. Second core value is: one team, one dream. This one we find again is incredibly important and it's designed to specifically reinforce the first. For us the "I got it" alone, could actually be weaponized and could be destructive as a core value. And for those of you who have led in increasingly large organizations can see some of the some of the risks of a core value like that.
And so one of the things that we do with our "one team, one dream" core value is to put in the checks and balances to align everybody. This keeps us all rowing in the same direction, and again, is really core to that belief structure around enabling individuals to achieve and affect outcomes on their own. This only works when everybody is rowing in the same direction. If you don't have a team that is well aligned and you have them rowing or pulling in different directions, teams are the product of their individual vectors.
And so if they're all pulling in different directions, we tend to not see very much overall progress. And so while we very much cherish and embrace the individual outcome, we need to make sure that everybody is properly aligned. And this really keeps us balanced here. The third core value that we have inside of Ascend that we've found tremendous success with is: "evolve with intent". Evolve with intent is specifically designed to keep us humble.
As we embrace solving what are some of the hardest and highest impact problems around automation for data pipelines, we have found, time and time again, that there are really similar patterns we could follow. We have found prior art and operating systems and databases and data science domains and distributed systems. We are certainly not the first to solve this category and style of problems. We may be the first to apply it and solve it to data pipelines in this way, but there's so much to learn from others. And so for us, when we take this core value to the max extent here, Where we really try and apply this is with all humility, embracing the fact that innovation is expensive. And so for us, when we apply our time and energy and effort to go and innovate, we want to be students of what has been done before, back to our polymath notion.
We want to deeply think about how can we apply our effort to the maximum impact, where it is unique and differentiated and will profoundly move the needle for our customers. So this core value truly helps us stay aligned to those principles. And then the last one. I think this is a really important one for for engineering teams in general, which is: "build for 10 x plan for 100x". We've described this in many ways in the past. Oftentimes think of this as: you may be building for the next horizon, but you kind of want to have a plan for the horizon after that.
And the reason why is, I think, pretty simple. It's really easy to make some short-sighted decisions to achieve quick wins. It's a lot harder to build a long-term stable foundation for startups to survive and thrive on. One of the things that we really find valuable in finding the balance between these two is knowing where you want to go, having a vision. Knowing what you're building towards.
For those who are interested in what we're doing in that regard at Ascend I have my talk tomorrow afternoon. I'd love to tell you more about it, then. but once you know where you're building towards, how do you construct a path to there and how do you put in the right balance of long-term vision with short-term powerful capabilities, that again deliver impact for your customers. This is one of the areas where we've put a tremendous amount of time and energy and investment at Ascend into to ensure that we can get the right features, the right capabilities for our customers, in the short term, to make sure that we solve for the challenges and the opportunities that you all are looking for today, while still being able to construct a longer term strategy that will deliver significant value for you in the years to come. When you look at things like our Gen2 architecture and our investments around multi-data planes and the data mesh itself, all of these have been part of a multi-year strategy, really build upon a powerful framework, that will continue to serve the needs of our customers for multiple years to come. So it's with these four core values that we have found reinforce our engineering best practices and really drive this this overall culture for how to maximize the impact for our engineering team.
And with that, I wanna thank you all for your time. Really want to thank you all for attending day one of our sessions. It's been a ton of fun getting to listen to all the sessions. I hope everybody else has had an equally fun time.
2023-05-22 06:55
