When I was a data scientist at Airbnb, I once received an ad-hoc request to identify Olympians that had accounts on Airbnb. No context given, just a colorless request in Slack. After a day of painstakingly scraping the web and munging data in SQL, I dumped the data into a table in our data warehouse, sent it off, then promptly moved onto other work.
Months later, I discovered that this work had contributed to the launch of a new business segment: Olympian and Paralympian online experiences. I felt proud that I’d been able to directly contribute to the business in this way, but I was bothered that no one had thought to let me know that the launch had occurred, let alone include me any more deeply in the launch process. I’d never felt more viscerally that my role in the product-building process was that of data vending machine: request in, data out.
It’s precisely this sort of transactional quality to our work that leads many of us to leave the industry. Put bluntly, it’s a big reason why it feels like analytics sucks sometimes. But it really doesn’t have to. In what follows, I’ll argue that a core problem with our workflow is that tools used for analytics — the SQL IDE, dashboards — are making this problem worse due to their inherently transactional nature. We need a new surface area built for sharing insights.
So why does the Olympian scenario I shared feel so bad? There are three key ingredients we need to feel motivated and fulfilled in any line of work: autonomy, mastery, and purpose. But all three are often absent in analytics:
And so I did some soul searching to figure out why this was. While at Airbnb, I tried to evangelize better processes around stakeholder alignment and delivery, better centralization of work, earlier involvement of analysts in conversations. But people were generally resistant to the scope creep this represented. It wasn’t that analysts didn’t want to be involved in the decision-making process, but few wanted to fight tooth and nail for a seat at the table, particularly when we’re barely keeping our heads above water to begin with. When you have a million queries to write, you don’t want to double your work by getting them peer reviewed and pushing them to git. Transactional behaviors were too entrenched, and data scientists and analysts were pigeonholed into their own little worlds.
This not only leads to low job satisfaction, but high levels of turnover: those who crave greater purpose (and to some extent, autonomy) become PMs. Those who crave mastery become increasingly technical, often moving into more engineering-heavy roles, laying claim to that thin slice of the world where we have full ownership and autonomy.
At the root of it, I realized the culprit was not process per se, but the way our existing tooling had inevitably shaped our processes. The way that we interact with data today is through SQL IDEs and dashboards, but they’re both inherently transactional in nature. Both expose the data itself, without room for qualification, perspective, discussion. And the subtle implication inherent in our usage of these tools is that our greatest value add as analysts is to simply expose the data rather than interpret it.
But analysts aren’t hired to simply get data in front of stakeholders. We’re hired because data is not always plainly comprehensible. Data is nuanced. In the right hands, data can be leveraged to construct narratives that can influence a company’s most consequential decisions. The analyst’s job is to help tease out these narratives, and to help make them understandable for the business. And the way that we share our work should reflect that.
The process change we finally landed on at Airbnb was stricter adherence to sharing SQL-based insights within docs. We put our SQL in context-rich docs and linked to our work from a central team homepage. This helped us share their work in a more meaningful, discoverable way, and it made our work more visible to stakeholders. And this workflow, when leveraged, was a huge improvement over the transactional question-response interaction that I’d often fallen victim to.
My one qualm with this, however, was that the friction was a bit too high, so only the most important reports were written up. Most ad-hoc work and insights were still buried in Slack. This is why we’ve built Hyperquery — a SQL-enabled doc workspace that makes it easy for data analysts to share their insights and analyses as soon as they’re done. With Hyperquery, data analysts can easily add perspective and nuance while pulling data. This not only enables you to easily elevate your work with added context, but it has the secondary effect of encouraging stakeholders to view you as a thought partner, not a SQL monkey.
The thin slice of the world where data analysts live is neither data- nor analytics-driven. It’s a transactional world, where data is sliced and diced according to the whims of whoever needs it at the moment. The data might be used to inform an executive decision, but it’s too often divorced from the actual business context that it was intended to support. We need to seriously reconsider how to make analytics part of the business process, not just an afterthought, and rethinking where we do our work is a great first step.
Tweet @imrobertyi / @hyperquery to say hi.👋
Follow us on LinkedIn. 🙂
To learn more about hyperquery, visit hyperquery.ai.
