Navigating the Data Tools Landscape: A Developer's Essential Guide
Guide to data tools landscape for developers
I joined Deepnote without a data background and quickly realized how vast the ecosystem is. This guide breaks down the four main data professions, from analysts to engineers, and explains the data lifecycle. I aim to help software engineers understand the buzzwords, tools, and workflows they encounter when working in data teams.
The data field revolves around data (no surprises here). Data big and small, ugly and beautiful. It all starts with getting it from somewhere, processing it somehow, and then putting results somewhere. That's really it, thank you for coming to my TED talk.
- aleda145
As a data engineering person I can say that this is a great write up!
Some thoughts:
A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/
I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with pandas.
In my network everyone is talking about DuckDB. As long as you are under a 1TB it will have everything you need. I think most people should start with that vs locking themselves into something like Snowflake
- jbonatakis
> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
- otter-in-a-suit
This is a fantastic primer.
A few suggestions:
- Add a "last updated" note, since this space changes often (see the prefect/dagster situation that just happened)
- Add a note about MCPs and other LLM-driven tools and features are becoming more and more important (e.g. hex.ai or the various MCPs shipped with some of the tools you mention, such as OpenMetadata)
- Maybe organize the various tools you mention briefly by their license/model (lots of them can be self hosted, some are SaaS only), since a fully self-hosted data platform is (at least for now...) very much feasible
I also wish more people would talk more about the "engineering" part of "data engineering". I've seen way too many people who claim a title like "data engineer" but lack the fundamentals of building software and are really just copy-paste scripts together.
What I'd love more DEs to think about are things like {unit,integration,e2e,performance} tests, deployments, infrastructure, networking, monitoring (you do touch on that), and all the other things a regular SWE is expected to have at least basic competency in at a certain level. For instance, tools like dbt natively support tests, but people need to write them. Or how you don't have to click-ops Airbyte, there's a terraform provider etc.
- di1eep
This is a great article covering the ecosystem but has not covered one of the key advancements in this area in the past 40 to 50 years., which is IVM (incremental view maintenance).
The topic could start from database “views” and how analytical queries are basically different views of the same underlying data and how IVM solves the batch+stream unification and real-time queries over data elegantly. Tying change data capture into the picture as well.
The company “Feldera” and its founding folks are the leading researchers/experts on IVM and it’s becoming increasingly relevant for all kinds of workloads.
- chrisweekly
Great post! Also, I dig your site -- it's attractive and highly usable, and the "personal" toggle in the footer is a clever affordance I haven't seen before for separating professional content.