AI erased the data team's backlog — now the hard part is trusting the answers
The Dawn of the Data Intelligence Engineer
MotherDuck's lead data engineer has no backlog for the first time in his career, but he's busier than ever: his job shifted from computing correct numbers to verifying the numbers an LLM or agent produces. The company proposes a new role, the Data Intelligence Engineer, who automates ingestion and modeling with agents, curates shared context and ontology, runs evals, and stays accountable for the surfaces the business relies on. Supervision should scale with each agent's blast radius.
Data eng, regardless of tooling or who does it, is an accountability function.
- agenticfish
It's frustrating how the data discipline is split up into so many different job titles. I regularly have to work with data engineers, data analysts, analytics engineers, data scientists, machine learning scientists, and machine learning engineers. Each view their role really narrowly, so you need to pull in a lot of people to get something done. It's often easier to just do the work yourself.
I get that some specialisation is useful, but I would like to see more T-shaped data folks.
- aleda145
I think there are big risks with just labelling it as "wild west" and let users do whatever they want there.
It's always been possible to massage your data to fit whatwver case you want to make. Previously you had to go through a data analyst and convince them to let go of their principles. Now you can just let an agent figure it out.
So how do you make sure insights are actually shareable (and trustable) with other parts of the company?
I think just exposing the metrics layer is probably the answer here.
It's kind of funny that the industry has tried to democratize data for so long, but now we scramble to restrict access to protect users from making incorrect decisions.
- zodwick
While in general I agree with most of what's said, some of these feel like "Horseless Carriages".
Like for example
>It doesn't know how your company counts revenue, or which of several similar-looking tables is the real one (it is revenue_cpx_new_revised_v2_use_this_one )
This assumption mostly only holds for non-clean data ( which is mostly a result of humans being lazy and backlog being clogged up ). As AI starts building more of the pipelines upstream - this becomes a non-issue.
Even for existing pipelines, I'd expect fixes to move upstream: when an agent encounters ambiguity at query time, the useful output isn't an edit of the context,but a PR that fixes it.
- epgui
I’m here using Opus 5.5 and Fable 5.2 and I wouldn’t say the code is generally very good.
- moezd
That's just data engineer enhanced with agents.