7 Crucial Barriers Between Data Teams and Self-Healing Data Architecture | Towards Data Science
Introduction, AI examples of data engineering revolve around one thing: fixing a pipeline. An engineer opens up Claude Code, pastes some logs, and a pull request is made. Semantics are fundamental here. . . .
Introduction, AI examples of data engineering revolve around one thing: fixing a pipeline. An engineer opens up Claude Code, pastes some logs, and a pull request is made. Semantics are fundamental here. Because when people say “self-healing” what they mean is “self-managing”. The key to success in AI is not defined by manual intervention and interaction — but the absence of it. The dream for data teams is a system whereby data pipelines and workflows generally succeed without any human intervention at all. However, there are barriers that lie in between us and this golden future. Agents require context — fixing a pipeline may be due to a transient error, upstream schema change, or something uncontrollable entirely like a human dropping a table. Experience provides engineering teams with the know-how of how to fix these; context agents are missing. A shift in mindset will also be apparent. The old pattern of “New branch, merge, re-run” is distinctlyslowand notagent-y. Unless we are to change our patterns and allow agents tomergePRs as well, this seems like a large mindset shift is required. Finally, data does not “branch” well. Projects like Lake FS promised to make “Git for data” mainstream, but it is not. I have been writing aboutzero-copy cloningfor years, but it is still not widely used. The distinctions betweencodeanddataare not obvious. In this article, we’ll cover 7 barriers in between the typical data stacks of today and the nirvana of self-healing data pipelines / autonomous data pipelines. Let’s dive in!Barrier 1 | Context and failure recallPipelines can fail for a plethora of reasons, and being able to fix pipelinesperiodis a requirement for an AI system. We can categorise failures into a few broad types:Infrastructure issuesCode issuesData IssuesTransient or third party issuesGenerally, the manner of fixing data requires knowledge of the system. For example, Acme’s Kubernetes Cluster may only be accessible by Mr. Bob, who is the only person who has access to Bob’s special access key hidden in AWS Secrets Manager with a non-standard header. AI doesn’t know about Bob’s key, so won’t be able to fix the cluster. Similarly, Analyst Sophie may know that the right thing to do in Widgets Incorporated is to simply gloss over the fact that sales are reported in multiple currencies, and to manipulate the numbers to be 10% higher than the ones yesterday. AI doesn’t know how to treat the numbers. AI may also not know that to failure handle the internal API, you simply need to try it again between 2. 47am and 3. 12am. These are ridiculous examples, but they illustrate the point that the knowledge to fix these different types of errors often exists within individuals’ heads. It is not enough to speak about “metadata context”. While gathering lineage, logs, code, documentation, and otherwritten-downcontext is undoubtedly imperative, AI is actually pretty good atjust working it out. As Data Folks, we’ve all been in a situation where we (or perhaps someone we’ve spoken to) has thought:“How on earth could I have known that?”At the end of the day, only humans know where the bodies are buried. This entire structure is tech debt and could be broken down with AI. SourceBarrier 2 | Elastic infrastructureConsidering issues of the infrastructure type specifically, I am coining a term “Elastic” infrastructure. “Elastic Infrastructure” does not just scale, but also has an API to manage it. An EC2 instance would not be elastic, as it does not scale beyond a certain point. A Kubernetes cluster on a locked-down machine would not be elastic w. r. t cloud as there would be no API to be managed. The reason is that AI will require access to Infrastructure in order to recover failures from it. SaaS providers should relish this opportunity. SAAS providers necessarily take the management burden of infrastructure from data teams away, for a fee. This is a very AI-friendly approach, but falls down in respect of Barrier 6, which we will get to. Barrier 3 | Operational Agents and Quality DataPete in Finance has overwritten the Supply and Operations Planning Google Sheet for the US again. The international forecasts are broken, and your pipeline is failing. There are 0 rows inus_forecast_dec_v1andforecasts_aggis stale. AI is telling you the connectors are fine but there was no data. It can’t do anything. What is the solution here? Let’s play a quiz. I’ll give you some ideas, and you pick the right answer. Option 1:let AI hallucinate the forecastsOption 2:let AI hallucinate the forecasts in your data warehouse, and re-run the Google Sheet Pipeline laterOption 3: AI tells Pete to upload the damn forecasts!Option 4: there is a warm pool ofrented humans. When this type of pipeline fails, the AI instructs the warm pool to bother Petein personuntil he fixes the pipeline himself, by handOf course, there is no right answer! All options are not great, ranging from bad to ludicrous. In fact, Option 4 doesn’t really require AI at all, but something called teamwork. Quality data is, as ever, the most important thing for a data engineer. Data teams should ask this question when they interview more “How good is your data?”. It is such a determinant of quality of life, it is surprising not to get more of a mention. That is not to say that operational agents have no place — for example, genuine fat finger errors could easily be corrected by an operational agent. For example, let’s say there is a new deal for $10m — perhaps the correct number is $1m. An agent with a Salesforce API Key could easily amend the data, and restart a pipeline. Barrier 4 | Git for DataThe previous example raises an important question, which is “Should AI Agents edit production?”If you’ve experienced multiple Salesforce environments in your career — I hear your pain. But the feature is designed to avoid the situation above. You see, perhaps the account executivehaslanded a whale deal and itisworth $10m. In that case, surely much better for the agent to edit the staging Salesforce instance rather than the Production one?Complex Version of how AI can take branching data in git and then you can automatically recover a pipelineThe above is a high-level rendering of what the process using a git-for-data like approach would work. There is a simple version below. Simple version of an AI WorkflowIn both cases, AI needs a new branch to do its work. That branch needs zero copy clones of the data, it needs a git for data approach, and you need to be able to efficiently “switch in” the data at the end. Simple git for data workflowWithout this structure in place, I struggle to see how AI will be trusted to reliably fix things, without creating a governance nightmare whereby it has write access to production data. In respect of this, companies like Snowflake are well-positioned as they have supported features like zero-copy cloning for a long time. Motherduck also supports this feature. The clearest winner, though — is iceberg. Iceberg supportstime travel, rollback, and git for data. Companies likeBauplanhave built compute engines around iceberg, which make for a nice, AI-friendly experience. AI should be a huge catalyst for iceberg. Barrier 5 | Pervasion through the industrySelf-healing architecture hits a problem when we talk about interoperability. Fivetrananddbtmade a big fuss about open data infrastructure in 2025 — it is not the same thing asopen source data infrastructure, but rather refers to an approach I think is better called theModular Data Architecture, whereby different functions get different tools. An exampleis included below. Modular Data Architecture. SourceThere is no point having a self-healing architecture if the underlying components do not support it. Underlying service providers most provide relevant APIs that support all the tenets of this paper, as well as self-healing functionality themselves for patterns to work. For example, suppose there is a silent failure in an ELT provider, whereby the sub-schema changes; the columns and types remain the same, but the values change. Perhaps now there are currencies reported in Yen, as well as in USD, but the two columnscurrencyandlocal_valueremain. The right thing to do may be to amend the ELT job in its staging environment, verify the rest of the pipeline from that staging data, switch out the data that is now correct, and then finally switch over the erroneously succeeding ELT job. Many ELT tools simply do not provide the APIs to get this functionality. However if you were doing this with a python script you controlled yourself — no problem. This will create massive pressure on the ETL players of today to change their structures or die. This is a massive barrier in between the modular systems of today and true self-healing autonomous architecture. The only other examples would be for the systems themselves to all become independently self-healing, as you would hope that if all parts of a system are self-healing, then so too is the whole. Barrier 6 | Agent Sandboxes and New OrchestratorsThe logical place to run agents that fix things is within an orchestration tool. This is because the orchestration tool has a few things the agent needs. The ability to run any code, and to replay any DAG with any sets of arbitrary parametersThe connections to the different parts of the system the agent may need (remember, an orchestrator orchestrates, so it has access to things)Alerting built-in, with monitoring, recovery, and scalable infrastructureHowever there is one huge enormous problem — and that is security. Companies likeCloudflare have built agent sandboxes. This is because models like Fable (which was recently banned) need sandboxes, as they canbreak out. This is especially the case when under attack fromprompt injection. The dangers of prompt injection when running AI Agents in the same infrastructure as your legacy OrchestratorLegacy orchestration tools are simply not made to handle agents in this way. The security risks are immense. Not to mention AI workloads could tread on the toes of data ones!It is quite clear agents will require access to orchestration frameworks. Whether that is Open AI and Anthropic providing an orchestrator, new age orchestrators with agent sandboxes, or some form of interoperability between the two — something has to give here. Because security. Barrier 7 | Standards for Proxy Servers and Agent DefinitionOne approach to security is to setup a proxy service for agents. Rather than install the secrets in the sandbox, the agent has access to a given number of tools / MCPs. The proxy service is then the only thing that has access to external systems. This means that even if the agent becomes a victim of a prompt injection attack, all it could do is limited by the endpoints in the MCPs it has access to. An illustration of a basic proxy service with an auth server and a credentials DBWhat this proxy service needs to look like is not obvious. MCP is big. Cloudflarereleased Code Mode. If you need to access multiple different endpoints, how the MCP Servers ought to be configured is not straightforward or obvious. Open standards should prevail — any agent looking to interact securely with multiple systems would benefit, from a security perspective, from interactive with a proxy service. These exist today, but in private SaaS tools likeFoundry. Frameworks for designing agents would also need to emerge. In the example above, a single agent requiring integration to hundreds of systems may not be feasible, as the context required to access hundreds of MCPs may be too large. Putting it all together | A Single Pane of Glass for AICollectively, achieving the above would allow data teams to build out a single pane of glass for AI. Context: provides the agents with the information to solve any problemElastic infrastructure: provides the foundation for fixing pipelinesQuality Data: eradicates the human side of the data inputsGit for Data: creates reliability and trust in AIMass Adoption: prevents industry collapseAgent Sandboxes and New Orchestrators: remove legacy architectureProxy Servers: do their best to guaranteee securityThis single pane of glass would allow AI Agents to operate in a secure way. They would execute when they needed to, and would have the context to achieve what they needed to as well. Core data primitives like git for data, elastic infrastructure, and support throughout the ecosystem would turn this from a theoretical idea into a practical reality. Data teams looking to implement autonomous architecture will impose significant pressure on existing vendors to support interoperability. This will exacerbate consolidation, as traditional walled-gardens like Salesforce, SAP, and ServiceNow roll out their own agentic products and data studios, capable of controlling the end-to-end without providing interoperability.