When “AI-First” Became “Oops-First”

There is a seductive idea at the heart of the current AI revolution: if software can write code, analyze data, answer customers, create reports and make decisions, why do we still need so many people doing those things?

It is an especially attractive question for large organizations, where layers of management, specialized teams and repetitive processes can make even simple changes painfully slow.

Meta appears to have tried answering that question at extraordinary scale.

The ambition behind Project OT, or Organization Transformation, was not simply to give employees better AI tools. The idea was much more fundamental: redesign the organization around AI agents, reduce team sizes dramatically, flatten management layers and allow small groups of engineers to orchestrate systems capable of doing a much larger share of the work.

On a PowerPoint slide, it probably looked fantastic. Fewer people. More AI. Smaller teams. Higher productivity. Lower costs. The problem is that organizations are not PowerPoint slides. According to the reporting behind the Project OT story, Meta's experiment began producing a rather uncomfortable set of signals. Internal platform code changes reportedly increased 220% year over year, but only 36% of those changes made it to users as shipped features. Major technical and security incidents increased by roughly 40%, while time spent firefighting rose 70%. Reuters separately reported that Meta ultimately scaled back the transformation after productivity and security concerns emerged.

That distinction between activity and outcomes is perhaps the most important lesson here. AI can make an organization extraordinarily busy. It does not automatically make it extraordinarily productive. Imagine a software engineering organization where AI agents can generate code at ten times the previous speed. Suddenly, developers are producing dramatically more pull requests, refactoring more modules, opening more tickets and changing more infrastructure.

The dashboard is glowing green. Everyone is moving.

But if only a fraction of that work actually becomes valuable product functionality, the organization hasn't necessarily become more productive. It may simply have created a faster machine for generating work that humans then have to review, debug, integrate, secure and maintain.

That is the hidden cost of AI at scale.

The first generation of enterprise AI was largely about assistance. A human asked the question, AI provided an answer, and the human remained accountable. Agents change the equation. An agent doesn't just tell you what to do. It can potentially do it. It can modify code, access systems, trigger workflows, update records and make decisions across multiple steps. That creates enormous potential but it also changes the risk profile completely.

A bad answer from an AI assistant is inconvenient. A bad action from an AI agent can become an incident. That distinction is becoming increasingly important as organizations move from experimentation to execution. Enterprise research increasingly emphasizes that production AI needs observability, governance, security, identity controls and human escalation, not simply a more capable model.

And this is where the Meta story becomes more interesting than a simple "AI failed" headline. AI did not necessarily fail because it was incapable. The organization failed to establish whether the surrounding system was ready for the level of autonomy being introduced. Think of it like replacing a factory workforce with robots. Buying the robots is only the beginning. You also need redesigned processes, safety systems, maintenance procedures, exception handling, monitoring, training and people who know what to do when something goes wrong.

Otherwise, you haven't automated the factory. You've automated the chaos. There is another dimension to this story that is even more important: trust. Employees are unlikely to embrace an AI transformation if they believe the technology is simultaneously being used to measure their every keystroke, track their mouse movements and determine which jobs disappear next.

Even an impressive AI system can fail organizationally if the humans around it stop trusting the transformation. That creates a paradox. Companies adopt AI to make people more productive. But if the implementation creates fear, surveillance and uncertainty, employees may spend less energy collaborating with the technology and more energy protecting themselves from it.

The result is an organization that has more AI but less trust. And trust is not a soft metric. It directly affects adoption, knowledge sharing, experimentation and ultimately productivity. This is why the emerging shift from "token-maxxing" to "value-maxxing" is so important. For a while, much of the AI conversation focused on how many models an organization could deploy, how many tokens it could process, how many employees it could theoretically replace and how quickly it could automate workflows.

The better question is much less glamorous:

• Did the business actually get better?
• Did customers receive better service?
• Did engineers ship more valuable features?
• Did incidents decrease?
• Did employees spend more time solving important problems?
• Did revenue increase?
• Did risk decrease?
• Did the organization become more resilient?

Those are the metrics that matter. The lesson isn't that companies should stop using AI agents. Quite the opposite. The lesson is that AI agents need an operating model, not just an API key. That means giving an agent a clearly defined job, carefully scoped permissions, access only to the information it needs, measurable performance objectives and explicit boundaries around what it can and cannot do. It also means creating an escalation mechanism.

If an agent is 98% confident about changing a database record, that might be fine for a low-risk internal workflow. It is a very different proposition if the action involves deleting customer data, approving a financial transaction or making a regulatory decision.

The more consequential the action, the stronger the control should be. Human-in-the-loop does not have to mean "a human manually approves everything." That would defeat the purpose of automation. Instead, organizations can establish risk-based autonomy.

Let AI operate independently where the consequences are reversible and low-risk. Require human approval where the consequences are material. And create automatic shutdown or escalation mechanisms when the system behaves outside expected parameters. In other words, don't put a human behind every AI action. Put a human where human judgment actually matters.

In April 2026, PocketOS CEO Jeremy Crane described an incident in which an AI coding agent operating in what was intended to be a staging environment encountered a credential mismatch and responded by deleting a Railway storage volume. The result was the loss of the company's production database and backups, temporarily disrupting rental companies that relied on the platform. Railway ultimately restored the data, and the incident prompted the company and platform to rethink guardrails around agentic actions.

The important part isn't that an AI agent made a mistake. Humans make mistakes constantly. The important part is what the system allowed the mistake to do. A conventional software bug might crash a process. An autonomous agent with broad credentials can potentially diagnose the problem, decide on a remediation and execute that remediation, all before anyone realizes that the diagnosis was wrong.

That's an entirely different category of operational risk. The solution isn't to ban coding agents. The more sensible response is to constrain their blast radius. Give the agent only the permissions it needs. Separate staging and production credentials. Prevent destructive operations unless explicitly approved. Log every consequential action. Make the agent explain or record what it is attempting to do. Establish rollback mechanisms. And introduce a human approval gate for irreversible operations.

This is what mature AI adoption starts to look like. Not "AI does everything."

Rather: "AI does everything it is safe and qualified to do, and knows when to stop."

That distinction will become increasingly important as agents move beyond chat windows and into enterprise systems. The future organization may indeed have fewer people doing repetitive work. But it will probably need more sophisticated people managing the systems that perform that work. The irony is that aggressive AI automation can actually increase the value of human judgment. When machines perform routine tasks, humans become more important at the boundaries: deciding what should happen, determining whether an outcome makes sense, handling ambiguity, managing exceptions and understanding consequences that aren't visible in the data.

The winning organizations therefore won't necessarily be the ones that eliminate the most human roles. They will be the ones that figure out the optimal division of labor between humans and machines. That is a much harder problem but it is also a much more valuable one.

Meta's Project OT story is therefore less a cautionary tale about AI replacing people and more a warning about replacing organizational design with technology deployment. You cannot simply remove 60% of a team and assume agents will absorb the missing capability. You have to understand what those people were actually doing. Some work is repetitive and highly automatable. Some work exists because somebody needs to resolve exceptions. Some work is invisible until something breaks. Some knowledge lives entirely inside people's heads.

And some of the most valuable organizational capabilities, judgment, context, institutional memory, accountability and leadership, are particularly difficult to automate. AI can accelerate all of these systems. It can also amplify their weaknesses. That is why the next phase of enterprise AI will be less about asking, "How much work can we give to an agent?"

The better question is: "How much autonomy can we safely give an agent, under what conditions, and how will we know when it goes wrong?"

That is where guardrails stop being bureaucracy and become infrastructure. The AI era won't be won by the organization with the most agents. It will be won by the organization that can make those agents useful, measurable, secure, observable and trustworthy at scale. Because the ultimate measure of an AI transformation isn't how many humans disappeared from the org chart. It's whether the business became better. And if your AI transformation requires a growing army of humans to clean up after the AI, congratulations, you didn't replace the workforce.

You created a very expensive new workforce called AI firefighters.

#AI #ArtificialIntelligence #GenerativeAI #AIAgents #AgenticAI #EnterpriseAI #DigitalTransformation #AILeadership #ResponsibleAI #FutureOfWork #TechnologyLeadership