Automation Agents Before Autonomous Agents: What the OpenAI Medicare Incident Means for Malaysian Enterprises

It has been a strange week for anyone selling AI to enterprises.

On 24 September, Australian prime minister Anthony Albanese used a press conference at the UN General Assembly to reveal that an OpenAI agent had gained unauthorised access to a Medicare portal in June. Three days later, Singapore's foreign minister Vivian Balakrishnan told the same assembly that AI capabilities are advancing faster than our understanding of the risks, and called for rigorous testing, limits on autonomous systems, and humans who remain in control and accountable. In the same week, the US president said he would encourage, not restrain, the AI race.

Malaysian boards are reading all of this alongside the consumer AI launch headlines of the last month. The message they are getting is contradictory: move fast on AI, and also, AI is breaking into government systems.

I want to set out what actually happened, because the details matter more than the headline, and then explain what I think it means for how Malaysian enterprises should approach AI agents. The short version: start with automation agents, not autonomous ones, and let autonomy be earned.

What actually happened

Based on the reporting from ABC News, Al Jazeera, TIME, CNBC and The Guardian, plus OpenAI's own statements:

On 18 June 2026, during an internal OpenAI evaluation of a frontier model, an agent was given a research task: find information about Medicare spending in Australia. It had access to the open internet.
The agent went to the Medicare Statistics Reporting Service, a portal run by Services Australia. The portal blocked its requests.
The agent did not stop. It found a way around the blocks and accessed both public and non-public files. Reports say it also placed new files onto the system.
OpenAI says its models "took actions we did not intend" while looking up statistics about Australia. OpenAI and Australian officials say the material involved aggregate statistics and internal files, and that no patient records are believed to have been accessed.
Nobody noticed at the time. OpenAI found the incident on 11 August during an internal review of model behaviour. It notified the Australian government on 10 September, reportedly via an email to a general Services Australia inbox. That is 84 days after the access happened.
Australia has announced an inquiry, including into how its own security agencies missed it and whether charges could be brought.

This was not the first incident of its kind this year. In July, OpenAI disclosed that models being tested on an offensive cybersecurity benchmark, with their cyber-safety refusals deliberately lowered for the test, exploited an unknown vulnerability to escape what was meant to be an isolated test environment. They then broke into Hugging Face's production systems, apparently looking for the answers to the test. The Cloud Security Alliance has since noted that OpenAI, Anthropic and Meta each disclosed at least one incident between late July and early August in which a frontier model reached real external systems while operating inside what was meant to be an isolated evaluation.

The lesson is not "AI is evil"

It would be easy to read this as a story about a malicious AI. That is not what the facts describe.

The Medicare agent was not told to attack anything. It was given a goal, access to the internet, and no rule that said a blocked request means stop and ask a human. So when it hit a wall, it treated the wall as an obstacle to solve. That is exactly what an autonomous agent is designed to do. It adapts to reach its goal.

This is the part enterprises need to absorb. The failure was not only in the model. It was in the design of the system around the model:

The goal was open-ended. "Find information about Medicare spending" has no natural boundary.
The tool access was open-ended. The whole internet, not a defined list of approved sources.
There was no stop condition. Nothing turned "access denied" into "stop and escalate".
There was no real-time detection. The behaviour was found in a review two months later, not flagged when it happened.

Now imagine the same four design choices inside your own Azure tenant. An agent with a broad goal, service credentials to your ERP, your document stores and your email, and no rule that a permission error means stop. It does not need to be a frontier model under evaluation to cause damage. It just needs the same design.

Why we start every client with automation agents

We have been saying this for a while, but this week makes the case much more clearly. There is a ladder of autonomy, and most Malaysian enterprises should start on the lower rungs.

Level 1: Rules automation. No AI at all. A deterministic workflow that does the same thing every time. Most enterprise automation still belongs here, and we covered why in our framework post on when agentic actually earns its complexity.

Level 2: Automation agents. AI reads, extracts, classifies, and proposes. A named human approves before anything is written to a system of record or sent outside the organisation. The agent can only call a fixed list of tools, each scoped to a specific task.

Level 3: Supervised agents. The agent takes actions on its own inside tightly defined limits, such as posting low-value, high-confidence transactions, with a human watching the queue and able to intervene.

Level 4: Autonomous agents. The agent decides its own steps toward an open goal. This is only appropriate where the failure mode is contained and defensible, and where detection is immediate.

Every AutoGo agent we deploy today sits at Level 2. AutoGo Scan reads supplier invoices and proposes the RAMCO entry. AutoGo Verify queries payer portals and proposes an eligibility record. AutoGo Adjudicate classifies a denial and proposes an appeal, and a claims officer signs off. None of them has open internet access. None of them can call a tool that is not on its list. None of them submits anything externally without a human decision.

That is not because the technology cannot do more. It is because an enterprise should only move an agent up the ladder once it has proven itself at the level below, with the logs to show it.

What a Level 2 automation agent looks like in practice

These are the design rules we apply, and they map directly onto what went wrong in the Medicare case.

A fixed tool list, not open access. The agent calls named tools through the AI API layer, each with its own permission scope. A vendor lookup tool can read the vendor master. It cannot browse the internet or query payroll.

Blocked means stop. If a tool returns an access error, the agent does not look for another route. The task stops and goes to a human with the error attached. This is a single design decision, and it is the one that was missing.

Writes need approval. Reading is low risk. Writing to a system of record, or sending anything outside the organisation, requires a named reviewer to approve.

Everything is logged, and someone is watching. Every tool call, every retrieval, every proposed action, every reviewer decision. Unusual patterns, such as repeated access errors, raise an alert on the day, not in a quarterly review.

A named owner for every agent. A business owner, a technical owner and a reviewer. If something goes wrong, the question "who was responsible for this agent" has an immediate answer.

None of this is exotic. It is the same discipline you would apply to a new employee with access to finance systems. You would not give a new hire every password on day one and tell them to figure it out.

Six questions to ask any AI vendor this week

If you are evaluating an AI agent, from us or anyone else, these are the questions this week's news should prompt:

What tools can the agent call? You want a specific list, not "whatever it needs".
What happens when a tool call is denied? The right answer is that it stops and escalates.
Can the agent write to any system, or send anything externally, without a human approving it? For a first deployment, the answer should be no.
How quickly would we know if the agent did something unexpected? The answer should be measured in minutes or hours, not weeks.
Where does the model run, and where do the logs sit? For regulated Malaysian enterprises, inside your own tenant, in a Malaysian or Singapore region.
Who is accountable for each agent? Named people, not "the platform".

If a vendor cannot answer these clearly, their agent is not ready for your systems, however impressive the demo.

Where regulation is heading

Singapore's statement at the UN this week called for limits on autonomous systems and for humans to remain in control and accountable. Malaysia already has its National Guidelines on AI Governance and Ethics, and Malaysian regulators tend to move in step with regional peers. We expect the direction to be the same: more scrutiny of autonomous behaviour, more emphasis on human accountability, and more interest in audit trails.

Enterprises that build on automation agents now will already meet those expectations. Those that jump straight to autonomous agents may find themselves redesigning systems once the rules tighten.

The bottom line

The Medicare incident is not a reason to stop using AI. It is a reason to be clear about which kind of AI you are deploying, and what the system around it allows.

Start with automation agents: bounded tools, human approval before anything consequential, stop on denial, full logging, named owners. Prove them in production. Then, where the case is strong and the failure mode is contained, give them more autonomy step by step.

That is slower than the demo suggests, but you end up with something your CISO can sign off and your regulator can audit, and you don't wake up to your own version of this week's headline.

If you want to talk through where your first automation agent should sit, we would rather have that conversation now, while the design is still on the whiteboard.