We Tested an AI Tool That Claimed to Be 'Read-Only.' It Wasn't.

Before we let any AI agent touch a real business process, we test it first. Not because we do not trust AI. Because we do not trust marketing claims, even the ones written into the tool's own instructions.

Here's a real example of why that matters.

The tool said it only reads. It could also write.

We were testing a review tool meant to give a second opinion on work before it ships, the AI equivalent of asking a colleague to check your work before you send it. The documentation for this tool described it as read-only. It reads, it comments, it does not change anything.

When we actually ran it and checked what permissions it had, that description turned out to be wrong. The way it was set up, the tool could also write and delete files. The instructions said one thing. The actual behavior said another.

This is the exact kind of gap you cannot catch by reading a product description. You catch it by testing.

It could see things it should never have seen

The bigger issue was what the tool had access to while it was working. It was pointed at a folder to review, but the permission setup did not block it from also reading other files sitting in that same folder, including a file holding sensitive access keys and login credentials for several outside services.

Those are the digital equivalent of the keys to your business. A password manager, a bank login, the codes that let your website talk to your email system. If a tool can read a folder, and your sensitive information happens to live in that folder, it can read that too, whether or not anyone intended it to.

We fixed this by testing in a folder with no sensitive information in it at all, and by explicitly blocking the tool from writing or saving anything outside the review itself. The fix was simple once we found the problem. Finding the problem is the part that actually takes discipline.

Even reviewing a fix can leak the exact thing you are trying to protect

Here is the part that surprised us most. Imagine you are removing a customer's information from your system, deleting an old record you no longer need. The change itself is good. It is the right thing to do.

But when an AI tool reviews that change, it sees the entire before-and-after picture, including the data that is being removed. In our test, a review like this nearly exposed a real list of customer email addresses, not because anyone did anything wrong, but because the record being deleted has to be shown in full for the tool to check the change makes sense.

Deleting something safely and reviewing that deletion safely are two different problems. Most people only think about the first one.

What this means if you are using AI in your business

None of this means AI tools are unsafe to use. It means their own descriptions of themselves are not a substitute for checking what they actually do with your real data.

Before you give any AI agent access to your files, your inbox, or your records, ask three plain questions:

• What can this actually read, not just the folder it is supposed to work in, but everything sitting next to it?
• Can it write or change anything, even if it is advertised as read-only?
• If it reviews a change involving customer data, what does it see in the process?

If you cannot answer those confidently, you are trusting a label, not a test.

How we handle this for the agents we build

Every agent we set up for a business goes through this same kind of check before it ever touches real data. We test what it can actually access, not what the documentation claims, and we scope its permissions down to exactly what the task needs. Nothing more.

This is part of why an Automation Audit starts with mapping your actual processes and data before anything gets built. It is also why we are upfront that AI agents make judgment calls, and judgment calls deserve scrutiny before they run unsupervised in your business.

Curious what agents actually running in production look like once they have been through this process? Take a look at the agents we have built, or get a free audit on your own setup before you hand anything real access to your business.