An agent reports success when its own code path finishes, not when the work lands in the destination system. Those are two different events, and the gap between them is where most production agent failures hide. The fix is to stop asking the agent what happened and start asserting on the destination — read back the channel, the row, or the record, and confirm the thing you asked for actually exists.
We learned this the hard way. We evaluated eight agent platforms on free or trial tiers, with identical documents and prompt, no vendor help, and a 60-minute cap per platform. Only Zapier Agents and Shogo completed the Slack write. Six did not: n8n could not finish the bot-invite step; CrewAI and LangGraph reported connected but delivered nothing; Stack AI, Mastra and AgentScope never reached an action-ready no-code prompt. LangGraph’s result includes two attempts totaling 48:50; the others had one. These findings describe this evaluation, not universal capability.
The failure that looks like success
Our test used one agent, five questions from uploaded documents, and one real action: post in Slack. The cap was 60 minutes per platform; LangGraph took two attempts totaling 48:50.
Two platforms completed the action. Six did not, and the six did not fail the same way.
The visible one was the easiest to trust. n8n answered all five questions correctly, then asked for a Slack token. We supplied it. It came back saying a bot had to be invited to the channel as a separate step, which we could not complete inside the time limit. Annoying, but honest: at every point we knew exactly where we stood, and the failure arrived with a message attached.
The other two were the problem. CrewAI and LangGraph both reported the Slack integration as connected, ran to completion, and delivered nothing. No error, no failed step, no message in the channel. If we had been running these unattended — a nightly report, an escalation trigger, a status digest — nothing would have told us anything was wrong. The job would have reported green every night while the channel stayed empty.
Here is the part worth sitting with: delivery was orthogonal to every other score in the test. n8n was perfect on retrieval and still failed the write. CrewAI connected to Slack with no friction at all and never arrived. Whether a platform could read our documents told us nothing about whether it could act on a real system.
We had built the whole test around retrieval because that is what we thought we were evaluating. Acting turned out to be the harder half, and it is the half that nothing in the platform’s own dashboard will flag for you.
The five ways an agent fails
Every failure in our run landed in one of five categories, but they are not equally visible. Setup friction and refusal tend to be obvious; retrieval depends on whether you know the source; hallucination and delivery can fail silently.
1. Setup friction
What it looks like: You never reach a prompt. The login hangs, creating a first flow throws an error, or the product turns out to be a library rather than an application.
How it fails loudly or silently: Loudly, usually — you are standing there when it happens.
How to catch it: Before evaluating anything, get one agent to a working prompt in one sitting. If you cannot, you have your answer and no other score matters.
Where it showed up in our run: Three of eight. Stack AI never got past signup — the login screen did not finish loading across several sessions on a clean browser. Mastra and AgentScope were not no-code products at all; AgentScope opened straight into a GitHub repository. None of the three ever reached a question, so they have no retrieval or hallucination score to report. A fourth, LangGraph, did get running — but it is a Python library rather than an application, and we ended up working through LangSmith, its observability product, re-uploading our documents across two attempts that totalled 48:50.
2. Retrieval
What it looks like: The agent answers confidently from the model’s prior knowledge instead of from your documents.
How it fails loudly or silently: Usually loudly, as long as you know the answers. You will only catch it if your test includes a fact the model could not guess.
How to catch it: Ask a question whose answer exists only inside the documents, and a second one whose answer exists nowhere. A platform that gets the first right and declines the second is behaving correctly. A platform that answers both is guessing.
Where it showed up in our run: A clean split. Zapier and n8n both scored five of five. Shogo scored five of five but missed one on the first ask and only produced it alongside the following question. CrewAI scored zero of five and classified a ringer question as an IT query before saying it could not find anything.
3. Refusal and escalation
What it looks like: The agent either acts when it should stop, or it stalls when it should act — asking you to confirm its understanding instead of doing the work.
How it fails loudly or silently: Loudly, but it is easy to misread as politeness.
How to catch it: Give it one instruction that should be refused outright, and one ordinary instruction that should be executed without a confirmation round-trip. Both behaviours matter.
Where it showed up in our run: CrewAI kept asking us to confirm its understanding was correct rather than acting on it. On the ringer question, everything that failed declined to answer, which is the correct behaviour and worth crediting.
4. Hallucination
What it looks like: The agent answers a question that has no answer in the documents. Not a wrong answer — an invented one.
How it fails loudly or silently: Silent by definition. It looks exactly like a correct answer.
How to catch it: Always include one question with no answer in the source material. This is the single cheapest test you can add, and it is the one most evaluations omit.
Where it showed up in our run: Exactly once. We asked each platform a question about a recent sporting event — nothing in our documents mentioned it, and no reasonable document would. One platform answered it. Everything else that got that question wrong got it wrong by refusing. LangGraph got it wrong by knowing, which is retrieval not constraining generation, and it is the failure mode a demo will never surface.
5. Delivery
What it looks like: The agent completes its reasoning, reports the action as done, and nothing arrives.
How it fails loudly or silently: Silently, which makes it the most dangerous category here.
How to catch it: Do not read the agent’s transcript. Read the destination. The section below is entirely about this.
Where it showed up in our run: Six of eight did not complete the write: three never reached an action-ready no-code prompt, n8n stopped at Slack’s bot-invite step, and CrewAI and LangGraph reported success without delivery. Only the latter two are silent write failures; the other cases do not establish a dropped write. Only two platforms completed an action against a real system.
Why “connected” is not a test
“Connected” is the most over-trusted word in agent tooling, and it usually means the weakest of three separate things:
- A credential exists. A token was stored. Nothing has been called with it.
- A scope was granted. The token can do something, not necessarily this thing, in this place.
- A destination is reachable. This specific bot can post into this specific channel right now.
An integration that reports “connected” has typically confirmed only the first. Slack is a clean example because its failure modes are so separate: taking a token proves nothing about whether the bot has been invited to the channel, and a bot that is not in the channel cannot post, regardless of how valid the token is.
This is also why the failure is invisible from the agent’s side. The platform asked the integration a question, the integration answered “yes, configured”, and the platform reported success. Nobody asked the channel.
How to verify a write: four tests
There are four techniques that actually catch this, in increasing order of how much they cost you. The first two we now run by default. The last two came from practitioners who replied to our write-up of the run, and both go further than anything we did.
1. Destination assertion correlated on a job id
What it catches: Everything in the delivery category, including a message silently dropped for reasons specific to that payload.
How to run it: Generate a unique id at send time, include it in the payload, then read the destination back and look for that id. For Slack, send via chat.postMessage and confirm via conversations.history — match on the id you generated, not on the message text, because text can be reformatted in transit.
What it costs: One extra read per write. Roughly the same amount of code as the write itself.
The trap: If the write times out, check the destination once before retrying. Retrying blind is how you double-post, and a duplicate is worse than a failure because it is silent in the other direction.
2. A canary plus a per-job assertion
What it catches: A systematically broken connection, on a faster clock than your real jobs would surface it.
How to run it: Post a test message to the channel every morning on a schedule, then query the channel API to confirm it exists. If it is not there within a short window, mark the pipeline unhealthy before any real job runs. Alongside that, assert per job — the canary proves the channel works, not that this job’s payload landed.
What it costs: One scheduled job and one read. Cheap enough that there is no good reason not to.
The trap: The canary alone is a false sense of safety. It passes while a specific job drops, because the failure was in that payload, not in the connection.
3. Precondition injection
What it catches: The case where the platform reports green because nobody ever tested the unhappy path.
How to run it: Break the precondition on purpose before go-live, then watch whether the agent still claims success. Revoke the bot’s channel permission. Expire the token. Point a write at a closed or already-settled record. Send a payload that exceeds a field limit.
What it costs: An afternoon, once, and it should be a permanent part of your test suite rather than a one-off.
The trap: Almost nothing in a demo state will surface this. If the agent reports success on a revoked permission, you have found the bug that would otherwise have found you.
4. Assert on the record, not on the destination
What it catches: Every write action, not just the one you happened to test.
How to run it: Generalise the pattern. “Assert on the destination” for Slack becomes “assert on the record” for a Salesforce or HubSpot write, an order update, or a payment. Read back the object you claimed to have created and confirm the fields you set are the fields that exist.
What it costs: Design time. It is a convention rather than a component.
The trap: Teams tend to build this for the integration that broke and not for the others, so the next write action ships with the original blind spot intact.
Quick reference: what to assert on
| Failure mode | Fails silently? | Assert on | Cheapest check |
|---|---|---|---|
| Setup friction | No | Time to a working prompt | One sitting, one agent |
| Retrieval | Sometimes | A fact only in the documents | One question, known answer |
| Refusal | No | Both directions | One refused, one executed |
| Hallucination | Yes | A question with no answer | One ringer question |
| Delivery | Yes | The destination | Read back by job id |
| Any write action | Yes | The record itself | Precondition injection |
The eight-platform run
Free or trial tiers, no vendor help, 60-minute cap per platform. LangGraph’s time combines two attempts; DNF entries were not scored for Slack delivery.
| Platform | Time to build | Answers (of 5) | Slack delivered | Verdict |
|---|---|---|---|---|
| Zapier Agents | 15:01 | 5 | Yes | Pass |
| Shogo | 17:33 | 5 | Yes | Pass |
| n8n | 13:14 | 5 | No | Answers pass, action fails |
| CrewAI | 8:15 | 0 | No | Fail |
| LangGraph | 48:50 across two attempts | 0 | No | Fail |
| Stack AI | DNF | — | — | Never reached a prompt |
| Mastra | DNF | — | — | Not a no-code product |
| AgentScope | DNF | — | — | Not a no-code product |
The fastest scored builds were CrewAI (8:15) and n8n (13:14); both failed part of the task. Zapier Agents (15:01) and Shogo (17:33) completed the write. LangGraph took 48:50 across two attempts and did not complete it. Build time did not predict success in this run; it does not show that slower is better.
Time-to-build is the number most comparisons lead with, but it did not predict the outcome here.
One platform we deliberately left out was Make. It requests the same Slack permission n8n did, so the same step is where we would expect it to surface — that is an expectation from the permission model, not a result from this run.
A pre-launch checklist for any agent that writes
-
☐ Identify every write action the agent performs, not just the one you are testing.
-
☐ For each, define what “landed” means in the destination — the row, the channel, the record.
-
☐ Send a unique id with each write and read the destination back to confirm it.
-
☐ Confirm the retry path checks the destination first, so it cannot double-post.
-
☐ Schedule a canary so a broken connection surfaces before a real job needs it.
-
☐ Add a per-job assertion, because the canary does not cover payload-specific drops.
-
☐ Inject at least one broken precondition before go-live and confirm the agent reports it.
-
☐ Keep a question with no answer in the source material, so hallucination stays visible.
-
☐ Read the destination in your monitoring, not the agent’s own success claim.
Frequently Asked Questions
How do I know if my AI agent actually completed a task?
Read the destination, not the agent. Generate a unique id at send time, include it in the write, then query the target system and confirm that id exists. The agent’s own report tells you its code path finished; only the destination tells you the work landed.
Why does my agent say it succeeded when nothing happened?
Because the agent is evaluating its own completion, not the outcome. When an integration reports itself as “connected” or a tool call returns a success code, the agent has satisfied its own condition for done. Nothing in that chain inspects whether the message, row, or record exists.
Can I rely on the platform’s integration status?
No. “Connected” typically means a credential was stored, which is the weakest of three separate checks. A credential can exist while the scope is wrong, and a scope can be right while the specific bot, user, or app is not a member of the destination.
What is the single cheapest test to add?
A question whose answer is not in your source documents. Every agent should decline it. If the agent answers it, retrieval is not constraining generation, and you have found a failure mode that no demo will surface.
What is precondition injection testing?
Deliberately breaking the conditions a write depends on before go-live, then checking whether the agent still reports success. Revoke the permission, expire the token, point the write at a closed record, and see what the agent says. If it reports green on a revoked permission, that is the bug you want to find before your users do.
How is a canary different from asserting on the destination for each job?
A canary proves the connection works on a schedule; a per-job assertion proves this particular payload landed. You need both, because a canary passes while an individual job drops for a reason that lives in the payload rather than the connection.
How do I stop a retry from double-posting?
Check the destination once before retrying. A timed-out write may have succeeded without returning, so a blind retry can create a duplicate. Reading the destination by id first costs one call and removes the risk.
Does this only apply to Slack?
No. Slack is just the case where it is easiest to see, because the destination is human-readable. The same pattern applies to any write action — a Salesforce or HubSpot record, an order update, a payment, a Postgres row. The rule generalises to “assert on the record” rather than “assert on Slack”.
Which platforms handled this best in your test?
Zapier and Shogo were the only two of eight that completed the write. Neither surfaced a missing permission on the platform’s side; we only knew because we checked the channel ourselves. That is the point — you cannot rely on any of them to tell you.
How much does it cost to run this kind of test?
The tests themselves are cheap — mostly a scheduling job and one extra read per write. The cost that matters is time: defining what “landed” means for every write action, and building the destination check once as a convention rather than per integration.
Sources
This limited evaluation covered eight platforms on free or trial tiers, with identical documents and prompt, no vendor help, and a 60-minute cap per platform. LangGraph took two attempts totaling 48:50; other platforms had one. Three did not reach an action-ready no-code prompt and were not scored for Slack delivery. This is not a universal ranking. Confirm current features and pricing with vendors.
Related Reading
