AgentRealTaskBench
About
Every assistant promises to run your life, and none can be compared, because each demo shows a different task. ARTB writes the tasks down, publishes them, and runs the same ones against every product we can get into.
How a round works
- 01
Write the task
Each task is something a real person would say, with rules you can check against the evidence.
- 02
A person runs it
A tester runs every product on test accounts, records the whole run, and sends one neutral nudge at most.
- 03
Two blind reviews
A second reviewer scores from the evidence alone. A third settles disagreements.
- 04
Publish everything
Vendors check facts 48 hours ahead. Recordings, parameters and corrections are all public.
House rules
- No SandBaseand no directly competing infrastructure products.
- As shippedNo tools or plugins added by us.
- No moneyNo sponsors, no paid placement. We pay for every subscription.
- Blank means untestedWe never guess a score.
Who runs it
Founder of SandBase, which builds infrastructure for AI agents. He uses these assistants every day and owns ARTB's tasks and verdicts.
Think a score is wrong?
Or shipped a fix since we tested? Send task feedback and we will retest.