Composio · AI Product Ops take-home

100 apps. Which ones can be agent toolkits today?

An agent researched all 100. Auth scheme, whether an outside developer can actually get a credential, API surface, existing MCP, and a buildability call with evidence behind each one. Then I graded it against 100 labels I wrote by hand, and against Composio's own catalog. Findings, agent, and the proof it's trustworthy are all on this page.

The answer

Five things the rows don't say on their own

Numbers come from the shipped pass. All of it is filterable in the table and callable over MCP.

Gap analysis

What Composio already ships, and what it doesn't

I joined the research set against Composio's live catalog through the v3 API. That turns a table into a build queue. What an engineer starts tomorrow, and what's a partnerships call instead.

The build queue

Apps missing from the catalog, ranked. Score = credential access (40) + API surface (25) + auth effort (15) + evidence confidence (10) + agent-readiness (10). Hover a score for the breakdown. The weights are my judgement, written down so you can argue with them.

The matrix

Ten categories, and how open each one is

Sorted by the share of apps where a developer can issue their own credential. Every cell prints its number, so colour never carries meaning alone.

Auth: these 100 vs Composio's whole catalog

The 100 skew enterprise by design, and it shows.

These 100 All Composio toolkits

How an outside developer gets in

Open at the top, closed at the bottom.

The findings

All 100, with the evidence behind each row

Every judgement carries the URL it came from. Rows I graded by hand are tagged gold. Rows a real browser visited are tagged browser.

The agent

Six passes. Four exist to catch the other two lying.

About 900 lines of Python. Pass 1 is the control, not a stage. It's what this assignment looks like if you just ask a model, and it's here so the final number has something to beat.

Where a human was required

What it cost

Measured on the final grounded run over all 100 apps. The control pass, the browser pass and the API probe aren't counted, and neither are the runs I threw away.

Verification

How I know this is trustworthy, and where it isn't

Three measurements, kept separate on purpose. 20 apps I labelled by hand from primary sources before writing any grading code. A cross-check against Composio's shipped auth config on every app it already covers. And a real browser on the pricing pages, for the one field docs never state.

Accuracy per field

Same 20 apps, same 100 labels, every stage.

Is the agent's confidence worth anything?

Bucketed by what the agent claimed for that field before it knew the answer.

Every label the shipped pass still got wrong

Nothing filtered. Abstained means it said unknown instead of guessing, which is the better way to be wrong.

Cross-check: Composio's own catalog

Browser evidence for the gate

The loop that touched the APIs instead of reading about them

Constraints and honesty

What defeated it

For agents

This page is also a tool

Composio turns apps into things agents can call. Handing in research about that as a PDF would be odd. The dataset behind this page runs as an MCP server, so you can ask it questions instead of reading it.

Connect it


      

Five tools: search_apps, get_app, build_queue, patterns, verification. Try: "which finance apps are gated, and what is the blocker on each?"

Or just take the data