{"uuid": "7ec3dbe3-0265-46df-b908-05c5f730b6eb", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2024-27980", "type": "seen", "source": "https://gist.github.com/ohad6k/586cd3e78282e37c1ecb930534637fb6", "content": "# What agents actually commit\n\nA three step agentic pipeline built for ctaio.dev. It reads commits that AI coding tools\nauthored in public repositories, identified by the trailers the tools write themselves\nrather than by anyone's claim, has an agent find one defensible claim in them, and renders\nthe result as a page.\n\n## The steps\n\n1. **ingest.py** shells out to `gh api search/commits` for five agent tool repos and keeps\n   commits carrying an agent trailer. Deterministic, no model involved. Output:\n   `out/1-ingested.json`.\n2. **transform.py** aggregates those rows into counts first, then hands only the counts to\n   Claude Code over stdin with a fixed question. The agent is asked for one non obvious\n   claim, the number behind it, and what would falsify it. Output: `out/2-facts.json`,\n   `out/2-finding.md`.\n3. **publish.py** renders the finding plus the underlying tables into a standalone page.\n   Output: `out/3-post.html`.\n\nThe handoff that matters is between 1 and 2. The agent never counts anything. Counting is\ndone in Python where it is checkable, and the model is only asked to interpret counts it\ncannot silently get wrong.\n\n## What it found\n\nIn the repositories of the agent tool vendors themselves, trailered commits skew to repair\nrather than construction: `fix` 40 against `feat` 30 across 97 commits, with only 6\ncarrying a `test:` prefix. 67 of the 97 come from repos that are themselves the tools.\n\nFalsifier, stated by the agent and not by me: pull the same prefixes from the same repos\nover the same window excluding trailered commits. If humans sit near the same ratio, the\nfinding is a property of these repos rather than of agent authored work, and it is dead.\n\n## What it does not show\n\nTrailer presence proves a tool touched a commit. It does not prove how much of it the tool\nwrote, and it undercounts every team that strips trailers before merge.\n\n## Three things broke\n\n**cp1252.** `subprocess` on Windows decoded `gh` output with the system codepage and died on\na UTF-8 commit message at byte 31254. Silent on four repos, fatal on the fifth. Fixed with\nan explicit `encoding=\"utf-8\", errors=\"replace\"`.\n\n**claude vs claude.cmd.** `subprocess.run([\"claude\", ...])` returns WinError 2 on Windows\nbecause the executable is `claude.cmd` and Node style shims are not resolved without a\nshell. Same class of bug I fixed in `cabinetai/cabinet` this morning, where `npm.cmd` was\nspawned without a shell and Node rejects that with EINVAL since CVE-2024-27980.\n\n**Prompt truncation through argv.** Passing a 6KB prompt as a command line argument silently\ntruncated it mid sentence, and the agent answered a question it had only half received. The\noutput looked like a model failure and was an argv limit. Fixed by piping the prompt over\nstdin, which is also how it should have been written the first time.\n\nThe third one is the interesting one, because nothing errored. A pipeline that hands work to\na model can fail by delivering a corrupted instruction and still return a confident,\nwell formed, wrong answer.\n\n## Run it\n\n    python ingest.py\n    python transform.py\n    python publish.py\n\nNeeds `gh` authenticated and `claude` on PATH. GitHub commit search is capped at 30 requests\nper hour, which is the first thing that will stop you adding repos.\n", "creation_timestamp": "2026-08-27T12:20:30.879621Z"}