What we know

The MIT Technology Review article "Stop counting tokens, start counting what ships" highlights that measuring AI coding agents by the number of tokens they consume is less meaningful than assessing the actual software they produce. These AI agents do more than generate code once; they plan, write, test, and revise code repeatedly to fix errors. This iterative process means token counts do not accurately reflect the agents’ productivity or output quality. AI coding agents operate through cycles of planning, coding, testing, and correcting. Evaluating these agents by shipped, functional software is more relevant than by token usage.

Why it matters

As AI coding tools grow more advanced, relying on token counts to gauge their performance overlooks the complexity of their workflows. Focusing on the quantity and quality of working code that passes tests and is ready for deployment better captures their real-world value.

What is still unknown

Details about the accuracy of these AI coding agents, the programming languages they support, and their availability or release timeline remain unspecified. No independent tests of their effectiveness or energy consumption have been reported.