Google is raising the standard for testing AI coding tools with Android Bench 2.0.
The new benchmark tests how well AI models handle large Android development projects. These projects can take human developers several days or even a full week to complete.
The benchmark follows the structure of the Harbor framework. It moves beyond simple bug fixes and small code changes. Instead, AI agents must handle several connected engineering tasks.
These tasks include updating software dependencies, building applications from scratch and adding new features. They also include converting cross-platform applications into native Android apps.
AI models face tougher coding tests
Earlier Android Bench tests focused mainly on small changes to existing code. Leading AI models often achieved pass rates close to 91 percent on those tasks.
Android Bench 2.0 tells a different story.
The new tasks require AI agents to work on projects for much longer periods. They must understand the existing code, make several changes and ensure that the final application still works.
The highest pass rate on these multi-day tasks is currently around 28 percent.
OpenAI’s GPT-6 Astra currently leads the public leaderboard for the new benchmark.
The results show that completing a small coding task is very different from managing a large software project.
Google changes how it scores AI coding agents
Google also changed the way Android Bench measures performance.
The earlier system used a simple pass-or-fail score. This approach could give an AI agent a zero even when it completed most of a project successfully.
For example, an AI agent could convert 40 screens to Jetpack Compose, organise database tables and complete 90 percent of the required features. However, one failed test could still cause the entire task to receive a failing result.
Android Bench 2.0 now uses a more detailed scoring system.
The benchmark looks at functional correctness, visual quality and regression prevention. It also removes points when an AI agent ignores instructions or breaks important project rules.
This approach gives developers a clearer view of how much work an AI model actually completed.
AI performs better when creating new code
The benchmark also shows a clear difference between creating new software and maintaining existing projects.
AI models tend to perform better when they create new files and features.
They face more problems when they must change an existing codebase. Large projects often have complex structures, dependencies and rules that the AI must understand before making changes.
Android Bench 2.0 found that AI agents can handle some large conversion tasks with strong consistency.
For example, models can convert Java classes to Kotlin. They can also replace Retrofit with Ktor and configure ViewModel architecture.
Some of these tasks involve more than 125 files and 8,000 lines of code.
However, the results become less reliable when the AI must test the application during runtime.
Runtime testing remains difficult
AI coding agents still struggle with problems that only appear when an application runs.
One example involves dependency injection. An AI agent may create the required code but fail to connect parts of the application correctly.
AI models can also struggle when a project uses a new framework version or an unreleased software library.
These problems show an important limit of AI coding tools.
Writing code is only one part of software development. Developers must also test that code and make sure all parts of the application work together.
Native Android development remains a challenge
Converting a cross-platform application into a native Android app is another difficult task for AI agents.
No model achieved a 100 percent pass rate on the cross-platform tasks tested by Android Bench 2.0.
The leading systems reached a completion score of about 80 percent.
Converting an application involves more than changing programming languages. The AI must understand Android’s architecture and platform requirements.
It must also maintain the original application’s features while adapting them to the Android environment.
This remains a major challenge for current AI coding agents.
The AI agent matters too
Android Bench 2.0 does not only test AI models.
Google also looks at how the models work inside developer tools.
Early tests paired AI models with tools from their respective providers. For example, OpenAI’s GPT-5.6 Sol was tested through Codex, while Google’s Gemini 3.8 Flash was paired with Google Antigravity.
The testing also shows that the design of these tools can affect performance.
Features such as prompt caching and smaller tool windows can reduce the number of tokens an AI agent uses during a long coding session.
This means that the AI model is only one part of the system. The development tools and environment can also affect how efficiently the agent completes a project.
More models join the benchmark
The updated Android Bench leaderboard includes several frontier AI models.
These include Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3 and Qwen 3.8-Max.
Google also plans to test AI models with agents from different providers.
These future tests could provide more information about the role of AI models, developer tools and agent systems in complex software development.
What Android Bench 2.0 means for developers
Android Bench 2.0 shows that AI coding has moved beyond simple code generation.
Modern AI agents can already create files, convert code and complete large parts of development tasks.
However, long-running engineering projects remain much harder.
AI agents must understand an entire codebase, manage dependencies, implement several features and test their changes. They must also avoid breaking parts of the application that already work.
The new benchmark gives developers a clearer way to measure these abilities.
For now, the results suggest that AI coding agents can handle significant parts of software development. Human developers still play an important role in reviewing architecture, testing applications and fixing complex problems.
As AI coding tools continue to improve, benchmarks such as Android Bench 2.0 will help show how close these systems are to handling real-world software engineering projects.
