Anthropic launched Fable 5 (Mythos), but faced backlash for silently degrading performance on AI research prompts without disclosure, raising trust and reproducibility concerns. Many critics, including researchers and builders, argued explicit refusals would be more defensible. Despite controversy, Fable 5 showed top-tier agentic coding benchmarks, leading Agent Arena and scoring 81.9% on SimpleBench. Distribution expanded quickly—Perplexity added it as an orchestrator, and Apple integrated Claude via Foundation Models. Concurrently, Google released DiffusionGemma, a 26B MoE diffusion LLM under Apache 2.0 that generates text blocks simultaneously, claiming 4x faster output and over 1000 tokens/s; it gained immediate vLLM support. The week also saw shifts toward trace-based agent evals and new agent memory/orchestration tools.
This newsletter highlights FrontierCode, a new benchmark from Cognition that evaluates code mergeability rather than just unit test passing, with top models scoring only 13% on the hardest subset. It covers the rise of 'loops' as an agent control metaphor, improvements in agent ergonomics, and new model releases like Kimi Code and Gemma 4. The article also discusses shifts in evaluation methodology toward real-world telemetry and the ongoing race in consumer AI platforms. Additionally, it notes research directions in continual learning and optimization.
Auriel Wright discusses common failures in reinforcement learning training harnesses that produce garbage data. She identifies three major error classes: stale cache, reward hacking, and false resolution. The post emphasizes that a flaky environment corrupts model training and advocates for traditional software engineering practices in RL research. It provides practical advice for building robust harnesses and suggests that teams should fix harness issues before addressing model problems.
This AI news roundup highlights NVIDIA's launch of the open-source Nemotron 3 Ultra, a 550B MoE model optimized for long-running agents, and Anthropic's internal data showing Claude now authors over 80% of merged code, indicating early signs of recursive self-improvement. Cloudflare acquired VoidZero to strengthen its agent-friendly developer platform, while OpenAI's ChatGPT surpassed 1 billion monthly active users. The update also covers new agent evaluation infrastructure, open image models like Ideogram 4.0, and frontier AI adoption signals including a joint letter on biosecurity screening.
This podcast episode discusses Andon Labs' work on real-world evals for AI agents, moving beyond traditional benchmarks to test models in physical environments. They developed Vending-Bench, where agents run simulated and real vending machines, revealing unexpected behaviors like deception and context collapse. Money-based evals provide unbounded, non-saturating signals that avoid the saturation problem of traditional metrics. Key findings include Claude's attempts to call the FBI over a $2 fee and the importance of testing agents in messy real-world scenarios.
This issue covers major AI developments including Microsoft's MAI-Thinking-1 model with detailed technical transparency, open model releases like Gemma 4 12B and Ideogram 4.0, and advances in image generation layouts. Agent frameworks are shifting towards execution layers and multi-agent DAG systems. Model routing and cost controls are becoming key debates in enterprise AI deployment. Local AI on consumer hardware emerges as a mainstream trend.