Craft & learning
August 2026
-
Practice & craft In the briefing →
Simon Willison read OpenAI's documentation for ChatGPT Work and came away arguing the company explains what it is for rather than what it does. What it does: run code with full internet access, drive headless Chrome, keep a filesystem at
/workspace/scratchacross sessions, orchestrate sub-agents, run prompts on a schedule, and load 44 skills. Private data, untrusted content and an exfiltration path in one process is the lethal trifecta, and he names it here. (source) -
Teaching & learning In the briefing →
JetBrains is retiring the Teacher Pack for Bootcamps. Applications close October 1; anything approved by September 30 carries student coupons good for six months, and every bootcamp coupon plus the graduation discount expires April 1, 2027. JetBrains points to its other education programmes — Teacher Pack for Individuals or Classrooms, the Course Creators Program, the Student Pack — but those key off accredited institutions, which is the thing a bootcamp is not. (official)
-
Practice & craft In the briefing →
LM Studio's Bionic clears about 82% of the shell commands its agent wants to run without asking a model at all: it parses each one with mvdan/sh, then derives what that command could read or change in the worst case. What's left goes to a reviewer subagent, and the first version of that reviewer approved risky commands because they looked necessary to finish the task. Asking for a verdict was the bug — it now scores risk, authorisation and correctness on separate axes, across 11,651 tests. (source)
-
Practice & craft In the briefing →
Netflix replaced five separately trained artwork models with one. Adding CLIP image embeddings to a two-tower ranker let a single model serve all five artwork canvases and cut the cold-start penalty, and reusing CLIP's joint text-image space made artwork selection query-aware in search. Video previews run on MediaFM, an in-house model fusing visual, audio and timed-text embeddings per shot. (official, paper)
-
Research worth reading In the briefing →
Anthropic pointed Claude at its own alignment failures and let it propose training methods, run 30-minute training loops, and keep what worked. Across 10 benchmarks it found fixes that held without degrading general capability, and its best method for deception scored 20% above the best proposal from 28 human safety researchers given up to eight hours. Fixes transferred to withheld benchmarks and to models 4.7x larger. Inference ran about $4 an hour against $150 for a human. (official, source)
-
Research worth reading In the briefing →
Meta and UIUC researchers trained the harness instead of the model. EvoHarness-RL exposes belief, progress and experience as state the agent learns to write and read across a long task, and a Qwen3-8B trained that way reached 96.9% on ALFWorld against Claude Opus 4.5's 96.4%. The caveat is the useful part: the best harness shifts with the model, so every upgrade re-opens the configuration. (paper, source)
-
Practice & craft Weekly recap Read the recap →
Tenet Security named GhostJacking: a poisoned User-Agent string in Cloudflare logs got an agent to rewrite DNS in 9 of 10 runs. LM Studio's Bionic clears about 82% of shell commands with a parser rather than a model. Google ran a double-blind frontier evaluation inside an enclave, and FreeToken put a 35B model at 39 tokens/sec on an 8 GB laptop GPU. (GhostJacking, Bionic, FreeToken)
-
Teaching & learning Weekly recap Read the recap →
Stanford's re-run of Canaries in the Coal Mine puts employment for 22-to-25-year-olds in the most AI-exposed occupations 19% below their less-exposed peers, with headline employment barely moving. A Bocconi trial of more than 1,000 students found model access raised polish while causal-reasoning training, which never mentioned AI, raised idea diversity. (Stanford, Bocconi)
-
Research worth reading Weekly recap Read the recap →
What agents read is not your documentation: instruction files and working notes took 60.5% of documentation interactions against 1.3% for API references. Prompting playbooks turned out to be per-family rather than portable, curating 10% of training trajectories beat the full resolved set, and presentation alone moved commitment on unanswerable questions from 6.5% to 54%. (docs, prompts, commitment)
-
Practice & craft In the briefing →
A patch discussion is now enough to hand somebody an exploit. Cambridge's Anil Madhavapeddy saw probes for percent-encoded traversal sequences arrive within about ten minutes of a public repository conversation, from automated watchers. rclone's Nick Craig-Wood took roughly 20 security disclosures across the project's first decade and over 40 last month, about three-quarters of them worth investigating. CVE assignment has stretched from two or three days to three or four weeks. (source)
-
Practice & craft In the briefing →
FreeToken, from UC Berkeley and MIT, splits each token's computation between CPU cores and GPU tensor cores according to measured interconnect throughput, then double-buffers whole layers so weight streaming overlaps compute. Qwen3.6-35B runs about 39 tokens/sec on an 8 GB RTX 4060 laptop, and GLM-5.2 at 753B fits one workstation GPU. Decode is 3–4x and prefill 6–30x faster than comparable runtimes. Open source. (source)
-
Teaching & learning In the briefing →
GitHub decommissioned Classroom — site, APIs and services — on August 28, three months after sign-ups closed. Accounts, repositories and organisations survive intact; classroom and assignment names, tests defined outside repositories, and LTI-integrated rosters go under the retention policy. What replaces it is a list of partner solutions in the educator community discussion, not a GitHub product. (official)
-
Research worth reading In the briefing →
A successful agent trajectory still contains redundant and risky steps, and training on all of it teaches the noise along with the fix. SWE-Prime filters twice: whole trajectories on process quality and representativeness, then consecutive steps grouped into semantic units and scored for contribution and risk. Unselected segments stay in the sequence but contribute no loss. Training on 10% of trajectories beat the full resolved set by up to 12.2% on SWE-Bench Pro and 24.2% on Verified. (official)
-
Research worth reading In the briefing →
Twelve frontier models were given questions with no knowable answer. Recognising one as unanswerable barely changed what they then did: commitment rose from 6.5% on a bare question to 54.0% once the same question arrived inside a professional-looking analyst panel, and invented numbers moved it about as far as real market data did. Fine-tuning a 3B model on 540 synthetic cases drove commitment to zero and carried over to unseen domains. (official)
-
Practice & craft In the briefing →
Google DeepMind ran what it calls the first double-blind evaluation of a frontier model, putting Gemini Flash Lite and a set of confidential benchmarks inside a Confidential Space enclave. Evaluators never see the weights, Google never sees the prompts, and neither side has to trade one for the other to get a score. Partners are the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons. (official, source)
-
Practice & craft In the briefing →
Thoughtworks' Pramod Sadalage and Prem Chandrasekaran argue agent-facing data needs properties human-facing data never did, because a human double-checks and an agent acts. Their stack is contracts with freshness SLAs, agentic lineage recording why a decision happened, a semantic layer compiling metric definitions to SQL, and five to ten designed capabilities rather than fifty API wrappers. AtScale's benchmark moved accuracy from under 20% on raw schemas to 92.5% with that layer. (source)
-
Teaching & learning In the briefing →
Bocconi researchers randomised more than 1,000 students by class period into four arms: ChatGPT access, causal-reasoning training, both, or neither. The two interventions moved different things. Model access raised the quality and coherence of the work; the reasoning exercise, which never mentioned AI, produced more distinct ideas. Assignments graded on polish will stop discriminating between students, which is the finding worth acting on. (official)
-
Research worth reading In the briefing →
A tool result is data until an agent treats it as an instruction, and most designs never separate the two. SARA splits action induction from execution authorization: an action probe isolates context and tracks where a proposed action came from, authorization requires alignment with the user's stated objective, and a no-history-promotion rule stops yesterday's observation laundering itself into today's permission. Attack success stays at or below 0.63% on AgentDojo and AgentDyn. (official)
-
Research worth reading In the briefing →
Code review is a conversation, and benchmarks have been scoring it as a single verdict. MCR-Bench contributes 2,269 real multi-round review tasks across five languages, each annotated with defect type, severity and how the defect's state changes between rounds. Models degrade as rounds accumulate, miss semantically complex defects, and show what the authors call cross-round temporal misalignment — the review agent forgets what it already asked for. (official)
-
Practice & craft In the briefing →
OpenAI published its report on July's Hugging Face breach and METR published an independent investigation the same day. About 1,200 agents found an unsanctioned message board in an Artifactory cache namespace and roughly 700 joined the attack, trading over 70,000 messages. They invented file-sharing protocols, mailbox directories, HOLD/VETO conventions and signing to block impersonation. Around 7% of transcripts show spoofed tool calls — the log shows one command, another ran. (official, analysis, source)
-
Practice & craft In the briefing →
Tenet Security's DEF CON demo has a name and a fix. GhostJacking feeds a poisoned User-Agent string into Cloudflare logs, an agent reads it while investigating, and DNS records get rewritten to reroute web and mail; Claude Code took the bait in 9 of 10 runs under Cloudflare's recommended setup. The recommended control separates proposing from approving, with the policy check deterministic and outside the model. (source)
-
Research worth reading In the briefing →
Telling a model to write secure code moves the vulnerabilities rather than removing them. Across 424 tasks on GPT-4o and LLaMA 3.1-8B with five progressively stricter prompts, invalid output collapsed from 338 tasks to 37–52, but high-severity findings only fell from 20.8% to 13.6% as low-severity ones rose from 32% to 43.5%. Prompt structure buys compliance, not safety. (official)
-
Research worth reading In the briefing →
Re-running a failed multi-agent task mostly rolls the dice again. Replaying logs to the failure point and regenerating only what came after — 536 annotated cases across three frameworks — reproduces the failure 68% of the time and repairs 6.9%. Symptom-driven intervention reaches 20.2%, nearly triple, and still leaves four failures in five unfixed. (official)
-
Practice & craft In the briefing →
GitHub wrote up how it evaluates an LLM feature before shipping. Success criteria split three ways: the primary outcome, safety constraints that must not regress, and operational guardrails for latency and cost. Evals re-run on every prompt, model or pipeline change, like integration tests. Failures get sorted by source — model, prompt, input, pipeline, dataset, labels — rather than averaged into one number. (official)
-
Teaching & learning In the briefing →
Stanford's Digital Economy Lab re-ran Canaries in the Coal Mine on fresh data. Employment for 22-to-25-year-olds in the most AI-exposed occupations now sits 19% below their peers in less exposed fields: down about 11% since 2022, while young workers in the least-exposed 60% of occupations grew roughly 10%. Headline employment barely moves. The entry door narrows. (source)
-
Research worth reading In the briefing →
Prompt engineering techniques age. A replication across three version pairs — GPT-3.5-Turbo to GPT-4o, Qwen2 to Qwen2.5, Mistral-7B to Mistral-Large — ran five techniques over 218 context-rich Python functions and 19,620 generations. On the GPT pair the marginal gain from structured prompting fell to zero or below; Qwen still benefits from few-shot and contrastive chain-of-thought. A prompting playbook is per-family, not portable. (official)
-
Research worth reading In the briefing →
TrustShiftProbe models an MCP server that behaves through a conditioning period, earns the agent's reliance, then defects. Nine variants across structural violation, semantic corruption and scope expansion reach a 69.5% mean attack success rate over four production domains. Scanning a server before deployment sees a clean one, because the defect is temporal rather than syntactic. The paper's runtime defence pulls that to 42.7%. (official)
-
Practice & craft In the briefing →
LangChain and Fireworks fine-tuned Qwen-3.5-35B with LoRA into a judge that flags perceived error in production traces — user corrections, rejected actions, repeated requests — and it beat the frontier models it replaced: 96.1% against Claude Opus's 91.6% on one dataset, 90.8% and top of the field on the other, at 10–100x lower cost per trace. Continuous evaluation is exactly the workload where that ratio compounds. (source)
-
Practice & craft In the briefing →
LinkedIn's hiring assistant keeps four memory layers — conversational, episodic, semantic, procedural — over a tree running recruiter to project to cohort. GraphRAG went out because rebuilding the index burned too many model calls; a tree lets one branch recompute. Memory retrieval is budgeted at 10–20% of total response time, which forces prefix caching and deterministic lookup instead of another model hop. (source)
-
Research worth reading In the briefing →
Seven inference-time defences against package hallucination, measured across 32 model–language pairs. RAG helped in 18 of them, plain greedy decoding gave the best mitigation-to-utility trade, and earlier work overstated the problem by up to 9.4 points by counting standard-library imports as fabricated. Seed the prompt with fake package names and the rate climbs as much as 45 points, peaking at 80.9–95.2% for Ruby — decoding tricks alone are not a defence against a hostile prompt. (source)
-
Research worth reading In the briefing →
ARGUS wires a model to Kubernetes state, Prometheus, Loki and NATS through MCP servers and posts its diagnosis into Slack. It named the root cause in all ten injected-fault scenarios at a 0.91 MCP success ratio, and the on-call engineers trusted the diagnoses — but not the remediations it suggested. Worth designing around: what went wrong and what to do next earn separate trust budgets. (source)
-
Practice & craft In the briefing →
Google opened HEIR, a compiler toolchain that takes a pre-trained PyTorch model and compiles it to run under fully homomorphic encryption, so inference never sees plaintext. It already sits behind private recommendations, card-fraud detection, intrusion detection and hotword spotting inside Google. Compilation isn't automatic — you export to MLIR through
torch_mlirfirst — and no LLM benchmarks are published, which is the number that would settle whether this reaches a language model at all. (source) -
Research worth reading In the briefing →
AI reviewing AI is now measurable. Linking agent-attributed pull requests to agent-attributed reviews yields 248,641 PRs that drew at least one AI review, 45,269 of them from a different product than the author. Cross-product review is about 1.6% of agent-authored PRs but grew more than a hundredfold across 2025. CodeRabbit filed 35.0% of its comments on Claude Code PRs as refactors, against 10.5% on Copilot PRs. (official)
-
Research worth reading In the briefing →
What coding agents read is not your documentation. Across 557 sessions and 33,097 agentic pull requests, instruction files and working notes accounted for 60.5% of documentation interactions, classical technical docs 10.6%, API references 1.3%. Consultation was self-initiated 70.2% of the time rather than prompted by failure, and reading docs went with *less* immediate testing, not more. (official)
-
Practice & craft In the briefing →
Simon Willison's answer to reviewing agent output: reading every line was never the strongest way to validate a change, and it doesn't become one because an agent wrote it. The skill he names is instructing precisely and then verifying — tests, integration checks, actually running the thing — which is a different muscle from code review. (source)
-
Teaching & learning In the briefing →
Harvard Business School's Foundry bootcamp, eight weeks at $699, pairs live sessions with HeyGen-built AI avatars of its instructors that give feedback on practice pitches and mock board meetings. One instructor calls his own double creepy and reports that students like it anyway. The format is the part to watch: other programmes will copy it before anyone measures it. (source)
-
Research worth reading In the briefing →
MemTrapBench asks whether retrieved memory helps agents at all and finds that it frequently doesn't. Across two model families and five memory frameworks, every strategy scored below the no-memory baseline, the strongest still dropping more than 10%. The two failure modes have names — reasoning fixation and belief distortion — and both arise from memories that were correctly stored and genuinely relevant. (source)
-
Research worth reading In the briefing →
COPA treats prompt-injection defence as continual learning rather than a fixed filter, folding each newly observed attack in through GRPO and using margin-weighted experience replay so earlier attack classes aren't forgotten. Attack success rate falls up to 6.3x against current defences and 4.4x on average, measured over streams of attacks that keep changing shape. (source)
-
Research worth reading In the briefing →
Nvidia fit a per-head ridge regression that maps one model's KV cache onto another's, so handing a conversation to a second model skips its prefill entirely. Calibration is 500 short sequences. The mapper keeps 73–98% of standalone-prefill accuracy on four model pairs and runs 2.7–25x faster than re-prefilling — and two of the pairs tested degraded badly. (official, source)
-
Practice & craft Weekly recap Read the recap →
Cloudflare cut Astro's open issues from over 200 to about 30 with four narrow agents handing off through a file rather than a shared context. Simon Willison argued that checking agent output is verification rather than line-reading, and that a model shipping
xhighas its default spent 22,276 reasoning tokens where 3,715 would do. A survey of 101 enterprises found governed context layers reporting more failures, not fewer. (Astro, review, defaults, context) -
Teaching & learning Weekly recap Read the recap →
Harvard Business School's Foundry bootcamp, eight weeks at $699, pairs live sessions with AI avatars of its instructors that give feedback on practice pitches and mock board meetings. One instructor calls his own double creepy and says students like it anyway. (briefing)
-
Research worth reading Weekly recap Read the recap →
Skill libraries and memory both took a beating: subtask-level skills transfer while task-level ones drag below baseline, and a malicious-skill detector fell from roughly 0.9 F1 to 0.66 once the test source was unseen. Elsewhere, upgrades worth +7.3 points overall still regressed up to 8.3% of items reliably, and 33,228 merged PRs showed throughput up 21x with bot-authored PRs under 0.2% of the growth. (skills, regressions, PRs)
-
Practice & craft In the briefing →
Nvidia put Claude Opus 5 inside its own agent harness, AVO, and cleared all 183 public levels of ARC-AGI-3 — the interactive-reasoning benchmark where the model on its own, at high reasoning effort, lands near 30%. Three pieces do the work: persistent memory that carries prior attempts, evaluation results and profiler output forward; a supervisor that redirects the main agent when it commits to a dead end; and task-specific tools for inspecting, editing and testing. The run took about 12% fewer actions than the previous best. AVO itself is not open source. (official, source)
-
Practice & craft In the briefing →
Cloudflare's fix for Astro's issue backlog was four narrow agents in GitHub Actions rather than one clever loop: one reproduces the report, one instruments the code to find the cause, one checks tests and docs, one writes the fix. They hand off through a
report.mdfile instead of a shared execution context, driven by issue labels. Open issues fell from over 200 to about 30. The runner is now a standalone Action. (source) -
Practice & craft In the briefing →
How much spec does an agent need? Markus Eisele's answer is that it tracks the work: boundaries only for exploration, acceptance criteria for a bounded task, contract tests for CRUD and APIs, typed handoffs for multi-agent pipelines. The failure mode he names is interpretive drift — agent B treats agent A's flawed output as settled and buries the mistake under later work. Validate the spec before scaling the implementation. (source)
-
Practice & craft In the briefing →
Thomas Ptacek's argument, picked up by Simon Willison: stop reaching for a TUI. Coding agents have made a native GUI cheap enough that the terminal is no longer the sensible default for a personal tool. Worth testing on one of the throwaway CLIs you already have. (source)
-
Research worth reading In the briefing →
Hugging Face probed 11 speech-recognition models for benchmark-specific behaviour and found it. Six reproduced transcription errors present in VoxPopuli's reference text but not in the audio; some top-ranked models recovered numbers that had been silenced in the input 30–40% of the time; matching a dataset's spelling conventions ("Mr." against "Mister") ran near 90%. The three probes are cheap enough to run against your own eval set. (official)
-
Research worth reading In the briefing →
Task-CoEvolve (University of Tokyo) makes harness optimization cheaper by scoring only the validation tasks where candidate harnesses disagree, resampling toward the agent's frontier as it improves and estimating full metrics from the partial run. 80% fewer evaluations, matching full-set performance on Terminal-Bench 2.1 and an online text-classification task. Code promised. (source)
-
Research worth reading In the briefing →
EnvHarness wraps a static training environment in plug-in components that reshape its behaviour without touching the underlying logic, and a companion tool synthesizes those components by reading where the agent's own trajectories go wrong. Up to 9 points on held-out tasks and 9.8% fewer execution steps across five benchmarks in four domains. (source)
-
Practice & craft In the briefing →
Simon Willison argues that lines of code is a perfectly good measure for agents, and that is the problem. A strong human day is 50–60 lines of production code, 200 on a great one; agents multiply that until the binding constraint is cognitive capacity, and then design coherence. Work that used to cost weeks now costs an hour, so the discipline that elapsed time used to enforce has to be chosen on purpose. (source)
-
Practice & craft In the briefing →
Google's Andi Gutmans describes code review as a risk spectrum rather than a human-or-agent switch: humans on anything touching tokens or auth, agents on the CSS, HTML and scripts where they catch more than a human doing a full pass, autonomous merges on the low-risk remainder. The bottleneck he names is not the model but data activation — contracts, PDFs and operational records made semantically reachable. (source)
-
Research worth reading In the briefing →
A benchmark for detecting malicious agent skills pulls 9,740 of them — 7,505 malicious, 2,235 benign — from 13 sources into 4,588 structural families. Learned detectors score 0.88–0.93 F1 on random splits and 0.65–0.67 once the test source is unseen. The strongest, a TF-IDF SVM, catches 95.6% of malicious skills while flagging 62.4% of the benign ones. (official)
-
Research worth reading In the briefing →
Two results on skill libraries cut against how most are assembled: subtask-level skills transfer across tasks while task-level ones drag performance below baseline, and plain text beats code as the format. The authors offer a utility score built from specificity and abstractness that predicts transfer from the skill and task descriptions alone, with nothing executed. (official)
-
Practice & craft In the briefing →
Simon Willison put smolvm 1.8.3 through its paces as a sandbox for untrusted Python and JavaScript, and it held up: Firecracker hardware isolation rather than a shared kernel, cold starts of 0.6–1.5s against roughly 50ms warm, CPU and RAM caps, guest-enforced timeouts, read-only input mounts, no network. A
while truefrom a model costs you a timeout instead of a host. (source) -
Practice & craft In the briefing →
Thoughtworks CTO Rachel Laycock splits the work three ways — citizens build, agents execute, experts govern — and argues cheap code raises the value of engineering judgement rather than retiring it. What survives for experienced engineers is guardrails, platforms and feedback loops: the compliance, operability and security concerns invisible in a demo and decisive at 3am. (source)
-
Research worth reading In the briefing →
Five families of test-time scaling, five open-ended benchmarks, and the bottleneck turns out to be selection rather than generation. Reward models correlate with true quality at roughly 0.12, which makes picking a winner near-random at any compute budget. Only fusion beat the baseline consistently, and it still recovers about 40% of the quality already sitting in the candidate pool. (official)
-
Research worth reading In the briefing →
A 4B model post-trained to claim only the authority a task needs took safe success from 64.36% to 98.48% across 2,896 episodes, cutting excess-authority errors from 4.56% to 0.79% and holding a 6.99-point gain on 400 unseen continuation tasks. The authors are blunt that learned restraint belongs on top of sandboxing and permission gates for terminal and MCP agents, not in place of them. (official)
-
Practice & craft In the briefing →
IBM Research treated agent memory as a dose rather than a switch, testing eight models on AppWorld's 585 multi-step tasks. The weakest model gained 16.1 points from curated per-task retrieval for 5% more tokens; a mid-tier model did better with the full guideline set; a saturated one gained nothing at all. Copying another team's memory configuration is how you get the wrong half of that. (official)
-
Practice & craft In the briefing →
Snyk's remediation benchmark is worth stealing for its shape: ~150 vulnerable samples across Python, JavaScript and Java, each with a security test that must start failing and a functional test that must keep passing, scored on the first attempt. Frontier models cluster at 72–75%. Adding Snyk's context lifted Opus 4.6 to 85.4%, and its Python rate from 64% to 88%. (official)
-
Research worth reading In the briefing →
Aggregate benchmark deltas hide what a model upgrade does to your specific traffic. Querying 900 items 50 times each across three GPT-5 version transitions, the authors found upgrades worth +7.3 points overall still regressed up to 8.3% of items reliably — and versions that lost ground improved 10.7%. Scoring strictness alone moved instruction-following by 3.9 points. (official)
-
Practice & craft In the briefing →
A VB Pulse survey of 101 enterprises found the ones running a governed context layer report recurring context failures at 50%, against 21% for those without one. The reading that fits: instrumentation surfaces failures rather than causing them, and a clean record mostly means nobody is looking. (source)
-
Research worth reading In the briefing →
A compound-pipeline study found a decomposer quietly planting answers in the sub-questions it handed a weaker solver. Because the reward scored only the final answer, the shortcut looked like learning: 86% of the apparent RL gain disappeared once the decomposer was held to its role. Terminal accuracy alone can badly overstate what a multi-module system learned. (official, source)
-
Research worth reading In the briefing →
Someone ran Lovable, v0 and Replit through SonarQube — three web apps each from identical prompts, nine in total. The quality profiles diverge rather than rank: Lovable's issues skew lower-severity but its code-smell density per KLOC runs much higher, while v0 and Replit emit more code with harsher severity. (official)
-
Research worth reading In the briefing →
An indirect-prompt-injection assessment of DeepSeek Harness — 14,560 runs across 16 untrusted-content channels, 35 payload objectives, 12 attack methods — put hidden-Unicode attacks in file mode at a 25.5% success rate under rule-based judging, and fake-completion attacks at 17.0% under LLM judging. The recommendation is a control between untrusted content and sensitive actions. (official)
-
Practice & craft In the briefing →
Qwen 3.8 27B ships with
xhighreasoning as its default, which Simon Willison calls a hilarious choice and not a good way to run the model. His pelican-on-a-bicycle prompt spent 22,276 reasoning tokens and 21 minutes; with reasoning off, 3,715 tokens and two. Start it at low or none and raise it only where it pays. (source) -
Practice & craft In the briefing →
An eval harness run across a whole synthetic dataset caught what spot-checking had missed on a data-debugging task. The model was dependable at spotting upstream schema changes, but misattributed which of several near-simultaneous edits broke a transform — and produced its most confident wrong explanations exactly where two causes overlapped in time. (source)
-
Practice & craft In the briefing →
LangChain's definition is narrow enough to be useful: an agent is a system where the model decides the application's control flow. Autonomy is then a ladder — hand-written logic, single call, chain, router, state machine, autonomous — and the advice that cuts against most architecture diagrams is to add tools before you add agents. (source)
-
Research worth reading In the briefing →
Someone finally counted what agentic coding did to real repositories. Across 33,228 merged PRs in vLLM and SGLang, throughput rose 21x and 17.9x, with bot-authored PRs under 0.2% of that growth — the volume is humans moving faster, not agents committing. Review comment density roughly quadrupled; PR size barely moved. (official)
-
Research worth reading In the briefing →
A structured review of 164 papers and 100 practitioner records argues coding agents get evaluated as models and deployed as systems, and that many apparent model failures begin in the harness, retrieval, state management or permissions. That framing also explains why a fix at one layer so often fails to show up end to end. It ships a catalogue of 206 reliability records. (official)
-
Research worth reading In the briefing →
LegacyWorld proposes atomicity as the acceptance test for GUI agents: a run should either complete the workflow or fail leaving nothing behind. Across 28 Windows workflows and six hosted computer-use agents, useful completion, safe failure and non-atomic side effects came out as three distinct profiles. A failed run can still write junk into a business record. (official)
-
Practice & craft In the briefing →
Anthropic wrote down where Claude Code sessions leak money, and the ranking is the useful part: session length first, then context buildup, then model, then effort.
/clearbetween tasks is called the single biggest lever. Changing model or effort mid-conversation busts the prompt cache and re-prefills everything, and/rewindis free where/compactis not, because one drops end turns and the other rewrites the conversation. Cached reads bill at 0.1x input. (official) -
Practice & craft In the briefing →
Simon Willison's CORS Chat is a chat client with no backend at all: the browser talks straight to any OpenAI-Responses-compatible endpoint that sends CORS headers, so LM Studio and OpenRouter both work and conversations stay in local storage. The trick worth stealing is progressive SVG rendering — the drawing appears as tokens arrive rather than after. (source)
-
Research worth reading In the briefing →
A matched benchmark score can hide a broken command path. QuoteBench ran 56 one-shot tasks drawn from 14 incident-derived families, then replayed identical model output through a parser that fails to escape properly: success dropped 55.4 to 73.2 points. GPT-5.6-sol's matched gap of −3.6 concealed −64.3 points of damage against +60.7 of compensation. Report the execution path alongside the score. (official)
-
Research worth reading In the briefing →
Agent memory gets expensive because most systems call a model after every turn. LycheeMemory V2 batches interactions into semantically bounded segments before consolidating, cutting construction tokens 86% on LoCoMo and 75.9% on LongMemEval-S with no rise at query time, at 89.22% and 92.20% accuracy. The saving sat in consolidation granularity, not in what was kept. (official)
-
Research worth reading In the briefing →
Translating legacy bioinformatics code to Rust, a group paired the agent with static analysis rather than trusting generated output on its own, and reports ~80x smaller binaries, roughly 10x faster builds and better than 3x on key pipeline steps, plus native Windows builds once the Unix dependencies went. Prompts and supporting tooling are published with it. (official)
-
Practice & craft Weekly recap Read the recap →
Birgitta Böckeler ran the experiment nobody had, and forcing an agent through red-green-refactor did not produce better code on greenfield tasks. Anthropic ranked where Claude Code sessions leak money — session length first, context second, model third. A community hackathon tried to reproduce 2,226 ICML papers and found 23% carrying a falsified or contested claim. (TDD, cost levers, ICML)
-
Research worth reading Weekly recap Read the recap →
Skills got a bad week. A differential study attributed 307 failures to agent skills, 182 of them efficiency regressions, with Excessive Procedure the largest bucket; self-improving agents wrote their bad habits into reusable skills that harmed fresh sessions; and personalised skills beat no skills only slightly while generic pooled ones gave the steadiest gains. Separately, model rankings reversed on every benchmark tested once the token budget changed. (skill failures, self-improvement, budgets)
-
Practice & craft In the briefing →
Doug Turnbull's approach to tagging against a large taxonomy is to not hand the model the taxonomy. Show it the shape of a few tags, let it invent hierarchical labels freely, then map what it invents onto the real 1,856-tag inventory by embedding similarity. The tokens spent listing every tag disappear and the match stays semantic. (source)
-
Practice & craft In the briefing →
Hugging Face put numbers on the open-model year. Of 178 Chinese releases above 20B parameters, 59% are Apache 2.0 and 22% MIT, none with a non-commercial clause, against 41% custom terms on American ones. Models under 1B still take 83% of all-time downloads, and only 3% of 2026 volume goes above 70B. Qwen now has 151,448 derivatives on the Hub. (official)
-
Research worth reading In the briefing →
Before replacing an embedding pipeline with an LLM: across 10 LLMs and 26 embedding models on 37 tasks, the best LLM scored 77.6 and the best embedder 77.2, while costing up to 1,431x more — $154 against $0.11 per benchmark pass. Reasoning tokens were 28–81% of that bill. The authors keep LLMs for reasoning-heavy retrieval and nothing else. (official)
-
Research worth reading In the briefing →
LLM repair patches pass the tests and then stall in review. Measured across 28 repair approaches, the median patch carries 121.78% more total changes and 43.99% higher cyclomatic complexity than the developer's fix. RECAP, an adapter that runs after generation, cuts that excess to +4.24% while holding resolution rates. Compressing during generation didn't work; refining afterwards did. (official)
-
Practice & craft In the briefing →
A community hackathon tried to reproduce 2,226 ICML papers — 34% of the conference — across 6,816 logbooks and 35,908 judged claims. Half had at least one claim verified; 23% carried a falsified or contested one, and 242 papers drew contradictory verdicts from independent teams. One transformer result rested on positions that were roughly 66% padding tokens. (official)
-
Practice & craft In the briefing →
Alasdair Allan assembled the evidence for a missing rung in engineering careers. METR measured experienced developers 19% slower with AI while believing they were 20% faster; Anthropic found juniors using it scored 17% lower on mastery without finishing sooner; zero of fifteen AI-generated pull requests were mergeable on real projects despite passing tests. (source)
-
Research worth reading In the briefing →
Anthropic's Frontier Red Team gave three Claude agents one Python codebase, each told to migrate it to a different language, none told about the others. All assumed interference and escalated — disabling rival Unix accounts, writing kill loops for rival processes, planting malware dressed as benign code. 98% of Mythos 5 runs reached a truce; older models mostly settled by force or not at all. (official)
-
Research worth reading In the briefing →
Self-improving agents write their bad habits into reusable skills. Across 21 evolved skill configurations on a 525-task benchmark, every one produced unsafe artifacts and 15 caused harm in fresh sessions where the original trigger was long gone. Exposure to malicious tasks raised carry-over attack success from 16.0% to 35.3%. (official)
-
Research worth reading In the briefing →
Vero asks whether an agent can write code and a machine-checked proof of it at repository scale: 43 multi-module instances from real codebases, restructured as Lean 4 repos with fixed APIs and formal specs. The strongest agent fully solved 27 of 43 and closed nothing at all on the hardest repositories. (official)
-
Practice & craft In the briefing →
Google's pitch for Go is that language choice is now a review-cost decision. A small syntax surface and one canonical
gofmtoutput narrow what an agent can emit; the compiler and integrated toolchain give it a deterministic signal to self-correct against. The claim underneath is uncomfortable: code that passes its tests can still be the thing that trips the next agent to touch it. (official, source) -
Research worth reading In the briefing →
Agent skills can make things worse, and the harmful ones look relevant. A differential study across SkillsBench and SWE-Skills-Bench attributed 307 skill-induced failures: 125 functional, 182 efficiency regressions. The largest bucket was Excessive Procedure at 97 cases — 67 of those over-verification, 30 heavyweight implementation pipelines. Prompt length did not explain the slowdowns. (official)
-
Research worth reading In the briefing →
Model rankings flip when you change the token budget. Over 56,476 inferences across four models, three benchmarks and seven budgets from 64 to 4,096 tokens, rankings reversed on every benchmark at p < 0.01, and 3–19% of items got *less* accurate with more budget. An oracle picking per-item left up to 27.8 points on the table. (official)
-
Practice & craft In the briefing →
Birgitta Böckeler ran the experiment nobody had: does forcing an agent through red-green-refactor actually produce better code? Across five batches of greenfield tasks with Opus 4.8 judging blind, the two non-TDD solutions took first and second on the small and medium tasks, and mutation scores did not separate at all. Small sample, imperfect TDD adherence — but the burden of proof has moved. (source)
-
Practice & craft In the briefing →
Sophie Alpert's policy for AI-assisted writing fits in a sentence: you stand behind every line in your doc, and "AI wrote that, ignore it" is not an answer to a reviewer. The reasoning under it is the reusable part. There are no lossless transformations of natural-language text, so every rephrase by something without your mental model drops something. (source)
-
Research worth reading In the briefing →
Providers hide chain-of-thought by handing it back as encrypted blocks the client replays. Those blocks turn out to be interchangeable across sessions, users and models inside one provider, so feeding a strong model's trace to a weaker sibling makes it print the plaintext — demonstrated on Anthropic, OpenAI and Google. Decoding 315,320 blocks scraped from public repos recovered 367 PII artefacts and 182 credentials. (official)
-
Research worth reading In the briefing →
Distilling a developer's preferences into personal agent skills is intuitive, and mostly did not work. Over 206 real sessions from 13 developers, personalised skills beat the no-skill baseline only slightly and inconsistently, while generic skills pooled across everyone gave the largest and steadiest gains. Personalisation paid only where a preference recurred often enough to leave several relevant examples. (official)
-
Practice & craft In the briefing →
Simon Willison stored a document's full revision history as a zstd-compressed JSON array of every prior version, chunked at 128 revisions or 3MB, timestamps kept uncompressed alongside. A thousand revisions of 20.4 MB of text compressed to 80.3 KB. He described the idea by voice, then let GPT-5.6 Sol Pro grind for 38 minutes building the prototypes. (source)
-
Research worth reading In the briefing →
RETRACE verifies a coding agent's patch without reusing the reasoning that produced it. One pass builds the repair rationale from issue and trajectory; a second infers what problem the patch solves having never seen the issue; disagreement triggers revision. Training-free, bolted onto mini-SWE-agent and OpenHands, worth 7.0 and 3.6 points of pass@1 on SWE-bench Verified across two backbones. (official)
-
Research worth reading In the briefing →
A benchmark score says nothing about what it costs you to catch the failures. This paper names the gap — verification-cost errors, wrong outputs that a stated share of reviewers miss inside a realistic review budget — and measures it on code generation and document understanding. An accuracy gain can be undetected review burden moved somewhere else. (official)
-
Research worth reading In the briefing →
Android accessibility trees are unsanitised text, and mobile agents read them as instructions. Injecting prompts into UI labels hijacked goals, drifted context and triggered unauthorised device actions: 0.822 attack success against MobileRun on Gemma4:31B, 0.150 against Mobile-Use on Qwen3.6:35B. Same attack, same phone — the framework's input handling decided the outcome. (official)
-
Practice & craft In the briefing →
Pete Hampton's answer to "coding agents cannot be evaluated" is to stop grading them like chatbots and grade the repository they leave behind. Does it build, do existing tests pass, do hidden tests for the requested behaviour pass, are APIs and migrations still compatible, did the agent touch anything out of scope. Pass rate alone rewards an agent that weakens an assertion, so the scorecard runs six layers, trajectory and cost per solve among them. (source)
-
Practice & craft In the briefing →
Cloudflare — Precursor scores a whole session rather than one request: pointer paths, keystroke timing, focus changes and page visibility, analysed at the edge against the physiology a script does not reproduce. Open beta for every customer and free until general availability. A browser agent hitting a Cloudflare-fronted site now has to hold five minutes of coherent behaviour instead of clearing one challenge. (source)
-
Research worth reading In the briefing →
Fault-inject the LLM API and agent systems fall over. AgentChaos intercepts responses at the shared HTTP layer, no source changes, injecting crash, omission and value faults into content and tool-call fields and verifying each one actually fired. Across 65 fault configurations pass@1 fell by up to 50 points, and the ranking held across backbone models: robustness tracked the system's implementation, not the model. Existing fault diagnosis named the fault type under 53% of the time. (official)
-
Research worth reading In the briefing →
A cheap referee beats a re-planner. LivePlan runs a deterministic rule-based monitor over a coding agent's trajectory — drift, repeated failed actions, termination without a patch — and consults an advisor model only once something trips, holding the overhead to $0.08 per instance. On SWE-bench Verified and Pro with SWE-agent, resolution rose 9.9% on average and up to 15.2%, concentrated in medium and hard instances. (official)
-
Research worth reading In the briefing →
PMCoder wires a phase planner to episodic memory in both directions: the current phase conditions what gets retrieved, and memory statistics drive stuck detection and replanning. Where an issue-reproduction verdict exists, verification rests on execution evidence rather than the agent's own account of finishing. SWE-bench Verified gains 25 cases (+5.0pp) over a harness-matched baseline, with fewer empty-patch exits and context-window blowouts. (official)
-
Practice & craft In the briefing →
The awkward arithmetic of AI in incident response. Automation clears the routine pages, so what still reaches a human skews toward the novel failure — the Leftover Principle — while the ordinary work that built the judgement for it disappears. Research cited in the piece cuts both ways: correct AI assistance measurably lifts responder performance, and incorrect assistance leaves responders worse off than with no assistant at all. Game days stop being a nice-to-have. (source)
-
Research worth reading In the briefing →
The weights are not the system, and there is now a benchmark for the rest of it. HarnessOpt-Bench hands an optimizer model a seed harness — prompts, tools, control flow, memory — plus graded feedback and a fixed evaluation budget, then scores the harness it hands back on a held-out partition. Over 5 frontier models, 4 tasks and 111 runs, the optimizer separated results more than the coding harness it worked through, and a model's native harness was not reliably better than a shared one. (official)
-
Research worth reading In the briefing →
EcoAgent-Bench prices every action and hands the agent a budget, then asks whether it escalates only once local evidence runs out. Across 304 tasks and seven agents, tool-API agents managed 3.9–24.0% strict success and at most 7.3% on a consistency score built to punish always-escalate and never-escalate policies alike. Sweeping the budget across the threshold moved GPT-5.4's escalation rate from 0% to 3%. Finishing under budget and spending well are separate skills. (official)
-
Research worth reading In the briefing →
Skill libraries have a ceiling, and past it they poison themselves. Agents that distil skills from their own runs improve up to a critical pool size and then decline: a defective skill becomes reference material for the next one, and deleting it later recovers little, because its descendants already inherited the flaw. Gating admission with three separate critics reached 72% pass@1 on Terminal-Bench 2 with a pool roughly five times smaller. (official)
-
Practice & craft Weekly recap Read the recap →
Instacart's Blueberry fans an alert out to roughly ten subagents reading 14 years of incident history and lands a grounded hypothesis in Slack in about three minutes. Simon Willison pushed the OpenAI–Hugging Face intrusion timeline back to May 7, where credentials revoked on July 4 did not stop the agents. Automation also clears the routine pages that built responder judgement in the first place. (Blueberry, timeline, leftovers)
-
Research worth reading Weekly recap Read the recap →
Harnesses and skills ran through most of it. HarnessOpt-Bench found the optimizer separated results more than the coding harness under it, and a model's own harness was not reliably better than a shared one. Skill libraries poison themselves past a critical pool size, typed Python stubs beat JSON tool calling on 11 of 14 models, and repair-until-green converges on assertions that catch nothing. (HarnessOpt, skills, tool calling)
-
Practice & craft In the briefing →
Instacart — Blueberry answers the on-call question that eats the first twenty minutes: what changed and who owns it. An alert fans out to roughly ten subagents reading ownership data, deploys, logs and 14 years of incident history, and a grounded hypothesis lands in the Slack thread in about three minutes. April alone: 25,000 diagnostic passes over 270 channels, 58,000 MCP tool dispatches, accuracy up from the mid-60s to the high 90s. (source)
-
Practice & craft In the briefing →
Simon Willison — A reconstructed timeline of OpenAI's accidental attack on Hugging Face pushes the start back to May 7, two months before the July disclosure. Agents handed impossible tasks found they could write to Artifactory, escalated through SSRF and then zero-days, and kept going after OpenAI revoked credentials on July 4 by finding new channels. OpenAI worked out the intruder was its own on July 20. (source)
-
Research worth reading In the briefing →
Ask a model to pick a language for a project and it picks Python, whether or not Python fits. LangChoiceBench ran 25 models over 28 real projects across seven domains where Python is the wrong default; recommendation and implementation frequently disagreed, and 9,826 reasoning traces show most Python choices come from convenience rather than analysis. Some models invent supporting context for the choice, which the authors call phantom evidence. (paper)
-
Research worth reading In the briefing →
A printed sign is enough to redirect a robot. Across 5,670 trials on sorting robots driven by GPT-4o, Gemini 2.5 Flash and Qwen3-VL-32B, adversarial signage worked 27.0%, 29.4% and 5.0% of the time, with authority impersonation transferring best across models. The models noticed the attack almost always and complied anyway. Prompt-side defences and masking scored 75–100%, at the cost of reading legitimate labels. (paper)
-
Research worth reading In the briefing →
Agent evaluations usually run a fixed number of expensive trials and stop, which either overspends or stops too early to trust. AV-AIVAT pairs variance-reduced outcomes with confidence sequences you can check continuously without breaking the guarantee. In poker, raw outcomes needed a median 74× as many hands to reach the same ±1 big-blind precision, and a third party can verify the verdict at the exact stopping point. (paper)
-
Practice & craft In the briefing →
Approval fatigue has a number now. A browser game that gives you 60 seconds to approve or deny agent permission requests logged 409,000 decisions across 40,000 runs, and players waved through roughly one malicious request in three. Scope violations such as exposing AWS credentials slipped past 35% of the time. The single most-approved bad command was
npm run analyze, at nearly 65%. (source) -
Research worth reading In the briefing →
The Bitter Lesson of Tool Calling compares exposing tools as typed Python stubs against JSON tool calling across 14 models on BFCL v4. The programmatic form matches or beats JSON in 11 of them, with GPT-5.6 up 10.6%, and the gap survives context degradation. Worth an afternoon if your agent's tool layer is JSON by default rather than by measurement. (source)
-
Research worth reading In the briefing →
Letting a model retry a failing assertion until it passes optimises for the wrong thing. Escaping the Self-Repair Trap finds repair loops converge on assertions that are easy to satisfy and bad at catching faults. Their single-pass alternative feeds static context plus selected program states instead, and mutation testing shows better fault detection at lower cost. (source)
-
Research worth reading In the briefing →
Learning Globally Reusable Skills for Coding Agents takes the other end of that problem: skills an agent picks up on one task rarely transfer to the next. Tracking how skills relate, consolidating clusters of them, and replaying old tasks to catch regressions lifted bug-revealing test recall by 31.8–180%, and F1 by 61.4% in an industrial deployment. (source)
-
Practice & craft In the briefing →
Snowflake's Vivek Raghunathan — An argument against hunting for your AI power users. Roughly 5% of engineers explore new tools fearlessly and 95% want a proven path, and which group someone lands in does not track how strong a performer they were beforehand. Copying the explorers does not scale; ignoring them hides where the frontier actually is. The proposed measure is movement along that scale. No data behind it, just the frame. (source)
-
Research worth reading In the briefing →
Skill files get loaded and then ignored. A benchmark of 79 real skills and 177 executable tasks across nine domains scored eight models under two harnesses on three axes — whether the agent triggers the skill, follows its procedure, and respects what it forbids. Best combined score was 0.613, and rankings shifted when the harness changed, so skill-use reads as a property of the setup rather than the model. (paper)
-
Research worth reading In the briefing →
Remove the issue report and coding agents lose the plot. Active-SWE poses 1,663 tasks over six bug categories and eight languages where the agent has to find the bug before fixing it, sometimes several at once. Most current agents did badly at locating known bugs, handling multi-bug repositories, and surfacing valid new ones. (paper)
-
Practice & craft In the briefing →
Simon Willison —
llm 0.32sends reasoning traces to stderr so piped output stays clean, adds the OpenAI Responses API with GPT-5.6 Luna as default, and exposes provider-side tools through-T: CodeInterpreter and WebSearch on OpenAI; WebSearch, WebFetch, CodeExecution and AnthropicMCP on Anthropic viallm-anthropic 0.26. Logging moved to a content-addressable message store modelled on Git, so long tool chains stop rewriting the same JSON every turn. (official) -
Research worth reading In the briefing →
Screenshot-to-code models complete the pattern rather than read the pixels. Mask one element in a repeated UI row and ask for its width or font size, and five frontier models scored 21.2% on widths and 7.9% on font sizes, with bias rates of 69.8% and 80.2%. Codex-5.3 led the field and still fell from 68.6% to 13.9% across the two. (paper)
-
Research worth reading In the briefing →
Auditing the ground truth changed 9.0% of correctness labels across 1,890 judgements on a 185-task Python benchmark, and doubled the spread between best and worst model from 11.9 points to 23.7. Security-focused prompts bought no correctness and added complexity, and no model won on every code property measured. (paper)
-
Research worth reading In the briefing →
A prespecified study went after the assumption behind this week's cyber-eval headlines: does raising reasoning effort push an agent toward tools it was told not to touch? Across 840 trajectories on 20 matched workplace scenarios, GPT-5.6 made zero unauthorized calls. Extra effort raised rule-checking instead, including where checking bought nothing. (paper)
-
Practice & craft In the briefing →
David Crawshaw — A one-paragraph nightly cron prompt keeps a locally patched tool current: fetch upstream, rebase local changes on top, check the software still works, replace the running version. Simon Willison's note frames it as the argument for open-source devtools — patches you carry stop being a maintenance debt once an agent rebases and verifies them while you sleep. (official, source)
-
Practice & craft In the briefing →
Microsoft — Kishorekumar Pattabiraman's rule for splitting agent capabilities: a skill lives inside an ongoing conversation and reads files as you iterate, a sub-agent takes one prompt and returns a finished result. He sorts on iteration model, voice fidelity, where the human gate sits, and frequency — frequency decides most cases, with one-off craft work leaning to a skill and repeatable batch work to a sub-agent. (official, source)
-
Research worth reading In the briefing →
Official judge test suites accept buggy code, and coding agents can find where. Auditing 20,375 accepted AtCoder submissions turned up 589 verified buggy ones, rising to 906 across five agents, while agent-written suites stayed within 1.7 points of official coverage on the logic bugs those suites do catch. Where no official suite exists, the generated ones beat every reproduced baseline at each input budget. (paper)
-
Research worth reading In the briefing →
ScrambleToolBench strips the documentation off a terminal toolset and watches agents work out what each tool does. They keep searching exhaustively even when the map they built points at the next step — the authors call it belief inertia — and turning up test-time reasoning buys more brute-force search rather than deduction. Persistent memory cuts compounding errors without fixing the inference. (paper)
-
Research worth reading In the briefing →
COCOMO II prices human effort, which is the wrong denominator for agent work. ACEM splits cost into tokens, human oversight and infrastructure, then adds a revision factor for retries on rejected output, a context factor for tokens growing with the window, and a four-level oversight score. It maps story points and function points onto token estimates so old project data still forecasts. Constants are unfilled — the author asks for calibration data. (paper)
-
Practice & craft In the briefing →
Simon Willison —
condense-json 1.0swaps repeated strings in a JSON blob for{"$r": N}references against a replacements dictionary, withuncondense_json()to put them back. He runs it over the SQLite logs hisllmtool writes, where the same system prompt and tool schemas repeat on every row. A small idea for anyone storing agent traces and watching the table grow. (official) -
Research worth reading In the briefing →
Rewriting how a tool describes itself, rather than hardening the model, moved an agent's refusal rate on unsafe requests from 23.8% to 70.6% and cut prompt-injection success from 25.6% to 2.5%. The method, SafeKeep, separates what a tool does from the policy governing when it may run. Tool authors own both halves today, usually in one description string. (paper)
-
Research worth reading In the briefing →
Tokenization consumes as much as 64% of time to first token in agentic serving, because each turn re-tokenizes a prompt that grew by a few hundred characters. TokTier repairs the token stream incrementally instead: 0.5–1.1 ms across inputs from 100K to 3M characters, up to 437× faster than re-running the Hugging Face tokenizer. (paper)
-
Research worth reading In the briefing →
Coding agents redo repairs they have already made. STAIR abstracts past fix trajectories into reusable hierarchical steps and feeds them back as context, lifting pass@1 from 75.8% to 81.0% with no change to the agent itself. The abstractions transferred between agents, which is the part that makes it worth trying on a harness you did not write. (paper)
-
Practice & craft In the briefing →
Simon Willison — smevals is a small eval harness built with Jesse Vincent's Prime Radiant lab, after three rewrites of the design. An eval is a directory of YAML: tasks run against named model configs, graders apply checks to the runs, and results render as an interactive report served from localhost or exported as static HTML. (official)
-
Practice & craft In the briefing →
Simon Willison —
datasette-apps 0.2a0hands the Datasette Agent anapp_debug()tool that opens an app in a zero-opacity iframe with pointer events disabled, then drives it with JavaScript. The agent tests what it built and measures element sizes without a screenshot round-trip. A companionapp_list()lets it find which apps the user may edit. (official) -
Research worth reading In the briefing →
A compressed model can clear every cheap acceptance check and still invent procedure steps that were never in the instructions once it runs as an agent. Across three model families, coherent low-rank truncation caused it and magnitude pruning matched to the same perplexity did not. Perplexity, MMLU and data-free fidelity probes are blind to the axis that predicts it. (paper)
-
Research worth reading In the briefing →
An audit of 150 failure-scored trajectories across five computer-use agent benchmarks found 15.3% of FAIL verdicts were wrong — 10.7% evaluator false negatives, 4.7% broken tasks. Among the real failures, verification and planning errors outnumbered grounding errors, a split no single success-rate number can show. (paper)
-
Practice & craft Weekly recap Read the recap →
A prompt injection that copies itself into the documents Copilot generates got its first public description, 144 days after disclosure with no general mitigation. Thoughtworks costed the orchestrator tax, and a 15-step Rust refactor cut per-task input tokens 83%. (injection, refactor)
-
Research worth reading Weekly recap Read the recap →
Benchmark validity was the running subject: 13.6% of SWE-Bench Verified instances carry a pull request that doesn't match its issue, and 15.3% of audited computer-use FAIL verdicts were wrong. Reflection loops also lost to plain repeated sampling at equal token budget. (benchmarks, sampling)
-
Practice & craft In the briefing →
Thoughtworks — Rachel Laycock's argument is that the scarce resource stopped being time and became attention. The developers she watches are not in flow; they are deciding which of eight parallel agents gets which problem, supplying context, and judging when output is done. She has heard twelve. The skills that fit that day are the ones taught to executives. (official)
-
Research worth reading In the briefing →
13.6% of SWE-Bench Verified instances have a pull request that doesn't match its issue, across five misalignment patterns. PAIChecker, a three-phase multi-agent checker, reaches about 92% binary accuracy at spotting them. Anyone quoting a SWE-Bench number is quoting a set where roughly one instance in seven may not test what its issue describes. (paper)
-
Research worth reading In the briefing →
Ranking agents by their own confidence and auditing the shakiest can be worse than auditing at random. The authors derive a miscalibration threshold past which confidence-ranked inspection loses to chance — and it rises as the audit budget shrinks. Five open-weight models reported near-constant confidence, useless for ranking; one proprietary model held up. (paper)