Running in production
August 2026
-
AI cost tracking & telemetry Weekly recap Read the recap →
Twelve harness configurations running identical tasks spread token consumption 70-fold, on prefix stability rather than model choice. Google Cloud shipped a hard monthly cap that pauses API calls when hit, Cohere's Parse 5 undercut the frontier at $1.50 per 1,000 pages, and Ramp's card data put Opus 4.8 at 28% of enterprise usage. (harnesses, cap, Parse 5)
-
AI cost tracking & telemetry In the briefing →
Cohere's Parse 5 loses the benchmark and wins the invoice. The 2.3B vision model scores 79.2 on ParseBench against GPT-5.5's 84.4 and Opus 4.8's 84.3, then charges $1.50 per 1,000 pages; on a modelled 750-million-document year Cohere puts the saving against GPT-5.5 above 98%. Output is Markdown with HTML tables, image descriptions and bounding boxes, on Cohere's API, Microsoft Foundry and SageMaker. (source)
-
AI cost tracking & telemetry In the briefing →
Agent traces are only as useful as the services they land beside, and Grafana Cloud now scores that. An instrumentation quality report runs server-side checks per service — does it emit logs, is
service.namewell formed, are span metrics present, what is the metric cardinality — and rolls them into one percentage plus a fleet-wide table of what is failing. No model involved in the scoring. (official) -
AI cost tracking & telemetry In the briefing →
Twelve configurations of Aider, Claude Code and OpenClaw ran the same tasks through OpenRouter against DeepSeek V4 Flash and Nemotron 3 Ultra. Token consumption spread 70-fold. The mechanism is prefix stability: a harness that keeps its system prompt and tool descriptions byte-identical between turns reads from cache, while one that rebuilds context pays full rate for the same text every turn. Picking a cheaper model is the smaller lever. (source)
-
AI cost tracking & telemetry In the briefing →
Google Cloud gave agent workloads a spend cap that actually stops them: a hard monthly project limit that pauses API calls when reached, with alerts at 50%, 80% and 100% and anomaly detection naming the top three SKUs behind a jump. Quotas pool across Gemini Enterprise, Antigravity and Android Studio. Savings plans cut 10% for a year and 20% for three. (official)
-
AI cost tracking & telemetry In the briefing →
Grafana open-sourced the AI SDK for Go its own teams had been rewriting per project: one interface over Anthropic, Bedrock and OpenAI's Responses API, with streaming, tool loops, structured output, timeouts, retries and model fallback. The part worth stealing is the agent-observability middleware — it records usage, errors, which provider actually served a routed request, and how multi-step calls relate to each other. (official)
-
AI cost tracking & telemetry In the briefing →
Ramp's card-billing data across 70,000 companies puts Opus 4.8 at 28.0% of enterprise model usage in July, Sonnet 4.6 at 8.3% and Fable 5 at 8.0%, with nothing else above 7%. Simon Willison reads Fable's price as what has kept it niche — a strange place for a lab's strongest model to sit. (source)
-
AI cost tracking & telemetry Weekly recap Read the recap →
Gartner expects inference cost per agentic workflow up more than fivefold through 2028, on cheaper tokens funding more elaborate loops. OpenAI cut GPT-5.6 Sol output to $20 per million from $30 through November 21, Snowflake's gateway will now pick the model itself for up to 3x, Dynatrace signed a definitive agreement for Arize, and a survey found 21% of enterprises with no real-time control over agent spend at all. (Gartner, Sol, Snowflake, Arize)
-
AI cost tracking & telemetry In the briefing →
OpenAI cut GPT-5.6 Sol to $4 per million input tokens from $5, cached input to $0.40, and output to $20 from $30 — a third off the output side. The pricing is promotional and holds at least through November 21. It reaches Codex credits and ChatGPT Work; Pro, Plus and Business subscription usage is unchanged. (official)
-
AI cost tracking & telemetry In the briefing →
Dynatrace has signed a definitive agreement to buy Arize, terms undisclosed. Arize's own post commits to nothing specific about Phoenix or OpenInference beyond calling the open-source community important — the sentence to read twice if Phoenix sits in your tracing stack. (official)
-
AI cost tracking & telemetry In the briefing →
A VentureBeat survey of 107 enterprises found 21% have no real-time control over agent spend at all: monitoring after the fact and nothing else. 30% rely on native budget caps, 25% built gateway middleware to intercept a runaway, 25% route down to cheaper models. Only 16% describe their deployed agents as genuinely multi-step and autonomous. (source)
-
AI cost tracking & telemetry In the briefing →
Ramp shipped Router, an API that switches between OpenAI, Anthropic, DeepSeek, Moonshot, Minimax, Nvidia, xAI and Z.ai on a strategy you choose — provider tier, benchmark score, or premium models for the hard queries — with a dashboard for spend, cost and latency. Free through 2026, US only. It records inputs, outputs and tool calls for a year by default, which is the line to read before pointing production at it. (source)
-
AI cost tracking & telemetry In the briefing →
Stripe confirmed it is buying OpenRouter, three days after Bloomberg had the agreement and neither side would comment. Stripe named no price; the New York Times reports $7.5B and Axios above $8B in mostly stock, so the number stays unofficial. OpenRouter says its product, mission and current commitments are unchanged and that it keeps operating independently once the deal closes in a few weeks. The founders' letter calls the rationale overlapping developer customers, having joked about the singularity first. (source)
-
AI cost tracking & telemetry In the briefing →
Snowflake's Cortex AI Gateway will now take
autoin place of a fixed model and route each call itself. A small model tries first and escalates to a larger one as a tool when it stalls; a classifier trained on past queries sends the easy ones straight to the cheap tier. Snowflake's internal testing puts the saving at up to 3x on some workloads. Customers can fence routing to an approved model set. (source) -
AI cost tracking & telemetry In the briefing →
LangSmith Tuned Evaluators are judge models LangChain trains and hosts, attaching scores to production traces without anyone maintaining a prompt or a grader. The first, Perceived Error, beat every frontier model in their benchmark at 82% lower evaluation cost, with some early partners reporting 98%. Threads qualify after two human-AI exchanges and score within 12 hours. (official)
-
AI cost tracking & telemetry In the briefing →
Gartner expects inference cost per agentic workflow to rise more than fivefold through 2028, and names the mechanism: cheaper tokens fund more elaborate workflows, whose consumption swallows the saving. Analyst Will Sommer's line is the one to quote at planning meetings — each generation of capability needs more, and often pricier, tokens. (source)
-
AI cost tracking & telemetry In the briefing →
Bloomberg reports Stripe has agreed to buy OpenRouter for more than $7 billion, roughly five times the valuation it raised at 82 days earlier; Stripe told TechCrunch it doesn't comment on speculation and the figure could still move. OpenRouter claims 8 million users and 400-plus models behind one interface, which for a lot of teams is where model spend is actually metered. (source, source) (unconfirmed)
-
AI cost tracking & telemetry In the briefing →
Cloudflare added agent-level spans to Workers tracing — invocations, model calls, tool execution, approvals, token counts and subagent nesting — following the OpenTelemetry GenAI conventions and exporting to any OTLP endpoint. Free through September 30, then Workers Observability rates: 200,000 events a day on the free tier, or 20M a month included and $0.60 per additional million. Metering counts individual spans, and payload capture is on by default in Flue but off in Think and
wrapAISDK(). (source) -
AI cost tracking & telemetry Weekly recap Read the recap →
Uber exhausted its 2026 AI budget by April and capped tools at $1,500 a month per engineer. GitHub retired GitHub Models outright, breaking CI that had free inference wired into a token it already had, then split its usage report into input, output, cache-read and cache-write lines. AWS documented per-caller Bedrock attribution, and Dynatrace is buying Arize for $915M. (Uber, GitHub, Arize)
-
AI cost tracking & telemetry In the briefing →
Dynatrace is buying Arize for $915M — about $815M cash plus replacement equity — to move AI evaluation upstream of production, closing this quarter or early next. Arize says Dynatrace intends to support Phoenix and keep stewarding OpenInference, the OpenTelemetry-based tracing spec a good deal of LLM instrumentation already emits. (official)
-
AI cost tracking & telemetry In the briefing →
AWS — AgentCore Observability now accepts telemetry from agents running anywhere, not only on its own runtime. The path is OpenTelemetry GenAI semantic conventions through the AWS Distro, over OTLP to CloudWatch with SigV4 auth. AWS demonstrated it on Strands from on-premises and from Google Cloud, and names LangGraph and CrewAI. Setup is seven environment variables and an
opentelemetry-instrumentprefix. (official) -
AI cost tracking & telemetry In the briefing →
Writer shipped Palmyra X6, post-trained from Z.ai's open-weights GLM-5.2, alongside a harness rework it says cuts customer cost by up to 50% on basic tasks. The split is the number worth keeping: its own testing put the harness changes at about 40% on their own, which the company reads as a steadier lever than swapping models. (source)
-
AI cost tracking & telemetry In the briefing →
AWS documented per-caller cost attribution for Bedrock using Cost and Usage Reports 2.0 with IAM principal data switched on, which fills
line_item_iam_principalwith the ARN behind each inference call. Athena queries then split spend by principal, role, model and token type — input, output, cache read, cache write — and IAM principal tags roll it up by team or project. CUDOS 5.8+ ships the dashboards. (official) -
AI cost tracking & telemetry In the briefing →
LangChain pushed its Deep Agents suite — 145 tasks, 6.3 model calls each — through Switchyard and measured where the money actually goes. 7% of turns reached Opus 4.8 and carried 68.4% of the spend; Nemotron Lightning handled 93% of calls for 10.4%. Routing cut cost 74% for about six points of accuracy. The judge model came second at 21.2%, since it runs every turn until escalation and gets no prompt caching. (official)
-
AI cost tracking & telemetry In the briefing →
GitHub — The AI usage report now splits each model's credits into input, output, cache-read and cache-write tokens, which is the detail it takes to explain a charge rather than merely observe one. Separately, MAI-Code-1.1-Flash lists 73% below the model it replaces and bills at a 0.25x premium multiplier for annual subscribers; Business and Enterprise admins have to switch it on. (official, official)
-
AI cost tracking & telemetry In the briefing →
Uber exhausted its 2026 AI budget by April, with individual engineers running $500 to $2,000 a month through Cursor and Claude Code. The response was a hard $1,500 monthly cap per tool and a dashboard engineers can watch. VentureBeat's argument around it: stuffing the context window was never a strategy, and curated agentic memory is the cheaper discipline. (source)
-
AI cost tracking & telemetry In the briefing →
GitHub Models is gone — playground, model catalog, inference API and bring-your-own-key, existing customers with live usage included. The changelog went up July 30 and the brownout reached Simon Willison's GitHub Actions this weekend, failing with a scheduled-retirement message. GitHub points at Microsoft Foundry or Copilot. Free inference attached to a token you already had was a convenient default in CI, and it now needs a budget line. (official, source)
-
AI cost tracking & telemetry In the briefing →
GitHub — The Copilot impact dashboard grew an ROI section: cost per developer per month, that cost as a share of payroll, and pull requests per developer, shown as paired cards for high- and low-adoption cohorts with a salary selector for your own compensation bands. GitHub calls the figures directional estimates derived from AI credit consumption, not measurements. Enterprise and organization levels. (official)
-
AI cost tracking & telemetry Weekly recap Read the recap →
Gateways were the shipping form. Azure API Management's new AI tier organises around models, MCP servers and tools instead of APIs, and LangSmith's LLM Gateway sets limits per organisation, user or API key behind a
base_urlswap. GitHub's Copilot dashboard grew an ROI section, and OpenTelemetry warned that a metric stream silently folds past 2,000 attribute combinations. (gateways, ROI, cardinality) -
AI cost tracking & telemetry In the briefing →
Rippling — The company was on track to spend 40% of its R&D headcount budget on tokens, growing 80% month over month, and hit 605 billion tokens the month its CFO objected. Analysis found 10–15% of employees driving 60% of spend and one engineer at $50,000 a month. AI Spend Console routes prompts to cheaper models per task and scores usage against output; July matched April's volume at 37% of the cost. (source)
-
AI cost tracking & telemetry In the briefing →
Microsoft — Azure API Management's new AI gateway tier is organised around models, MCP servers and tools instead of APIs. It publishes models from OpenAI, Anthropic, Mistral, Bedrock and Vertex behind one name, federates MCP backends from a URL, an OpenAPI spec or a built-in connector, and exports token metrics as OpenTelemetry. Policies are portal cards — token limits, quotas, content screening, model fallback — not XML. Public preview. (source)
-
AI cost tracking & telemetry In the briefing →
LangChain — The LangSmith LLM Gateway is the spend-control half of the same release: swap your
base_url, add provider keys, and set limits per organisation, workspace, user or API key, with PII and secrets redacted before a request leaves. Violations land as traced events next to the run that caused them. Rate limiting and MCP enforcement are not in the beta. (official) -
AI cost tracking & telemetry In the briefing →
OpenTelemetry — A guide to metric cardinality, and the failure mode is quieter than an out-of-memory kill. The SDK caps a stream at 2,000 attribute combinations and folds everything past that into a single point tagged
otel.metric.overflow=true. Totals stay correct; the breakdowns your dashboards and SLOs filter on quietly undercount. Agent telemetry is where this bites, because session ids and raw prompts look like perfectly good attributes. (official) -
AI cost tracking & telemetry In the briefing →
Microsoft — An internal memo from EVP Jay Parikh told engineers that "tokenmaxxing is not what we are optimizing for", with individual divisions handed spending targets and warned about restrictions; CoreAI has not set per-team or per-person budgets yet. GPT-5.6 Sol becomes the default model in internal GitHub Copilot on cost grounds. Some engineers were reportedly running to thousands of dollars a month. (source, source)
-
AI cost tracking & telemetry In the briefing →
Linux Foundation — The Tokenomics Foundation launched with 30 members including IBM, Oracle, SAP, ServiceNow, JPMorganChase and Accenture, two months after June's intent-to-launch notice. The deliverable worth tracking is token cost telemetry landing in the FOCUS billing spec, which would make agent spend comparable across vendors rather than something each dashboard defines its own way. (official)
-
AI cost tracking & telemetry In the briefing →
JetBrains — Internal AI spend rose about 10x in six months and doubled most months, reaching 150 Claude Code seats before the company moved to an enterprise API plan. Consolidating one snapshot by department took four days of spreadsheet work. What replaced it started as one developer's debugging CLI: JetBrains Central now routes installed agents through the company's own AI platform, meters them in shared "AI credits", and sets limits per person, team or group. Over 1,000 developers picked it up within weeks, and the approval queue for terminal agents is gone. (official)
-
AI cost tracking & telemetry In the briefing →
Temporal — The company put 5x more into AI spend and doubled revenue over the same stretch, and CEO Samar Abbas declines to say the second came from the first. Features ship 20–30% faster; CTO Maxim Fateev did in under thirty days with coding agents what had been a six-month job. Whether that speed reaches customers as value is what he says he cannot yet answer. (source)
-
AI cost tracking & telemetry Weekly recap Read the recap →
Databricks split its agent budgets into a self-serve daily circuit breaker and an approval-gated monthly cap. Grafana's Agent Observability reached GA with cost sitting beside latency, and Bedrock added explicit cache breakpoints at a 90% read discount. (Databricks, Grafana)
-
AI cost tracking & telemetry In the briefing →
AWS — Bedrock AgentCore Observability gets the pieces a spend owner asks for: per-session token metrics you can alarm on in CloudWatch, P95 latency thresholds, and an OpenTelemetry trace timeline down to individual tool spans. Two additions are new — Evaluators scoring behaviour against criteria as runs happen, and Insights in preview doing failure triage. (official)