This deep dive expands on the Loop Asia conversation with Aki Ranin.

In his apidays Singapore 2026 keynote, Aki Ranin cited a number everyone repeats: Shopify took a use case from roughly $5 million a year on a frontier model down to about $70,000 a year on a 70-billion-parameter open model, without losing accuracy. The episode gives that number a sentence. The number is a result — the process behind it is what's worth stealing.

Start at the top, not in the middle

Aki's framing: "You start from the frontier model — whatever it is — and if it works well, then you just keep going down. The optimisation techniques are there." Most cost-conscious teams do the opposite — start cheap, upgrade if it isn't good enough. Aki inverts it: establish your ceiling first, on your own evaluation set.

Bottom-up testing asks "is this cheap model good enough." Top-down testing asks "how much am I giving up, and is it worth what I'm saving." Only the second is something you can defend to a CFO.

Evals are not a nice-to-have — they're the whole mechanism

Aki was blunt about the actual gap: "The step up from something you can whip up on Claude to something that needs to run your business is specifically the measurability aspect." Consumer AI tools create "a false sense of confidence" — a plausible output with no structured way to know if it's right, repeatable, or safe to depend on.

An eval, as Aki uses the term, is close to a test suite for a model's behaviour on your use case — "basically just software testing in a way for AI." Three parts: representative inputs drawn from real cases, not assembled afterward to justify whichever model you'd already picked; a defined standard for what counts as a correct output; and a scoring method that compares versions over time. Built before you compare models, not after.

Without it, a question has no answer: "What is the percentage risk of hallucinations? If you can't tell me that, it's a complete no-go." That's what a CFO, CIO, or board risk committee is entitled to ask before signing off on anything touching customer-facing work, financial data, or regulated processes. No number, no governance — there's nothing yet to govern.

Cascades, not a single giant model

The end state isn't "replace the frontier model with one smaller model." It's a cascade of task-specific, right-sized models — one for email triage, another for document sorting, the big model reserved for the genuinely hard cases. Beyond cost, Aki argued this buys auditability: "there's a cascade of different agents doing certain things, which gives you more transparency and trustworthiness" than "one giant model that can tell you whatever it wants and fool you." A bounded, specialised model doing one job is easier to eval, monitor, and explain to a risk committee than one generalist making judgment calls across the whole business.

Where this connects to the AI mandate problem

This is the gap Blue Connector sees most often: growth-stage companies with a board directive to "do something with AI" and no framework for what that means operationally. The mandate exists; the evaluation discipline that lets a technical leader say "here's our minimum viable model, here's the validated accuracy, here's what we're saving" usually doesn't. Building it is what lets a technical champion walk into a board conversation with a number instead of a demo.