This deep dive expands on the Loop Asia conversation with Koo Ping Shung, Practicum Director at Data Science Rex.

Koo Ping Shung made an admission most vendors won't say out loud: if your organisation uses machine learning — particularly supervised learning — false positives and false negatives are built into the method, not an edge case you'll engineer away. A model making decisions at scale will, with certainty, make some wrong ones. The only question is what happens to the person on the receiving end.

For whoever owns the outcome of an AI-enabled process — not the model, but the business result and reputational exposure — this is the part of governance that lands on your desk: not model accuracy, but what happens after the model is wrong.

The robodebt lesson

I raised Australia's robodebt scheme because it's the clearest case of what happens when this mechanism is missing: an automated system decided people had been overpaid government benefits and needed to repay the difference, with no meaningful way to contest it before the harm was done. The scandal wasn't about the algorithm's error rate — it was the absent path to say "this is wrong."

The commercial version is quieter but structurally identical: a credit decision, a claims denial, a churn-risk flag, a frozen account. Each will occasionally be wrong. The organisations that get hurt aren't the ones with the least accurate models — they're the ones where a wrongly-affected customer had nowhere credible to go.

Koo's point about consumer psychology

Here's the nuance that's easy to miss: a customer who calls in and says "your algorithm is wrong" has already assumed the machine is at fault, before any investigation happens — itself a kind of bias. That shapes the feedback channel: not one that assumes the company is always wrong when challenged, but one that treats every challenge as worth a second look, without conceding the outcome in advance. It's a different brief than a complaints form: the review needs independence to catch a real error, and rigour to close out one that doesn't hold up, without the customer feeling dismissed.

The filtering problem is its own risk

Make the appeals channel too easy to use, Koo noted, and you invite the vexatious-complaint pattern legal systems have wrestled with for centuries — people appealing regardless of whether the decision was wrong. Handled naively, that drowns the review team in noise or filters out legitimate complaints along with the vexatious ones. The human review layer has its own false-positive and false-negative problem, structurally identical to the model underneath it — designing it well is an operational problem, not a policy statement.

What this means before you deploy, not after

Organisations that get this right treat the feedback mechanism as a launch requirement, not a support-desk afterthought. Before an AI-enabled process goes live, you should be able to answer plainly: where does a wrongly-affected person go, how fast, and who can reverse the machine's call. If those answers don't exist, the system isn't ready for production, however accurate the model tests out to be.