Skip to Content

The Loop Asia

Insights

Analysis and commentary from The Loop Asia — AI, APIs, and technology leadership across Asia-Pacific.

Thanks for subscribing!

No spam. Unsubscribe any time.

Diagnosis is not the bottleneck — access is

AI can tell you exactly what's wrong with your Kubernetes cluster faster than ever. It can't fix the org chart that decides who's allowed to act on it.
October 7, 2024 by
Diagnosis is not the bottleneck — access is
Jon Scheele

This deep dive expands on the Loop Asia conversation with Nilesh Gule, Enterprise Architect, Microsoft MVP, and Docker Captain.


Ask most technology leaders why a Kubernetes incident took four hours instead of forty minutes and they'll reach for a technical answer — the ingress config was wrong, the tag didn't publish. Nilesh Gule gave those answers too, talking to Jon Scheele — but partway through, said something that had nothing to do with YAML.

He was describing what happens when a change from development breaks something operations owns: "The development team might not have the right access to make some changes," he said, "and it could result in a scenario where a change needs to be done, but only operations team can do it." Jon pushed: even with access, a developer may never have touched that component and need help. Nilesh agreed: "It could be quite complex."

Two gaps, and only one closes with better tooling

The first gap is knowledge: nobody fully understands the layer where the problem lives. K8sGPT closes it — reading cluster state and explaining it in plain language, in any of its eighteen-plus supported languages. Nilesh sees this as the tool's biggest win: a newly onboarded engineer finds root cause faster and learns the system's shape too.

The second gap is access: the person who understands the problem isn't the person permitted to fix it. No amount of plain-language diagnosis touches this. K8sGPT can tell a developer exactly why the ingress is misrouting traffic — but if the fix sits inside operations' RBAC boundary, the diagnosis just becomes a better-written ticket in the same queue.

Where the two gaps compound

The CI/CD detail Nilesh raised earlier is the compounding case: a tag never publishes to the registry, the Helm chart pulls an image that was never pushed. That's usually a developer's own pipeline problem — until the fix needs a Helm release or namespace in operations' territory, and the same access boundary resurfaces, this time inside a deployment failure instead of a runtime one.

Organisations moving fast on an AI diagnostic tool tend to make a quiet mistake: they measure success by mean-time-to-diagnosis, the number the tool visibly improves, and skip mean-time-to-remediation — the number that depends on whether the right person has the right access once diagnosis lands. A team can get far better at knowing what's wrong while staying just as slow at being allowed to fix it, and nobody notices until someone asks why on-call still feels just as bad.

The fix isn't a tool. It's an ownership map.

Closing the access gap is an org design decision, independent of any tool: which team owns which namespace, what RBAC follows, whether the boundary matches how incidents flow, not the reporting lines. Teams that do this well draw the boundary around blast radius, not who wrote the code — a developer gets standing access to redeploy their own service, but a shared ingress or cluster-wide policy stays gated behind whoever owns the shared risk.


Buying visibility into what's broken but not fixing how fast incidents close?

Deciding who's allowed to act — and rebuilding access grants around that, not last year's org chart — is exactly the structural work I do alongside technical leaders.

See how I can help →