← Documentation
Text size
Innovation · License Commander

The Self-Healing Engine Room

How License Commander watches its own machinery, fixes the failures that are safe to fix on its own, and tells a human, in plain language, about the ones that are not.

Last revised:

Most automation is invisible until it breaks, and then it breaks silently. License Commander is built the other way around: it treats its own reliability as a feature, reading its logs, healing what it safely can, and surfacing the rest where a person will actually see it.

License Commander is not one program. It is a few dozen small automations, cloud functions, an ETL container, and scheduled jobs and browser robots on a campus machine, each doing one job. Any one of them can fail: a vendor changes a page, a login expires, a network call times out.

Left alone, those failures hide. A cloud function’s error scrolls past in a log nobody reads. A browser robot wedges at two in the morning and just stops. By the time anyone notices, a week of orders is stuck.

So the Engine Room does three things in order: it reads its own logs for recurring trouble, it fixes the failures that are safe to fix without a human, and for anything weightier it writes a plain-language note where the operator will see it, once, not a storm of alerts.

By the numbers

28

cloud functions and robots the Engine Room watches as one system.

1

page, called Sick Bay, for what broke, what got fixed, and what needs you.

Once

a persistent failure is escalated exactly once, after the platform tries the real fix itself.

Self-relogin

an expired vendor login is repaired from a fresh session, with no human required.

Cloud + floor

the box robots now report their health right beside the cloud functions.

Every day

each approved Adobe and Microsoft seat is checked against the vendor, and a seat that never landed is put back.

Reading its own logs

Mining the machinery for the errors that repeat, not the ones that do not.

The platform continuously mines its own logs across every cloud function for the errors that repeat, the ones that signal a real, recurring problem rather than a one-off blip. A pattern raises an alert; a single hiccup does not.

Why it is clever

It distinguishes a pattern from noise. A one-time timeout is ignored; the same error three nights running is surfaced. Hard failures trip an alarm immediately, and everything else is rolled into a weekly digest, so the operator is paged for genuine emergencies and briefed, calmly, on the rest.

The alarm has earned its place. In August it caught the free-software eligibility check failing on every call, and the fix shipped that morning. In October it caught a handful of empty requests crashing the order endpoint; the endpoint now turns an incomplete request away with a clear error instead of falling over.

✓ Toil removed reading raw cloud logs by hand, hoping to spot a pattern before it becomes an outage.

Sick Bay, the honest ledger

One page for what broke, what got fixed, and what still needs you.

Everything the Engine Room learns lands in Sick Bay, a living log inside License Commander. Each entry is one of a few plain kinds: fixed on its own, shipped, watching, or needs you. It is the single place to see the platform’s health, and the weekly digest links straight to it.

Why it is clever

Status is never a color alone. Every entry carries a word, “Fixed,” “Needs you,” so it reads at a glance and for anyone. Nothing important is buried in an inbox, and the record is deliberately honest, including about the things that still need a person.

Since October 17, an entry can also close itself. When the daily provisioning check comes back clean, it marks its own earlier “Needs you” entries as fixed, with the date it resolved them, so a problem that was already solved stops asking for attention.

✓ Toil removed reconstructing what went wrong from scattered emails and memory.

It treats its own reliability as a feature. It reads its logs, heals what it safely can, and tells you plainly about the rest.

Fixing the safe things itself

The repairs a machine can make without asking a human.

For a well-understood, low-risk failure, the Engine Room simply fixes it. The clearest example: when a browser robot’s vendor login quietly expires, the platform logs itself back in from a fresh session, with no human re-login, and carries on. Expected, harmless noise gets silenced; a missing guard gets added.

The list of safe repairs keeps growing. The daily provisioning check puts back any approved Microsoft seat that never reached its group, and since October 13 any Adobe seat too, once Adobe confirms the miss live. The job that keeps License Commander’s copy of the Adobe roster now retries a failed page instead of throwing away the whole pull, waits patiently when Adobe asks it to slow down, and refuses to retire more than a fifth of the roster in a single run, so one bad run can never wipe it out. And a nightly job finds records where a number was saved in a field the database expects as text, a small mismatch that quietly blocks every later update to that record, and converts them, staying silent unless it actually fixed something.

Why it is clever

The dividing line is deliberate. Trivial and safe, it does, and then reports. Anything that changes behavior, touches money, or is fuzzy, it does not touch, it escalates. The automation earns trust precisely by never quietly doing something risky.

✓ Toil removed a person waking up to re-login a robot or clear a stuck job by hand.

Escalating the weighty things, once

A single clear note when a human is genuinely needed, not a wall of alerts.

When something genuinely needs a person, the Engine Room says so plainly and only once. The always-on Canva browser robot used to email an operator every time a watchdog restarted it, six or seven times in a bad night. The first fix was to try the real repair itself and raise one clear “needs a hand” note only if that could not get through. Then, in late August, Canva released an official API, and the robot was retired altogether: Canva work now runs as a cloud function, with no browser and no login to lose. The same rule carries on for the jobs still on the campus machine. The Adobe device robot’s early-morning run stays quiet when Adobe stumbles, a second attempt runs later that morning, and only that last attempt emails if the problem is still there.

Why it is clever

It knows the difference between a transient wobble and a persistent problem. A first failure gets a quiet note; a repeating one is escalated exactly once, after a real repair has been attempted. The result is an operator who trusts the alerts, because they are rare and they are real.

✓ Toil removed an inbox full of restart notices that trains you to ignore all of them.

One pane for the whole fleet

The cloud functions and the box robots, watched in the same place.

The trickiest failures were the ones outside the cloud: the browser robots that run on a physical machine and log to that machine, invisible to the cloud’s own monitoring. The Engine Room bridges them in, so a box robot’s health shows up in Sick Bay right beside the cloud functions. One pane, whether the trouble is in the cloud or on the floor.

Why it is clever

Two monitoring worlds that never used to talk now report to the same ledger. A stuck browser robot no longer hides on a machine nobody is watching; it surfaces where the cloud failures do, with the same calm cadence, one watch, then needs-you if it persists.

✓ Toil removed a whole class of failures that lived on a box and never reached anyone.

Seeing the changes nobody logged

Every write to a license record, no matter who or what made it.

License Commander writes a history entry for every change it makes. But other writers touch the same records, such as the nightly campus directory sync and the occasional one-off script, and their changes used to leave no trace. In October a seat flipped from revoked back to active with no record of who or why. So since October 14 the platform listens to the database’s own change stream, which sees every write, and records changes to the fields that matter: status, account, billing stamps, transfers, fiscal officer, unit, and removal markers. Directory noise, like names and HR fields, is ignored.

Why it is clever

It only speaks up about what nobody explained. A change the app already logged within five minutes is not repeated. Anything left over shows as an “Untracked” row in that license’s activity timeline, and every Monday morning a digest emails the week’s untracked changes, or sends nothing at all on a quiet week. The stream can tell what changed and when, but not who made the change, so the record says “Untracked” rather than guessing at a name.

✓ Toil removed piecing together an unexplained change from guesswork and memory.

An engine room that watches itself

Reading its own logs, checking every day that each seat really landed, fixing the safe failures without a human, noticing changes nobody logged, escalating the weighty ones exactly once, and keeping one honest ledger of it all. The platform does not wait for someone to notice it is broken. It notices first, fixes what it can, and asks for help, in plain language, when it cannot.