The Camera Never Blinks: Validating Automated Colony Counting Before It Signs Off a Result
Walk into a growing number of UK microbiology departments in 2026 and you will find a camera on a stand, a plinth-sized incubation carousel or a benchtop reader with a screen showing plates rendered as clean, countable dots. The proposition is attractive: fewer repetitive reads, more consistent counting, shorter turnaround and a digital record of every plate image. None of that is false. What is often thin is the paperwork that sits between the sales demonstration and a result released into a patient's record.
The purpose of this piece is not to talk laboratory managers out of adoption. It is to set out what a defensible verification and governance package should contain, where algorithmic failures tend to concentrate, and how to keep accountability where it belongs — with a named, competent human — without freezing the workflow in the process.
What the vendor is actually selling you
Start by separating three functions that are frequently bundled in a single pitch. Image capture and colony counting is one thing. Classification of colonial morphology is a second. Species-level identification, or an interpretive statement about resistance, is a third. Each carries a different burden.
Under the UK Medical Devices Regulations, software intended by its manufacturer to be used for diagnostic purposes is a medical device, and you should expect a UKCA mark, a stated intended purpose and instructions for use that match what you plan to do with it. Ask directly whether the tool is presented as decision support or as an autonomous reader, and get the answer in writing. If the manufacturer's intended purpose extends to identification, treat that as a diagnostic claim and interrogate the evidence behind it.
There is a second trap. An uncertified or over-claimed tool does not transfer its regulatory risk to the vendor once it is in the workflow. If your laboratory releases results derived from it, the laboratory carries that in its accreditation scope. A useful rule of thumb: the sales contract should tell you precisely when the vendor stops being responsible, and your verification plan should begin exactly there.
Verification under UKAS expectations
ISO 15189 places the obligation on the laboratory to demonstrate that equipment and methods are fit for the intended use before being placed into service, and to keep that assurance under review. Where a commercial, marked device is used strictly according to its instructions for use, verification against the laboratory's own performance specification is normally the proportionate route. Where the laboratory configures the tool — setting size thresholds, excluding artefacts, choosing which plates route to auto-release — it is arguably shaping the method, and the documentation should reflect that.
In practice, a credible package contains:
- A written intended-use statement in the laboratory's own words, not the vendor's.
- Pre-defined acceptance criteria, agreed and signed before any data are collected.
- A description of the reference method and reader competence for the comparator arm.
- Records of installation and operational checks, plus a performance qualification set drawn from your own specimen mix.
- A change-control procedure covering firmware, algorithm version and interface updates.
- A named individual accountable for the tool within the quality system.
The ordering matters more than it sounds. Acceptance criteria written after the data are seen are not criteria; they are rationalisations.
Where the misses hide
Aggregate accuracy figures are poor guides to clinical risk, because the errors are not evenly distributed. Colony counting failures cluster at the boundaries, and the boundaries are where clinical decisions are made.
Expect trouble with counts near the lower limit of detection, where a faint colony or a small translucent one sits close to the segmentation threshold. Expect trouble with spreading organisms — Proteus swarming across an entire plate, moulds radiating from a single point — where the algorithm may log one object where a human sees a lawn. Expect trouble with mixed growth, with pinpoint colonies at an early read that would have been visible at the intended incubation endpoint, and with chromogenic media where colour shift rather than size carries the meaning.
Practical artefacts matter too: condensation on the lid, reflections from overhead lighting, marker pen across the base, a cracked plate, a pour plate with uneven depth. None of these trouble an experienced biomedical scientist for long. Several will trouble a model trained largely on clean reference images.
Most important is the asymmetry of the failure modes. Overcounting a urine culture is an irritation. Missing a single colony in a sterile-site specimen, or failing to flag a mixed culture that warrants further work, is a clinical event. A verification protocol should therefore report positive and negative agreement separately, with the false-negative rate at clinically actionable thresholds stated explicitly rather than buried in a confusion matrix.
Building a challenge set that earns its name
A set of two hundred plates drawn at random from routine work will tell you how the tool performs on the easy majority and very little else. Deliberate enrichment is required. Compose the set so that it contains a substantial proportion of negatives, a band of low-count positives, known discordant or atypical plates held back for exactly this purpose, and a spread of media types and specimen sources relevant to your service.
Blind the reads. Have two experienced readers score the plates independently and adjudicate disagreements by consensus before comparing with the algorithm. Record count bias as well as categorical agreement, and note reading time — a tool that is accurate but slow will quietly erode the case for it.
Two further elements belong in the same protocol. First, data integrity: image retention periods, audit trail completeness, and the route by which a count reaches the laboratory information system without manual transcription. Second, the interface between software and the quality system, so that an algorithm change is treated as a method change rather than an unnoticed download.
Keeping the human accountable
"Human in the loop" has become a phrase that reassures without specifying anything. Replace it with concrete arrangements. Which results release automatically, and on what criteria? What does the reviewer actually see — the image, the overlay, the count, the previous result? How is a disagreement recorded, and does that record feed back into the vendor or the quality review? What training and competence assessment applies to reviewers, and how is their agreement with the algorithm monitored over time?
Automation bias is the quiet risk here. A plausible number on a screen attracts less scrutiny than a plate held up to the light, particularly at the end of a long shift. Periodic re-reading of a sample of auto-released plates by a competent scientist is a low-cost countermeasure, and it generates the discordance data that external assessment will eventually ask for.
The ongoing audit and the exit clause
Set out from the start how the tool will be watched: a rolling re-read sample, a discordance rate reviewed at defined intervals, a route for reporting suspected misses as incidents, and an annual review of performance against the original specification. When a major software version arrives, run a reduced verification set before it goes live and record the outcome.
Commercially, three clauses are worth arguing over. Notification of algorithm changes. Continued access to your own images in a usable format. And a workable exit plan, because equipment that cannot be replaced without losing its archive is equipment you no longer control.
Automated reading is likely to become unremarkable in UK laboratories, and it should. The technology is not the difficulty. The difficulty is proving, on paper and in practice, that a machine's count is a result your laboratory is prepared to defend — and that when it is wrong, a person notices.
