Proof
Does it catch harmful changes, and only those?
We seeded realistic dosing bugs and harmless refactors into oref0 and ran each through four checks: what exists today, two simpler checks, and Virtual Trial. A good check catches every harmful change and flags no harmless one.
Run 2026-10-17 15:39:23 UTC on commit ea81d331: 6 virtual patients x 10 scenariosScenariosWhat it is. Eight simulated days, each with meals and at most one fault: a normal day, a forgotten bolus, a large dinner, a sensor dropout, a false low, a sensor jump, a pump history that starts with a resume, and a bolus missing from history.Why it matters. Each fault is modeled on a failure reported in real dosing software or a recall, so the trial tests what has actually gone wrong before. per version, 27830 seconds. Reproduce with python -m bench.run. Worst-day hunter on commit 86c948f8 (4922 seconds; python -m bench.hunter).
HeadlineBase and headWhat it is. Base is the dosing code before the change; head is the code with the change. The trial runs both on the same patients, meals, faults, and random seed.Why it matters. Because everything else is identical, any difference in outcomes comes from the code change.
oref0's own unit testsoref0's own tests already failWhat it is. Run as published, 9 of oref0's 243 tests fail on its current development branch (7 on the 0.7.1 release), because the code changed and the tests did not.Why it matters. Before Virtual Trial these failures were ignored, so a new failure would have been ignored too. Now any new failure blocks the change.
What exists today: fixed inputs, expected outputs.
Open-loop replayOpen-loop testingWhat it is. Feeding fixed inputs to the algorithm and comparing its output with an expected answer, with no body in between.Why it matters. It catches typos but cannot show harm: a dose that looks reasonable once can stack up into a dangerous low over hours.
Replays thousands of real inputs from the trial; flags any changed decision, harmful or not.
Release safety envelopeMax safe basalWhat it is. oref0's hard cap on a temporary basal rate: the lowest of the configured maximum, 3 times the highest daily basal, and 4 times the current basal.Why it matters. Every deployed engine release is checked to never exceed it.
Our engine release gate: oref0's own hard limits on 150 inputs.
Virtual TrialClosed loopWhat it is. The algorithm doses, the virtual body responds, the next reading feeds the next decision, every 5 minutes.Why it matters. Mistakes compound over hours. Checking one decision at a time misses that.
Closed loop on virtual patients: 10 fixed scenarios plus a search for the worst day, human-set limits.
Every change, every checkSafety gateWhat it is. The pipeline step that reads the trial's verdict and passes or blocks the change.Why it matters. It turns the trial into a decision: risky changes cannot merge or ship.
| Change | Kind | Unit tests | Open loop | Envelope | Virtual Trial |
|---|---|---|---|---|---|
| Insulin-on-board safety cap raised 4x A moved safety limit: oref0 may stack four times the configured insulin. | Harmful: too much insulin | Stops it 1 new failures | Stops it 356 of 4968 decisions changed | Lets it through | Stops it lbgi (paired) |
| Insulin sensitivity halved A wrong factor: each unit is assumed to lower glucose half as much, so more insulin is given. | Harmful: too much insulin | Stops it 6 new failures | Stops it 1596 of 4968 decisions changed | Lets it through | Stops it time_below_70_pct, lbgi (paired) |
| Low-glucose suspend threshold lowered A moved safety threshold: insulin keeps flowing closer to a low before oref0 stops it. | Harmful: too much insulin | Lets it through | Stops it 185 of 4968 decisions changed | Lets it through | Stops it worst day found (severe lows), worst day found (severe lows) |
| Insulin on board undercounted by half The failure class behind Trio #898 and AndroidAPS #2580: insulin already given is missed, so more is given. | Harmful: too much insulin | Stops it 12 new failures | Stops it 1260 of 4968 decisions changed | Lets it through | Stops it time_below_70_pct, lbgi (paired) |
| Carb effect doubled A unit or ratio error: carbs are predicted to raise glucose twice as much, so meals get extra insulin. | Harmful: too much insulin | Lets it through | Stops it 320 of 4968 decisions changed | Lets it through | Stops it lbgi (paired) |
| Max safe basal doubled A moved hard cap: temporary basal rates may go twice as high as oref0's own limit. | Harmful: too much insulin | Stops it 10 new failures | Stops it 544 of 4968 decisions changed | Stops it 25 outside limits | Stops it time_below_70_pct, severe_low_events, lbgi (paired) |
| Falling glucose trend ignored An ignored input: the most optimistic recent trend is used, so falls are noticed late. | Harmful: too much insulin | Lets it through | Stops it 942 of 4968 decisions changed | Lets it through | Stops it time_below_70_pct, lbgi (paired) |
| Insulin sensitivity doubled A wrong factor in the other direction: too little insulin, glucose runs high. | Harmful: too little insulin | Stops it 10 new failures | Stops it 2433 of 4968 decisions changed | Lets it through | Stops it hbgi (paired) |
| Insulin-on-board cap cut to a quarter An over-tight limit: oref0 cannot give enough insulin to correct highs. | Harmful: too little insulin | Stops it 5 new failures | Stops it 1245 of 4968 decisions changed | Lets it through | Stops it time_in_range_70_180_pct, hbgi (paired) |
| Refactor: rename a variable Same behavior, different name. | Harmless | Lets it through | Lets it through 0 of 4968 decisions changed | Lets it through | Lets it through |
| Refactor: equivalent arithmetic min_bg - 0.5*(min_bg-40) rewritten as (min_bg + 40) / 2, which is the same number. | Harmless | Lets it through | Lets it through 0 of 4968 decisions changed | Lets it through | Lets it through |
| Refactor: reorder a multiplication Same factors in a different order. | Harmless | Lets it through | Stops it 1 of 4968 decisions changed | Lets it through | Lets it through |
| Change an explanation message Only the text oref0 writes in its reason string changes. | Harmless | Stops it 3 new failures | Lets it through 0 of 4968 decisions changed | Lets it through | Lets it through |
| Add a comment No code changes at all. | Harmless | Lets it through | Lets it through 0 of 4968 decisions changed | Lets it through | Lets it through |
| Finer temp basal rounding (0.025 U/hr) Supports pumps with a finer increment: many rates change by up to 0.025 U/hr, a clinically negligible amount. | Harmless | Stops it 5 new failures | Stops it 463 of 4968 decisions changed | Lets it through | Lets it through |
| Low-glucose suspend threshold raised slightly A more cautious setting: oref0 suspends a little earlier when a low is predicted. | Harmless | Stops it 1 new failures | Stops it 154 of 4968 decisions changed | Lets it through | Lets it through |
Green means the check made the right call: it stops a harmful change, or lets a harmless one through. Red means it got it wrong.
Real bugs from oref0's historyReal bugs from oref0's historyWhat it is. Dosing bugs that oref0's own maintainers found and fixed. We undo each fix on today's code and run every check, as if the bug were a new merge request.Why it matters. Seeded bugs show the method works on bugs we invented. Real bugs show what it would have done on bugs that actually shipped.Source
Bugs that oref0's maintainers found and fixed, years after they shipped. We undid each fix on today's code and ran every check, as if the bug were a new merge request. 6 virtual patients x 10 scenarios per version, plus the worst-day hunterWorst-day hunterWhat it is. Besides the 10 fixed scenarios, the trial searches dozens of varied days (meal times and sizes, a mis-counted meal, a sensor gap, a false low, a basal rate set wrong) for the day that hurts the new code most compared with the old.Why it matters. Fixed scenarios test days someone thought of. The hunter looks for the day nobody did, If it finds a day where the new code causes a severe low the old code did not, it replays that day 12 slightly different ways and blocks only if the harm holds up, so a lucky borderline day cannot block a change. If the change also clearly lowers low risk overall, it is a trade-off and a person decides.Source.
| Bug | Unit tests | Open loop | Virtual Trial | What it means |
|---|---|---|---|---|
| Low-glucose suspend skipped (bug fixed in 2017) Before oref0 PR #476, suspending insulin at low glucose also required insulin on board above a mis-scaled bound (basal x sensitivity x 20/60), so oref0 could keep insulin flowing while glucose was already low. The fix, 2017 (647a35f9) · original logic ported to today's code | Lets it through | Stops it 184 of 4968 decisions changed | Stops it worst day found (severe lows) | Blocked. The hunter found a day with a new severe low and confirmed it on 12 variations of that day. LBGI +9%, worse in 33 of 60 runs, p=0.035 HBGI +0%, worse in 20 of 60 runs, p=0.98 Hunter: 59 days searched; worst: child#001, lowest 55.4 -> 53.1 mg/dL; suspect day (child#001) rerun 12 ways: head lower on 12, p=0.0002, confirmed |
| Last temp age wrong when the last temp is unknown (bug fixed in 2023) With no recorded last temp basal, oref0 assumed it had just been set (age 0) and could refuse to change a running temp when glucose data was stale. The fix, 2023 (0cf223c7) · fix reverted with git | Lets it through | Stops it 3 of 4968 decisions changed | Lets it through | Passed: no change in outcomes measured. The hunter's one bad day did not hold up on 12 variations, so it did not block. LBGI +0%, worse in 2 of 60 runs, p=0.5 HBGI -0%, worse in 1 of 60 runs, p=0.97 Hunter: 59 days searched; worst: child#001, lowest 55.4 -> 51.8 mg/dL; suspect day (child#001) rerun 12 ways: head lower on 6, p=0.6128, not confirmed |
| Overreacting to negative deviations after a big drop (bug fixed in 2017) Before oref0 PR #615, one large drop in glucose made oref0 project the drop forward too strongly and cut insulin too much, which can leave glucose high. The fix, 2017 (dd6e1a64) · fix reverted with git | Lets it through | Stops it 426 of 4968 decisions changed | Lets it through | Passed, with the harm measured: high blood sugar risk +8%, worse in 59 of 60 runs, p=5.3e-17. That is under the 15% block limit, so it is reported for a person, not blocked. LBGI -13%, worse in 1 of 60 runs, p=1 HBGI +8%, worse in 59 of 60 runs, p=5.3e-17 Hunter: 59 days searched; worst: child#001, lowest 55.4 -> 53.4 mg/dL; suspect day (child#001) rerun 12 ways: head lower on 6, p=0.6128, not confirmed |
| Rate at the safety cap returned as text (bug fixed in 2016) Before oref0 PR #177, when the requested rate hit max safe basal oref0 returned it with toFixed(1), a text value instead of a number rounded for the pump. The fix, 2016 (244ba9dd) · original logic ported to today's code | Stops it | Stops it 112 of 4968 decisions changed | Lets it through | Passed: no change in outcomes measured. LBGI -2%, worse in 2 of 60 runs, p=1 HBGI +0%, worse in 13 of 60 runs, p=0.26 Hunter: 48 days searched; worst: adolescent#003, lowest 72.1 -> 69.9 mg/dL |
Of four dosing bugs that shipped, oref0's own tests caught one (a value-type bug with no effect on outcomes). Virtual Trial blocked the one that let insulin flow during a low, which the unit tests missed, and measured the consistent rise in highs from a second. Open-loop replay flagged all four but cannot say which ones harm anyone.
In a real pipelinePipelineWhat it is. The automatic sequence of jobs GitLab runs for each change: tests, security scans, the trial, the gate, and deployment.Why it matters. It makes the safety check impossible to skip: a blocked gate stops the change.
A merge request that raised oref0's insulin-on-board cap 4x ran the full trial (30 patients, 10 scenarios, 6 parallel jobs) in GitLab CI. Low blood sugar risk rose 30%, in 222 of 300 matched runs (p = 1.5e-50): blocked. A minute later the explain agent posted why, on its own.
On another real codebaseReusable CI componentWhat it is. Virtual Trial packaged as a GitLab CI/CD component. A project whose dosing code is oref0 or a fork of it adds the trial and the gate with one include line.Why it matters. Safety checks only help if projects can adopt them. We proved it on Trio's dosing code, which ran without changes.Source
Trio's dosing code (trio-oref) ran through our reusable CI component, unchanged: before vs after Trio's November 2025 dosing changes. Full trial in 6 parallel jobs plus 48 hunted days: passed, no limit crossed. Adopting it is one include: line.
When production regressesIncident response agentsWhat it is. When the virtual clinic's patients do worse on a new engine version, an hourly job opens a GitLab incident and calls two agents. One finds the dosing change behind it; the other undoes exactly that change in a new merge request.Why it matters. Fixing a bad release starts within the hour, with no one on call. The undo still goes through the full trial, and a person approves it by merging.Source
In a drill, a harmful change shipped on a test branch. The clinic's incident opened, and the incident agents found the dosing commit behind it and opened a merge request undoing exactly that commit, in about three minutes, on their own. The full trial passed the undo (low blood sugar risk down 29%), and it waits for a person to merge.
Open the incident · Open the revert merge request · Open its gate job
In productionCanary releaseWhat it is. A new engine version is deployed but receives no traffic until it passes a safety check on 151 dosing decisions. Then it gets all traffic and is checked again.Why it matters. A broken release never serves anyone. This caught a real crash on Oct 13: the bad version never received traffic.
On Oct 13 a new engine build crashed at startup. The canary never gave it traffic; the public engine kept serving the previous version.