Lab Middleware Reliability: What Happens When the Connection Drops

Every middleware demonstration runs on a healthy network. The analyzer is on the bench, the cable is seated, nothing has rebooted in an hour, and results flow across the screen exactly as the brochure said they would. That demonstration tells you the software can do the easy part.

The interesting question is what it does on a bad day. A cleaner catches a serial cable. A network switch takes its scheduled reboot at 2am. The building loses power mid-write. Somebody double-clicks the desktop icon on a machine where the application is already running and does not realise it. None of these are exotic. In a hospital, all four will happen inside a year.

Lab middleware reliability is defined entirely by what happens in those moments, and it is the part of an evaluation that almost never gets tested before purchase. This post walks through the four failure modes that actually occur, what a correct response to each looks like, and the questions that will tell you whether a vendor has thought about any of it.

The Failure Modes Nobody Specs For

Requirements documents for lab integration are full of protocol coverage and instrument lists. They are almost never explicit about failure. So the four that matter, stated plainly:

• The link drops and comes back. A cable is knocked, a switch reboots, an analyzer is power-cycled for service. The connection is gone for somewhere between two seconds and twenty minutes, then the far end is available again.
• The process dies without warning. Power cut, forced restart, operating system update, an unhandled crash. There was no chance to close cleanly.
• Two copies run at once. An autostart entry plus a user double-click. Both instances try to bind the same ports and open the same database.
• Something fails silently. The socket is open, the indicator is green, and nothing is arriving. This is the worst one, because every surface you would check says the system is fine.

Each has a specific correct answer.

Reconnect: Backoff With Jitter, Not a Retry Loop

When a TCP link drops, the software has to decide when to try again. Two naive approaches both fail.

Retrying immediately and continuously turns one dropped analyzer into a machine spinning at full tilt against a dead address, generating log volume nobody will read and consuming resources the other instruments need. Retrying on a long fixed timer means a link that recovered in five seconds stays down for a minute because that is when the next attempt was scheduled.

The correct behaviour is exponential backoff: retry quickly at first, and lengthen the interval as failures accumulate. A blip is recovered almost immediately; a genuinely dead endpoint is polled at a sensible, low rate rather than hammered.

The refinement is jitter — randomising each interval slightly. Without it, a lab whose analyzers all lost their link to the same switch will have every one of them retry at the same instant, repeatedly, in a synchronised wave. That thundering herd is a real effect, and the fix is a few milliseconds of randomness per device. LISBridge implements reconnect with both backoff and jitter on its TCP client connections.

There is also a state question underneath the timing. During a reconnect, the connection is neither up nor down; if the interface reports it as connected, monitoring is lying, and if it reports it as failed, staff will start troubleshooting something that is already fixing itself. Reconnecting is its own state and should be shown as one — a point that matters for real-time monitoring generally.

Crash Safety: Write-Ahead Logging

A power cut during a database write is the classic way to lose data — and worse, the classic way to end up with a file that will not open at all the next morning.

LISBridge stores results locally in SQLite with write-ahead logging enabled. The mechanism is worth understanding because it is the difference between a bad morning and a very bad one. In WAL mode, changes are appended to a separate log file before being folded into the main database. If the process dies mid-transaction, the database is not left half-modified: on next open, the engine replays what was committed and discards what was not. You get a consistent database and, at worst, the loss of a transaction that had not completed.

Compare that with the alternative, which is a database file that was being rewritten in place at the moment the lights went out. That is the scenario where a lab discovers that its results are on disk but unreadable.

WAL has a second benefit that shows up under load rather than under failure: readers do not block the writer. In a lab where results are arriving from several analyzers while somebody is running a report, that is the difference between a responsive application and one that stutters at exactly the busiest hour.

Liveness and Panic Logging: Knowing That It Failed

Silent failure is the expensive failure. A lab that knows an instrument is down loses an hour. A lab that does not know loses a morning and then has to work out which results are missing.

Two kinds of logging address this, and they are different things.

Panic logging captures the moment a thread dies unexpectedly, with enough context to say what it was doing. Without it, an unhandled failure inside a background worker can take out one analyzer's handler while the application as a whole continues to look perfectly healthy — and the only symptom is that one instrument went quiet.

Liveness logging is the positive signal: periodic evidence that a component is still running and still doing work. It is what lets you distinguish "this analyzer has sent nothing because nothing was ordered" from "this analyzer has sent nothing because its handler stopped existing at 11:40."

Queue state belongs in the same category. A forwarding queue that is full, or one that has been closed underneath a producer, is a condition that must be logged explicitly rather than silently swallowed — a dropped result that nobody records is indistinguishable from a result that was never produced. LISBridge logs full and closed states on both its serial and TCP queues for exactly this reason.

Single-Instance Locking

Two copies of the same middleware running simultaneously is a surprisingly common accident, and its symptoms are bizarre enough to waste a day. Both instances try to bind the same listening ports, so one succeeds and one fails in ways that may or may not be visible. Both open the same database. Results appear twice, or appear once and are attributed unpredictably.

The fix is a single-instance lock, and the word that matters is crash-safe. A naive lock — a file written at startup and deleted at shutdown — becomes a trap the first time the application is killed rather than closed, because the stale lock file now prevents a legitimate restart until somebody deletes it by hand. On the morning after a power cut, that is precisely the wrong behaviour. LISBridge implements a crash-safe single-instance lock on Windows: a second launch is refused, but a stale lock from a previous crash does not block a genuine restart.

A related instance of the same principle applies to licensing. A device that crashed used to be able to hold its licence slot indefinitely, so a lab could hit a device limit because of a machine that no longer exists. Crashed devices now release their slots, and a licence can be unbound from a device from within the application when hardware is replaced.

What to Ask a Middleware Vendor

Reliability claims are unfalsifiable in a sales conversation unless you ask for specifics. Six questions that produce useful answers:

1. What is your reconnect strategy? "It reconnects" is not an answer. Ask about the interval, whether it backs off, and whether it jitters.
2. What happens to a result that arrives while the LIS is unreachable? Queued and retried, or lost? Where is the queue, and does it survive a restart?
3. What happens if the machine loses power mid-write? Ask about the storage engine and its durability mode by name.
4. How do I find out that something failed? Ask to see a log from a real failure, not a feature list.
5. What happens if the application is launched twice? If the answer is a pause, that is informative.
6. Can I unbind a licence myself when a machine dies? If this requires a support ticket, your recovery time depends on their business hours.

You can also test most of this during a trial, and it is the most valuable thing to do with one. Pull a cable mid-run. Kill the process. Launch it twice. A vendor confident in their failure behaviour will suggest it before you do.

For the underlying transports these behaviours protect — serial, network and file-based — see device connectivity. The same reliability model governs the polling cycle described in LIS order pull, where an unreachable endpoint triggers a per-node cooldown rather than an endless retry.

FAQ

How long does a dropped analyzer take to reconnect?

A brief interruption typically recovers within seconds, since backoff starts short. A longer outage lengthens the retry interval, so recovery after the far end returns is on the order of the current backoff window rather than instant — a deliberate trade to avoid hammering an endpoint that is genuinely down.

Are results lost if the LIS is unavailable?

No. Results are parsed and stored locally first, then delivered. If delivery cannot complete, the result remains in local storage — which is crash-safe — and delivery is retried. Local-first storage is also what allows LISBridge to keep working during an internet outage.

Does reliability differ between serial and TCP connections?

The failure modes differ but the principles do not. Serial links fail physically and their queue behaviour is logged explicitly; TCP links fail at the network layer and use reconnect with backoff and jitter. Both are surfaced in the same monitoring view.

Does any of this depend on the cloud?

No. LISBridge is a native desktop application built in Rust, running locally with local storage and offline-enforced licensing. Reliability is not contingent on a remote service being available.

Start your free trial — no credit card required. lisbridge.com/pricing