Highly Available FIX: From “Just Start Another Server” to Safe Failover

If one FIX server dies, why not just start another one?

For a stateless HTTP service that question is almost boring. Put two instances behind a load balancer, add health checks, let the platform route around the failure. Done.

FIX doesn’t work like that.

A FIX connection is long-lived, stateful, ordered, and tied to a session identity. The engine tracks sequence numbers, heartbeats, resend history, Logon/Logout state, and messages that may need replaying after a reconnect. The FIX Session Layer explicitly requires the next inbound and outbound sequence numbers to survive across connections so a session can recover correctly.¹

So high availability here isn’t really about keeping two processes alive. The actual question is much narrower, and much harder:

How do I prove that exactly one node owns a FIX session during every failure scenario?

That question dragged me through HAProxy, Pacemaker, Corosync, quorum, fencing, persistent FIX state, and eventually etcd. This is the architecture I landed on, and more importantly why.

Click the diagram for the full-size version.

Initiator and acceptor are roles, not different protocols

A FIX session has two sides.

The initiator opens the TCP connection. The acceptor listens for it. With QuickFIX/J that distinction is just a config line — ConnectionType=initiator or ConnectionType=acceptor.²

Simplified, traffic flows like this:

Broker / Client
FIX Initiator
      |
      | TCP + FIX
      v
FIX Acceptor
      |
      +--> OMS
      +--> Risk Engine
      +--> Buying Power
      +--> Market Data / Reference Data
      +--> Internal APIs

The acceptor usually isn’t the whole trading platform. It’s the FIX-facing gateway that translates session traffic into calls or events for everything behind it. A NewOrderSingle arrives through the acceptor, but validation, risk checks, routing, persistence and lifecycle are handled by services further in.

The failure that breaks the easy answer

Two FIX nodes:

FIX-01 = ACTIVE
FIX-02 = STANDBY

The active node has a session with a broker and has just sent:

MsgSeqNum(34)=72881

Then FIX-01 goes unreachable from FIX-02.

Now FIX-02 has a genuinely dangerous question to answer:

Is FIX-01 actually dead?

or

Can I simply not reach it?

Those are very different failures. Suppose the network has partitioned like this:

              cluster network
FIX-01  XXXXXXXXXXXXXXXXXX  FIX-02
   |
   |
   | FIX connection still healthy
   v
Exchange / Broker

FIX-01 is alive and still connected to the counterparty. It just can’t talk to its peer.

If FIX-02 starts the same session now, you get:

FIX-01 -> same FIX session -> Counterparty
FIX-02 -> same FIX session -> Counterparty

That’s split brain. For a trading system, a short outage is almost always safer than two machines independently believing they’re allowed to send orders on the same session.

So what the architecture needs isn’t failover. It’s safe ownership transfer.

The architecture

For a relatively small or static set of FIX sessions, this is where I’d start:

              FIX Initiator / Counterparty
                          |
                          | FIX over TCP
                          v
                     Virtual IP
                          |
                          v
                  HAProxy (TCP mode)
                     |           :
             active  |           : standby / unhealthy
                     v           v
               +----------+  +----------+
               |  FIX-01  |  |  FIX-02  |
               |QuickFIX/J|  |QuickFIX/J|
               |  ACTIVE  |  | STANDBY  |
               +----+-----+  +-----+----+
                    |              |
                    +------+-------+
                           |
             +-------------+-------------+
             |                           |
             v                           v
   +-------------------+     +-----------------------+
   |   HA PostgreSQL   |     | OMS / Risk / internal |
   | session + message |     |       services        |
   |       store       |     +-----------------------+
   +-------------------+

   Corosync / Pacemaker runs between FIX-01 and FIX-02.
   Both also hold a quorum link to qnetd, which sits in a
   third failure domain and does nothing except vote.

Several layers, and each one answers a different question:

Keeping those responsibilities separate is what made the whole thing tractable for me. Every time I tried to collapse two of them into one component, the failure analysis got worse.

What HAProxy is actually doing here

HAProxy does not understand FIX semantics in this design. It’s operating at Layer 4, in TCP mode:

FIX client
    |
    | TCP connection
    v
HAProxy
    |
    v
FIX Acceptor

It supports TCP health checks and drops a backend from rotation when those checks fail.³ A minimal config:

frontend fix_frontend
    bind *:9876
    mode tcp
    default_backend fix_acceptors

backend fix_acceptors
    mode tcp
    option tcp-check

    server fix01 10.10.10.11:9876 check inter 2s fall 3 rise 2
    server fix02 10.10.10.12:9876 check inter 2s fall 3 rise 2

There’s one rule I wouldn’t bend here: HAProxy must not be responsible for deciding FIX ownership. Pacemaker decides which node is allowed to run the FIX resource. HAProxy only routes traffic to the node that’s actually ready.

The simplest way to implement that is to let the standby be genuinely down:

FIX-01 ACTIVE  -> QuickFIX/J listening on :9876
FIX-02 STANDBY -> QuickFIX/J stopped; :9876 closed

So HAProxy sees:

fix01 -> UP
fix02 -> DOWN

and sends everything to FIX-01. After a failover it sees the reverse, and new connections go to FIX-02.

Why not keep both FIX processes listening?

Because a TCP health check only answers one question: can I open this port? It says nothing about whether the node currently owns the session.

If both acceptors listen on :9876, HAProxy happily sees both as healthy even though only one is allowed to process the session. You’ve just moved split brain one layer down.

You could expose an ownership-aware readiness endpoint and check that instead:

READY =
  Pacemaker resource owned
  AND FIX engine initialized
  AND persistent store reachable
  AND session state loaded
  AND node permitted to accept traffic

That works, but it’s more moving parts to get wrong. I’d keep the first version: the standby FIX service is stopped, full stop.

One more thing worth being clear about — HAProxy can’t move an existing TCP connection from FIX-01 to FIX-02. When FIX-01 dies, that connection dies with it. The initiator reconnects, and HAProxy hands the new connection to the newly active node.

Which is exactly where FIX’s own recovery machinery starts earning its keep.

Pacemaker and Corosync: who’s alive, and where should FIX run?

I find it easiest to keep the two straight like this:

Corosync:
    Who is in the cluster?
    Who can communicate?
    Do we have quorum?

Pacemaker:
    Where should the FIX resource run?
    Should it be started?
    Should it be stopped?
    Should a failed node be fenced?

Pacemaker’s documentation describes Corosync as the membership layer and Pacemaker as the resource manager that reacts to node and resource events, starts and stops resources, and can fence nodes.⁴

The nice consequence: the FIX application doesn’t need any leader-election logic of its own. Conceptually Pacemaker manages a resource group:

fix-stack
  |
  +-- VIP
  +-- HAProxy
  +-- QuickFIX/J acceptor

Depending on the deployment, HAProxy might live in its own HA pair, or the VIP might point straight at the FIX listener. What matters is that the resources defining the active path move together, and that the FIX service has exactly one owner.

Quorum: why there’s a third vote

A two-node cluster has an awkward problem. If FIX-01 and FIX-02 stop seeing each other, each node knows exactly one thing for certain:

"I am alive."

Neither can tell whether the peer crashed or whether the link between them broke.

That’s why a two-node Pacemaker/Corosync design usually adds a QDevice backed by qnetd, running outside the pair. Corosync’s docs describe it as an external voting mechanism and specifically recommend it for even-numbered clusters, two-node ones especially.⁵

               qnetd
                 |
          +------+------+
          |             |
       FIX-01         FIX-02

The witness doesn’t process orders and doesn’t store FIX state. It exists to help decide which partition gets to stay quorate.

If the witness itself dies while FIX-01 and FIX-02 can still talk, the active service doesn’t need to panic. The dangerous case is a second failure that cuts the two FIX nodes off from each other while arbitration is also unavailable.

For a financial system the policy should be boringly conservative:

Cannot prove safe ownership
        |
        v
Do not promote another FIX owner

Failing closed beats split brain.

Hard fencing: the part I wouldn’t make optional

This is probably the most important idea in the whole design.

Cluster membership tells you a node is unresponsive. It does not tell you the node has stopped running. Those are not the same claim, and the gap between them is where trading systems get hurt.

Pacemaker defines fencing — STONITH — as the ability to ensure a node cannot continue running a service, and its docs explicitly call out conditions where safe recovery is impossible without it.⁴

Back to the partition:

                 Cluster traffic
FIX-01  XXXXXXXXXXXXXXXXXXXXX  FIX-02
   |
   | still alive
   |
   +-----------------------> Exchange

What FIX-02 must not do is reason like this:

"I cannot see FIX-01, so I will assume it is dead."

What it should do instead:

FIX-02 / Pacemaker
       |
       v
Establish quorum
       |
       v
Fence FIX-01
       |
       v
Verify FIX-01 cannot run
       |
       v
Start FIX on FIX-02

On real infrastructure the fencing device is something out-of-band: IPMI, iDRAC, iLO, Redfish, a managed PDU, some other power controller.

The goal isn’t a graceful shutdown. It’s certainty. Before promoting the new owner, make the old owner physically incapable of sending another FIX message. That’s a fundamentally different guarantee from a health check, and you can’t substitute one for the other.

Persistent FIX state: failing over without forgetting where the session was

Say you’ve safely fenced FIX-01 and promoted FIX-02. You still have a problem if FIX-02 thinks the session starts at sequence number 1.

Before the failure:

NextNumOut = 72882
NextNumIn  = 98114

FIX-02 needs those numbers back. The FIX Session Layer requires session processors to persist NextNumOut and NextNumIn across connections.¹

QuickFIX/J supports persistent message stores including FileStore and JdbcStore; the architecture docs note the message store persists messages and sequence numbers to support delivery and ResendRequest recovery.⁶

For HA I’d take a centralized durable store — PostgreSQL via JdbcStore — over node-local files, because both nodes need the same authoritative session state:

              +-----------------------+
FIX-01 ------>|                       |
              | PostgreSQL FIX Store  |
FIX-02 ------>|                       |
              +-----------------------+

sessions:
  SenderCompID
  TargetCompID
  NextSenderMsgSeqNum
  NextTargetMsgSeqNum

messages:
  seq 72878 -> FIX message
  seq 72879 -> FIX message
  seq 72880 -> FIX message
  seq 72881 -> FIX message

QuickFIX/J also has RefreshOnLogon, which refreshes persistent session state when a Logon arrives — described in the docs as enabling a simple form of failover when the store is persistent.²

[DEFAULT]
PersistMessages=Y
RefreshOnLogon=Y

JdbcURL=jdbc:postgresql://fix-db:5432/fix
JdbcUser=quickfix
JdbcPassword=********

Worth flagging: these config properties aren’t the whole story. Your application still has to actually select JdbcStoreFactory. Setting the JDBC properties without wiring the factory gets you a file store and a confusing afternoon.

What actually happens when the active FIX node dies

Worth walking the whole failure as a timeline, because the ordering is the interesting part.

Before the failure

FIX-01 = ACTIVE
FIX-02 = STANDBY

Counterparty -> HAProxy -> FIX-01

NextNumOut = 72882
NextNumIn  = 98114

FIX-01 has just transmitted message 72881.

1. FIX-01 becomes unresponsive

Corosync notices the membership change.

FIX-02 cannot communicate with FIX-01

2. The cluster works out whether FIX-02 has quorum

The QDevice contributes to that decision. If safe quorum can’t be established, nothing gets promoted.

3. Pacemaker fences FIX-01

Powered off, reset, or otherwise made incapable of running the service. Recovery doesn’t continue until fencing succeeds.

4. The resources move

Pacemaker starts the FIX stack on FIX-02.

VIP / HAProxy path -> FIX-02
QuickFIX/J          -> running

5. HAProxy marks FIX-02 healthy

The listener is up, so FIX-02 becomes the eligible backend.

6. The initiator reconnects

The old TCP connection is gone and isn’t coming back. The initiator opens a new one and sends Logon(35=A).

7. FIX-02 restores session state

It reads the persistent store:

NextNumOut = 72882
NextNumIn  = 98114

8. FIX reconciles sequence numbers

Here’s where the protocol carries its own weight. FIX spots gaps through sequence numbers. ResendRequest(35=2) asks for what’s missing, and retransmitted messages reuse their original MsgSeqNum(34) with PossDupFlag(43)=Y.¹

So the counterparty either says “I expected 72882” and you carry on, or it effectively says “I only processed up to 72880” and the engines reconcile the gap between them.

That split is the thing I’d want a team to internalize: the HA layer transfers ownership, FIX recovers protocol continuity. Trying to make Pacemaker or HAProxy solve sequence recovery is the wrong abstraction, and you’ll build something fragile trying.

What about duplicate orders?

There’s a layer underneath all this that infrastructure genuinely cannot solve for you.

Say FIX-01 sent an order immediately before it died. It may be unclear whether the counterparty ever processed it. FIX gives you PossDupFlag and retransmission semantics, but the specification puts responsibility for handling duplicate application messages on the application layer.¹

Which means your OMS still needs idempotency:

ClOrdID = VT-BROKER-A-20260826-000184

The business layer treats the appropriate identifiers as unique, according to whatever the counterparty’s Rules of Engagement actually say.

So the safety model ends up in three tiers:

Cluster layer
    -> exactly one active FIX owner

FIX session layer
    -> ordering, sequence recovery, resend handling

Application layer
    -> business idempotency and duplicate protection

Skip any one of them and the other two won’t cover for it.

Do I actually need HAProxy?

Not always.

For a pure active/passive pair on a normal Layer-2 network, Pacemaker can move a floating VIP directly with the FIX service:

Counterparty
     |
     v
Floating VIP :9876
     |
     v
Active FIX Acceptor

Simpler, and one fewer thing to operate.

HAProxy earns its place when you want a stable proxy tier, richer health checking, better operational visibility, or when the backend topology will eventually include multiple FIX workers. I think of it as an optional traffic-management layer, not a requirement of FIX HA.

And if I do run it, I still don’t let it make the ownership decision.

Then I looked at etcd

The Pacemaker design works well when the unit of failover is roughly “this FIX service runs on node A or node B.”

But what happens with hundreds of FIX sessions?

broker-a -> FIX-01
broker-b -> FIX-02
broker-c -> FIX-03
broker-d -> FIX-01
broker-e -> FIX-03

Moving an entire server as one active/passive resource starts looking clumsy. You’re failing over at the wrong granularity.

That’s where etcd gets interesting. It replicates key-value data through Raft and provides leases with TTL and keepalive semantics, aimed at exactly this kind of coordination.⁷⁸

Represent ownership as keys:

/fix/sessions/broker-a -> fix-01
/fix/sessions/broker-b -> fix-02
/fix/sessions/broker-c -> fix-03

with each key attached to a lease. Now you’re active-active at the platform level while staying single-owner at the session level:

       +---------+     +---------+     +---------+
       | etcd-01 |-----| etcd-02 |-----| etcd-03 |
       +----+----+     +----+----+     +----+----+
            |               |               |
            | lease         | lease         | lease
            |               |               |
       +----+-----+    +----+-----+    +----+-----+
       |  FIX-01  |    |  FIX-02  |    |  FIX-03  |
       | broker-a |    | broker-b |    | broker-c |
       | broker-d |    |          |    | broker-e |
       +----------+    +----------+    +----------+

       (etcd-03 also peers back to etcd-01 — Raft quorum
        across all three)

If FIX-01 genuinely dies, its leases expire and another worker picks up the orphaned sessions. That scales far more naturally than moving whole servers around.

But etcd doesn’t remove the hardest problem.

etcd can transfer ownership, but it can’t kill the old owner

Another partition, slightly different shape:

                    etcd cluster
                       ^
                       |
                       X
                       |
FIX-01 ----------------+
   |
   | FIX connection still healthy
   v
Exchange

FIX-01 can’t reach etcd. Its lease expires. FIX-02 acquires the session. As far as the coordination system is concerned:

owner = FIX-02

Except FIX-01 may still be alive and still connected to the exchange. etcd cannot physically stop that process. It was never able to.

So the application has to enforce a strict fail-closed rule itself:

lease ownership lost
      |
      v
immediately disable FIX sends
      |
      v
disconnect the FIX session
      |
      v
fence/self-fence if safe shutdown cannot be guaranteed

Which led me to the distinction I keep coming back to: consensus determines who should own the resource; fencing guarantees a previous owner can’t continue acting as the owner. etcd is an excellent coordination layer. It is not, on its own, a fencing mechanism.

Pacemaker or etcd?

I don’t think there’s a universally correct answer. It depends on what you’re failing over.

Pacemaker + Corosync

I’d reach for it when the FIX deployment is small or static, when I want active/passive service ownership, when I want the FIX application to contain no distributed-systems logic at all, when hard fencing is a first-class requirement, and when the infrastructure team is comfortable running Linux HA clusters.

The model states in three lines:

Pacemaker owns the service.
Only the active node runs it.
The old node is fenced before the service moves.

etcd

I’d explore etcd when there are many independent FIX sessions, when I want session-level placement rather than server-level failover, when multiple workers should be active at once, when I need dynamic redistribution — and when I’m prepared to build ownership, lease-loss and fencing behaviour into the control plane myself.

More flexible architecture, more sophisticated software. That trade is real, and I wouldn’t take it before I needed it.

The production design I’d build first

Before building a distributed per-session scheduler, I’d start with whatever is easiest to prove correct:

                         FIX COUNTERPARTY
                                |
                                | FIX / TCP
                                v
                         +--------------+
                         | VIP / HAProxy|
                         +------+-------+
                                |
                     +----------+----------+
                     |                     |
                     v                     v
               +-----------+         +-----------+
               | FIX-01    |         | FIX-02    |
               | ACTIVE    |         | STANDBY   |
               | QuickFIX/J|         | QuickFIX/J|
               +-----+-----+         +-----+-----+
                     |                     |
                     +----------+----------+
                                |
                       +--------v---------+
                       | PostgreSQL HA    |
                       | FIX state/store  |
                       +------------------+

                 Corosync + Pacemaker
                  between FIX nodes

                 QDevice in a separate
                    failure domain

                  Hard STONITH fencing

The rules I’d hold it to:

1. Only one node may run a given FIX acceptor resource.
2. No promotion without quorum.
3. No promotion until the old node is fenced, where fencing is required.
4. The FIX listener isn’t ready unless the node owns the resource and session state is usable.
5. Sequence numbers and resendable messages are durable.
6. The new TCP connection performs normal FIX Logon and sequence recovery.
7. The business layer still enforces idempotency.
8. If ownership is uncertain, stop trading rather than guess.

That last one matters more than the other seven combined.

What I walked away with

I started out thinking the problem was “how do I keep the FIX server available?”

Having worked through the failure modes, that framing is incomplete. The real problem is how to make ownership unambiguous while preserving enough state for the next owner to recover safely.

Which splits cleanly into three responsibilities:

Consensus / cluster management
    -> Who should own the session?

Fencing
    -> Can the previous owner still act?

FIX
    -> How do the peers recover the session after reconnect?

And underneath all of it sits the business layer, still needing to make order processing idempotent.

The interesting part of FIX high availability was never keeping two servers running. It’s proving that under every failure condition there is exactly one valid owner of the session. That’s the guarantee I’d design the rest of the system around.

References