Compliance Engineering in Trading Systems: From Logs to Trust

Part I — Theory before the hands-on implementation

A trading system can execute an order correctly and still leave behind a poor explanation of what happened.

That distinction matters.

In a normal web platform, logs are often treated primarily as an operational tool. Engineers use them to answer questions such as:

• Why did this request fail?
• Which pod restarted?
• Why did latency increase?
• Which downstream dependency returned an error?
• What changed after the last deployment?

In financial trading infrastructure, the same technical events can carry a second responsibility.

They may need to help explain, reconstruct, preserve, and defend what happened to an order long after the incident dashboard has disappeared.

That changes the engineering problem.

The question is no longer only:

Do we have logs?

It becomes:

Can we prove what happened, in what order, using records whose integrity, timestamps, retention and access history we can trust?

This article develops that idea from the perspective of a DevOps/SRE/platform engineer. It brings together six themes:

1. When logs become regulatory evidence.
2. Reconstructing an order end-to-end.
3. Time synchronization as a compliance dependency.
4. Searchable logs versus regulatory archives.
5. Access control around the audit trail.
6. Privacy versus retention.

The goal is not to turn an infrastructure engineer into a lawyer. It is to translate regulatory concepts into engineering properties that can be designed, monitored and tested.

Important: this article is an engineering discussion, not legal or regulatory advice. Applicability, required records, retention periods and implementation choices depend on the regulated entity, jurisdiction, registration status, asset class and activity.

1. Compliance Becomes an Engineering Problem

Compliance requirements are often written in language such as:

• make and preserve records,
• maintain integrity,
• retain records for a defined period,
• reconstruct order activity,
• synchronize business clocks,
• restrict access,
• maintain an audit trail,
• support regulatory inspection,
• protect personal data.

None of those phrases names Kubernetes, Kafka, OpenTelemetry, Elasticsearch or Amazon S3.

But all of them eventually become technical questions.

For example:

That translation—from obligation to system property—is what I mean by compliance engineering.

Compliance is not a storage bucket, a dashboard, a SIEM license, or a retention number.

It is a collection of system guarantees.

2. The Regulatory Lenses Behind the Architecture

Before discussing architecture, it helps to understand why these properties exist.

The examples below are intentionally cross-jurisdictional because modern trading platforms frequently operate across more than one regulatory environment.

2.1 U.S. broker-dealer recordkeeping: SEC Rule 17a-4 and FINRA Rule 4511

FINRA Rule 4511 requires members to make and preserve books and records required under FINRA rules, the Exchange Act and applicable Exchange Act rules. Where FINRA requires a record but no specific retention period is otherwise given, the rule provides a default period of at least six years.¹

Electronic recordkeeping is especially interesting from an infrastructure perspective.

Historically, SEC Rule 17a-4 was closely associated with non-rewriteable, non-erasable storage—commonly described as WORM, or write once read many.

The SEC’s 2022 amendments introduced an audit-trail alternative. Under that alternative, an electronic recordkeeping system may preserve records in a way that permits recreation of an original record if it is modified or deleted.²

This is an important engineering nuance:

“Compliant recordkeeping” does not simply mean “put everything in WORM storage.”

The underlying property is stronger and more useful:

The system must preserve required records in a manner that supports integrity, recreation and production.

That gives engineers multiple possible designs—but also makes architecture and controls much more important.

2.2 Order lifecycle traceability: SEC Rule 613 and CAT

The Consolidated Audit Trail is a powerful example of why order reconstruction matters.

SEC Rule 613 and the CAT framework are designed so reportable events can be linked through the lifecycle of an order—from generation through routing, modification, cancellation or execution.³

Think about what that means technically.

One order may generate state in:

• the client/API layer,
• pre-trade risk,
• the OMS,
• a smart-order router,
• the FIX gateway,
• the broker or venue connection,
• post-trade services,
• databases,
• message queues,
• reconciliation systems.

No single component necessarily contains the entire story.

Order reconstruction is therefore fundamentally a distributed-systems correlation problem.

2.3 Clock synchronization: FINRA CAT rules

FINRA Rule 6820 requires applicable CAT business clocks to be synchronized within specified tolerances. For many computerized business clocks used for reportable events, the tolerance is 50 milliseconds relative to NIST time. The rule also requires synchronization before market open, checks and re-synchronization during the day as necessary, documented procedures, and synchronization logs including excessive drift. Those synchronization logs must generally cover at least five years.⁴

This turns something that looks like a mundane operating-system setting into part of the evidentiary chain.

If two systems disagree about time, the audit trail can disagree about causality.

2.4 EU order records: MiFIR

Article 25 of MiFIR requires investment firms to keep relevant data relating to orders and transactions available to competent authorities for five years.⁵

Again, the engineering implication is not simply “retain data for five years.”

The real questions are:

• Which records?
• In what schema?
• With what identifiers?
• How will they be retrieved?
• Can they still be interpreted after applications have changed?
• Can corrections be distinguished from original history?
• Can an investigator reconstruct the sequence later?

2.5 EU clock synchronization

The EU clock-synchronization framework requires business clocks used for reportable events to be traceable to UTC, with accuracy requirements depending on the trading activity. Under Commission Delegated Regulation (EU) 2025/1155, high-frequency algorithmic trading activity can be subject to a maximum divergence from UTC of 100 microseconds.⁶

The important engineering idea is not the number alone.

Different trading environments require different accuracy—and therefore different infrastructure.

A simple NTP client that is acceptable for one workflow may be inappropriate for a latency-sensitive trading system.

2.6 Operational resilience and access: DORA

DORA’s technical standards push financial entities toward strong identity, access, logging and control practices. The detailed rules include unique identities for users, lifecycle management of accounts, access-right management and logging around access-related events.⁷

This reinforces a point that is often missed:

Record integrity is not only a storage problem. It is also an identity and privilege problem.

2.7 GIFT IFSC: IFSCA cyber security and resilience

IFSCA’s Cyber Security and Cyber Resilience Guidelines for Regulated Entities in IFSCs explicitly address access control and audit trails.

The guidelines call for need-to-know access, least privilege and segregation of duties. They also state that audit trails for IT assets should support business continuity and recovery, regulatory and legal obligations, audits, forensic evidence and dispute resolution.⁸

For a DevOps engineer, this is a direct bridge between regulatory language and platform controls:

• IAM,
• RBAC,
• privileged access,
• immutable records,
• security event logging,
• recovery testing,
• access reviews,
• auditability.

3. Principle One: Logs Are Not Automatically Evidence

A centralized logging platform is useful.

It is not automatically a trustworthy recordkeeping system.

Suppose a production incident occurs and an engineer searches:

order.id = "ORD-78421"

Elasticsearch returns:

10:00:00.120  oms          ORDER_CREATED
10:00:00.126  risk-engine  RISK_APPROVED
10:00:00.132  fix-gateway  ORDER_ROUTED
10:00:00.160  fix-gateway  EXECUTION_RECEIVED

Operationally, this is excellent.

But an investigator may ask questions that an operational dashboard does not answer:

• Could these events have been altered?
• Could an administrator delete them?
• Was the original record overwritten during re-indexing?
• Was the retention policy changed later?
• Is there a history of modifications?
• Who accessed or exported the records?
• Can the original data still be produced after several years?

That is the difference between observability and evidentiary trust.

3.1 Three related but different concepts

It is useful to separate three concepts.

Operational logs

Designed primarily for:

• debugging,
• monitoring,
• incident response,
• performance analysis,
• search.

They prioritize convenience and speed.

Audit records

Designed to answer:

• who did what,
• when it happened,
• what changed,
• what identity performed the action.

They prioritize accountability.

Regulatory records

Records that a regulated entity is required to make, preserve or produce.

They prioritize:

• correctness,
• integrity,
• retention,
• retrievability,
• controlled access,
• defensibility.

The same event may participate in all three—but the systems that store and protect it do not have to be identical.

3.2 Why “we have it in Elasticsearch” is incomplete

Elasticsearch is optimized for indexing and search.

Operational teams may:

• re-index data,
• alter mappings,
• roll indices,
• delete old indices,
• transform documents,
• change retention policies,
• grant administrators broad access.

None of these things is inherently bad.

They are normal operational features.

But normal operational flexibility can conflict with the guarantees needed for long-term evidence.

That is why a good design begins with the purpose of the data.

4. Principle Two: The System Must Be Able to Tell the Story of an Order

A trade is not one database row.

It is a sequence of decisions and messages.

Consider a simplified lifecycle:

Client / API
     |
     v
Pre-Trade Risk
     |
     v
Order Management System
     |
     v
FIX Gateway
     |
     v
Broker / Venue
     |
     v
Execution / Reject / Cancel
     |
     v
Post-Trade / Reconciliation

Each step produces a fragment of the story.

A mature trading platform should be able to connect those fragments later.

4.1 Questions the system should be able to answer

For a given order, an investigation may need to answer:

1. Who or what submitted the order?
2. What were the original economic terms?
3. Which account or client reference was involved?
4. Which risk rules were evaluated?
5. What was the result of the risk decision?
6. When did the OMS accept or reject the order?
7. What identifiers were assigned?
8. What FIX message was sent?
9. To which broker, venue or route was it sent?
10. Was the order replaced?
11. Was it cancelled?
12. Was it rejected?
13. What execution reports came back?
14. What was the final state?
15. Which application version processed the event?
16. Which configuration was active?
17. Can the records supporting the story be trusted?

Some of these may be regulatory requirements in a particular context; others are additional engineering context that makes investigations substantially easier.

Do not confuse those categories.

A useful architecture preserves both required records and useful provenance, while clearly classifying them.

4.2 Correlation identifiers

Correlation starts with identifiers.

Examples include:

internal_order_id
client_order_reference
fix.cl_ord_id
fix.orig_cl_ord_id
execution_id
account_reference
trace_id
correlation_id
venue
session_id

A common failure mode is to let each service invent its own identifier without preserving mappings.

Then the data exists, but reconstruction becomes manual archaeology.

A better design treats identifier relationships as first-class data.

For example:

{
  "event_type": "ORDER_ROUTED",
  "internal_order_id": "ORD-78421",
  "client_order_id": "CLIENT-9831",
  "fix_cl_ord_id": "FIX-20260907-000912",
  "venue": "BROKER_X",
  "timestamp": "2026-09-07T10:00:00.132Z"
}

If the order is replaced:

{
  "event_type": "ORDER_REPLACED",
  "internal_order_id": "ORD-78421",
  "fix_orig_cl_ord_id": "FIX-20260907-000912",
  "fix_cl_ord_id": "FIX-20260907-001033",
  "timestamp": "2026-09-07T10:00:04.381Z"
}

The relationship survives.

4.3 Structured events beat forensic parsing

This:

INFO order sent successfully for user abc qty 10 at 103.25

is convenient for humans but fragile for long-term reconstruction.

This:

{
  "event_type": "ORDER_ROUTED",
  "order_id": "ORD-78421",
  "quantity": 10,
  "limit_price": "103.25",
  "side": "BUY",
  "venue": "BROKER_X",
  "timestamp": "2026-09-07T10:00:00.132Z",
  "schema_version": 1
}

is much easier to:

• validate,
• index,
• retain,
• migrate,
• correlate,
• mask,
• query,
• archive.

The long-term lesson is:

Do not make future investigators reverse-engineer important business events from arbitrary log strings.

5. Principle Three: Time Is Part of the Data Model

Distributed systems already have a difficult relationship with time.

Trading systems make the consequences more visible.

Imagine:

OMS           Order created      10:00:00.120
Risk Engine   Risk approved      10:00:00.085
FIX Gateway   Order routed       10:00:00.130
Broker        Order received     09:59:59.980

The timestamps imply:

Broker received order
        ↓
Risk approved order
        ↓
OMS created order

That is impossible.

The trading flow may have worked correctly.

The clocks did not.

5.1 Why timestamps establish causality

Timestamps help answer:

• what happened first,
• how long a risk decision took,
• when an order left the firm,
• whether a cancel was sent before an execution,
• whether a deployment coincided with a failure,
• when a privileged configuration change happened.

If machines disagree about time, those conclusions become weaker.

5.2 UTC everywhere

For multi-region and cross-border trading systems, a good baseline is:

• store event timestamps in UTC,
• preserve sufficiently precise timestamps,
• render local time only at presentation boundaries,
• avoid local-time ambiguity in persisted events,
• include timezone/offset information where appropriate.

“10:00:00” is weaker evidence than:

2026-09-07T10:00:00.132Z

5.3 Synchronization architecture

A simplified design might be:

Authoritative / traceable time sources
              |
        redundant sources
              |
              v
       NTP / PTP layer
              |
   +----------+-----------+
   |          |           |
 OMS        Risk       FIX Gateway
   |          |           |
   +----------+-----------+
              |
       drift telemetry
              |
       Prometheus / SIEM
              |
          alerts

The exact implementation depends on latency and accuracy requirements.

Potential technologies include:

• chrony / NTP,
• PTP,
• cloud-provider time services,
• GNSS-backed time infrastructure,
• hardware timestamping for environments that require it.

5.4 Monitor time like a dependency

A system can be “up” while its clock is wrong.

Therefore monitor:

clock_offset_seconds
clock_sync_status
last_successful_sync_timestamp
time_source
frequency_error
stratum
ptp_offset_ns

Then alert according to the actual tolerance relevant to the system.

An infrastructure dashboard where CPU, memory and FIX sessions are green but clock drift is outside the required tolerance is not a healthy trading dashboard.

6. Principle Four: Search and Preservation Are Different Storage Problems

This is one of the most important architectural distinctions in the entire series.

Operational engineers want:

• low-latency search,
• dashboards,
• flexible queries,
• index enrichment,
• recent data online,
• reasonable cost.

Long-term record preservation wants:

• integrity,
• controlled mutability,
• enforced retention,
• durable storage,
• reproducible retrieval,
• auditable access,
• economical multi-year retention.

Trying to make one platform optimize equally for both goals often creates expensive and fragile compromises.

6.1 A dual-path reference architecture

One practical pattern looks like this:

 Trading App       OMS       Risk Engine      FIX Gateway
     |              |             |                |
     +--------------+-------------+----------------+
                            |
                     events / logs
                            |
                            v
                 OpenTelemetry Collector
              receive / parse / enrich / route
                            |
                            v
                      Kafka / MSK
                  durable event buffer
                            |
              +-------------+-------------+
              |                           |
              v                           v
     OPERATIONAL PATH              PRESERVATION PATH
              |                           |
      OTel / Logstash                    Sink
              |                           |
              v                           v
        Elasticsearch                 Amazon S3
              |                      + Versioning
              v                      + Object Lock
            Kibana                   + KMS
              |                      + Retention
     Search / dashboards                  |
                                          v
                                  lifecycle tiers
                                          |
                                          v
                                controlled retrieval

This is an engineering reference architecture, not a statement that any regulation mandates these specific products.

6.2 OpenTelemetry’s role

OpenTelemetry is useful as a normalization and routing layer.

The OpenTelemetry Collector is designed to receive, process and export telemetry to one or more backends.⁹

That makes it a sensible place to do the work every downstream consumer would otherwise duplicate: attaching service.name and environment metadata, enriching with Kubernetes context, normalizing timestamps into UTC, and routing different classes of event to different destinations.

It is a poor place to keep anything.

The reason is worth stating carefully, because it reads like a criticism of the Collector and is not one. A telemetry pipeline is built to protect the pipeline. Under pressure it batches, retries and queues, and when the queue is full it sheds load. Processors drop and reshape fields because they were configured to. That is correct behaviour for observability data, where a missing span costs some resolution on a dashboard and nothing else.

Apply the identical behaviour to a record the business is obliged to preserve and the cost changes shape completely. A dropped span is a gap in a graph. A dropped order event is a gap in the evidence, found years later by somebody who has no way to distinguish “this did not happen” from “we did not keep it.”

So the Collector belongs in the data plane. It moves and normalizes records on their way to durable storage; it is not itself the source of evidentiary truth. The reconciliation control in Section 10.2 exists precisely because that path can lose things, and because losing them silently is the failure that matters.

6.3 Why Kafka or another durable broker can help

A durable message layer decouples event production from downstream storage.

Without a buffer:

Application -> Elasticsearch

If Elasticsearch is unavailable, you need every producer to implement failure handling correctly.

With a durable broker:

Application -> Collector -> Kafka -> consumers

you gain:

• buffering,
• replay,
• consumer independence,
• back-pressure handling,
• easier fan-out,
• a place to measure consumer lag.

Kafka is not itself the regulatory archive either.

It is an architectural reliability mechanism.

6.4 Elasticsearch for operational search

Elasticsearch is well suited for the operational path. An engineer working an incident wants to ask one question across OMS events, FIX gateway logs, Kubernetes logs, audit events and deployment metadata at once, and have an answer in seconds:

order.id:"ORD-78421"

Modern Elasticsearch also supports OpenTelemetry ingestion, and Elastic recommends a Collector/gateway architecture for many production scenarios.¹⁰

The tension shows up in how that speed is sustained.

An index lifecycle policy exists to keep a search cluster fast and affordable, and it achieves that by continuously changing the data underneath:

hot     -> rollover at 50GB or 30d
warm    -> shrink, force-merge, mark read-only
cold    -> reduce replicas
delete  -> after 90d

Every phase in that policy is a mutation. Rollover splits a single order’s events across indices. Force-merge rewrites segments. Re-indexing to correct a mapping produces new documents from old ones. The delete phase removes data on a schedule chosen for storage cost, not for a retention obligation.

None of this is a defect. It is the system working as designed, and it is precisely why the operational path is fast.

But the same behaviours are the inverse of what preservation asks for: fixed content, a retention period set by obligation rather than budget, and deletion that cannot happen early by accident or by a misapplied policy template.

A search cluster is optimized for the question being asked right now. An archive is optimized for a question somebody asks in five years, possibly about an order nobody remembers. Asking a single system to do both means one of the two jobs is being done badly, and it will usually be the one nobody is testing.

6.5 S3 Object Lock as a preservation primitive

Amazon S3 Object Lock supports a WORM model and can protect object versions from deletion or overwrite using retention periods or legal holds.¹¹

It provides:

• Governance mode,
• Compliance mode,
• legal holds,
• retention dates.

In Compliance mode, protected versions cannot be overwritten or deleted by users—including the root user—during the retention period.¹¹

This makes Object Lock a useful building block for immutable archival designs.

But another critical statement belongs here:

S3 Object Lock does not make an architecture compliant by itself.

You still need to solve:

• correct record selection,
• complete ingestion,
• retention configuration,
• IAM,
• key availability,
• retrieval,
• indexing/cataloging,
• audit procedures,
• legal applicability,
• monitoring.

A perfectly immutable archive containing incomplete or incorrectly classified data is still a bad recordkeeping system.

7. Principle Five: Access Control Is Part of Record Integrity

Suppose your archive is immutable.

Now ask:

Who can change the policy that makes it immutable?

Or:

Who can export every record?

Or:

Who can assume the role that bypasses governance retention?

Or:

Who can disable the pipeline before an event reaches the archive?

Trust depends on identities and privileges.

7.1 The shared-admin anti-pattern

Avoid designs such as:

DevOpsAdmin
  ├── administer production
  ├── administer Elasticsearch
  ├── change archive policy
  ├── read archive
  ├── export records
  ├── modify KMS
  └── delete storage

The problem is not that DevOps engineers are untrustworthy.

The problem is that the control model cannot strongly demonstrate independence or accountability.

7.2 Separate duties by purpose

A better conceptual model:

Exact roles depend on the organization.

The design principle is:

Separate the ability to operate the system from the ability to silently rewrite its evidence.

7.3 Least privilege

Least privilege should apply to:

• people,
• CI/CD identities,
• Kubernetes service accounts,
• OTel Collectors,
• Kafka producers,
• Kafka consumers,
• archive writers,
• retrieval tooling.

For example, the archival writer may need:

s3:PutObject

without needing:

s3:DeleteObject
s3:PutObjectRetention
s3:PutBucketObjectLockConfiguration

The exact permissions depend on the design, but write-only or narrowly scoped identities can reduce the blast radius.

7.4 Privileged activity must produce its own evidence

Monitor and log events such as:

• privileged role assumed,
• archive read,
• bulk export,
• retention changed,
• bucket policy changed,
• KMS policy changed,
• delete attempted,
• Object Lock settings changed,
• logging pipeline changed,
• collector configuration changed.

In AWS, this naturally points toward:

• IAM,
• CloudTrail,
• KMS audit events,
• S3 data events where appropriate,
• alerts on security-sensitive API calls.

Then send the security trail somewhere that is difficult for the same actor to erase.

7.5 Break-glass access

There will always be emergencies.

Instead of permanent super-admin access, design a break-glass path:

request
  -> approval
  -> temporary privilege
  -> MFA / strong identity
  -> session logging
  -> action audit
  -> automatic expiry
  -> review

Emergency access becomes observable rather than invisible.

8. Principle Six: Retention and Privacy Are Not Opposites

One of the easiest mistakes in compliance-heavy systems is:

“We may need this later, so log everything.”

That is not a mature data strategy.

Privacy regimes such as GDPR include principles of data minimisation and storage limitation.¹²

At the same time, financial regulation may require specific records to be preserved for years.

The correct architecture is not:

KEEP EVERYTHING

or:

DELETE EVERYTHING

It is:

CLASSIFY -> MINIMIZE -> PROTECT -> RETAIN INTENTIONALLY

8.1 The over-logging problem

Bad:

{
  "event": "ORDER_ROUTED",
  "order_id": "ORD-78421",
  "email": "amit@example.com",
  "phone": "+91xxxxxxxxxx",
  "tax_identifier": "ABCDE1234F",
  "account_number": "00123456789",
  "ip_address": "203.0.113.22"
}

Then replicate this to:

• ELK,
• SIEM,
• dashboards,
• alert notifications,
• support tools,
• backups,
• archives.

Now dozens of systems and people potentially handle sensitive data.

8.2 Prefer references where possible

A safer operational event might be:

{
  "event": "ORDER_ROUTED",
  "order_id": "ORD-78421",
  "client_ref": "USR_442",
  "account_ref": "ACC_TOKEN_91",
  "venue": "BROKER_X",
  "timestamp": "2026-09-07T10:00:00.132Z"
}

The mapping from:

USR_442 -> actual customer

can live in a more tightly controlled system.

This reduces exposure without necessarily preventing authorized reconstruction.

8.3 Data classification

Not all data has the same purpose.

Example classification:

This table should be defined deliberately rather than discovered accidentally years later.

8.4 Apply privacy controls before broad distribution

A privacy-aware pipeline can look like:

Trading systems
       |
       v
OTel / Fluent Bit / Collector
       |
       v
classification
       |
       v
redaction / masking / tokenization
       |
       +----------------------+
       |                      |
       v                      v
Operational search       Required archive
minimized fields         required record set
       |                      |
       v                      v
ELK / Kibana          immutable controlled store

The important part is the placement.

If sensitive data is sent everywhere and masked only in the UI, the blast radius already exists.

9. The Unified Compliance Data Architecture

Putting the six ideas together gives us a more complete model.

+------------------------------------------------------------------+
|                        TRADING SYSTEMS                            |
|                                                                  |
| Client/API   Risk   OMS   FIX Gateway   Post-Trade   Infra Audit |
+-------------------------------+----------------------------------+
                                |
                                v
                 +------------------------------+
                 | Telemetry / Event Collection |
                 | OTel / Fluent Bit / agents   |
                 +------------------------------+
                                |
                   normalize / validate / enrich
                    classify / redact / tokenize
                                |
                                v
                     +-------------------+
                     | Durable Event Bus |
                     | Kafka / AWS MSK   |
                     +-------------------+
                         |             |
           +-------------+             +------------------+
           |                                               |
           v                                               v
+---------------------------+                +---------------------------+
| OPERATIONAL SEARCH        |                | PRESERVATION             |
|                           |                |                           |
| Elasticsearch / Kibana    |                | S3 Object Lock           |
| SIEM                      |                | Versioning                |
| dashboards                |                | KMS                       |
| alerting                  |                | retention/lifecycle       |
| short/medium retention    |                | controlled retrieval      |
+-------------+-------------+                +-------------+-------------+
              |                                            |
              v                                            v
     Engineers / SRE / SOC                      Compliance / investigation
              |                                            |
              +------------------+-------------------------+
                                 |
                                 v
                        ACCESS AUDIT LAYER
                  IAM / CloudTrail / security logs
                                 |
                                 v
                        immutable security trail

Supporting control planes:

* clock synchronization + drift monitoring
* schema registry / versioning
* record classification
* retention policy
* KMS/key management
* break-glass workflow
* DR and restore testing
* pipeline SLOs

This architecture encodes a key idea:

The compliance story is a chain.

A trustworthy archive cannot repair an event that was never captured.

Perfect timestamps cannot help if order identifiers cannot be correlated.

Immutable storage cannot help if the wrong people can change the pipeline before archival.

Retention cannot help if nobody can retrieve the data.

Privacy controls cannot help if PII has already been replicated to every observability backend.

Every layer contributes to the final level of trust.

10. Reliability: Monitor the Evidence Pipeline

A regulatory pipeline deserves its own operational objectives.

Do not assume that because the application is trading, evidence is being preserved correctly.

Monitor the pipeline itself.

10.1 Useful signals

Collector

accepted_log_records
refused_log_records
send_failed_log_records
exporter_queue_size
exporter_queue_capacity

Kafka

producer_errors
under_replicated_partitions
consumer_lag
oldest_unconsumed_record_age

Search

indexing_failures
ingest_latency
rejected_documents
cluster_health

Archive

archive_write_failures
last_successful_archive_timestamp
objects_written
retention_configuration_status
retrieval_test_status

Time

clock_offset_seconds
clock_sync_status
last_clock_sync

Security

archive_exports
privileged_role_assumptions
retention_changes
failed_delete_attempts
policy_changes

10.2 The missing-record alert

One particularly useful concept:

Alert not only when a component is down, but when expected evidence is missing.

For example:

orders_routed = 10,000
archived_order_routed_events = 9,997

That discrepancy may deserve immediate investigation.

Reconciliation is a powerful compliance control.

11. Availability Is Not Enough: Design for Retrieval

There is little value in preserving records for years if nobody can successfully recover them.

The failure mode worth designing against is not the one people expect.

A deleted encryption key or a corrupted bucket is an obvious disaster, and it announces itself immediately. The dangerous case is quieter, because the retrieval succeeds.

Consider a request for the full lifecycle of an order routed four years ago. The objects are located. Object Lock held. Checksums match. Everything decrypts and decompresses exactly as designed.

Then somebody opens the records:

archived 2022-03-14, schema v1:
  {"order_id":"ORD-41822","status":"ROUTED"}

current tooling expects schema v3:
  {"order":{"id":"ORD-41822","state":"ROUTED"}}

The reconstruction tooling understands the second shape and returns nothing for the first. The parser for v1 was retired along with the service that wrote it. Nobody can now say with confidence whether status and state drew on the same vocabulary, or whether ROUTED marked the same moment in the order lifecycle in both versions.

Storage did its job. Retrieval did its job. The archive is intact, durable, and provably unaltered.

There is still no usable evidence, because nothing in the pipeline preserved the ability to interpret what was preserved. That is the schema problem from Section 12 arriving four years late, at the worst possible moment to discover it.

Retrievability is therefore a property of several things surviving together:

• the bytes,
• the keys,
• the schema definitions and their parsers,
• the identifier mappings,
• somebody who knows the procedure.

A design that guarantees only the first is a backup. It is not yet a recordkeeping system.

11.1 Retrieval tests

The only way to know retrieval works is to perform it on a schedule, against real archived data, as a drill rather than a code path nobody has executed since it was written.

A useful drill:

1. Select a historical order at random, weighted toward the oldest retention tiers.
2. Retrieve, decrypt and decompress its archived records.
3. Validate hashes and metadata where applicable.
4. Reconstruct the lifecycle across every schema version that order touched.
5. Produce a human-readable investigation package.

What the drill should emit is not a pass/fail bit:

order:                 ORD-41822
retention tier:        archive (4y)
schema versions seen:  v1, v3
parsers available:     v3 only          <-- gap
records retrieved:     14 / 14
lifecycle complete:    no
time to package:       6h 40m
performed by:          <identity>

Two of those lines carry most of the value. parsers available catches the interpretation gap while it is still cheap to close. time to package is the number worth trending, because requests of this kind tend to arrive with a deadline attached, and “we can retrieve it eventually” is a materially different capability from “we can retrieve it this week.”

Run the drill often enough that both numbers are real, and the compliance system has a tested capability.

Not just a storage configuration.

12. Schema Evolution Is a Compliance Problem

Trading applications change.

Suppose version 1 stored:

{
  "order_id": "...",
  "status": "ROUTED"
}

Version 3 stores:

{
  "order": {
    "id": "...",
    "state": "ROUTED"
  }
}

Five years later, can the team interpret both?

Treat schema evolution intentionally.

Useful fields include:

schema.name
schema.version
application.version
event.type
event.version

Maintain:

• schema definitions,
• compatibility rules,
• migration documentation,
• parsers for historical formats.

A record whose meaning has been forgotten is not very useful evidence.

13. Corrections Should Not Silently Rewrite History

Trading records can contain errors.

The instinct may be to update the original row or document.

For auditability, consider append-oriented correction models.

Instead of:

UPDATE order SET price = 101.50

the event history can preserve:

ORDER_CREATED price=100.50
CORRECTION old=100.50 new=101.50 reason=...

The current business state can still show 101.50.

The historical trail explains how it became 101.50.

This is a fundamental distinction:

Current state answers “what is true now?”
Event history answers “how did we get here?”

Trading compliance frequently cares about the second question.

14. A Practical Control Matrix

Before implementation, build a matrix like this.

This matrix connects abstract objectives to measurable controls.

15. What DevOps/SRE Owns—and What It Does Not

Compliance engineering is cross-functional.

DevOps/SRE/platform teams can own or influence:

• event transport,
• centralized collection,
• durability,
• observability,
• archive infrastructure,
• IAM,
• encryption,
• backup and DR,
• time synchronization,
• infrastructure audit trails,
• automated policy controls,
• retrieval tooling.

But engineers should not independently decide:

• which exact records the legal entity is required to retain,
• the legally required retention period,
• regulatory reporting scope,
• lawful basis for processing personal information,
• interpretation of regulator-specific obligations.

Those decisions need compliance, legal, security, business and engineering together.

A useful workflow is:

Compliance / Legal:
    "This record must exist and be preserved for X."

Security:
    "These identities may access it under these controls."

Engineering:
    "Here is the architecture that guarantees capture,
     integrity, retention, monitoring and retrieval."

SRE:
    "Here is how we prove the architecture remains healthy."

Audit:
    "Here is how we test that those controls actually work."

That is compliance engineering as a team sport.

16. The Six Principles in One View

Principle 1 — Logs are not automatically evidence

Centralized logs are useful, but evidentiary trust requires integrity, preservation, controlled access and retrievability.

Principle 2 — Every order needs a reconstructable story

Design identifiers, structured events and mappings so the order lifecycle can be correlated across distributed components.

Principle 3 — Time is part of correctness

Clock synchronization, drift monitoring and synchronization evidence are part of the audit trail.

Principle 4 — Search and archive optimize for different goals

Operational search optimizes speed and usability. Preservation optimizes long-term integrity and trust.

Principle 5 — Access control is part of integrity

Least privilege, segregation of duties, strong identity and privileged-action auditing protect the credibility of the record.

Principle 6 — Retention and privacy must be designed together

Keep required evidence, but minimize unnecessary exposure using classification, structured schemas, masking and tokenization.

17. The Most Important Mental Model

If there is one idea to take away from this article, it is this:

           CAPTURE
              |
              v
          CORRELATE
              |
              v
         TIMESTAMP
              |
              v
          PRESERVE
              |
              v
           PROTECT
              |
              v
          RETRIEVE
              |
              v
            PROVE

If any stage fails, the confidence of the final answer falls.

A mature compliance architecture therefore asks:

Can we capture the event?

Can we correlate it?

Can we trust the time?

Can we preserve it?

Can we control who touches it?

Can we retrieve it years later?

Can we demonstrate that all of the above controls actually worked?

That is a much stronger engineering model than:

“We keep logs for seven years.”

18. What Comes Next: Build the PoC

Theory is useful only up to a point.

The next part of this project will build a small production-like trading environment and test these ideas with real components.

The planned proof of concept will include:

Dummy User / Trading Client
             |
             v
          Dummy OMS
             |
             v
        Risk Engine
             |
             v
        FIX Initiator
             |
             v
        FIX Exchange

with free market data for simulated trading.

Around the trading path, we will build the compliance and observability layer:

Trading services
      |
      v
OpenTelemetry
      |
      v
Kafka / durable event pipeline
     / \
    /   \
   v     v
 ELK     S3 Object Lock
search   archive
   |       |
   +---+---+
       |
       v
order reconstruction

The PoC will aim to demonstrate:

• structured business/audit events,
• correlation IDs,
• FIX order lifecycle mapping,
• time synchronization and drift monitoring,
• operational search,
• immutable archival,
• retention controls,
• encryption,
• IAM and separation of duties,
• privileged-access auditing,
• privacy-aware event schemas,
• masking/tokenization,
• pipeline monitoring,
• failure/replay behavior,
• historical order reconstruction.

And the final test will be simple:

Pick one dummy order after the fact and reconstruct its entire story—from client submission to risk decision, OMS processing, FIX routing and execution—using preserved records.

At that point, compliance is no longer a diagram.

It becomes a demonstrable system property.

Conclusion

Financial compliance often looks like documentation from a distance.

Up close, a surprising amount of it becomes distributed systems engineering.

Records have to survive.

Events have to correlate.

Clocks have to agree.

Privileges have to be constrained.

Sensitive data has to be minimized.

Archives have to be retrievable.

And the entire pipeline has to remain observable.

The strongest trading systems do not merely process orders.

They preserve enough trustworthy context to explain those orders later.

That is the transition from:

logs

to:

auditability

to:

evidence

to:

trust.

Author’s Note

This is Part I of a larger project on compliance engineering in trading systems.

Part II will move from theory to implementation: a production-like proof of concept using a FIX-based trading flow, dummy OMS and risk services, OpenTelemetry, a durable event pipeline, operational search, immutable archival, access controls and end-to-end order reconstruction.

References