Most trading firms discover the state of their data the same way: a margin number that nobody can reproduce, a week before the board meeting.

Where firms actually sit

In our diagnostic work, most commodity trading firms operate at what we would call Level 2 of 5: data exists, but it is siloed by function, reconciled manually, and consumed through spreadsheet archaeology. Trading has positions, treasury has cash, logistics has cargo, finance has the ledger - and the firm's official view of itself is a reconciliation of extracts, days old. The gap between Level 2 and the real-time intelligence the business assumes it has is where risk lives: unpriced credit exposure, collateral surprises at T+1, margin reconstructed after the fact.

Five dimensions that decide the outcome

  • Assimilation. Are all source systems feeding one governed layer - or does every question start with an extract? The single most predictive dimension.
  • Lineage. Can the firm trace any reported number back to source trades, unaided, in minutes? Lineage is what converts data from output to evidence.
  • Quality controls. Are completeness, freshness and plausibility enforced at ingestion with measured SLAs - or discovered downstream, in a P&L variance?
  • Latency. Batch T+1, intraday, or streaming? Latency determines which decisions data can serve at all; a T+1 estate cannot support intraday limit monitoring regardless of model quality.
  • Consumption. Does the business read dashboards wired to the governed layer, or a shadow spreadsheet economy that quietly re-implements the truth?

The maturity trap: buying tools before the layer

The expensive failure mode is skipping to analytics - dashboards and ML on top of unreconciled extracts. Tools amplify whatever they are fed: a real-time dashboard on a T+1, manual-reconciliation estate simply makes the wrong number arrive faster. The order of operations is assimilate first, govern second, model third, visualise last.

The honest assessment

A maturity diagnostic is not an IT audit; it is a risk assessment. The questions that matter are operational: how many systems must agree before margin is known, how many manual touches a number passes through, how long a source-to-report drill takes. Firms that measure those three numbers usually find the case for a unified feed writes itself - and firms that fix the layer find that half their model problems were data problems wearing a costume.

Sources & further readingDatabricks - Transforming regulatory data management and risk analyticsDatabricks - Unity Catalog governance value levers