Canonical Parquet / Arrow schema¶

The pipeline reads and writes a small set of tables. This page is the written contract for those tables: for every column it gives the Arrow type, the unit, whether the column can be null, and what the value means. Other tools can read the Parquet files directly against this spec, without going through ob-analytics or pandas.

The spec is versioned. Each Parquet file carries a schema version in its metadata, so the on-disk format is a fixed contract, not a detail that can change without notice.

Schema version¶

The current version is 3.0 (the constant ob_analytics.schemas.SCHEMA_VERSION). Version 3.0 (issue #155) stores every price column as int64 ticks (see Price policy); 2.0 and 1.0 stored price as a double in the quote currency. Version 2.0 (issue #154) also makes both timestamp clocks tz-aware UTC nanoseconds; 1.0 wrote them tz-naive in each venue's native clock. All three still read — Parquet is self-describing, so a 1.0 / 2.0 file loads as the float-price frame it stored (a pre-tick file, whose prices are not directly comparable to a 3.0 file's ticks); re-save it with this build to move it onto the tick model.

save_data(..., fmt="parquet") writes the version into each file's Arrow key-value metadata under the key ob_analytics_schema_version (bytes, because Arrow metadata keys and values are raw bytes), alongside the tick size under ob_analytics_tick_size (see Price policy). load_data reads the version back and checks it before returning the frame:

• A supported version loads normally.
• An unsupported version raises ConfigError.
• A file with no version key is treated as legacy data. It loads, with a warning. This keeps files written before the version existed — and the bundled sample — working.

Bump the version when a change would make an older reader misread a newer file: a renamed or removed column, a changed unit or dtype, or a new required column.

Read the version from another tool through pyarrow:

import pyarrow.parquet as pq

metadata = pq.read_schema("out/events.parquet").metadata
version = metadata[b"ob_analytics_schema_version"].decode()  # "2.0"

The tables¶

Two groups of tables exist.

Persisted tables — the four core outputs a pipeline run produces and save_data writes to disk:

Derived records — engine output computed on demand from events. They are not written by default, but they follow a fixed shape and you can save them with save_data like any frame:

For a Level-2 (price-level) feed there is no per-order identity, so events is an empty but schema-valid frame and the derived records do not apply. Read depth, depth_summary, and trades instead.

Timestamp policy¶

All timestamps are timestamp[ns, tz=UTC] — tz-aware UTC nanoseconds (int64 nanoseconds since the Unix epoch, with the UTC zone attached). Both clocks use this one type, so frames from different venues sit on the same clock and are comparable: you can join or concatenate them directly. A source in a venue-local clock (LOBSTER, US/Eastern) is converted to UTC on load, using the session date and the venue time zone; a source already in epoch-UTC (Bitstamp, CCXT) keeps its wall-clock instants and only gains the zone and the nanosecond unit.

timestamp is the local receive time. exchange_timestamp is the venue's matching-engine time. For LOBSTER the two are equal, because only exchange time exists there.

Same-instant order¶

A timestamp alone cannot order two events that share an instant, so the schema defines a total order: timestamp, then — as tie-breaks, when the frame carries them — the venue sequence, then event_id (the dense per-order key on the L3 path), then the local ingest_seq. ob_analytics.schemas.time_order_keys returns this key list for a frame; the per-order reconstructions (queue_positions) sort by it, and every loader builds its frame with fixed, stable sorts, so a rebuild is deterministic run-to-run. On a price-level (L2) feed there is no event_id; ties there fall to sequence / ingest_seq when tracked, otherwise to the loader's stable arrival order. Enforcing the full key inside the price-level depth engine — so an alternate engine reproduces it bit-for-bit — lands with the engine separation and rewrite (#136 / #104 / #138), when it can be checked against that second backend.

Price policy¶

Every price column is a whole number of ticks (int64), not a float in the quote currency. The quote-currency price is ticks * tick_size, where tick_size is the instrument's minimum price increment (PipelineConfig.tick_size, default 0.01). Storing the exact integer removes the float rounding that made small-tick and 0-1 instruments show crossed levels that were not real, and lets the depth engine bin and compare levels on exact integers. A loader converts a raw price to ticks on the way in; the plots and the round-trip writers convert back for display.

tick_size is a property of the instrument, not of any one row, so it is stored once per file in Arrow key-value metadata under the key ob_analytics_tick_size — a JSON object mapping an instrument key to its tick size. The single-instrument pipeline writes one entry, {"default": 0.01}; a multi-instrument file (issue #147) adds entries keyed by "venue|symbol". Read it back from another tool through pyarrow:

import json
import pyarrow.parquet as pq

metadata = pq.read_schema("out/events.parquet").metadata
tick_sizes = json.loads(metadata[b"ob_analytics_tick_size"])  # {"default": 0.01}
tick_size = tick_sizes["default"]
# quote-currency price = price_column * tick_size

load_data surfaces it on each returned frame's attrs: df.attrs["tick_size"] is the resolved default and df.attrs["tick_sizes"] the full map. A legacy (pre-3.0) file carries neither and stores float prices already in the quote currency.

Volume and fill¶

Per-order size follows one convention across every loader:

• volume is the order's outstanding size after the event for created and changed rows, and the size removed (the outstanding size just before the delete) for deleted rows.
• fill is the executed size at this event (0 when nothing traded). A changed row is either an execution (fill > 0, outstanding drops by exactly fill) or a non-executed reduction (fill == 0), never both.

Sizes are integer lots and prices integer ticks — multiply by lot_size and tick_size respectively (see Size policy and Price policy).

Size policy¶

Every volume and fill column is a whole number of lots (int64), not a float in the base asset. The base-asset size is lots * lot_size, where lot_size is the instrument's minimum size increment (PipelineConfig.lot_size, default 1e-8, a satoshi grid; LOBSTER sets 1, whole shares). A loader converts a raw size to lots on the way in; the plots and the round-trip writers convert back for display.

This is the size half of the Price policy, and it exists for the same reason. A price level is a running sum of adds, cancels and fills, and a float sum does not return to exactly zero when the last order leaves it. It lands on residue such as 5.55e-17, the level stays live, and it is reported as the best bid or ask ahead of the real one. On the bundled Bitstamp sample that corrupted the reported best bid on 8.2% of rows and the best ask on 9.6%, and so the reported spread on about one row in eleven. Integer lots cancel exactly, so a level empties or it does not.

lot_size is stored the same way tick_size is: once per file in Arrow key-value metadata under ob_analytics_lot_size, a JSON object mapping an instrument key to its lot size.

import json
import pyarrow.parquet as pq

metadata = pq.read_schema("out/events.parquet").metadata
lot_sizes = json.loads(metadata[b"ob_analytics_lot_size"])  # {"default": 1e-08}
lot_size = lot_sizes["default"]
# base-asset size = volume_column * lot_size

load_data surfaces it on each returned frame's attrs: df.attrs["lot_size"] is the resolved default and df.attrs["lot_sizes"] the full map. A pre-4.0 file carries neither and stores float sizes already in the base asset.

Nullable integer columns¶

pandas has no built-in null for a plain int64 column, so the loaders carry nullable integer ids (order and event ids on trades) as an object column holding Python ints or NA. On write, pyarrow infers the Arrow type from the values: a column with no nulls becomes int64; a column that contains nulls may become int64 with a null mask or double. The tables below name the logical type (int64) and mark the column nullable. Cleaning this up is one of the data-model decisions below.

events¶

Required columns (the EVENT_COLUMNS contract) plus the provenance columns every loader carries.

raw_event_type is present but all-null on a Bitstamp frame; pyarrow writes an all-null column as the null type. raw_size and aggressiveness_bps are present only after the stages that add them.

trades¶

Required columns (the TRADE_COLUMNS contract) plus maker / taker provenance.

See Nullable integer columns for how the id columns materialize on disk.

depth¶

Output of price_level_volume. Required columns are the DEPTH_COLUMNS contract; the L3 path adds event_id.

volume here is the level's absolute resting size, not a signed delta. An L2 loader yields this frame directly with the same four required columns.

depth_summary (book states)¶

Output of depth_metrics. One row per depth event, holding the reconstructed best bid and offer plus cumulative resting volume in basis-point bins on each side. This is the reconstructed book state over time.

The bin columns depend on configuration. With the defaults (depth_bps = 25, depth_bins = 20) there are 20 bins per side at N = 25, 50, …, 500: bid_vol25bps … bid_vol500bps and ask_vol25bps … ask_vol500bps. Each bin holds the volume in that bps slice of the book.

* event_id is always present on a standard pipeline run, which feeds L3 depth carrying it. It is omitted only if you call depth_metrics on a depth frame without an event_id column.

Order lifecycles¶

Output of order_lifecycles. One row per order, collapsing its events into a placement and an outcome. Orders with no created row (a pre-existing opening book, hidden executions) are excluded.

Book snapshot¶

Output of order_book. A snapshot of the resting book at one point in time, returned as two frames (bids and asks), each with these columns.

Categorical value domains¶

Three categorical columns are ordered; the order is the category order, not alphabetical.

An ordered categorical is stored as an Arrow dictionary<string> (the categories plus small integer indices). A reader that does not want dictionary encoding can cast the column to plain strings.

Frame types: pandas in, pandas out¶

The frame type is part of the contract, so it is written down here:

• Every public function takes and returns pandas. Pipeline.run, the metrics, the plot helpers, load_data and save_data all speak pandas.DataFrame.
• Plug-ins are handed pandas. The protocols you implement to add a source or an export format — EventLoader, TradeSource, DepthSource, OfflineSource.compute_depth, DataWriter — take and return pandas.DataFrame. See Extending.
• The file format is how other tools read the output. The versioned Parquet described on this page is the interop surface, not the Python type. Polars, DuckDB and anything else Arrow-aware read the files directly, as shown below.

ob-analytics does not depend on Narwhals, Polars or DuckDB, and its public functions never return one of their types. The reasoning is recorded in ADR 0002.

From a result in memory¶

You do not have to write files first. A PipelineResult converts itself:

result = Pipeline().run("data/orders.csv")

tables = result.to_arrow()    # dict[str, pyarrow.Table]
frames = result.to_polars()   # dict[str, polars.DataFrame]

tables["depth_summary"].num_rows

Both return the same four keys — events, trades, depth and depth_summary — for every run. On a price-level (L2) run events is an empty table rather than a missing key, so the keys do not change with the level of the data.

The Arrow tables carry the same key-value metadata the Parquet files carry: the schema version under ob_analytics_schema_version and the tick size under ob_analytics_tick_size. A reader handed a table in memory is no worse off than one reading a file.

to_polars() needs polars, which ob-analytics does not install. Without it the call raises ImportError telling you to pip install polars. Polars keeps no schema-level metadata, so the schema version and tick size do not survive that conversion — read the tick size from result.config.tick_size, or use to_arrow() when the metadata has to travel with the tables.

Zero-copy reads in Polars and DuckDB¶

The files are plain Parquet, so any Arrow-aware tool reads them directly, with no pandas step. The examples below are illustrative. Polars and DuckDB are not dependencies of ob-analytics and are not installed by default; install them yourself to run these.

Polars¶

import polars as pl

# Eager read — one file, Arrow-native, no pandas.
events = pl.read_parquet("out/events.parquet")

# Lazy scan — push filters and column selection into the Parquet reader,
# so only the needed row groups and columns are read.
touch = (
    pl.scan_parquet("out/depth_summary.parquet")
    .select(["timestamp", "best_bid_price", "best_ask_price"])
    .filter(pl.col("best_bid_price") > 0)
    .collect()
)

# The ordered categoricals arrive as Polars Categorical / Enum columns.

DuckDB¶

import duckdb

# Query the Parquet files in place; DuckDB reads them through Arrow.
# Prices are integer ticks, so the spread here is in ticks; read the file's
# `ob_analytics_tick_size` metadata and multiply for the quote currency.
spread = duckdb.sql(
    """
    SELECT timestamp,
           best_ask_price - best_bid_price AS spread_ticks
    FROM 'out/depth_summary.parquet'
    WHERE best_bid_price > 0
    """
).arrow()  # hand back an Arrow table, zero-copy

# Filter by an ordered categorical value using its string form.
bids = duckdb.sql(
    "SELECT price, volume FROM 'out/depth.parquet' WHERE direction = 'bid'"
).df()

Data-model decisions¶

The four cross-cutting choices the schema left open now have a decided direction (2026-08-23): follow the Databento / Nautilus conventions where they are the standard, expressed in Arrow / Parquet, not the DBN binary format. All four are implemented: sequence numbers (#146), instrument identity (#147), the time model (#154), and integer-tick prices (#155). See the roadmap (#124).

1. Time model: tz-aware UTC nanoseconds and event order — implemented (#154)¶

• Decision, built here. Both clocks — timestamp (receive) and exchange_timestamp (matching engine) — are timestamp[ns, tz=UTC]: int64 nanoseconds since the Unix epoch, with the UTC zone attached (see Timestamp policy). A venue-local source (LOBSTER, US/Eastern) is converted to UTC on load from the session date and venue time zone; an epoch-UTC source (Bitstamp, CCXT) keeps its wall-clock instants and only gains the zone and the nanosecond unit. The same-instant total order is timestamp, then the venue sequence, then event_id, then ingest_seq (ob_analytics.schemas.time_order_keys).
• Comparable across venues. Because every frame is on one UTC clock, frames from different venues can be joined or concatenated directly — the earlier "not comparable across venues" rule is gone.
• Deferred. Enforcing the full same-instant key inside the price-level depth engine (so an alternate backend reproduces the rebuild bit-for-bit) lands with the engine separation and rewrite (#136 / #104 / #138), validated against that second backend. Today the per-order reconstructions sort by the total order and the engine plays events back in a stable receive-clock order, deterministic run-to-run.

2. Source sequence numbers — implemented (#146)¶

• Decision, built here. Two optional columns: sequence (nullable Int64, the venue's per-event sequence number, populated where a source provides one — the CCXT nonce, or an optional column in a Bitstamp / L2 capture CSV) and ingest_seq (int64, a local monotonic counter in arrival order, the stable replay key). detect_sequence_gaps() reports dropped and out-of-order messages, surfaced in DataQualitySummary and the audit command (which loads with track_sequence on for exactly this reason).
• Non-breaking. The loader-attached columns are gated behind a default-off PipelineConfig.track_sequence flag, so existing frames are byte-for-byte unchanged.
• Later. Live re-sync on a detected gap (refetch a snapshot) is the non-additive half and is not built yet.

3. Integer-tick prices instead of floats — implemented (#155)¶

• Decision, built here. Every price column is an integer tick count (int64) plus a per-instrument tick_size — not float, and not one global scale (see Price policy). The quote-currency price is ticks * tick_size; the plots and the round-trip writers convert back for display only. tick_size lives on PipelineConfig at run time and in each Parquet file's ob_analytics_tick_size metadata on disk. This matches the depth engine's internal integer binning and the backtester-export targets (#113).
• Simpler engine, exact comparisons. LOBSTER's price_divisor is now just the raw-feed encoding scale; the loaders quantise to ticks once, so the depth engine bins and compares levels on exact integers instead of multiplying and rounding each event. Crossed-level detection is exact, removing false crosses from float rounding.
• Breaking, by design. This is the deepest of the four changes — the price column in every table, all price arithmetic, every loader, and the plots. It changes numbers on purpose (prices are re-expressed as ticks, and price-valued analytics such as VWAP and Kyle's λ are now in tick units), so the golden outputs were re-baselined behind the correctness gate (#143).

4. Instrument identity — implemented, first stage (#147)¶

• Decision, built here. Readable per-row symbol and venue columns (nullable string, dictionary-encoded) — not Databento's numeric id plus a definitions join, which is built for a far larger feed than this project has. Each loader tags its frame when identity is supplied; venue defaults to the source name (bitstamp, lobster, the CCXT exchange id). group_by_instrument() splits a frame by whichever of (venue, symbol) are present.
• Non-breaking. Identity is opt-in, so a default single-instrument run adds no columns and existing output is unchanged.
• Later. The full multi-symbol, cross-venue pipeline (one run over several instruments) is the larger part of #147 and is not built yet.