Changelog¶

All notable changes to ob-analytics are documented in this file.

The format is based on Keep a Changelog.

[Unreleased]¶

Moving from 0.1¶

Each break, old -> new. The package keeps no aliases for old names before 1.0.

Pickle is gone (#409). A pickle runs code when it is loaded.
save_data(data, "run.pkl", fmt="pickle") -> save_data(data, "run"), which writes a Parquet folder.
PickleWriter -> removed; use ParquetWriter.
load_data("run.pkl") -> removed. To move an old pickle, read it with pandas.read_pickle (only a file you wrote yourself) and save it with save_data.
save_data takes a folder or a file, by format (#409). It checks path before it writes, and says which it expected.
save_data(r, "out/orders.csv", fmt="bitstamp") -> save_data(r, "out", fmt="bitstamp"), which writes out/orders.csv and out/trades.csv. BitstampWriter.write(data, dest) takes the folder too.
save_data(r, "out/", fmt="databento") -> save_data(r, "out/events.dbn", fmt="databento"). DatabentoWriter.write no longer fills a folder with events.dbn.
save_data(r, "out/session", fmt="hftbacktest") -> save_data(r, "out/session.npz", fmt="hftbacktest"). The writer writes the exact path it is given, and no longer adds .npz.
save_data(r, "run.parquet") -> save_data(r, "run"): the default Parquet format takes a folder, and now refuses a name that ends in a file extension instead of making a folder called run.parquet.
A writer declares target (#409). A DataWriter without target = WriteTarget.FOLDER or target = WriteTarget.FILE -> add one. WriteTarget is in ob_analytics.protocols and at the top level.
OfflineSource.create_writer is optional (#409). A source whose create_writer returns None -> leave the method out. list_writers() lists a source as a format when it has the method.
list_writers() lists every format save_data accepts (#409): the registered writers and the sources with a writer of their own (bitstamp, databento, depth_csv, lobster). The unknown-format error offers the same list, so it no longer offers ccxt and cryptofeed.
load_data and load_result refuse a path that is not a folder with ConfigError (#409). load_data raised ValueError for a file.
One error type for one kind of mistake (#408). except ObAnalyticsError now catches every mistake the package reports.
A bad value (an argument out of range, empty or incomplete input data) raises ConfigError, which is now also a ValueError, so except ValueError still catches it. An empty frame raised the base ObAnalyticsError; it now raises ConfigError.
An unknown name raises the new UnknownNameError, which is also a KeyError. This covers sources, metrics, features, bar rules, formats, plots, plot backends, gallery views, theme styles and contexts, and CCXT and cryptofeed venues. Its message names the closest known names, not every registration. Several of these raised ValueError: except ValueError around save_data(fmt=...), plot(backend=...), PlotTheme(style=...), Pipeline.from_source or a venue id -> except UnknownNameError (or KeyError).
A plot a backend does not draw names the backends that do draw it.
UnmappedMessageError is now also an ObAnalyticsError.
read_manifest refuses a bad manifest.json with ConfigError (#418). It raised ValueError, KeyError or TypeError, depending on what was wrong. except ValueError still catches it, since ConfigError is one; except TypeError or except KeyError around it -> except ConfigError.
Every plot entry point takes the same arguments (#410). plot(), plot_result() and PipelineResult.plot() decide the level by one rule, and all three take ax=.
plot_result(result, concept) and result.plot(concept) for a concept the result has at both L2 and L3 (on an L3 run: trade_tape, order_activity, cancellations, book_snapshot, depth_chart, book_replay, liquidity_at_touch) drew the L2 plot. They now raise ConfigError, as plot() did -> pass level="L2" (or "L3"). A concept the result has at one level still needs no level.
plot_result(..., level=None) and result.plot(..., level=None) meant "pick the level". None now means a level-less plot (a metric), as in plot() -> leave level out instead.
result.plot(..., ax=ax) and plot_result(..., ax=ax) are new; they draw on your Matplotlib axes, as plot(..., ax=ax) does.
plot(..., ax=ax, backend="plotly") (or "bokeh") ignored ax. It now raises ConfigError -> leave ax out for those backends.
The error for a missing level says pass level="L2" or level="L3"; it no longer says the concept is "comparable".
plot_result(result, metric, level="L2") ignored the level. It now raises UnknownNameError, since a metric has no L2 or L3 plot -> leave level out.
The ccxt and cryptofeed source and settings classes are in ob_analytics.live (#410). The module paths still work; import from the package instead.
from ob_analytics.live.ccxt_source import CcxtSource -> from ob_analytics.live import CcxtSource.
from ob_analytics.live.ccxt_source import CcxtSettings -> from ob_analytics.live import CcxtSettings.
from ob_analytics.live.cryptofeed_source import CryptofeedSource -> from ob_analytics.live import CryptofeedSource.
from ob_analytics.live.cryptofeed_source import CryptofeedSettings -> from ob_analytics.live import CryptofeedSettings.
The capture folder format has one module (#401). The sink that writes a capture moved out of the private runner module: ob_analytics.live._runner.FileCaptureSink -> ob_analytics.capture_files.FileCaptureSink.
Palette.flashed and Palette.resting are gone (#415). No plot read them. Palette(flashed=..., resting=...) -> leave both out.
The command line has one name per thing (#411). The input is path and the output is -o/--output on every subcommand.
ob-analytics capture ... --out DIR -> ob-analytics capture ... -o DIR (or --output DIR).
ob-analytics capture ccxt --exchange kraken -> ob-analytics capture ccxt --venue kraken; the same for cryptofeed.
ob-analytics capture <source> --help lists the source's flags, which are now read after the source's name: ob-analytics capture --exchange kraken ccxt -> ob-analytics capture ccxt --venue kraken. ob-analytics capture --help no longer lists them.
ob-analytics bitstamp-demo --input PATH -> ob-analytics bitstamp-demo PATH.
A plug-in source's settings are flags of capture (#411). A SourceSettings field gets a --<field> flag when its type is one the command line can read. A plug-in whose settings have a field named like a capture flag (pair, minutes, output, roll_minutes, roll_mb, no_raw) -> give the field CliOption(flag=...) with another name, or capture stops with a message naming the field.
A function that needs per-order events refuses an L2 run (#399). On a run with no events (what an L2 run has), set_order_types, order_lifecycles, order_book, order_aggressiveness, price_level_volume, queue_positions, queue_age_grid and hidden_trades raise ConfigError naming L3. queue_positions, queue_age_grid and hidden_trades returned an empty result, which read as "nothing there". The Bitstamp, Databento, hftbacktest and Nautilus writers raise it too instead of writing an empty file. The LOBSTER writer already refused; its message is now the same as the others'.
A metric declares needs (#399). A Metric without it -> add needs: tuple[str, ...], the run's tables it cannot be computed without, e.g. ("trades",). A metric is skipped on a run where one of them has no rows.
A lot_size finer than volume_decimals is refused (#399). PipelineConfig(lot_size=1e-9) -> PipelineConfig(lot_size=1e-9, volume_decimals=9). Sizes were shown rounded to a coarser step than they have. PipelineConfig now raises ConfigError from its own checks, as SourceSettings does.

Added¶

The command line sets what the Python API sets (#411).
process and audit take --tick-size, --lot-size, --depth-bins, --depth-bps and --session-tz. Without --tick-size and --lot-size a capture's own still come from its meta.json.
process --format saves in any format save_data writes, such as --format bitstamp or --format hftbacktest -o run.npz.
process --cut TIME runs one time window at a time, as Pipeline.run_windows does; give it once for each cut, and --no-carry to start each window from an empty book.
gallery --backend and process --gallery --backend pick the plot backends; give it once for each.
capture <source> has one flag for each field of the source's settings class, so a plug-in source can be set up from the command line. A field marked with the new ob_analytics.sources.CliOption gets its help text, flag name or value type from the marker; a Credential field never gets a flag.
The docs code runs in CI (#420). scripts/check_doc_examples.py runs the Python blocks of every docs page outside the tutorial, page by page and in order, and a new examples job in the docs workflow runs it. A block that cannot run as written, such as one that needs an API key, the network or the reader's own file, carries a docs-test: skip comment with the reason. Quoted outputs, such as the audit report and the bar and feature tables, are compared with what the code prints, so a change that moves a figure fails the job. The examples that did not run are fixed: the uncross= example on Data quality, the custom metric that called a Resampler.skew pandas does not have, the custom source that took no settings, and blocks with missing imports or undefined names. The quoted audit report, episode counts, the bar and feature tables, and the quickstart's table shapes are brought up to date.
CI tests what a user installs, and a release waits for CI (#416). A new wheel job installs the built wheel with no extras and runs scripts/smoke_wheel.py: the sample pipeline, a Matplotlib plot, and a ConfigError naming the extra for a Plotly or Bokeh plot. The release workflow runs the same script on the wheel it publishes, and refuses a commit whose CI run did not pass. New jobs also run the tests on Python 3.14 (now in the classifiers), on Windows (core and offline tests only), with each direct dependency at its lowest allowed version, and with the backtest-engines group, so the hftbacktest and Nautilus Trader parity tests run. The lowest versions found floors that were wrong. The bokeh extra now needs Bokeh 3.1, because 3.0 does not import with NumPy 2. The ccxt extra needs ccxt 4.5.90 and the cryptofeed extra cryptofeed 2.4.1, the oldest versions CI tests: ccxt 4.0 has no Coinbase in ccxt.pro, and cryptofeed 2.0 has no Independent Reserve. The dev group needs nbmake 1.5.1, the first that loads under pytest 9. Coverage now counts the CLI tests' subprocess runs. CONTRIBUTING lists the checks to mark as required on main; the speed check still only reports. CI on main no longer cancels or replaces a commit's run when a newer commit lands, and the docs build runs on every pull request, skipping itself when no site input changed, so it can be a required check.
Every registry loads plug-ins from entry points (#403). Bar rules, features, writers and plot backends now load from the entry-point groups ob_analytics.bar_rules, ob_analytics.features, ob_analytics.writers and ob_analytics.plot_backends, as sources and metrics already did. One loader serves all six groups, and load_bar_rule_plugins, load_feature_plugins, load_writer_plugins and load_plot_backend_plugins re-scan a group. A plug-in that fails to load is now skipped with a UserWarning, not a log message that is off by default. The renderer registry moved to visualization._renderers, which removes an import cycle that depended on line order.
InstrumentGrid and DepthCsvSettings (#348). A source states the tick and lot size its data is usually on as grid, an InstrumentGrid, read with protocols.grid_of; a run's config starts from it, and a field the caller sets wins. DepthCsvSource(settings=DepthCsvSettings(...)) sets the timestamp_unit and price_divisor of a price-level CSV, which were PipelineConfig fields. Pipeline.run_windows takes ctx=, as run does.
audit compares the rebuilt book with the venue's own book. The new venue_book check rebuilds the book from the LOBSTER messages and compares it with the orderbook file, row by row, on every price and size of the levels the file states. Each event is matched with the row of its own message, by original_number. The report gives the number of rows that differ and the first difference (a BookDifference); the summary has the fields venue_book_price_levels, venue_book_rows, venue_book_differing_rows and venue_book_first_difference. data_quality_summary takes the venue's book as venue_book, read by LobsterSource.venue_book or lobster_venue_book. A source with no book of its own shows the check as skipped. QualityCheck has a new skipped field: a skipped check is info, never fails a run, and is not counted as passed, as the clock checks on data with one clock now are too. On the AAPL samples the two books differ in 99.81% (50 price levels) to 100% (10 price levels) of rows, because the message file does not cover the orders resting before it starts or outside the requested levels; see the LOBSTER how-to (#392).
audit reports how long the per-order book is crossed. A LOBSTER run with an orderbook file reads its depth table from that file, so the crossed_book check passed even when the book rebuilt from the messages was crossed: 99.2% of the time on the 10-level AAPL sample. The summary has two new fields, per_order_crossed_pct and per_order_crossed_episodes, a per-order book crossed line, and a per_order_book_crossed warning when that book is crossed for longer than the depth table.
One dtype map for the events and trades tables (#351). ob_analytics.schemas now holds EVENT_DTYPES and TRADE_DTYPES, the pandas dtype of every column, next to the column names, with the four ordered categoricals they use, UTC_NS_DTYPE, and two helpers, cast_columns and empty_frame. The empty frames, the toy datasets and the synthetic session are built from the maps, and every trade reader (Bitstamp, LOBSTER, L2, Databento) casts its output through TRADE_DTYPES. load_data casts a current-version trades file back through it. _utils._EMPTY_EVENT_DTYPES and EMPTY_TRADES_COLUMNS are removed; UTC_NS_DTYPE is still importable from _utils.
load_result(path) reads a saved run back as a PipelineResult (#346). It is the inverse of save_data(result, path): the four tables come back, and so does the run's PipelineConfig. save_data now takes the result itself, and each Parquet file records the run's whole config under the new key ob_analytics_config. A folder that records only the tick and lot size (a windowed run, or one saved by an older version) gets those two, with the display precision that shows one step of each; a folder of whole-number prices or sizes that records neither is warned about. load_result reads Parquet folders only, and refuses a pickle, which keeps no config. The key is optional, so the schema version stays 4.0 and older readers ignore it.
ob_analytics.capture_record owns a capture's meta.json (#346). read_record(path) returns a typed CaptureRecord: the source that made the capture, its feed type, trade attribution, sequence kind and clocks, the sequence restarts, and the instrument's tick size and lot size. read_instrument(path) returns only the tick and lot size, as config fields. write_record and source_declarations are what the capture runner and the segment supervisor write it with. The module is outside ob_analytics.live, so code that only replays files does not import the live package.
A ccxt capture records the venue's size step as lot_size (#346), from ccxt's market data, beside the tick size. If a size arrives between two steps, the capture makes the step finer until the size fits, as it does for the tick, and counts the changes in lot_size_changes. The recorded lot is never finer than 1e-8, the default grid.
audit counts trades matched by order id (#352). A new summary field, trades_matched_by_order, counts the trades whose taker is an order created with size 0. Bitstamp sends an instant order bought by quote amount that way and never reports what it bought, so its trades are matched to it by its order id, not by size. audit prints the count as matched by order and reports it as information; it never fails a run. A feed that does not show takers prints not checked. The bundled Bitstamp sample has 2. schemas.unsized_order_ids(events) returns the ids of orders created with size 0; the trade reader and audit both use it.
A source can take an API key (#239). A source declares each key it needs as a SecretStr | None field of its settings, marked with Credential(env=..., issued_at=...). The settings read an unset key from that environment variable; with from_file=True the variable holds the path of a key file. A key is never a command-line option. run_capture stops before it writes anything when a key is missing, and names the variable and where the venue issues keys; a key shorter than eight characters is refused. No file a capture writes holds a key, and neither does the CLI log: a key that reaches one, in a venue's message or an error, is replaced with **********. ob-analytics sources lists the variables each source needs. No built-in source needs a key yet. See "Use an API key".
Exercises and an editable toy book in the tutorial (#121). Chapters 2, 4 and 6 end with exercises: where a new order queues and which order keeps the Bitstamp book crossed (chapter 2), what happens if Eve does not cancel (chapter 4), and how the VPIN bucket size changes the result (chapter 6). Each solution is collapsed until the reader opens it, and runs when the documentation is built, so a wrong answer fails the build. The toy session page now has a book replay to step through, and shows how to rewrite the session: toy_orders() returns its twelve orders as ToyOrders keyed by actor, and match_toy_orders(orders) matches an edited script by price–time priority and returns its events and trades in the layout of toy_events() and toy_trades().
Run a large input one time window at a time (#116). Pipeline.run_windows(source, boundaries, output) cuts one input at the given times and runs the depth stages on one window at a time, so their peak memory is set by the largest window. Each window is written to output as it finishes, as one Parquet file per table, the same folder save_data writes; load_data reads it back. With carry=True, the default, each window starts from the book the previous one ended with, and the output matches a single run row for row. The exceptions are aggressiveness_bps on a Bitstamp input, and depth where the Databento loader already warns that it is off. A run that fails part-way leaves the output folder as it was. On 1.26M events, eight windows peak at 771 MiB against 1,507 MiB for a single run. The loader still reads the whole input. See "Scale and chunking".
The flow-toxicity faces and the L1 quote are built-in metrics (#118). vpin, kyle_lambda, order_flow_imbalance and ofi_horizon are now registered metrics, so available_concepts lists them and result.plot("vpin") computes and draws VPIN with the defaults of compute_vpin. A keyword goes to the calculation or to the picture, whichever names it: result.plot("vpin", n_buckets=20, threshold=0.8). result.metric(name, **settings) takes the calculation's settings too. The new l1_ticker metric is the Level 1 quote — best bid, best ask and last trade. It draws the three prices over time, or the quote card for one instant with at=; plot("l1_ticker", bid=99, ask=101, last=100) draws a card from plain numbers. Each built-in metric gets a gallery card. ofi_by_horizon is the multi-horizon order flow imbalance that the ofi_horizon face draws, now a function in flow_toxicity.
A book replay (#117). The book_replay concept steps through the book over part of a run, on Plotly: the ladder at one instant per second beside the trades and the mid, with a slider and play buttons at 1×, 5× and 20× real time. Click a trade to move to it: the price level it took is marked in the trade's colour. The L2 face replays the depth table, so its touch equals the depth summary's at every frame; the L3 face shows each order, with stale crossed orders removed so both faces show the same market. The gallery shows it for the zoom window, and prepare.book_replay builds it directly. The price-level book the depth summary is built on is now its own class, ob_analytics.depth.PriceLevelBook, and price_level_snapshots returns that book at many instants in one pass.
A page on what each feed shows (#288). "What each feed shows" describes 16 properties that decide what a capture can tell you, such as depth shown, what can be missed, sequence, clocks and taker side. It gives each property's value for every source and venue, checked with live captures. Every source's how-to page now opens with its values and when to use it.
FeedType.PRICE_LEVELS, the crossing value for L2 feeds (#288). The ccxt source, cryptofeed at L2 and depth_csv declare it instead of MATCHED_BOOK. The venue does not publish a crossed price-level book, so audit scores a crossed one as an error, as for a matched book: most often, the capture kept a level the venue removed. Captures made before this record matched_book and are scored the same way. In the Source protocol, level and feed_type are now read-only, so a source can work either one out, as the cryptofeed source does from its venue.
A live capture can run for days (#150). ob-analytics capture now writes a directory of segments with a manifest.json. A lost connection ends the segment; the capture waits (1 s, doubling to 60 s) and starts a new one from a fresh snapshot. --roll-minutes and --roll-mb start a new segment by time or size, and the new segment is streaming before the old one stops, so a roll loses nothing. Running the same command again continues the capture, and closes a segment a crashed process left open; a lock on the directory stops a second process from writing to it at once. The manifest records every segment, why it ended, and every gap with its cause. process and audit read the whole capture, and audit adds the checks capture_gaps, unfinished_segments and dropped_messages. From Python: ob_analytics.live.run_capture and read_manifest.
Each source declares which orders of a trade it can name (#284). The new Source.trade_attribution (TradeAttribution.BOTH, MAKER_ONLY or NONE) says whether a feed's order events show the trade's maker, its taker, or neither. bitstamp names both. lobster, databento and cryptofeed at L3 name the maker only: their feeds show resting orders, and a taker trades on arrival. The L2 sources name neither. The unmatched_trades check now counts only the orders the feed can name, and says which. Before, a Databento run could only report every trade as unmatched, and LOBSTER's guessed takers counted as matches. A source that does not declare it is read as BOTH, the old check.
A capture records what its source declares, and audit holds it to that (#284). meta.json now records source, feed_type and trade_attribution. ob-analytics process copies meta.json into its output. audit checks a capture against the declarations of the source that made it, even when --source names another: a cryptofeed L3 capture is read with --source bitstamp, whose feed shows more. Read the record with recorded_source, recorded_feed_type and recorded_trade_attribution.
A cryptofeed reconnect is counted (#309). cryptofeed reconnects by itself when it loses a message and starts again from a new opening book. The capture counts each new opening book as book_resyncs in meta.json, as the ccxt source already did. On Bitfinex and Blockchain.com the venue's count starts again too; the capture counts that step back as sequence_restarts, and audit leaves it out of sequence_out_of_order, so a reconnect is scored the same on every venue. Read it with recorded_sequence_restarts and pass it to data_quality_summary(sequence_restarts=...).
A Bokeh plot backend (#123). result.plot(concept, backend="bokeh") renders the core concepts — trade_tape, depth_heatmap, book_snapshot, depth_chart — as interactive Bokeh figures, alongside the static Matplotlib default and the Plotly backend. Suited to Bokeh / Panel server dashboards and streaming views. Ships in the new bokeh extra: pip install "ob-analytics[bokeh]".
Binance venue notes and a market-data mirror (#101). A new how-to page covers capturing Binance spot through ccxt: the location block, the binanceus alternative, the depth a 100-level book reaches, and trade sides. capture ccxt --market-data-mirror (CcxtSettings.market_data_mirror) reads Binance spot from Binance's market-data-only endpoints. New SequenceKind and recorded_sequence_kind(), a kind argument on detect_sequence_gaps(), and a sequence_kind argument on data_quality_summary().
Find hidden liquidity (#111). detect_icebergs(events, trades) finds iceberg orders from their refills: a resting order filled out, then a new order at the same side and price within one millisecond. It chains refills into one suspected iceberg and gives each a confidence. hidden_trades(events, trades, depth_summary) returns the trades that printed strictly inside the visible spread. On one day of LOBSTER AAPL it finds 85% of the type-5 hidden executions, with no false ones. The synthetic generator now labels its iceberg slices in SynthSession.icebergs, so the detector can be scored exactly. See the hidden liquidity how-to.
Iceberg refills and hidden trades drawn on the depth heatmap and order activity map (#272). Both L3 faces now overlay detect_icebergs and hidden_trades: a diamond marks each refill, joined by a line per iceberg (opacity = confidence); a star marks a hidden trade, with a thin line to the standing best bid and best ask so the print reads as inside that spread. A filled star is a confirmed hidden order; an open star is a trade to check, where the maker order was actually visible or its maker identity did not resolve at all — the diff-feed case the how-to guide describes. Both overlays clip to the gallery's zoom window, and are absent without error when a run has neither. New prepare.hidden_liquidity_overlay builds the same overlay for a custom plot. hidden_trades now returns best_bid_price/best_ask_price in depth_summary's own dtype instead of always casting to int64, so a caller already holding display-unit floats gets floats back rather than a silently truncated value.
A feature table for models (#149). features(trades, quotes) returns one tidy table: a point in time on each row and a microstructure feature in each column — the shape a model or a study wants. It replaces a join per measurement.

Two decisions make the table, and they are separate. Where the rows fall is a bar rule, so features() takes the same sampling arguments bars() does and the same arguments give the same cut in both; a time grid is the time rule. What each column measures is a Feature, and ten ship: price, returns, flow, spread, mid_price, micro_price, imbalance, depth, vpin and kyle_lambda, writing 20 columns between them. register_feature adds one of your own, usable by name with no edit to the package.

Every row is stated as of the close of its bar. The trade columns hold what happened inside the bar, and the book columns hold the book as it stood at the close, a backward as-of join. Nothing from after that instant reaches the row, so the table carries no look-ahead — which is tested by truncating the inputs and checking that the rows that survive are unchanged, for every rule and every feature. The table holds no target either: a target looks forward, and building one is a shift the caller makes deliberately.

Two quote states are not books anything could have traded against, and both would otherwise arrive as ordinary numbers: a side with nothing resting on it, which the depth engine marks with a price of 0, and a crossed book, which a diff feed can genuinely hold. readable_quotes() drops them from the reference series, so a row reaches back to the last quote it could read — the same test transaction_costs already applied before measuring against a mid. A locked book, bid equal to ask, is a real state at a spread of zero and is kept.

Without a quotes frame the five book features are skipped and the table holds the trade features alone; naming one explicitly raises instead. Two features that would write the same column are an error rather than a silent overwrite, and so is a name listed twice in include; a trailing window a feature cannot use is refused when the feature is built. See the "Build a feature table" how-to, which ends in a baseline model.

Databento market-by-order files (#100). DatabentoSource reads Databento's DBN files in the MBO schema, which is a per-order feed: every record carries an order id, so a file replays through the full L3 path with order lifetimes, queue position and order classification. It reaches many venues that no other source in the package does, US equities and futures among them.

Databento reports an execution as a fill record that does not change the book, followed by the cancel or modify that takes the size off it. The loader pairs the two, so fill tells an execution apart from a cancel the trader asked for, and each trade names the resting order it hit. The aggressor's side comes from the venue rather than a classifier. A book clear deletes the orders still resting, and the record's two clocks are kept apart: Databento's receive time orders the events, the venue's own becomes exchange_timestamp.

A modify that moves an order to another price or makes it bigger is recorded as a changed event with the new price and size, and the depth follows it (see Changed). The loss of queue priority is not modelled. The one modify the depth still cannot follow is one that carries a fill and also moves the order or changes its size by more than the fill; the loader says how many rows the depth will be off by.

A feed the loader does not understand is refused: a publisher that only sends top-of-book or price-level data, because its order ids mean nothing; a price-level schema; a file covering more than one book; an action outside DBN's own alphabet; an order id too big for the schema's signed 64-bit id. A malformed record inside a feed it does understand — no price, or no side on a book action — is dropped and counted in a warning, because refusing a whole session over a handful of them would be worse.

Trades are built from the fill records by default, so each names the resting order it hit. A trade the publisher sent no fill for — an auction, a trade against a non-displayed order, an off-exchange print — is then left out, and the loader warns with the volume. DatabentoSettings(trades_from="prints") builds the trades from the whole tape instead, without makers; use it for VWAP, bars, flow toxicity and costs. DatabentoWriter writes an events frame back out as DBN. scripts/databento_window.py sizes a query against the in-memory envelope before downloading it, then runs one window at a time. databento is an optional extra (pip install "ob-analytics[databento]"). See the "Process Databento MBO files" how-to.

Flow-toxicity results say when they rest on too little data (#119). VPIN and Kyle's λ were designed for markets that trade thousands of times a minute, and on a thin tape they used to return a number with no sign that it meant little. KyleLambdaResult now has significant and diagnostics: λ is flagged when there are fewer than 30 regression windows (KYLE_MIN_WINDOWS), when |t| is below 2 (KYLE_MIN_T_STAT), or when the fit is undefined. It also carries a confidence interval, ci_low / ci_high, from a block bootstrap over the windows (1000 resamples by default, under a millisecond on the bundled sample, seeded with seed=0 so the same trades give the same interval; n_boot=0 skips it).

compute_vpin records how it ran in the frame's attrs: bucket_volume, bucket_volume_rule, n_buckets, and diagnostics, which flags a result with fewer complete buckets than n_buckets. bucket_volume is now optional. Left out, it is picked by the new vpin_bucket_volume(trades), which applies the common rule of average daily volume ÷ 50; a session shorter than a day is scaled up to a day at the rate it traded (trading_day="24h" by default). Existing calls are unchanged.

On the bundled sample, λ at 5-minute windows is flagged on both counts (7 windows, t = 1.45) and its interval spans zero; VPIN with the default bucket fills one bucket and says so. Tutorial chapter 6 and the flow-toxicity how-to now show these checks in place of the hand-written caveats.

Bars: the trade stream resampled into OHLCV rows (#148). bars(trades, rule, threshold) cuts a trades frame into bars and returns one row each with open, high, low, close, volume, turnover, VWAP, and the buy/sell split of that volume. Five rules ship with it: time (a fixed span of the clock), tick (a fixed number of trades), volume and dollar (a fixed amount of size, or of price × size), and imbalance (signed size drifting a set amount from where the bar opened). Leave the threshold out and the rule picks one aiming at about 50 bars.

What differs between bar types is only where the boundaries fall, so that is all a rule decides: BarRule states a name, how it reads its threshold, and which bar each trade belongs to. register_bar_rule adds one of your own, usable by name with no edit to the package. Feeds that don't label the aggressor are classified the same way the flow-toxicity metrics do.

Bars draw as a "bars" plot face on both backends — candles over a strip of volume coloured by the net aggressor — and bars_panel() puts them in a gallery. The demos show a clock cut and a volume cut side by side. See the "Build bars from trades" how-to.

Transaction cost and price impact (#110). A new cost module answers what trading cost, rather than what the book advertised. transaction_costs(trades, quotes) returns one row per trade with the effective spread — what the taker paid to cross — split into the realized spread the liquidity provider kept and the price impact the trade caused, in price units and in basis points. The three add up exactly, trade by trade. cost_summary() reduces that to volume-weighted session figures and reports how many trades each one could be measured on.

amihud() and roll_spread() read liquidity from the trade prices alone, so they run on a tape with no quotes and no aggressor side: the price move a unit of turnover buys, and the spread implied by bid-ask bounce. Roll also returns the lag-1 autocorrelation of the price changes, which its model puts at exactly -0.5; how far the number sits from that is how little of the price movement the bounce explains. On the bundled capture it is +0.197 and the estimate has no real root, so it is NaN rather than a number the model does not support. The diagnostic matters in the other direction too: when the autocovariance lands negative by chance Roll returns a spread that is not there, and the autocorrelation is what catches it.

The mid a trade is measured against is the last quote strictly before it, skipping crossed quotes: on a frame built from the same event stream, the quote sharing a trade's instant is the book after that trade took the touch, and a crossed book has no midpoint at all. A trade in the last horizon of the capture has no future mid, so its realized spread is NaN rather than the final quote reused.

transaction_costs takes mid_column to measure against a reference other than the plain mid — "micro_price" for the size-weighted mid, which on the bundled capture reads 1.24 bps against the plain mid's 1.45.

The decomposition draws as a level-less transaction_costs face on both backends — two lines with the impact as the band between them — and transaction_costs_panel() puts it in a gallery. Both demos now include it. See the "Measure transaction costs" how-to.

Polymarket prediction markets, through the ccxt source (#103). ob-analytics capture ccxt --exchange polymarket --pair <token id> streams one outcome's order book and trades over Polymarket's public websocket, with no account or API key. --pair is Polymarket's token id for the outcome, which the Gamma API lists as clobTokenIds. Each outcome is its own book, and its trades are priced in that outcome.

Polymarket makes a market's tick finer as the price nears 0 or 1, so a ccxt capture now makes its recorded tick size finer when a price arrives between two ticks, and counts each change in tick_size_changes in meta.json. The replay then reads every price exactly. See the "Capture Polymarket prediction markets" how-to.

Kalshi prediction markets, through the ccxt source (#102). ob-analytics capture ccxt --exchange kalshi --pair <market ticker> records a Kalshi market's order book and trades from Kalshi's public API, with no account or API key, and ob-analytics process replays it through the L2 path. The ccxt source now looks up CCXT's prediction markets (ccxt.prediction: Kalshi, Polymarket and others) as well as its crypto exchanges; before, --exchange kalshi failed with "Unknown CCXT exchange". binance and hyperliquid are in both lists, so the plain id keeps meaning the crypto exchange and prediction/<id> picks the prediction market.

The captured book is the market's Yes book: a bid to buy No at p is recorded as an offer to sell Yes at 1 - p, and a trade is priced in Yes and signed from the Yes side. A ccxt capture also records its market's tick size in meta.json, which process and audit use. See the "Capture Kalshi prediction markets" how-to.

A capture records which rows came from its opening snapshot (#237). orders.csv and depth.csv gain an origin column: snapshot for the opening book, stream for a live message, shutdown for a synthetic close-out. The capture runner fills it in, so every live source gets it without a change, and the loaders carry it through to events. Before this, the only way to tell a snapshot row from a live one was to compare its exchange_timestamp with snapshot_microtimestamp in meta.json.

meta.json also reports n_snapshot_unconfirmed: how many orders in the opening book no later order event or trade mentioned. The bundled Bitstamp sample has 6,294 of 6,512. Almost all of them sit far from the touch and did not trade, but two stale asks among them held the best ask for most of the session. See "Capture live data".

audit names stale resting orders (#234). A trade above a resting ask, or below a resting bid, shows that the order has gone. An order the venue then does not report again within one second is now reported as a stale_orders warning, and the worst one is named with its id, side, price and how long it held the touch. On the bundled Bitstamp sample this names ask 2002347646152704, which held the ask touch for 27 minutes and causes almost all of the 91.6% crossed time. The crossing note no longer calls a diff feed's crossing normal when the run has stale orders. Nothing is removed: order_book() still replays what the feed said. New public names: detect_stale_orders, StaleOrder, DataQualitySummary.stale_orders, and a tick_size= argument on data_quality_summary.
A metric registry, so a user metric runs and plots with no core edit (#140). A metric is a plain object with a name, a title, the levels it applies to, compute(result) and prepare(frame) — no base class to inherit, the same structural typing sources and writers use. Register it with register_metric(metric), or ship it in your own package under the ob_analytics.metrics entry-point group and load_metric_plugins() finds it at import ob_analytics.

A registered metric is a level-less plot concept under its own name, so a renderer at (name, None, backend) is its face. It then appears in available_concepts(result), renders through result.plot(name), and gets its own gallery card with no extra_panels=. Metrics run when asked for, not during Pipeline.run: result.metric(name) computes one and result.metrics() computes every metric whose levels include the run's resolution — so an L3-only metric is skipped on an L2 run instead of failing on its empty events table, and a metric that raises is logged and its card dropped, so one broken metric cannot stop the gallery being built. New public names: Metric, register_metric, list_metrics, get_metric, load_metric_plugins, PipelineResult.metric / .metrics. See the "A new metric" how-to.

ob-analytics audit, a data-quality gate (#108). The old validate verb is now audit (the old name still works), it scores the run against named checks, and it exits non-zero when one fails — so a script can stop before trusting a feed. Five checks are new: orphan orders (changed or deleted with no created row), non-positive prices, negative volumes or fills, and the two clock-order defects — a venue timestamp later than the receive timestamp, and messages that arrived out of venue order. audit also loads with track_sequence on, so the dropped-message check (#146) reads a venue sequence whenever the feed carries one.

Each check carries a Severity: an error fails the run, a warning fails it only under --strict, and info never does. A crossed resting book is scored by feed type, not by size — an error on a matched book, a faithful replay on a diff feed. --json emits every check plus an ok verdict; --from-parquet audits a saved process output without re-running the pipeline. New public names: Severity, QualityCheck, and DataQualitySummary.ok / .errors / .warnings / .checks. See the "Check data quality with audit" how-to.

PipelineResult.to_arrow() and PipelineResult.to_polars() (#104). Both return the run's four tables — events, trades, depth, depth_summary — keyed by name, with the same keys on every run: on an L2 run events is an empty table, not a missing key. The Arrow tables carry the schema version and tick size in their metadata, the same key-value metadata the Parquet files carry, so a reader handed tables in memory is no worse off than one reading files. Polars is not a dependency and is not installed; to_polars() raises ImportError with an install hint when it is missing, and Polars keeps no schema metadata, so the version and tick size do not survive that conversion.
The frame-type contract is written down in Frame types: pandas in, pandas out: public functions take and return pandas, plug-ins are handed pandas, and the versioned Parquet is how other tools read the output. The reasoning is in adr/0002-dataframe-library.md.
cryptofeed source for live L2 and L3 capture (ob-analytics capture cryptofeed --exchange <venue> --pair <symbol>). The per-order complement to the CCXT source: venues publishing an order-by-order book record orders.csv with the venue's own ids and replay through the full reconstruction pipeline; the rest record depth.csv for the L2 path. The level is discovered from the venue's declared channels rather than a hardcoded list, and --level forces it — except L3 on a venue that publishes none, which raises. Ships as the optional [cryptofeed] extra, imported lazily. See the new "Capture cryptofeed venues" how-to.
sequence is now written to orders.csv. The capture sink dropped the venue sequence on the L3 path, so detect_sequence_gaps had nothing to read; L2 already kept it. cryptofeed captures also report sequence_gaps / sequence_missing in meta.json.
L2 (price-level) depth-native ingestion path. Price-level feeds (Binance, Kalshi, Polymarket, most CCXT sources) publish [price, quantity] levels and diffs with no order IDs; ob-analytics now ingests them as a first-class L2 resolution instead of faking per-order state. A format declares its resolution (Level.L2 / Level.L3, exposed as ob_analytics.Level); an L2 format's loader is a DepthSource that yields the depth frame directly, and Pipeline.run takes the price-level path — depth metrics / spread and trade-sign classification run, while the per-order stages (set_order_types, order_aggressiveness, queue reconstruction) are skipped. PipelineResult gains a resolution field and, on an L2 run, returns an empty (schema-valid) events frame. Ships the depth_csv format (L2DepthLoader, L2TradeReader, DepthCsvWriter, DepthCsvFormat) for the canonical L2 CSV schema, a toy_l2_depth() / toy_l2_trades() synthetic snapshot+delta fixture, and data_quality_summary + the gallery degrade gracefully (L3-only faces skipped, not errored). ob-analytics process|validate --format depth_csv works from the CLI. Unblocks the aggregated venue connectors. Documented in a new "Process L2 feeds" how-to.
Trade-sign classification (ob_analytics.trade_sign) for feeds that don't label the aggressor side. tick_rule (last-price-change sign), lee_ready (quote-midpoint test with a tick-rule fallback), and bulk_volume_classification (BVC — the buy fraction of a volume bar via the standardized-price-change normal CDF). classify_trade_sign(trades, method=..., quotes=...) is the per-trade entry point. compute_vpin and order_flow_imbalance gain sign_method / quotes arguments and now synthesize direction automatically when the trades frame has none — so VPIN and OFI run on L2 / aggregated captures, not just L3. A native direction is still honored unchanged (sign_method=None). On the bundled Bitstamp L3 sample the classifiers agree with the true maker/taker side ~0.83 (tick) / ~0.79 (Lee–Ready) — validated by a test harness.
Feed classification. Every format declares a FeedType (matched_book vs diff_feed) through a feed_type attribute — BitstampFormat → diff_feed, LobsterFormat → matched_book — so downstream code reasons about crossed books by coordinate, not by format name. Exposed as ob_analytics.FeedType.
order_book(..., uncross=True) evicts crossed resting orders for display, mirroring the depth engine's crossed-level eviction. The default stays faithful, so a diff feed's genuinely crossed resting orders are replayed as-is. Threaded through prepare.book_snapshot(..., uncross=True) (also drives depth_chart) and available frame-level as analytics.uncross_book_sides.
Per-run data-quality summary. data_quality_summary() and the new ob-analytics validate <source> CLI verb report the crossed-resting %, unmatched-trades %, duplicate ids, and pre-existing-order count. A new "Data quality: matched book vs diff feed" explanation page and a validate how-to document the distinction.

Changed¶

Every measure reports in the instrument's own units (#400). VPIN, Kyle's λ, order flow imbalance, transaction_costs, cost_summary, amihud, roll_spread, trade_impacts and the l1_ticker and book_signals metrics read tick_size and lot_size from the frame's attrs and return prices in the quote currency and sizes in BTC, shares or contracts, whether they are given a run's integer ticks and lots or a display-unit frame. result.metric(name), result.plot(name) and the bare function now give the same numbers: result.plot and the gallery compute a metric on the run as stored instead of on display_result, which made Amihud differ by 10^10 between the two. A frame with integer prices or sizes and no tick_size or lot_size on its attrs now raises ConfigError. amihud, roll_spread, transaction_costs and book_signals are registered metrics, so result.metric("amihud") and result.plot("transaction_costs") work and each is a gallery card; the demos no longer add their own cost card. compute_kyle_lambda takes sign_method and quotes like compute_vpin and order_flow_imbalance, so it no longer needs a direction column.
Each core helper is written once (#402). Datetimes become int64 nanoseconds through one epoch_ns, and the loaders parse timestamp columns through one to_utc_ns. One validate_columns, in schemas, checks required columns, so validate_events_df, validate_trades_df and validate_depth_df now name the missing columns sorted, followed by the columns available. Each tick helper shares its body with its lot twin. trade_sign.volume_buckets cuts trades into the equal-volume buckets of compute_vpin and bulk_volume_classification. depth.imbalance_levels clamps a depth level count to the bins a summary carries. The new PipelineResult.tables() returns the four core tables, and the demos and process save a result through it. Pipeline.run and Pipeline.run_windows load their inputs through one path, and scripts/bench_scale.py times the pipeline's own stages (pipeline.STAGES). Outputs are unchanged, except that a sign-based VPIN can differ in the last bit of a float, because a bucket's volume is now added up from its parts. A compute_vpin(sign_method="bvc") call with a bad bucket_volume now reports the error under compute_vpin.
Every event loader casts through EVENT_DTYPES, and the validators check dtypes (#379). The Bitstamp, LOBSTER and Databento loaders build their events through the new schemas.cast_events, which keeps text order ids (UUIDs) as object. validate_events_df and validate_trades_df now also fail a column whose dtype differs from the map, and the message names the column and both dtypes, so a float volume or an object direction no longer passes. A custom loader must return the canonical dtypes. LOBSTER's raw_event_type is now object, as the map says. The side, action and order-type category lists are written only in schemas.py.
A loader knows its own feed format (#348). PipelineConfig no longer has price_divisor or timestamp_unit, and refuses an unknown field rather than dropping it. LOBSTER's ten-thousandths of a dollar (lobster.LOBSTER_PRICE_DIVISOR), Databento's fixed-point scale and Bitstamp's millisecond epochs are constants of their loaders and writers. A saved run whose config records either field still loads. A LOBSTER or Databento loader or trade reader takes the grid fields its config does not set from that source's grid, a cent tick and whole shares, as a pipeline run does; a writer built with no config uses that grid. A LobsterWriter refuses a tick that is not a whole number of ten-thousandths of a dollar.
The source factories take the run's loader (#348). OfflineSource.create_trade_source(config, ctx, loader) and compute_depth(events, config, source, ctx, loader) are handed the loader that read the run's events. A plug-in source adds the parameter.
A price-level (L2) run's gallery has the book snapshot and depth chart (#344). They were skipped as per-order faces. Their L2 faces now read the depth table, so build_gallery_model and available_concepts include book_snapshot and depth_chart at L2 for every L2 source.
The L3 book snapshot and depth chart are cut as the L2 face is (#344). The L3 face showed only the orders within 150 bps of each side's best price, then the 40 price levels nearest the touch. It now shows the 40 price levels nearest the touch, as the L2 face does, so the two faces of one concept cover the same levels. On a thin book or a coarse tick the L3 face can now reach further than 150 bps.
Every time window starts from the Unix epoch (#350). order_flow_imbalance and compute_kyle_lambda counted their windows from the start of the first day, while time bars and ofi_by_horizon counted from the epoch, so a window that does not divide a day cut different windows: on the bundled sample, "7s" gave 92 OFI rows against 94 bars. Both are now built on bars(trades, "time", window), and _utils.fixed_window and _utils.window_start are the one place a window is read and cut. amihud, roll_spread, the transaction-cost plot data, the clock bar rule and ofi_by_horizon use the same helpers. Windows that divide a day ("30s", "1min", "5min") give the same values as before. At "7s" on the bundled sample, OFI has 94 rows and Kyle's λ moves from 1.0007e-05 to 1.0746e-05. Also:
order_flow_imbalance now needs a price column.
A window whose trades all have size zero is kept, with ofi NaN; it was dropped.
"1W" is 7 days counted from the epoch, not a calendar week. A calendar step such as "1ME", or a bare number (pandas reads 5 as 5 ns), raises ConfigError naming the function.
Windows are cut in UTC, so a frame in a local time zone, or a daylight-saving change, gives the same windows as in UTC.
amihud's first_price / last_price and the OFI metric's last_price stay int64 ticks; they turned float64 when a window had no trades.
Trade event ids are nullable Int64 (#351). maker_event_id, taker_event_id, maker_og and taker_og were int64, float64 or object depending on the reader, and a column with a gap could be saved as double. They are now Int64 everywhere and are saved as Arrow int64 with a null mask. maker and taker stay object, because an order id can be text (Independent Reserve uses UUIDs). No value changed.
Pipeline.run reads a capture's tick and lot size (#346). Given a capture folder or a file in one, Pipeline.run and Pipeline.run_windows use the tick_size and lot_size in its meta.json when the config does not set them. So Pipeline.from_source("depth_csv").run(capture) and ob-analytics process capture --source depth_csv give the same result; the first used to raise ConfigError on a capture with a 0.001 tick. A tick size or lot size you set still wins, and a pipeline given its own loader or trade source does not use the record.
The recorded_* readers in ob_analytics.depth_l2 are gone (#346). This can break code that imports them. Use ob_analytics.capture_record.read_record(path) and its fields instead: recorded_tick_size(p) is read_record(p).tick_size, recorded_sequence_kind(p) is read_record(p).sequence_kind, and so on for source, feed_type, trade_attribution, clocks and sequence_restarts. A declaration with a value this version does not know is now left out with a warning, where the old readers raised ValueError.
Bar turnover is a float (#346). It was the integer product of ticks and lots on a pipeline result.
The quote helpers in trade_sign changed (#345). This can break code that called them directly:
prevailing_mid is the mid at an instant, the instant included, from the readable quotes only. Its allow_exact and skip_crossed settings are gone. For the mid a trade arrived into, use the new mid_before(rows, quotes); for the row position of that quote, quote_before(rows, quotes).
readable_quotes moved from ob_analytics.features to ob_analytics.depth.
classify_trade_sign and resolve_direction return the ordered buy/sell categorical of the trades schema.
resolve_direction takes fallback="tick" (the default) or fallback=None, which leaves the unlabelled trades empty when there are no quotes.
A plot drawn on your own axes leaves your figure's layout alone (#120). With ax=, a matplotlib face used to call tight_layout() on the figure that holds the axes. That moved panels you had placed, and warned when the figure held axes tight_layout cannot place, such as a gridspec strip or a colorbar. A face now lays out only a figure it creates. If you draw panels with ax=, call fig.tight_layout() yourself once every panel is drawn. On your axes, the cancellations colorbar now shares your gridspec, so your tight_layout() can place it.
A metric is computed when its plot is drawn (#118). Building the gallery model, available_concepts and result.plot of another concept no longer run every registered metric. A metric that raises now keeps its gallery card, which says why it has no plot, instead of being left out.
A plot concept is level-less or drawn at a level, on every backend (#302). Registering a concept at None when it is registered at a level on any backend, or the reverse, raises a ValueError that says which to use. So does a renderer key that is not (concept, level, backend). RENDERERS.placements(concept) lists how a concept is registered. This can break code that worked before:
Code that registered one plot both ways fails at the second registration.
A metric plug-in that does so is skipped when ob_analytics is imported, with a logged warning.
A backend module loaded with register_plot_backend that registers a built-in concept as the other kind fails to load, so no plot on that backend draws until it is fixed.
A metric named like a built-in plot drawn at a level, such as trade_size, cannot register its level-less renderer; give it another name.
The capture directory layout (#150). A capture's files are now in seg-0001/, seg-0002/, ... under --out, next to manifest.json. Read one segment as before, from its orders.csv or depth.csv; give process and audit the capture directory to read them all. --out must be new, empty, or a capture of the same venue and pair.
Live sources no longer reconnect by themselves (#150). The Bitstamp source reconnected without a new snapshot, so an order deleted while it was disconnected stayed on the book to the end of the run, and only a reconnects count (now removed) recorded it. The ccxt source stopped its book or trade loop on a network error and ran on with half its feed. Both now end the stream with the error, and the capture starts a new segment.
manifest.json counts resyncs, not missing sequence numbers (#309). Each segment, and the capture as a whole, records book_resyncs in place of sequence_missing, and audit warns about them in a new book_resyncs check. dropped now counts only the messages a source could not use. recorded_sequence_kind returns None when a capture records no kind, as recorded_feed_type does, in place of taking a default.
The price-level depth follows an order that moves or grows. price_level_volume used to count every one of an order's rows on the price of its first row, and ignored a rise in its size. It now reads a changed row that reports no execution and carries a new price as a move: the order's volume leaves the old level and joins the new one. A changed row that reports no execution and a larger size adds the difference where the order rests. Deletes and rows that report an execution are still taken off where the order rests, whatever price they carry, so the fix for Bitstamp's deletes at the wrong price is kept. The price-level rebuild now agrees with the per-order rebuild (book_state) on these orders. This matters for Databento, whose modify can move an order or make it bigger. The Bitstamp and LOBSTER outputs are unchanged: neither feed moves or grows an order this way.
A trade the venue left unlabelled is now classified on the L3 path too. Pipeline.run labels any trade with no aggressor side against the reconstructed quotes, filling one subset at a time instead of all or nothing, and never overwriting a side the venue did state. This was already what the price-level path did; it now also covers a per-order feed that states the aggressor on most trades but not all, which is what Databento does for auctions, non-displayed orders and off-exchange prints. No change for a feed that labels every trade, which is every other source in the package.
depth.bin_volume_columns() is public. It returns the per-bps depth-bin volume columns a depth summary carries, ordered from the touch outward, and it was already the answer book_imbalance and depth_signals needed. The feature table needs the same answer, and so does anyone writing a depth feature of their own, so it is no longer private. Behaviour is unchanged.
trade_sign.resolve_direction() now makes the guarantee its docstring already claimed: the direction column it returns holds only "buy" and "sell". A native column was previously passed back untouched however it was filled, and every consumer reads it as == "buy" and takes the rest as a sell — so a partly-labelled feed did not lose its unlabelled trades, it counted them on the wrong side. compute_vpin, order_flow_imbalance, bars and the cost metrics were all affected. Rows that are neither side are now inferred the same way a wholly unlabelled feed is, with a warning saying how many. A feed that labels every trade is passed through unchanged. compute_kyle_lambda reaches the same guarantee: it still requires a direction column rather than inferring one, but a column being present no longer means every row in it is trusted.
trade_sign.prevailing_mid() is now public, and takes allow_exact, skip_crossed, mid_column and require_covered — the last quote strictly before an instant, crossed books skipped, a named reference column such as micro_price, and NaN rather than the final quote reused once the quotes stop reaching. All four default to the previous behaviour, so classify_trade_sign is unchanged. An empty quote frame now returns all NaN instead of raising a pandas MergeError.
Sizes are integer lots plus a lot_size, not floats (issue #226). Breaking: the on-disk schema goes 3.0 → 4.0. Every volume and fill column is now a whole number of lots (int64) instead of a double in the base asset. The base-asset size is lots * lot_size, where lot_size is the instrument's minimum size increment (PipelineConfig.lot_size, default 1e-8; LOBSTER sets 1, whole shares). This is the size half of the integer-tick decision (issue #155) and it fixes a real defect rather than only re-expressing the data.

A price level is a running sum of adds, cancels and fills. A float sum does not return to exactly zero when the last order leaves, so a level landed on residue such as 5.55e-17, stayed live, and was reported as the best bid or ask ahead of the real one. On the bundled Bitstamp sample that corrupted the reported best bid on 25,611 of 313,565 rows (8.2%) and the best ask on 30,096 (9.6%) — the spread on about one row in eleven. Integer lots cancel exactly, so a level empties or it does not, and those counts are now zero.

It was found by the new cross-check against hftbacktest (issue #224), and that is what confirms the fix: replaying an exported session through hftbacktest's own L3 reconstruction now agrees with depth_summary on the best bid and ask for every row across five synthetic seeds, and Nautilus' book agrees too. Before the fix the two disagreed on up to 78 rows a seed.

The change reaches every size-valued column — depth_summary's per-bin volumes, placed_vol and filled_vol, the book snapshot's liquidity, and the queue's ahead_volume and remaining — so their sums are exact as well. Three float-era workarounds went with it: the Kahan compensation behind filled_vol, the simulator's _vol_eps exhaustion tolerance, and the LOBSTER book replay's 1e-12 level cutoff. Loaders convert on the way in; the plots and the round-trip and export writers convert back, so what a user sees and what another tool reads are unchanged. lot_size travels in each Parquet file's key-value metadata under ob_analytics_lot_size, next to ob_analytics_tick_size, and load_data surfaces it as df.attrs["lot_size"]. Files written at 1.0–3.0 still read, as the float-size frames they are. Golden outputs were re-baselined on purpose.

The export writers leave out orders that never rested (issue #224). A marketable order is recorded as a transient add on its own side at the touch, then the fill, then a delete; ob_analytics.depth has always excluded these from the book, but the hftbacktest and Nautilus writers were sending them. A backtesting engine reads an add as real liquidity, so its book crossed at the touch and dropped the resting level the order traded against — its reconstruction drifted permanently thinner than ours. Both writers now exclude them, which is what makes the two books agree.
The order-book engine is its own module (issue #136). The rebuild (order_book), the per-order lifecycles, and the FIFO queue reconstruction moved out of analytics.py / queue.py into ob_analytics/engine/, behind one input and one output: order events in, book states and order lifecycles out, and nothing else. The engine imports no pandas — everything crosses its interface as NumPy arrays, with the shared schema (issue #112) as the input, timestamps as int64 UTC nanoseconds (issue #154) and prices as integer ticks (issue #155). Results carry a row index back into the caller's event arrays instead of copying columns out, so adding a column to the schema does not widen the interface and the engine never learns a vocabulary — order types, venue names — belonging to the layer above. ob_analytics/_engine_frames.py is the one place pandas and the engine meet; analytics.order_book, analytics.order_lifecycles, and the ob_analytics.queue functions are now its frame adapters and keep their exact signatures, dtypes, column order, and index behaviour. Output is unchanged byte for byte — the golden-output gates from issue #143 pass on their recorded fingerprints. Two things did move: the display window (max_levels, bps_range) and the queue sampling window are set by the frame adapters rather than the engine, which reconstructs the whole book and replays to the instants it is given. A new import test (tests/test_engine_boundary.py) keeps the engine free of pandas and of every layer above it. This is what lets the inside be replaced with a faster implementation (#138) or fed one event at a time (#139) without touching anything else. Direction, Action, and Outcome are IntEnum code vocabularies that derive their schema strings from their own member names, so a code and its label cannot drift apart. Two details of the frame code are reproduced deliberately rather than rewritten: an order's executed total is accumulated with compensated (Kahan) summation, as the pandas aggregation it replaced did, and placement values are taken per column as the first non-null among an order's created rows. The lifecycle table is now covered by tests/test_golden_synth.py, which it was not before.
BitstampTradeReader no longer requires integer order ids. It keyed its maker/taker lookup on int(order_id), which crashed on a public trade tape carrying no ids (int(NaN)) and on venues publishing UUIDs. Integer ids behave exactly as before; other ids match on their string form, and a missing id resolves to NaN instead of raising.
One Source shape for every data source, file or live (issue #137; settled #145 as "optional extras plus entry-point plug-ins"). File loaders and live capturers were two separate designs with two registries; they are now one Source protocol with two capability refinements — OfflineSource (replay stored files: the loader / trade-source / writer / depth factories) and LiveSource (capture a venue: snapshot / stream / shutdown_synthetic_events). A source states its level (L2/L3) and feed_type, carries typed settings, and registers in the single SOURCES registry via register_source. A source can be both: BitstampSource now covers offline replay and live capture in one descriptor. This is a breaking API change with no back-compat shims:
- Format → OfflineSource; LiveCapturer → LiveSource; BitstampFormat / LobsterFormat / DepthCsvFormat → BitstampSource / LobsterSource / DepthCsvSource; CcxtCapturer → CcxtSource.
- Pipeline(format=...) → Pipeline(source=...); Pipeline.from_format → Pipeline.from_source.
- The FORMATS / CAPTURERS registries and their register_format / register_capturer / list_formats / list_capturers / get_capturer helpers are replaced by SOURCES / register_source / list_sources / get_source (in ob_analytics.sources).
- PipelineResult.resolution → PipelineResult.level (one coordinate name across the codebase; Source.level, matching the visualization layer).
- CaptureConfig.extras (the untyped settings dict) is removed. Per-source settings are now typed SourceSettings on the source itself, e.g. CcxtSource(settings=CcxtSettings(exchange="binance", depth_limit=100)).
- CLI: process / validate take --source (was --format), and the formats verb is now sources (it also shows each source's capability and required context).
Third-party sources load through entry points. A source can ship in its own package and advertise itself under the ob_analytics.sources entry-point group; ob_analytics.sources.load_source_plugins() discovers and registers it at import time, with no edit to ob-analytics. The built-in sources (bitstamp, lobster, depth_csv, ccxt) self-register on import and stay behind today's [live] / [ccxt] extras.
Prices are now integer ticks, not floats (issue #155). Every price column — events, trades, depth, depth_summary, book snapshot, and order lifecycles — is a whole number of ticks (int64); the quote-currency price is ticks * tick_size, where tick_size is the instrument's minimum price increment (PipelineConfig.tick_size, default 0.01). Loaders convert a raw price to ticks on load; the plots and the round-trip writers convert back for display, so figures and CSV output are unchanged. Storing the exact integer removes the float rounding that made small-tick and 0-1 instruments show crossed levels that were not real, and the depth engine now bins and compares levels on exact integers instead of multiplying and rounding each event — LOBSTER's price_divisor is now just the raw-feed encoding scale. tick_size is written to each Parquet file's ob_analytics_tick_size key-value metadata (a JSON map keyed by instrument, ready for per-(venue, symbol) ticks in #147) and surfaced on load_data frames' attrs. Breaking: the dtype of every price column changed from double to int64 and the stored numbers changed (prices re-expressed as ticks; price-valued analytics such as trade_impacts VWAP and Kyle's λ are now in tick units — multiply by tick_size for the quote currency; scale-free metrics such as bps depth and order-book imbalance are unchanged). The canonical Parquet schema version is now 3.0 (a 1.0 / 2.0 file still reads — Parquet is self-describing — as the float-price frame it stored, whose prices are not directly comparable to a 3.0 file's ticks; re-save it to move it onto the tick model). Golden-output baselines were re-recorded behind the correctness gate (#143).
Timestamps are now tz-aware UTC nanoseconds (timestamp[ns, tz=UTC]) on both clocks — timestamp (receive) and exchange_timestamp (matching engine) — across every table, loader, the synthetic generator, and the toy datasets (issue #154). Before, they were tz-naive and in each venue's native clock (millisecond-resolution UTC for Bitstamp, US/Eastern for LOBSTER), and frames from different venues were declared not comparable. Now every frame sits on one UTC clock, so cross-venue frames can be joined or concatenated directly. LOBSTER's seconds-after-midnight are converted to UTC from the session date and a venue time zone (RunContext(session_tz=...), default America/New_York); Bitstamp / CCXT keep their wall-clock instants and only gain the zone and the nanosecond unit, so their values do not move. The schema also documents a same-instant total order — timestamp, then sequence, then event_id, then ingest_seq (ob_analytics.schemas.time_order_keys), which the per-order reconstructions sort by. Breaking: the dtype of every timestamp column changed, so the canonical Parquet schema version is now 2.0 (a 1.0 file still reads — Parquet is self-describing — as the tz-naive frame it stored; re-save it to move it onto the UTC clock). Consumers that compared pipeline timestamps against tz-naive pandas.Timestamps must now use tz-aware (UTC) ones.

Removed¶

The sortedcontainers dependency, three dev tools and three scripts (#415). Nothing imported sortedcontainers, and the dev group no longer installs safety, codecov or pytest-mock, which nothing ran. scripts/bitstamp_demo.py and scripts/lobster_demo.py copied the bitstamp-demo and lobster-demo CLI commands and had fallen behind them; run python -m ob_analytics bitstamp-demo from a clone instead. scripts/parse_bitstamp_log.sh wrote a format that the Bitstamp loader now reads 10^8 times too large. The README and the docs home page share one hero image, docs/assets/hero-depth-heatmap.png, which scripts/make_hero.py writes. The unused volume_marker_areas helper in ob_analytics.visualization._data is gone.
The compiled-engine compare tests (#416). tests/test_compare_backends.py waited for a compiled depth engine behind a factory that no plan names, and always skipped. A faster engine, if one comes (#138), must pass the golden tests, which already fix the output.
The pandas-vs-Polars compare tests (#390). Three tests in tests/test_compare_backends.py waited for a Polars analytics backend and always skipped. ADR 0002 keeps the public API on pandas and does not add Polars, so they could never run. The module docstring now describes only the compiled-engine gate.
Source.config_defaults(), Pipeline.loader, Pipeline.trade_source, Pipeline.writer and LobsterWriter(price_divisor=...) (#348). A source's grid replaces config_defaults(). The pipeline builds a loader and trade source for each run, so it holds none; to read LOBSTER's trading halts, build a LobsterLoader, pass it as Pipeline(loader=...) and read it after the run. For a writer, call source.create_writer(result.config, ctx).
scripts/collect_bitstamp_btcusd.py (#349). It failed on import since #137. Use ob-analytics capture bitstamp; the error messages and documents that named the script now name the command.
The "check" category for hidden trades (#343). It worked around the depth summary dropping crossed levels, which is fixed. prepare.hidden_liquidity_overlay no longer takes events (call it as hidden_liquidity_overlay(icebergs, slices, hidden)), its output loses the category column, and Palette loses check_trade.
analytics.uncross_book_sides (#342). It uncrossed a book a second way and broke a tie at one price differently from order_book(..., uncross=True), so the book snapshot and the book replay could show different books for one instant. prepare.book_snapshot loses its uncross option with it: it draws the book it is given, so pass it order_book(..., uncross=True) for an uncrossed ladder or depth chart.
order_book(min_bid=, max_ask=) (#342). Neither setting was ever applied. Use bps_range or max_levels, or filter the returned frames.

Fixed¶

The Matplotlib book_signals plot no longer runs out of memory (#462). It drew the touch OBI strip as one bar per depth-summary row, about 313,000 bars on the bundled sample, and went past 3 GB. The strip is now two stepped fills, buy above zero and sell below, each reduced to the highest and lowest value in each of 4,000 time columns. On the sample the plot peaks at about 675 MiB, and ob-analytics process ... --gallery finishes under a 3 GB cap.
Data problems reach a Python user (#399). The package turns its loguru logger off, so these went unseen; each is now a UserWarning: Databento records dropped for no price or no side, Databento fills that do not account for all the printed trade volume, orders set_order_types cannot classify, Bitstamp duplicate cancellations, a LOBSTER run with no orderbook file, and a Parquet file with no schema version.
A run with no trades offers no trade metric (#399). available_concepts listed VPIN, Kyle's λ, order flow imbalance and the cost metrics, which cannot be drawn, and result.metrics() raised. result.metrics(), the gallery and plot_result now use one test of whether a metric applies (ob_analytics.metrics.skip_reason), and result.plot("vpin") on such a run says that it needs trades.
The Bitstamp-format loader numbers events in time order (#456). It numbered them by order id, so rows of two orders at one instant came in the order their ids sort. Bitstamp's ids rise with time; Independent Reserve's UUIDs do not, and the order of those rows then changed the rebuilt depth and the icebergs found (the bundled sample with its ids hashed lost one of its two icebergs). Rows of different orders at one instant now follow the venue's sequence when the file has one, then their file order. Each order's own rows are unchanged.
detect_icebergs no longer stops on UUID order ids (#314). On an Independent Reserve capture, a refill made detect_icebergs raise ValueError, and the gallery left out its hidden-liquidity overlay. The slices now keep each order id as the feed writes it.
The Bokeh depth heatmap draws the hidden-liquidity overlay (#410). With backend="bokeh", the iceberg chains and refill markers and the trades against hidden orders were left out without a word. Bokeh now draws them as Matplotlib and Plotly do.
BitstampWriter keeps a capture's origin and sequence columns (#401). It wrote only the order columns, so a capture read, run and written again lost which part of the capture wrote each row, and the clock checks then counted the opening book's rows. DepthCsvWriter keeps sequence too, as it already kept origin. The capture's file names and columns are now in ob_analytics.capture_files, which the capture, the crash repair, the readers and both writers share.
A wheel holds only the package (#415). The wheel and sdist include patterns now start with /, so hatchling no longer packs an ob_analytics folder found below the root, such as one in an agent worktree, and they leave out hidden files and __pycache__. A wheel built from a checkout with agent worktrees under .claude/ held 45 MB of them; .claude/ is now ignored by git too. CI fails when a wheel holds a file outside ob_analytics/ and its metadata, or a hidden file or cache inside the package (scripts/check_wheel_contents.py).
CI lints and type-checks scripts/ (#415), as the pre-commit ty hook now does too, and the scripts pass. build_gallery.py, build_tutorial.py, check_links.py and smoke_wheel.py read their arguments with argparse, so --help prints usage instead of running the script; build_gallery.py takes --output. Ruff now checks for broad except Exception, except: pass and exec (BLE001, S110, S102), so each noqa comment for those rules marks a reviewed case and a new one fails the lint. A registry's in check now catches only the TypeError of an unhashable key.
Every reader refuses a bad file with ConfigError (#418). The Bitstamp, LOBSTER, depth_csv, DBN and Parquet readers and read_manifest returned a frame built from some bad files, or raised an error from pandas, pyarrow, databento or json. Each now raises ConfigError when a file is not text, is not laid out as its format says, ends part way through a row or record, or has a required cell that is empty or holds something other than a number, a time, a side or an action where one belongs. Some of these the readers used to take: a LOBSTER row with an unknown event type was left out, a Bitstamp order with no time read as NaT, a trade side that was not buy or sell read as unknown, and a DBN file cut inside a record gave the records before the cut. An empty side, venue time or Bitstamp trade order id is still allowed, and an empty trades.csv reads as no trades.
A Bitstamp order message that cannot be parsed ends the segment. The Bitstamp capture left such a message out and carried on with a book that no longer matched the venue's, as cryptofeed did before #398. A snapshot row or live order event it cannot parse, or one with an unknown order_type, now raises UnmappedMessageError, so the segment ends and the next one starts from a fresh snapshot. A trade it cannot parse is still left out and counted in dropped. UnmappedMessageError moved to ob_analytics.live, so a plug-in L3 source can raise it too.
A segment that closes by itself after a second signal has no error. When a second Ctrl-C or SIGTERM arrived just as a segment was finishing on its own, the manifest still said a second signal had stopped it, and capture_errors failed. A segment that is late to stop after the time limit is still recorded as such.
Reading a .dbn.zst file closes it on Python 3.14 (#416). From 3.14 databento decompresses with the standard library, whose reader leaves the file it wraps open, so read_mbo_frame leaked one file handle per compressed file. It now closes the file itself.
The gallery page is written as UTF-8 on every platform (#416). The page declares UTF-8, but it was written in the locale's encoding, so on Windows a non-ASCII title could show wrongly or fail to write.
Live capture reports what it lost, and stops on a second signal (#398).
ccxt and cryptofeed counted lost messages as errors, but the manifest and audit read dropped, so dropped_messages passed a capture that lost frames. cryptofeed now counts them in dropped, as Bitstamp does. ccxt's errors only counted the error that ends the stream, which is already capture_error, so it is gone, and the runner no longer adds a stream error to errors.
audit has a new capture_errors warning for a segment that ended on an error or was slow to stop, and runs dropped_messages and capture_errors on a single capture from its meta.json too.
A source whose diagnostics() raises lost its counters with only a debug message. The capture now warns and records diagnostics_error in meta.json and the manifest, and dropped_messages fails, since the count is not known. audit also warns, not only logs at debug level, when it cannot read the sequence kind a capture's source declares.
A cryptofeed source with no venue said L2 but declared an L3 feed type and trade attribution. All three now say L2 until the venue is known.
An L3 cryptofeed message that cannot be mapped ends the segment, as in ccxt, so the next segment starts from a new opening book. Before, the message was left out and the capture went on with a wrong book.
A second SIGINT or SIGTERM stops a capture at once: a segment still closing is cancelled and closed from its files. Every file is closed even when one fails to flush (recorded as capture_error with the phase close), and meta.json is written last. The row counts in meta.json also survive a cancelled segment.
A Databento file with records after an unrepaired gap warns, and audit says so. Databento marks the records after a gap it could not repair with F_MAYBE_BAD_BOOK. The loader counted them in a loguru message, which the package switches off, so a user never saw it and audit passed the file. The loader now calls warnings.warn, keeps the count as DatabentoLoader.maybe_bad_book_records and on the events' attrs (under schemas.MAYBE_BAD_BOOK_ATTR), and data_quality_summary reports it as maybe_bad_book_records, with a records after a gap line and a maybe_bad_book warning when it is above zero. A source whose feed has no such flag shows the check as not checked (#389).
"What each feed shows" says why Kraken book drift is not noticed (#391). Kraken sends a checksum of its book, but the checks in ccxt and cryptofeed are both off by default, and the package turns on neither. Kraken sends no sequence number, so the only sign of drift is a crossed book, which the crossed_book check reports.
A source keeps nothing from a run, and run(ctx=...) reaches the loader (#348). LobsterSource and DatabentoSource kept the last loader they built, and LOBSTER read its depth from that loader's orderbook file. With one LobsterSource() serving two pipelines, a run near $100 came out with a best bid of 150.0, from the other run's file. The loader and trade source were built when the pipeline was, so a trading_date, symbol or venue given to run(ctx=...) was dropped, and the LOBSTER how-to's Pipeline(source=LobsterSource()).run(path, ctx=ctx) raised trading_date is required. Both are now built for each run, from that run's context. A LobsterLoader built by hand read prices 10,000 times too large (100000000 ticks for $100.00); it now reads the prices and sizes a pipeline run reads.
The L2 book snapshot and depth chart read the depth table (#344). Both faces of the gallery's book_snapshot and depth_chart were built from order_book(events), the book rebuilt from the per-order events. The L2 face is now built from the depth table with price_level_snapshots, as the L2 book replay already was, and only the L3 face reads the per-order book. On a LOBSTER file with few levels the per-order book keeps orders the file lost sight of and is crossed: on the 10-level AAPL sample of 21 June 2012 the L2 snapshot showed a best bid of 587.85 over a best ask of 582.39, and now shows 582.28 and 582.39, as the orderbook file does. On a diff feed such as Bitstamp, the L2 face now leaves out a stale crossed level, as the depth summary does.
A LOBSTER run warns when the book rebuilt from its message file is crossed (#344). A message file holds only the events inside the requested price levels, so an order cancelled or filled outside them keeps resting in that book. When it is crossed for more than 0.05% of the session time, LobsterSource warns with warnings.warn and gives the share and the cause. The 10-level AAPL sample is crossed 99.2% of the time and warns; the 50-level sample is never crossed and does not. The depth table, read from the orderbook file, is not affected, so the crossed_book check of audit does not count this crossing; audit reports it separately as per_order_crossed_pct. Without an orderbook file the depth table is rebuilt from the same messages, and the warning says so. The LOBSTER how-to and "What each feed shows" now state the limit: the per-order views are complete only when the requested levels cover every price an order reaches.
The matplotlib OFI horizon face no longer crashes (#349). Its time axis came only from the filled bands, so a grid of zeros drew none and saving failed with RuntimeError: ... raster overflow; one non-zero value widened the axis to about four years. The axis now spans the grid's own times (one second each side when the grid has one step), and an all-missing grid shows the "no data" title.
process and audit build their pipeline in one place (#349). The two copies had drifted: process printed a literal %s instead of the source name when --trading-date was missing, and capture printed %r instead of the source name for an offline-only source. Both messages now name the source.
A quote that cannot be read as a price gives no mid anywhere (#355). depth.readable_quotes decides when a quote can be read: both sides are above zero and the bid is not above the ask. Some readers still used their own test or none. micro_price, and so the micro_price column of depth_signals, gave 0 for a quote with an empty side, and the mid_price column gave a mid halfway to zero. The price view and the book signals face drew those mids, and the trade tape hung its stems off them. Now each of these is NaN (no line is drawn) for a quote with an empty side or a crossed book, and the trade tape takes its mid from the nearest quote that can be read. A locked book (bid equal to ask) is still read. The imbalance columns are unchanged, since all the volume on one side is a true imbalance. The Level 1 quote table reads an empty side with the same test, and blanks only the empty side. On the bundled Bitstamp sample 16 of the 313,561 depth summary rows cannot be read; the transaction-cost numbers are unchanged.
The Bitstamp-format loader puts an order's rows in time order (#354). It put the created row first and the rest by size, largest first, which assumed an order's size only falls. Some captures in this format raise an order's size, so the rows came out of order and the fills were wrong: an order of 5 lots that fell to 3 and then rose to 8 showed its 8 before its 3. The loader now keeps the created row first and puts the rest in order by timestamp. Rows that share an instant follow the file's sequence column when it has one; a deleted row then goes last, and the rest keep their order in the file. A row with no sequence number, such as a fill a cryptofeed capture took from the trade tape, keeps its place after the row before it in the file. fill is still taken only from drops in size, so a rise has fill 0. The bundled Bitstamp sample gives the same events, trades, depth and depth_summary as before.
The depth summary keeps crossed price levels (#343). When a new quote crossed a level on the other side, PriceLevelBook deleted that level for good, so the depth summary lost it for the rest of the run while the depth table kept it. It now keeps every level, as the depth table does, and leaves crossed levels out only of the touch it reports, by the rule of order_book(..., uncross=True): the level updated less recently is left out until the newer quote leaves. A locked touch is uncrossed the same way. PriceLevelBook.quoted_bids() and quoted_asks() give the levels the touch is read from; the depth summary, price_level_snapshots and LobsterWriter read them. On the bundled Bitstamp sample the summary's touch now matches order_book(uncross=True) at 299 of 300 sampled instants on each side (241 and 165 before), no row is locked (53,498 before), hidden_trades flags 0 trades where it flagged 29 that all had a visible maker, and 48 trades have a negative effective spread (73 before). A matched book (LOBSTER, Databento) is never crossed and does not change. depth_metrics is also faster: 9.35 s against 11.44 s on the sample.
The empty events and trades tables have a loader's dtypes (#351). The empty events table of an L2 run had volume and fill as float64 and direction, action and type as object, so save_data wrote double size columns for every L2 run, and pd.concat of an L3 and an L2 run's events turned every L3 size into a float. The empty trades table, which a trade reader returns for a capture with no trades, had every column as object. Both now match the maps.
The Databento loader writes one row per kind of change (#353). A cancel or modify that follows a fill and leaves the order resting can also take off more than the fill, move the order, or make it bigger. The loader wrote that as one changed row that was both an execution and a reduction, which the schema does not allow, so the depth took off only the fill and kept the rest on the level. On an order of 8, F 3 then C 5 left 2 lots on the level after the order had gone. The loader now writes two rows for such a record: one with the fill and the size left after it, at the price where the order rests, then one with no fill and the record's own price and size. Both share the record's original_number, and the trade names the first as its maker. The warning that a modify with a fill left the depth off is gone, since the depth now follows it. A cancel that takes off less than the fill charged to it, or a fill larger than the order had resting, is warned about; the book keeps the size the cancel or modify leaves, and an oversized fill is capped on the execution row.
read_mbo_frame closes the DBN file it opens (#353). A store the caller opened is left open. The run_windows docstring no longer mentions the removed warning.
Every rebuild puts an order at one price (#342). An order rests at the price of its created row, and after that only a changed row with no fill moves it. The depth table followed this rule, but the per-order book (order_book) and the queue tables put an order at the price of its latest row. Bitstamp reports a taker's fill at the price it traded at, so an order that crossed and then rested showed at one price on the L2 faces and at another on the L3 faces. The rule is now worked out in one place, _engine_frames.resting_price, and order_book, queue_positions, queue_age_grid, price_level_volume, the windowed run's carry and DatabentoWriter all read it. An id that is cancelled and placed again now rests at its new price in every rebuild; the depth table kept it at its first price. A changed row with no fill and a new price (a Databento modify) now moves the order to the back of its new queue in queue_positions and queue_age_grid, as it already moved it in the depth table. queue_positions reports the move as a deleted row at the old price and a created row at the new one. order_book, queue_positions and queue_age_grid now need the fill column, and the two queue functions also need type: classify the events with set_order_types first.
The touch-queue faces leave market orders out (#342). queue_age_grid and queue_positions kept market orders, which never rest. A Bitstamp taker is created at its limit and fills at other prices, so the grid added it and never removed it. On the bundled sample one such order was the liquidity_at_touch L3 bid queue from 02:36 to the end: the grid's touch queue differed from the book at 184 of 200 bid samples and 173 of 200 ask samples. It now matches the book at every sample. On the synthetic golden session, queue_positions loses the 251 rows of its 92 market orders and no other row changes.
A saved run reloads with its own units (#346). ob-analytics gallery and audit --from-parquet rebuilt a saved run with the default lot size and price decimals, so a LOBSTER or Databento run (lot size 1) showed every size 10^8 times too small, and a 0.001 tick was rounded to the cent. They now use load_result.
The L2 loaders warn about a size between two lots (#346). Such a size is rounded to the nearest lot, which is not the venue's size; the warning names lot_size.
A large price or size on its grid is no longer read as off it (#346). The grid test allowed a fixed 1e-6 of a step, but dividing a large value by a small step misses a whole number by more: 1251.81488791 at a lot of 1e-8 is 125181488790.99998 lots. The allowance now grows with the number of steps.
A size too large for int64 lots is refused (#346). Converting a size to lots wrapped a level above about 9.2e10 units (at the default lot of 1e-8) to a negative number, and the L2 loader then dropped the level. It now raises ConfigError that names lot_size.
Bar turnover and VWAP, and trade_impacts' VWAP, no longer wrap (#346). Price in ticks times size in lots passed int64 on a busy day: four trades of 4,000 BTC at 78,000.00 gave a negative turnover and VWAP. They are computed as floats, as amihud already was.
Every Bitstamp trade finds its taker (#352). An instant order bought by quote amount is created with size 0 and then shows 0 or one lot, never what it bought. The trade reader matched a trade to a row only by its fill size, so these trades had no taker and their orders were classed unknown. When no row shows the trade's size and the taker is an order created with size 0, the reader now takes that order's deleted row, else its latest row. Every other trade is still matched on size only: an order with no row of the trade's size has lost a message, and the trade stays unmatched so unmatched_trades reports it. On the bundled sample all 284 takers resolve (282 before), and orders 2002354918830080 and 2002351232385027 are classed market. As market orders they no longer rest in the price-level rebuild, so depth and depth_summary lose their 4 rows (313,565 to 313,561), including the one row whose best bid was a single lot from one of them.
A trade is read against the quote it arrived into, everywhere (#345). Lee–Ready read the quote stamped at the trade's own instant. On a feed whose trades and book events share one clock (LOBSTER, Databento) that quote is the book after the trade took the touch, which pulls the mid toward the trade price and flips signs. With quotes 99/101, then 99/104 at the trade's instant, an unlabelled buy at 101 was labelled sell, and transaction_costs then gave it an effective spread of -2.0. It is now a buy at +2.0. Lee–Ready, transaction_costs, hidden_trades, order_aggressiveness and the windowed run all use one rule, trade_sign.quote_before: the last quote strictly before the row, in event order. On a LOBSTER AAPL hour (30 levels), Lee–Ready now agrees with the feed's own side on 95.9% of trades, against 85.2%; on the bundled Bitstamp sample it is 72.9% against 73.2%. The pipeline's own tables do not change on the bundled Bitstamp sample or the synthetic session, where every trade is labelled by the venue.
hidden_trades reads the spread just before the maker's fill in event order (#345). It read the spread before the fill's whole instant, so a visible order placed or filled earlier in that instant was missed. On the LOBSTER AAPL day (10 levels) it now flags 11,304 of the 11,332 executions against hidden orders, against 9,645, still with none that is not one. On the bundled Bitstamp sample it flags 29 trades, against 41: the 12 it drops each had a visible maker placed at the trade price earlier in the same instant.
One test of a quote that can be read (#345). prevailing_mid and the effective spread kept a quote with an empty side, so a buy at the ask just after the bid side emptied cost 20,000 bps; it is now measured against the last quote with both sides. depth.readable_quotes (moved from features) is the one test: both sides above zero and the bid not above the ask. prevailing_mid, mid_before, hidden_trades, the feature table and the depth heatmap's spread line all use it.
The pipeline fills a missing trade direction by the same rule as the metrics (#345). The pipeline and resolve_direction disagreed on which trades needed a direction (missing, against neither buy nor sell) and on the dtype (ordered against unordered). The pipeline now calls resolve_direction(..., fallback=None), and both write the trades schema's ordered buy/sell categorical. A run with no quotes still leaves the unlabelled trades empty, now with a UserWarning rather than a log line.
LOBSTER depth is stamped with the right event after a cross trade or a halt (#340). The orderbook file has a row for every message, type 6 (cross trade) and type 7 (halt) included, but the events keep only types 1 to 5. lobster_depth_from_orderbook paired the two by position, so after the first type 6 or 7 message every book state carried a later event's time and event_id, and the last states were lost. Each event now reads the orderbook row of its own message, named by original_number. A LobsterSource run raises ConfigError on an orderbook file with more or fewer rows than the message file, where before it logged a warning a Python user never saw. lobster_depth_from_orderbook called on its own checks the count only when given message_rows.
LOBSTER depth sizes are int64 lots (#340). Sizes read from the orderbook file were float64 share counts, so depth.volume and every size column of depth_summary were floats, against the schema, and with a lot_size other than 1 they were in different units from the events. They are now converted with lot_size, as the message sizes are.
LobsterWriter writes the run's own book (#340). The writer rebuilt the orderbook file from the events with rules of its own that only fitted frames read from LOBSTER. On a frame from any other source it raised TypeError (the missing raw_event_type), and without that column it wrote a book that differed from the run's depth. It now writes the book after each event from the depth table it is given, or from price_level_volume when there is none, through PriceLevelBook, the book the depth summary is built on. Reading the files back gives the run's book after every event: the depth summary read back equals the run's, as far as the written levels reach. Both files are written in time order, and an empty level's dummy price is no longer multiplied by the price divisor. The writer refuses, with ConfigError, a run with no events, repeated event_ids, events or depth in display units, and a price_divisor too coarse for the tick (with a cent tick, a divisor of 1 wrote 100.01 and 100.02 as the same price). Events with no depth table now need the type column that set_order_types adds, so events straight from LobsterLoader.load are refused: pass the run's depth, or run set_order_types first. Prices are encoded as exact integers, and a writer built with no config uses LOBSTER's own grid (price_divisor=10_000, whole shares) instead of the general defaults.
One rule for a deleted row, and the loaders and writer that broke it (#341). The schema now says what a deleted row holds when its order also traded: fill is the executed part, volume the part removed without trading, and the two add up to what the order had resting. An order filled in full ends with volume 0. See "Volume and fill" in the schema page.
The Databento loader put the size in both columns, so an order filled in full came off its price level twice and other orders at that price lost their size from depth and depth_summary. Bids of 5 and 3 at 100.00, with the 5 filled, now leave 3 at 100.00, not 0. A cancel that asks to remove more than its order holds now removes only what the order holds.
DatabentoWriter is now the inverse of that rule. A deleted row is a C for everything the order had resting. It writes an F only for a fill the order took while resting. An order that took liquidity when it arrived is written from when it rests, as an A for the size it had left, or not at all if nothing was left. Pass "trades" with the events so the writer knows every fill an aggressor took; without them it knows only the market orders' fills. A synthetic session written to DBN and read back now has the same trades and the same final book; before, every trade came back twice and filled orders stayed resting. A size that is not a whole number of units now raises a ValueError instead of being rounded.
The Bitstamp loader now puts each order's created row first. It sorted an order's rows by size, so an instant order bought by quote amount, created with size 0, had its created row last and a phantom 1-lot fill. A rise in size is no longer read as a fill.
price_level_volume keeps int64 sizes after a partial cancel. One LOBSTER type 2 or Databento partial C made the whole depth.volume column, and the depth_summary sizes, float64.
price_level_volume no longer hides a level the events overdraw. It clipped only the size it reported, so the level stayed short for the rest of the run: overdrawn by 5 and then given an order of 4, it read 0. It now holds the level's running total at zero, so the same level reads 4, and it warns with the level and the first event. audit counts those rows as negative_level (a warning). The bundled sample, the toy session and a synthetic session have none.
The book replay keeps its click script in a notebook (#121). IPython displayed a BookReplayFigure through Plotly's own renderer, which drops the script that makes trades clickable, and the tutorial build dropped the figure entirely. The figure now displays as HTML in a notebook, as documented.
Stale orders and crossing are measured correctly on display tables. display_result converts sizes to base-asset floats. price_level_volume summed those floats, and a level whose last order left could keep a remainder such as 3e-17 in place of 0. The level then stayed in the faithful book. On the bundled sample, 22,915 depth rows kept a remainder. detect_stale_orders reported a touch_seconds of 0.0 for the stale ask at 78,333, which holds the touch for 1,645.7 s, and data_quality_summary reported the book crossed 92.03% of the time against 91.61%. price_level_volume now sums float sizes that sit on a decimal grid as whole multiples of that grid, the way it sums integer lots, so the display tables give the same results as the pipeline's own tables. Float sizes on no decimal grid, such as a size worked out by float subtraction, are still summed as floats, and price_level_volume now warns when it does that.
The price-level depth is replayed in event order. price_level_volume and depth_metrics sorted rows on the timestamp alone, so the rows of one instant were replayed bids first, then asks, by price, and a depth_summary row was not always the book after the event it names. On the bundled Bitstamp sample, for 44,043 of 156,719 new limit orders the last row before the order was replayed after it, so "the book before the order" already held the order. Both now use the order the per-order rebuild uses (schemas.time_order_keys: the timestamp, then sequence, event_id and ingest_seq where present). Each level's running total is also taken in that order, so every depth row holds the level's volume after its own event, and depth carries the sequence and ingest_seq columns over from the events when they have them. The book replay (price_level_snapshots) uses the same order, so its touch still equals the depth summary's. On the sample, the book at the end of an instant changes at 2,634 of 130,244 instants (the best bid or ask at 2,453), because removing crossed levels depends on the order rows arrive in, and aggressiveness_bps changes on 28,898 of 314,057 events. On a synthetic session, which is never crossed, every event's last depth_summary row now matches the per-order rebuild.
aggressiveness_bps is measured against the book at the order's own time. order_aggressiveness found the book standing before each order by taking the depth_summary row with the next-lower event_id. That is only correct when event ids are in time order. The Bitstamp loader numbers events after sorting them by order id, so on Bitstamp that row could come from any time in the session. On the bundled sample, 51,805 of 156,718 new orders were measured against a book from after the order, and 7,399 against a book more than a minute away, up to 30 minutes. The lookup now uses the documented event order (time_order_keys): the last depth_summary row with an earlier timestamp, or with the same timestamp and a lower event_id. aggressiveness_bps changes on 45,087 of the sample's 314,057 event rows (14.4%). Databento, LOBSTER and the synthetic generator number events in time order, so their values do not change.
compute_vpin refuses a bucket size in the wrong units. Sizes are integer lots, so bucket_volume=5.0 on the Bitstamp sample means 5e-8 BTC and asked for about 300 million buckets, enough to run the machine out of memory. compute_vpin and bulk_volume_classification now raise ValueError when a bucket would make more than MAX_VOLUME_BUCKETS (one million) buckets, and say which units bucket_volume is in. The how-to pages that passed bucket_volume=5.0 now size the bucket from the trades.
Docs written before lots and UTC timestamps are corrected. The schema page gives the current version, 4.0. The step-by-step and synthetic-data pages say timestamps are tz-aware UTC. Tutorial chapter 3's LOBSTER round trip now passes lot_size=1.0; without it every size was written as a fraction of a lot, and the round trip read back no executed volume.
The tutorial pages show no warnings (#120). Chapter 3 now reads its LOBSTER files back with the toy's tick size. With LOBSTER's one-cent default, $99 came back as 9900 ticks against the original 99, so the books before and after the round trip did not match. An empty book snapshot no longer draws an empty legend. The docs build now fails when a tutorial cell prints a warning or a figure fails to draw, and when a gallery recipe raises a warning.
An Independent Reserve capture no longer keeps orders cancelled before it started (#315). cryptofeed takes the opening book from the venue's REST interface when the first stream message arrives. The venue serves that book from a cache, and its orders are older than its CreatedTimestampUtc, so the book could be a few seconds older than the stream. An order cancelled in between stayed in the book until the capture ended, and a trade at a price past it made the book crossed: 13 orders in a 12-minute capture, two of them at the best ask. The cryptofeed source now uses its own Independent Reserve feed, which fetches the book again, once a second, until it was created at least 3 s after the first message. The stream waits meanwhile and loses nothing. After 15 tries it uses the last book and logs a warning.
Coinbase captures through ccxt record the taker's side (#308). Coinbase reports each trade with the side of the maker, and ccxt passes it on unchanged, so every Coinbase capture had its trade signs reversed, and so did order flow imbalance, VPIN and everything else built on them. The ccxt source now reverses the side for the coinbase venue, on both its websocket and REST trades, and records trade_side_reversed in meta.json. A Coinbase capture without that key was made before this change: reverse its signs. coinbaseexchange needs no change, because ccxt already reverses its side.
An Independent Reserve order no longer stays in the book after its cancel (#313). Before 3.0, cryptofeed forgets an Independent Reserve order after its first change, even when the order still has size, and skips every later message for it. So the cancel after a partial fill never reached the capture, the order stayed in the book until the capture ended, and audit reported it as a stale resting order. The cryptofeed source now uses a subclass of cryptofeed's Independent Reserve feed that keeps the order while it rests. cryptofeed 3.0 fixes this too, but needs Python 3.13.
audit no longer stops on UUID order ids (#311). A stale resting order from an Independent Reserve capture made audit raise ValueError. StaleOrder.id now holds the id as the feed writes it (int | str), and the text and JSON reports print it.
cryptofeed's Independent Reserve trades are linked to their orders (#311). The trades now carry both order ids (BidGuid and OfferGuid) in buy_order_id and sell_order_id, and each fill is recorded from the trade, as on Bitstamp. On a per-change book, a fill that the trade and the book both report is recorded once, and a book message that repeats an order's size writes no row.
An Independent Reserve capture's trades.csv no longer holds other markets' trades (#316). The venue sends a market's trade channel the trades of its other markets for the same coin, so a BTC-AUD capture also got BTC-NZD and BTC-SGD trades, priced in NZD and SGD. audit read them as trades through the book and reported resting orders as stale. The cryptofeed source now writes a trade to trades.csv only when it is for the capture's pair, and counts the others as other_market_trades in meta.json. The venue keeps one book for all its markets, so a trade left out still reports the fill of any order it names in the capture, at the order's price.
audit no longer reports lost messages on cryptofeed captures that lost none (#309). On Bitfinex and Blockchain.com cryptofeed's sequence number counts every message on the connection, trades and heartbeats too, and on Independent Reserve cryptofeed passes on no change to an order it does not hold. So the book rows skip numbers, and audit reported each skip as a sequence_gaps error. The cryptofeed source now declares sequence_kind = SequenceKind.MONOTONIC and records it in meta.json, so audit checks only that the number never goes back. meta.json reports sequence_out_of_order in place of sequence_gaps and sequence_missing, so a long capture's manifest no longer counts the skips as dropped messages. A capture whose meta.json records no sequence_kind, such as one made before this change, is checked the way the source that made it declares now, not the way --source does.
A quiet book no longer gives a blank depth heatmap (#303). The heatmap leaves out each price level that does not change in the window. When no level changed, as in a short capture of a quiet prediction market, that left out every level and drew blank axes, with only Not enough data for any price level in the log. Now the heatmap draws every level in that case, with a note under the title that no level changed. When the heatmap really is empty, the note says why: no depth data, no level between two named prices, no level in the chosen volume range, or no resting orders in the time window. All three backends put the note on its own line under the title and write it to the log. An empty heatmap still draws the midprice and trades. A spread or trades frame with no rows in the window no longer sets a NaN price range that left out every level. The gallery's depth-heatmap card names plot_result(result, "depth_heatmap", show_all_depth=True), and the Kalshi and Polymarket how-tos describe the case. The matplotlib heatmap also no longer goes blank when one level has a single row in the window.
The gallery prepares each card's data once (#303). The backend columns of a card now share one prepared payload, so the prepare step runs, and logs, once per card instead of once per backend.
The clock checks compare two real clocks, and run on L2 (#310). Where a venue sent no time, the cryptofeed and ccxt sources wrote the time they handled the message into exchange_timestamp, which is later than the receive time. On cryptofeed's Bitfinex and Blockchain.com books, and on the Independent Reserve opening book, audit then failed the capture on exchange_time_after_receive. Such rows now copy the receive time, as LOBSTER's one clock fills both columns. The new Clocks declaration (BOTH, RECEIVE_ONLY or VENUE_ONLY) says which clocks a source's rows carry. LOBSTER declares VENUE_ONLY. A live capture records clocks and books_without_venue_time in meta.json; read it with recorded_clocks. With one clock, audit does not run the two clock checks and says why. The L2 depth frame now keeps exchange_timestamp when depth.csv has it, so the clock checks run on L2 captures, which before were never checked. The clock checks leave out the opening book's rows (origin snapshot), whose clocks the capture may not have measured; the L2 depth frame now keeps origin for this.
A cryptofeed REST opening book replays before the message that changed it (#310). On Binance and Independent Reserve, cryptofeed fetches the opening book while it handles the first live message, and hands the book over first with a later receipt time. On Binance (L2), replay applied the message and then the book's older size. On Independent Reserve (L3), with sequences tracked, the numbered message sorted before the book. The capture now holds a book with no delta until the next one arrives, and places it 1 ms before a message received earlier. The message keeps its own receipt time, and the book's rows are marked as the opening book.
A Bitstamp capture no longer keeps trades from before its snapshot (#301). It already skipped order messages from before the REST snapshot, but kept the trades from the same time. The orders those trades filled are not in the capture, so audit counted the trades as unmatched: up to 52% of a segment's trades with 30-second segments. The trades are now skipped and counted as pre_snapshot_trades_skipped in meta.json. At a roll, the previous segment already has these trades.
raw.jsonl no longer stops a cryptofeed capture of independent_reserve or blockchain. cryptofeed reads these venues' date and time strings as datetime, date and time values, which raw.jsonl could not write, so every segment failed unless --no-raw was passed. They are now written as ISO 8601 strings. Any other value or dict key that JSON cannot hold is written as text (str(value)), and a frame JSON cannot hold at all, such as one that refers to itself, is skipped. Neither stops the capture. Each segment's meta.json names the types written as text (raw_text_types) and counts the skipped frames (n_raw_frames_skipped); the manifest adds up raw_frames_skipped, and the capture logs each kind of warning once.
A custom plot keeps working after it is added to the gallery (#302). The "A new plot" guide registered a plot at Level.L2, then registered it again at None to put it in the gallery. After the second step, plot("cumvol") raised and told you to pass Level.L3, which was not registered. The guide now registers its example once, at None, draws the gallery on the one backend it registered, and shows how a plot drawn at a level goes into the gallery as a PlotConcept. A test runs the guide's plot section as written. See Changed for the rule that replaces the second registration.
A gallery card added the wrong way says how to fix it (#312). A level-less plot added to GalleryModel.concepts, a plot drawn at a level added to GalleryModel.analytics, or a PlotConcept variant at a level no renderer is registered at, showed a bare "Not available", the same as a renderer that raised. The gallery now checks each card against the levels its renderers are registered at. When they disagree, the card says how the plot is registered and how to change the model, and the log has the same text once. A backend that cannot be loaded says so on its cards, and is tried once instead of once for each card. A backend with no renderer for a plot that other backends draw still shows a bare "Not available".
On Python 3.11, a capture stops each segment when asked (#296). The live sources waited for messages with asyncio.wait_for, which on Python 3.11 can drop a cancel that arrives with a message. A roll then left the old segment streaming next to the new one to the end of the capture, with no further rolls, and SIGTERM or Ctrl-C did nothing. The sources now use asyncio.timeout, and ruff bans asyncio.wait_for in this repository. The runner cancels a stream again while it keeps yielding items, so a plug-in source with the same pattern cannot block a stop; a stream that is closing its connection is left to finish. Python 3.12 and later were not affected.
A segment that does not stop cannot hold up the capture (#296). A segment asked to stop has 20 seconds to close. After that it is cancelled and closed from its files, like a segment a crash left open. The manifest keeps why it was stopped and records the delay as its error, not as a gap. If closing its files fails, the error says so and the capture carries on. A segment covers the market only until it is asked to stop, and one asked to stop before its first live event covers nothing, so it no longer adds a second gap next to the real one. At the end of a capture all running segments are stopped together, so SIGTERM ends a capture in under a minute even when a source hangs.
audit no longer fails a complete Databento file (#298). Databento numbers every message on the venue's channel, so the numbers in one instrument's events skip the other instruments' messages and the trade and fill records, which become trades. audit read every skip as a lost message and failed a sound file with a sequence_gaps error. DatabentoSource now declares sequence_kind = SequenceKind.MONOTONIC, so audit checks only that the numbers never go back. audit reads a source's declared sequence_kind when no capture meta.json records one; read it with the new sequence_kind_of. recorded_sequence_kind returns None when the capture records none, as recorded_feed_type does.
A Bitstamp L3 capture through cryptofeed passes audit (#284). cryptofeed's Bitstamp L3 channel is detail_order_book: a picture of the top 100 bids and top 100 asks about 10 times a second, not every order. Four defects followed from that:
An order that dropped past the 100th place was recorded as deleted, and as created again when it came back. It now stays tracked at its last size until the book shows its price again. This removes the duplicate created ids, and the clock errors the Bitstamp reader made from them.
Book rows carried the venue's time in both clock columns. timestamp is now the time the capture received the message, like the trades.
Trades dropped the maker and taker order ids that Bitstamp sends. They are now read from the raw message.
A fill showed only as a size change between two pictures, so a fully filled order read as cancelled. A trade that names a tracked order now reports its fill as it arrives, and a filled order is deleted at size 0, as the native Bitstamp feed reports it. A picture that shows an order gone before its trade arrives holds the delete for up to 2 seconds for it.

On a 90-second capture, makers now link on 40 of 42 trades, up from 16. For analysis, the native bitstamp source still shows more: every order, takers included.

A capture that fails now exits non-zero and says why (#283). ob-analytics capture used to exit 0 and write "errors": 0 to meta.json when the stream raised, or when the source's optional extra was missing. A source is now checked before the capture starts: a missing extra, or an unknown ccxt or cryptofeed venue, stops the run with status 1 and creates no output directory. An error in the snapshot, the stream or the shutdown events keeps the rows already written, is recorded in meta.json as capture_error and capture_error_phase and counted in errors, and makes the command exit 1. A source can take part in the early check by adding a preflight() method (SupportsPreflight).
A Binance capture no longer loses price levels near the top of the book (#101). ccxt deletes the levels of a Binance book that fall past the depth it is given, and Binance sends a level again only when it changes. After the price moved away and back, the captured book had holes in its top 20 levels. The ccxt source now asks for the whole Binance book (see #275 below for what it does with that book) and never gives ccxt less than 1,000 levels a side to track, so a level rarely falls out of what ccxt itself knows. ccxt's opening snapshot is now as deep as --depth-limit (never less than 1,000 levels), so a Binance capture can record up to the 5,000 levels a side that Binance sends, about 1% from the price. A deeper --depth-limit is refused.
audit no longer fails every ccxt capture with missing sequence numbers (#101). The ccxt nonce only rises: a Binance diff covers a range of update IDs, and ccxt can apply several diffs before it returns a book. A capture now records "sequence_kind": "monotonic" in meta.json, and audit then checks only that the number never goes back. Captures made before this change record no kind and are still read as contiguous.
ccxt book rows are stamped with the time they arrived (#101). The timestamp of a ccxt depth.csv row was the venue's book time, while its trades used the receive time. ccxt stamps its first Binance book with its own snapshot's time and then applies older diffs, so sorting on that time swapped two updates and left a stale level: audit reported the book crossed for 90% of a session. timestamp is now the receive time, and the venue's time is kept in a new exchange_timestamp column.
A ccxt book that loses sync is fetched again (#101). When ccxt finds a missing Binance diff it drops its book and raises an error, which used to end the book for the rest of the capture while trades went on. The capture now asks for the book again (up to 10 times) and counts it as book_resyncs in meta.json.
A venue that refuses your location gives a one-line error. Binance answers HTTP 451 from some countries. capture ccxt now says so and names --exchange binanceus and --market-data-mirror, instead of printing a traceback.
A level that only left --depth-limit, not the book, is no longer recorded as cancelled (#275). Every ccxt capture used to crop depth.csv/raw.jsonl to the top --depth-limit levels a side, so a level that was still resting just outside that crop read as a 0 row when the price moved, the same as a real cancel — 28% of the removals in a five-minute Binance BTC/USDT capture. The capture now records whatever ccxt reports, uncropped: Coinbase, Bitstamp and OKX ignore --depth-limit and always hand ccxt their whole book (over 20,000 levels a side on Coinbase); Binance and its family are asked for their whole book too (see #101 above). Kraken is unaffected, because it subscribes at exactly --depth-limit levels and drops a level from its own book once the price moves it out of that window — a 0 row there can still be either a real cancel or Kraken's own window exit, which is inherent to how Kraken reports its book. docs/howto/ccxt.md now says what a 0 row means, per venue.
compute_vpin now warns when its default bucket size is too big for the capture (#274). The default bucket_volume (average daily volume ÷ 50) scales a short capture up to a full day, so the bundled sample fills only one bucket and the trailing vpin_avg never covers a full window. That was already recorded in attrs["diagnostics"], easy to miss on a frame that otherwise looks fine — compute_vpin now also raises a UserWarning when there are fewer complete buckets than n_buckets, naming a smaller bucket_volume or n_buckets as the fix. The flow-toxicity how-to gains a short-captures section with a working example on the sample.
The VPIN chart draws an empty panel when no bucket is complete. A capture with less volume than one bucket gives compute_vpin zero rows, and plot("vpin", ...) then raised: a TypeError with matplotlib, a KeyError with both backends when the frame had no columns. Both backends now draw the axes with "(no complete buckets)" in the title. compute_vpin now returns its usual columns and dtypes when it has no rows, where before it returned a frame with no columns (or, with sign_method="bvc", columns of object dtype).
A price-level file no longer has its prices rounded to the tick size. L2DepthLoader and L2TradeReader converted each price to the nearest whole number of ticks, so a price finer than tick_size moved without a warning: a Kalshi price of 0.036 loaded as 0.04 at the default 0.01 tick, and the most traded Kalshi markets quote in tenths of a cent. Both now raise ConfigError when a price is not a whole number of ticks, and say which tick size to set. A ccxt capture records its market's tick size in meta.json, and ob-analytics process and ob-analytics audit read it from there (recorded_tick_size), so a CLI replay needs no extra option.
A streamed ccxt capture no longer writes a trade twice. ccxt's Polymarket websocket handed back a trade it had already delivered, together with the next new one, and only a polled capture skipped repeats. Every ccxt capture now skips a trade identical to one it has written: same id, time, price, size and side. Two fills that share a Polymarket id (the settling transaction) are both kept. meta.json counts the skipped repeats in duplicate_trades.
A polled ccxt capture no longer records trades from before it started. On a venue without websockets, the first poll of the trade tape returns the venue's recent history, which on Kalshi reached back nine hours. Those trades were written with the capture's receive time, as if they had just happened. The capture now drops trades older than its opening book.
A Bitstamp capture no longer starts from a snapshot older than its stream (#237). The capturer subscribes to the WebSocket, then fetches the REST book. It assumed the stream already covered the moment the book describes, but it often does not: in a live test the first order message came 0.7 s after the snapshot's microtimestamp, and the bundled sample shows the same 0.74 s gap. An order deleted in that gap stayed in the capture until the synthetic deleted at shutdown. In the bundled sample, one such ask was the best ask for 89% of the session.

The capturer now fetches the book again, a second apart and up to 10 times, until some buffered order message is at or before the snapshot's microtimestamp. meta.json gains snapshot_fetches and snapshot_overlap. In the live test, the second fetch no longer listed any of the 15 orders that were gone. Eight of those were orders that trades printed through.

A price level now empties when the order resting on it goes away. price_level_volume added an order's volume at the price on its created row and subtracted it at the price on whichever later row removed it. Those two prices are not always the same: Bitstamp reports a deleted carrying a price the order never rested at for 1.3% of orders, and the subtraction then landed on a level the volume was never added to, leaving the created level holding it for the rest of the session. Every later row now subtracts at the order's created price, so +v and -v always cancel on one level.

On the bundled Bitstamp sample this removed 104 price levels holding 29.95 BTC that no order was resting on. They were the reported touch on both sides — best bid $78,495.00 against a real best bid of $78,350.00, and best ask $78,324.00 against a real best ask of $78,333.00 — so best_bid_price moves on 35.7% of depth_summary rows and best_ask_vol on 68.9%. The per-order rebuild (engine.book_state) tracks orders by id and never had this problem; the two rebuilds now agree on how long that book is crossed.

aggressiveness_bps is NaN, not an infinity, against a zero touch. The depth engine reports a zero price for an empty side, and the Bitstamp sample also carries orders priced at zero (audit reports these as nonpositive_price). Dividing by that produced a signed infinity that travelled through every downstream mean. A distance from a price that is not tradeable has no value, so it is now NaN.
Order lifecycles read every filled order as cancelled when sizes were floats (#226 regression). order_lifecycles summed each order's fills and cast the total to int64. On integer lots that is exact, but the function also accepts base-asset floats, and it is handed them on every gallery run: display_result converts a whole result to display units before any face builds. Base-asset sizes are mostly below 1, so a 0.121 BTC fill truncated to 0, the order read as never executed, and the three lifecycle-derived L3 faces — Order Activity, Order Outcome and Queue Position — drew a book of nothing but cancellations. On the bundled Bitstamp sample the Order Activity face lost 224 of its 226 filled spans. The sum now keeps the units it was given, integer lots summing exactly and base-asset floats with the compensation that was dropped as part of #226.

LOBSTER was never affected: its lot size is 1, so a truncated size equals the size. Every LOBSTER face is pixel-identical across the change.

LOBSTER's fill column was float64, not integer lots (#226). A 0.0 literal in the expression that built it widened the whole column, so a schema-4.0 LOBSTER run wrote base-asset-looking floats that were really lot counts. Nothing raised; the values only differ from the correct ones once the lot size is not 1.

[0.1.0] - 2026-06-26¶

First public release (PyPI). The Python port of the R obAnalytics package, reworked into a pipeline API with pluggable formats, flow-toxicity metrics, L2/L3 visualization, and Matplotlib/Plotly backends — plus the packaging, documentation, and distribution that make it installable. The sections below also record how the API was deliberately de-bloated and unified during the port (the pipeline's numeric output is unchanged — the regression fingerprints pass; only the shape of the public API moved). See Extending ob-analytics.

Packaging & distribution¶

The bundled Bitstamp sample ships gzip-compressed (orders.csv.gz, ~23 MB → ~2.9 MB installed); sample_csv_path() returns the .gz path, read transparently by pandas. No API change.
Published documentation site (GitHub Pages), CITATION.cff, an explicit GPL-2.0-or-later license section, and a "Scale envelope" doc.
PyPI release workflow (release.yml, trusted publishing), package classifiers and project URLs, and ob_analytics.__version__.
Fixed quickstart and API-reference documentation drift.

Breaking¶

Pydantic models removed. ob_analytics.models (OrderEvent, Trade, DepthLevel, OrderBookSnapshot) deleted; the data contract is now column-list constants + validate_events_df / validate_trades_df / validate_depth_df in ob_analytics.schemas.
metrics/ package removed. ToxicityMetric, Vpin, Ofi, KyleLambda, register_metric, and list_metrics are gone. Call compute_vpin, compute_kyle_lambda, and order_flow_imbalance on result.trades directly.
Pipeline(metrics=...) removed. Metrics are no longer a pipeline stage — compute them after the run.
PipelineConfig.vpin_bucket_volume removed — pass bucket_volume= to compute_vpin.
PipelineResult slimmed to exactly events, trades, depth, depth_summary, and config. The vpin, ofi, metrics, metadata, and extras attributes are gone.
The thirteen plot_* wrappers removed → one plot(name, *, backend="matplotlib", ax=None, **data) dispatcher keyed by (plot_name, backend); renderers self-register into RENDERERS.
Global theme state removed. set_plot_theme / get_plot_theme / _current_theme deleted; pass theme=PlotTheme(...) to plot().
Exception hierarchy collapsed to ObAnalyticsError + ConfigError. InvalidDataError, MatchingError, InsufficientDataError, and ConfigurationError are removed.
Top-level __all__ trimmed to ~22 orchestration names. Low-level helpers now import from their submodules — ob_analytics.bitstamp, ob_analytics.lobster, ob_analytics.analytics, ob_analytics.depth, ob_analytics.data, ob_analytics.visualization, ob_analytics.flow_toxicity.
Format is now a typing.Protocol — there is no base class to inherit; any conforming object is recognised structurally.
Low-level helpers no longer re-exported from the package root (e.g. depth_metrics is now from ob_analytics.depth import depth_metrics).
RunContext.extras and Format.collect_extras removed. LOBSTER trading halts are read from LobsterLoader.trading_halts and composed into the gallery via extra_panels=.
DepthMetricsEngine.update() removed → the public hot-path method is update_side(price, volume, side, out).

Added¶

ob_analytics.schemas — the single data contract: column-list constants (EVENT_COLUMNS, TRADE_COLUMNS, DEPTH_COLUMNS) plus the validate_* functions, run at the pipeline's Protocol boundaries. Replaces the Pydantic model layer.
One generic Registry[K, V] (ob_analytics._registry) backs the format, writer, capturer, and renderer registries. Register through the public helpers register_format, register_writer, register_capturer, and RENDERERS.register / register_plot_backend.
Unified plot() dispatcher + RENDERERS registry keyed by (plot_name, backend), so new plots and backends plug in without a wrapper function. The HTML gallery composes custom panels via extra_panels=.
ob_analytics.live — optional sub-package for live order-book capture: the LiveCapturer protocol (with an optional SupportsDiagnostics capability), CaptureConfig, CaptureResult, CaptureSink, and a generic asyncio runner. Capture output drops straight into the pipeline (orders.csv schema unchanged). Install with pip install "ob-analytics[live]".
ob-analytics capture <venue> CLI verb with a built-in bitstamp capturer (ob_analytics/live/bitstamp.py); --list shows registered capturers. scripts/collect_bitstamp_btcusd.py is now a thin wrapper around it.
TradeSource protocol and BitstampTradeReader — read an authoritative companion trades.csv and join it to events via the fill column.
RunContext dataclass (ob_analytics.protocols, re-exported at the top level) for per-run parameters such as LOBSTER trading_date that don't belong on long-lived Format instances.
Docs — docs/extending.md (add a data source / writer / plot / metric / capturer).
Tests — test_bitstamp.py, test_cli.py (subprocess smoke tests for all CLI subcommands), test_exceptions.py, test_data_registry.py, a regression snapshot suite pinning demo Parquet hashes + the Kyle-λ baseline, and ob_analytics/__main__.py (python -m ob_analytics).

Changed¶

Bundled sample — ob_analytics/_sample_data/ now ships orders.csv and trades.csv from a modern BTC/USD live capture (replaces the legacy 2015 orders-only slice).
Demos consolidated into ob_analytics._demos; scripts/bitstamp_demo.py, scripts/lobster_demo.py, and the bitstamp-demo / lobster-demo CLI subcommands are now thin argparse wrappers. Behaviour unchanged.
Performance — the LOBSTER book is maintained as a SortedDict (no per-event re-sort), Bitstamp trade→event resolution is indexed, LOBSTER depth uses a single strategy, the Plotly import is memoised, and depth metrics sum active levels into bps bins. Numeric output is unchanged (pinned by the regression snapshots).
compute_kyle_lambda computes its OLS via np.linalg.lstsq (was hand-rolled; agrees with the prior implementation to rtol=1e-10).
Internal modules reorganized (renames from the 0.x line): e.g. event_processing.py → bitstamp.py, validation/time helpers → _utils.py, and the visualization modules split into a visualization/ subpackage.
Type checking is Astral's ty (not mypy); lint and format are Ruff.

Removed¶

pacman order type. A legacy artifact of the 2015 Bitstamp HTTP API, where a single order_id could appear at multiple prices over its lifetime. Modern Bitstamp WS v2 and LOBSTER do not produce this pattern (price-modifies become cancel + new id). The type Categorical no longer includes "pacman", the set-subtraction classification path is gone, and LobsterLoader no longer renumbers hidden-execution ids (raw type 5 now retains the native LOBSTER id=0).
Bitstamp trade inference. A companion trades.csv next to orders.csv is now required. Removed: Needleman–Wunsch matching, BitstampMatcher, BitstampTradeInferrer, the MatchingEngine / TradeInferrer protocols, NeedlemanWunschMatcher, and the match_cutoff_ms / price_jump_threshold fields on PipelineConfig.
Zombie detection — get_zombie_ids and the zombie_offset_seconds / skip_zombie_detection config fields.
LOBSTER LobsterMatcher — removed; LobsterTradeInferrer renamed to LobsterTradeReader with load(events, source).
Legacy Bitstamp-only wrappers load_event_data, event_match, match_trades, process_data, and plot_price_levels_faster.
12 unused runtime dependencies (scikit-learn, scipy, jupyter, bokeh, …) and stale dev dependencies (black, flake8 + plugins, darglint).

Fixed¶

depth_metrics no longer overflows for prices > $9,999.99 — dynamic dict[int, int] state replaces the fixed array.
best_bid / best_ask are tracked correctly from the first event (were initialised with dataset-wide max/min).
datetime_to_epoch uses .astype("int64") instead of the deprecated .view("int64").
All print() replaced with loguru logging; all bare assert statements replaced with raised exceptions; plt.show() removed from plot functions (callers control display).