Skip to content

LOBSTER

Support for LOBSTER message files (event types 1–7), orderbook-backed depth, and round-trip writers.

Use via Pipeline(source=LobsterSource(...)) or Pipeline.from_source("lobster", ...).

Key differences from Bitstamp:

  • Executions in the message file — LobsterTradeReader builds trades directly from type 4/5 rows in the events DataFrame (no separate trades file).
  • Orderbook-backed depth — LobsterSource.compute_depth reads the official orderbook file for ground-truth depth instead of reconstructing from events
  • Integer prices — raw prices are in ten-thousandths of a dollar (price_divisor=10000)

LobsterLoader 

LobsterLoader(
    config: PipelineConfig | None = None,
    *,
    trading_date: str | Timestamp,
    session_tz: str = LOBSTER_DEFAULT_TZ,
    venue: str | None = None,
    symbol: str | None = None,
)

Load raw limit-order events from LOBSTER message files.

Satisfies the :class:~ob_analytics.protocols.EventLoader protocol.

Parameters:

Name Type Description Default
config PipelineConfig

Pipeline configuration.

 None
trading_date str or Timestamp

The calendar date of the trading session (LOBSTER timestamps are seconds after midnight and need a date anchor).

 required
session_tz str

The venue's local time zone, used to place the session's seconds after midnight on the shared UTC clock. Defaults to :data:LOBSTER_DEFAULT_TZ ("America/New_York"), correct for LOBSTER's US equity data.

 LOBSTER_DEFAULT_TZ
venue str

Optional instrument identity. When either is supplied, the loaded frame gains per-row venue / symbol columns; venue falls back to "lobster" when only symbol is given. Both None (the default) leaves the frame untagged. The instrument ticker lives in the LOBSTER filename, so pass it as symbol when you want it on the rows.

 None
symbol str

Optional instrument identity. When either is supplied, the loaded frame gains per-row venue / symbol columns; venue falls back to "lobster" when only symbol is given. Both None (the default) leaves the frame untagged. The instrument ticker lives in the LOBSTER filename, so pass it as symbol when you want it on the rows.

 None

load 

load(source: Any) -> pd.DataFrame

Load LOBSTER message data and return a cleaned events DataFrame.

Parameters:

Name Type Description Default
source str, Path, or directory

Path to a LOBSTER message CSV, or a directory containing message/orderbook file pairs. When a directory is given the loader auto-discovers files by the LOBSTER naming convention.

 required

Returns:

Type Description
DataFrame

LobsterTradeReader 

LobsterTradeReader(config: PipelineConfig | None = None)

Build trades directly from LOBSTER execution events.

In LOBSTER, each execution event (type 4 or 5) represents the resting (maker) side of a trade. This reader builds trade records directly from those rows in the events frame; no matching is needed because the data already pairs maker rows with executions.

Satisfies the :class:~ob_analytics.protocols.TradeSource protocol.

load 

load(events: DataFrame, source: Any) -> pd.DataFrame

Build a trades DataFrame from LOBSTER execution events.

Parameters:

Name Type Description Default
events DataFrame

Events with raw_event_type column populated.

 required
source Any

Unused; trade information is embedded in events.

 required

Returns:

Type Description
DataFrame

Trades with timestamp, price, volume, direction, maker_event_id, taker_event_id, maker, taker.

LobsterWriter 

LobsterWriter(
    config: PipelineConfig | None = None,
    *,
    trading_date: str | Timestamp,
    session_tz: str = LOBSTER_DEFAULT_TZ,
    price_divisor: int | None = None,
)

Write a run as a LOBSTER message file and orderbook file.

The message file has one row per event. The orderbook file has one row per message: the book after that event, taken from the run's depth table. So the file holds the book the run computed, whatever source the run came from. Reading the files back with :class:LobsterSource gives the run's book after every event: the depth summary read back equals the run's, as far as the written levels reach.

Satisfies the :class:~ob_analytics.protocols.DataWriter protocol.

Parameters:

Name Type Description Default
config PipelineConfig

The run's configuration: its tick_size, lot_size and price_divisor. Defaults to LOBSTER's own grid (a cent tick, price_divisor=10_000, whole shares), as :class:LobsterSource sets it.

 None
trading_date str or Timestamp

Calendar date of the session.

 required
session_tz str

The venue's local time zone, used to convert the shared UTC clock back to LOBSTER's seconds after local midnight. Defaults to :data:LOBSTER_DEFAULT_TZ; must match the value used on load.

 LOBSTER_DEFAULT_TZ
price_divisor int

Multiplier to convert decimal prices back to LOBSTER integers.

 None

write 

write(
    data: dict[str, DataFrame],
    dest: str | Path,
    *,
    ticker: str = "DATA",
    num_levels: int = 10,
    **kwargs: Any,
) -> tuple[Path, Path]

Write a run to a LOBSTER message file and orderbook file.

Parameters:

Name Type Description Default
data dict

The run's tables. "events" is required. "depth" is the depth table the orderbook file is written from; when it is missing, it is computed from the events with :func:~ob_analytics.depth.price_level_volume, which needs the type column that :func:~ob_analytics.analytics.set_order_types adds.

 required
dest str or Path

Output directory.

 required
ticker str

Ticker symbol for filename.

 'DATA'
num_levels int

Number of price levels per side in the orderbook file. A book with fewer levels is padded with LOBSTER's dummy prices and zero sizes; a book with more is cut to the levels nearest the touch.

 10

Returns:

Type Description
tuple of Path

(message_path, orderbook_path)

Raises:

Type Description
ConfigError

If the run has no events, the events lack a column, two events share an event_id, the depth table names an event the events do not have, the prices and sizes of either table are not whole ticks and lots, num_levels is below 1, or one tick is not a whole number of 1 / price_divisor units.

LobsterSource dataclass 

LobsterSource(settings: SourceSettings = SourceSettings())

The LOBSTER source: offline replay of message/orderbook files (L3).

trading_date is taken from the per-run :class:~ob_analytics.protocols.RunContext, not the source constructor — so the same LobsterSource() instance can be reused across runs with different sessions. Offline only: LOBSTER ships as data files, so it satisfies :class:~ob_analytics.protocols.OfflineSource and has no live capability.

lobster_depth_from_orderbook 

lobster_depth_from_orderbook(
    events: DataFrame,
    orderbook_path: Path,
    config: PipelineConfig,
    *,
    message_rows: int | None = None,
) -> tuple[pd.DataFrame, pd.DataFrame]

Compute depth and depth summary from the LOBSTER orderbook file.

The LOBSTER orderbook file is ground truth: it records the complete visible book state after every message event. This function converts it into the (depth, depth_summary) pair the pipeline expects, avoiding the need to reconstruct depth from message events (which fails when events reference pre-market orders absent from the message file).

Parameters:

Name Type Description Default
events DataFrame

The events the :class:LobsterLoader read from the message file. Each event takes its book from the orderbook row of its own message, named by original_number; the time and event_id of the depth rows come from the event. The orderbook rows of type 6 (cross trade) and type 7 (halt) messages, which have no event, are skipped.

 required
orderbook_path Path

Path to the LOBSTER orderbook CSV.

 required
config PipelineConfig

Pipeline configuration. The file's prices are read on price_divisor and tick_size, and its sizes are converted to whole lots of lot_size, as the message sizes are.

 required
message_rows int

The number of rows in the message file, every event type included. When given, the orderbook file must have exactly this many rows. :class:LobsterSource passes the count its loader read.

 None

Returns:

Type Description
tuple of (depth DataFrame, depth_summary DataFrame)

Sizes in both are int64 lots.

Raises:

Type Description
ConfigError

If events lacks a column this needs, an original_number is missing, not a whole number, below 1 or repeated, or the orderbook file does not have one row per message: fewer rows than the events name, or a count other than message_rows.