Data I/O¶

Parquet serialization and writer registry.

load_data ¶

load_data(path: str | Path) -> dict[str, pd.DataFrame]

Load pre-processed pipeline data from a Parquet directory or pickle file.

Parameters:

Name	Type	Description	Default
`path`	`str or Path`	If path is a directory, each `.parquet` file inside is loaded as a DataFrame keyed by its stem (`events.parquet` → `"events"`). If path is a single file with a `.pkl` / `.pickle` extension, it is loaded via :func:`pandas.read_pickle` for backward compatibility (not recommended for untrusted data).	required

Returns:

Type	Description
`dict of str to pandas.DataFrame`

Raises:

Type	Description
`ConfigError`	If a Parquet file declares a schema version this build does not support. A file written with no version key (legacy data, or the bundled sample) loads with a warning — see :func:`ob_analytics.schemas.check_schema_version`.

save_data ¶

save_data(
    lob_data: PipelineResult | dict[str, DataFrame],
    path: str | Path,
    *,
    fmt: str = "parquet",
    writer: DataWriter | None = None,
    config: Any = None,
    ctx: Any = None,
    **write_kwargs: Any,
) -> None

Save pipeline data to disk.

Parameters:

Name	Type	Description	Default
`lob_data`	`PipelineResult or dict of str to pandas.DataFrame`	A pipeline result, or the DataFrames to save (keys become file stems). A result is saved as its four tables, with its own config unless config is given. :func:`load_result` reads a Parquet folder back as the same result.	required
`path`	`str or Path`	Destination directory (Parquet) or file (pickle).	required
`fmt`	`str`	Serialisation format. Built-in values are `"parquet"` (default) and `"pickle"`. The `"parquet"` path writes one file per key and tags each with :data:`SCHEMA_VERSION` in its metadata (checked by :func:`load_data`). A source name (e.g. `"bitstamp"`, `"lobster"`) round-trips through that source's own writer (its `create_writer` capability, so no separate writer registration is needed); a source that needs run state — LOBSTER's `trading_date` — reads it from ctx. A generic, source-independent writer registered via :func:`register_writer` is also resolved by name.	`'parquet'`
`writer`	`DataWriter`	A pre-constructed writer instance. When provided, fmt is ignored and the writer is used directly. This is the preferred path when saving from a :class:`Pipeline` that already holds a configured writer.	`None`
`config`	`Any`	Forwarded to a registered writer factory when `fmt` names one. The `"parquet"` writer records config in each file, so the files keep the run's tick size, lot size and display precision. `ctx` defaults to an empty :class:`~ob_analytics.protocols.RunContext`.	`None`
`ctx`	`Any`	Forwarded to a registered writer factory when `fmt` names one. The `"parquet"` writer records config in each file, so the files keep the run's tick size, lot size and display precision. `ctx` defaults to an empty :class:`~ob_analytics.protocols.RunContext`.	`None`
`**write_kwargs`	`Any`	Extra keyword arguments forwarded to `writer.write()`.	`{}`

load_result ¶

load_result(path: str | Path) -> PipelineResult

Load a saved run back into a :class:~ob_analytics.pipeline.PipelineResult.

The inverse of save_data(result, path): the four tables come back as they were saved, and so does the run's :class:~ob_analytics.config.PipelineConfig, which each Parquet file records. A result rebuilt this way shows the same prices and sizes as the run did, for example in :func:~ob_analytics.visualization.display_result or a gallery.

A folder that records only the tick and lot size (written by :meth:~ob_analytics.pipeline.Pipeline.run_windows, or by an older version) gets those two, with the display precision that shows one step of each. A folder with neither (prices and sizes stored as floats) gets a tick size of 1, so its prices read as they are.

The resolution comes from the tables: an :attr:~ob_analytics.protocols.Level.L2 run has an empty events table, and an L3 run never does.

Parameters:

Name	Type	Description	Default
`path`	`str or Path`	A Parquet folder written by :func:`save_data` or by `ob-analytics process`.	required

Returns:

Type	Description
`PipelineResult`	The saved run.

Raises:

Type	Description
`ConfigError`	If path is not a folder (it does not exist, or it is a pickle, which keeps no config, so its prices and sizes could not be read back), if the folder lacks one of the four tables, or if a Parquet file declares a schema version this build does not support.
`FileNotFoundError`	If the folder holds no Parquet file at all.

register_writer ¶

register_writer(name: str, factory: WriterFactory) -> None

Register a writer factory under name for use with save_data(fmt=name, ctx=...).

The factory is called as factory(config, ctx) and must return a :class:DataWriter. This is what lets format-specific writers (e.g. :class:~ob_analytics.lobster.LobsterWriter, which needs trading_date) participate in the registry — they pull required parameters from the :class:~ob_analytics.protocols.RunContext.

list_writers ¶

list_writers() -> list[str]

Return a sorted list of registered writer names.