CemirCol

CemirCol is a data storage library that keeps numeric data column by column and is used from Python. I wrote its core in Rust and bound it to Python through PyO3. The library can be installed from PyPI with pip install cemircol. I developed the project together with Cem Emir Yüksektepe.

Purpose

When I analyse data, I usually need only a few columns of a table rather than the whole thing. In row-based formats such as CSV or JSON, even reading a single column means scanning the entire file from start to finish. Parquet solves this with its columnar layout, but I wanted to see whether I could get a simpler file format and smaller files.

That is why I built CemirCol. My goal was to store large numeric tables in as little disk space as possible and to return a requested column directly as a numpy array without unnecessary copying.

How it works

When writing, I give the library a dictionary of column names and value lists. Each column is first turned into a raw byte array and then compressed independently with zstd. I use level 22, the highest zstd compression level. Thanks to the rayon library, the columns are compressed in parallel, so several processor cores work at the same time.

The file is saved with the .cemir extension and its layout is quite simple. It starts with the CEM1 signature, followed by the compressed column blocks one after another. After them comes a metadata section in JSON that records each column's position in the file, its compressed and uncompressed size and its data type. The file ends with the length of the metadata and the CEM1 signature again.

On the reading side, the file is opened with mmap. mmap lets a program read a file directly through the operating system's virtual memory without copying it into memory first. The reader first reads the signature and metadata at the end of the file. When a column is requested, only that column's compressed slice is touched. The slice is decompressed straight into a byte array owned by Python and turned into a numpy array with numpy.frombuffer without copying. No separate temporary buffer is created on the Rust side.

If numpy is not installed, the library returns the result with Python's built-in array module, and if that is not possible either, as a plain Python list.

Key features

• Every column is compressed separately at the highest zstd level, which keeps the files very small.
• Reading uses mmap and a zero-copy approach, so memory usage stays low.
• All columns are compressed in parallel during writing, using every processor core on the machine.
• The query() method returns a numpy array directly, so the data is ready for scientific computing right away.
• The columns() and num_rows() methods report the column names and row count without decompressing any data.
• The from_csv and from_parquet helpers convert existing CSV and Parquet files to the .cemir format in a single line.
• Files compressed with zlib by earlier versions of the library are recognised and read automatically.
• If column lengths do not match or an unsupported data type is given, writing stops with a clear error message.

Technical stack

The core is written in Rust and the Python extension is built with PyO3. I use maturin for packaging and building. The library relies on memmap2 for memory mapping, zstd for compression, flate2 for reading older files, rayon for parallel work, and serde with serde_json for writing and reading the metadata as JSON.

For release builds I enabled the highest optimisation level, link-time optimisation (LTO, where the compiler optimises the whole program as one unit) and a single code generation unit. The supported data types are currently 64-bit integers (int64) and 64-bit floating-point numbers (float64). The CSV and Parquet converters optionally need pandas and pyarrow, which are installed with the cemircol[froms] extra. The library works with Python 3.8 and later and is released under the MIT licence.

With the benchmark.py script in the repository, I ran a comparison on a dataset of 10 million rows with two columns, one integer and one floating-point. In that measurement the CemirCol file took about 5.5 MB, while the same data took about 85 MB as Parquet and about 183 MB as CSV. Reading a single column took 0.108 seconds with CemirCol and 0.113 seconds with Parquet. These results belong to that dataset and that machine; the ratios may differ for other data.

Use cases

• Archiving large numeric tables such as sensor readings, financial data or log statistics.
• Reducing read time and memory usage in data science work where only certain columns are needed.
• Storing data in far less space than Parquet or CSV where disk space is limited.
• Shrinking old data archives by converting existing CSV and Parquet files.

Status

Version 1.0.1 of the library is available on PyPI. I also prepared the cemircol-go project to use the same Rust core from Go.