CemirCol-Go

CemirCol-Go is the library I prepared for using the columnar CemirCol data format from Go. It binds CemirCol's Rust core to Go through a C-compatible interface. This lets me use the same mmap and zero-copy reading approach from the Python version in applications written in Go.

Purpose

I originally built CemirCol for Python. However, I write a large share of my backend services and data pipelines in Go, and I wanted to read and write the same file format there too. Instead of rewriting the format from scratch in Go, I chose to share the Rust core that was already working and tested.

The idea is to leave memory management and concurrency to Go while doing the heavy compression and decompression work in Rust. Go's goroutines, lightweight threads that let many tasks run at once, are a good fit for processing the data in parallel after it has been read.

How it works

The Rust core is compiled as both a static and a shared library. It exposes a handful of functions through the C ABI, the common binary interface that lets different languages call each other's functions. These functions open and close a reader, return the row count, read a float64 or int64 column and write a single-column file.

On the Go side I use cgo, the tool that lets Go code call C functions directly. NewReader opens the file with mmap on the Rust side and returns a reader. When QueryFloat64 or QueryInt64 is called, Rust decompresses that column's compressed section and hands the result to Go as a block of memory. Go copies the block into its own slice and then frees the memory on the Rust side. This way Go's garbage collector only manages memory that belongs to Go, and ownership never gets mixed up between the two languages.

The Close method closes the reader and removes the file's memory mapping. For writing, WriteFloat64 and WriteInt64 compress the given column with zstd and write it to a .cemir file.

Key features

• It uses the same Rust core and the same .cemir file format as the Python version.
• Files are opened with mmap, so only the requested column's data is accessed.
• It provides both read and write functions for float64 and int64 columns.
• Memory allocated on the Rust side is freed right after the data is copied into Go.
• Calls made after a reader is closed return a clear error instead of crashing.
• The setup.sh script checks the Rust and Go installations, builds the Rust core and tidies the Go modules.

Technical stack

The Go side of the library is written with cgo, and the Rust side adds a C-compatible API layer using the libc crate. As in the Python version, the core uses memmap2, zstd, flate2, rayon, serde and serde_json. The Rust library is built with the highest optimisation level and LTO.

The repository also contains example and benchmark programs for different kinds of data, such as sensor readings, financial data, web logs and application logs. For the comparison with Parquet I used the xitongsys/parquet-go library. In the measurement in the README, 1 million float64 rows were written in about 22 ms and read in about 12 ms with CemirCol-Go, while the same operations took about 75 ms and 130 ms with parquet-go. The file was about 1 MB with CemirCol and about 3.9 MB with Parquet. These figures come from one machine and one dataset; results may differ under other conditions.

The Rust core has to be built before the library can be used, so the Rust toolchain and Go must both be installed.

Use cases

• Storing large numeric datasets in a compressed, fast-to-read form inside backend services written in Go.
• Reading .cemir files produced in Python from Go in pipelines that process logs, sensor data or financial data.
• Providing a shared data format between the two languages in projects that use both Python and Go.
• Serving as an example of how to bind a Rust library to Go.

Status

The library currently supports single-column writing and reading of float64 and int64 columns. Multi-column writing and the converters from the Python version are not available on the Go side yet.