Get the latest tech news

Why Polars rewrote its Arrow string data type

DataFrames for the new era

End of last year we forked parts of Arrow2 in the crate polars-arrow, which is a trimmed down implementation of the Arrow spec and is tuned for Polars’ needs. As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. And to keep track of the start and end of the strings, an extra buffer was allocated containing the offsets (Yes signed integers, because Arrow needs to interact with Java).

Get the Android app