Get the latest tech news
Spiral
Data 3.0, backed by the best
Your AI engineers are stuck in a Sisyphean loop: Read Parquet → Explode to Arrow (10x memory) → Convert to tensors → Cache intermediate results → (Finally) train -> Repeat. WebDataset solved an immediate need for AI teams, but it's essentially equivalent to CSV files for the deep learning era: convenient for simple cases but lacking the performance, expressivity, & governance features that production systems demand. Spiral is our database built on Vortex: object store native from day one; unified governance across all data types; machine-scale throughput that actually saturates your GPUs; and one API that handles everything from tiny embeddings to massive video files.
Or read this on Hacker News