Get the latest tech news
Polars Cloud: The Distributed Cloud Architecture to Run Polars Anywhere
DataFrames for the new era
However PySpark is written in Scala, requires the JVM to run locally, has very poor unpythonic UX (java backtraces for one), and is very sensitive to OOMs. Our initial (and achieved) goal was offering an alternative for pandas with a flexible API that does enable query optimization, and parallel streaming execution. It will be very seamless to spin up hardware and run Polars queries remotely, either in batch mode for production ETL jobs, or interactively doing data exploration.
Or read this on Hacker News
