Get the latest tech news
Loading Pydantic models from JSON without running out of memory
Pydantic’s JSON loading uses a huge amount of memory; here’s how to reduce it.
Essentially, slots are a more efficient in-memory representation for Python objects, where the list of possible attributes is fixed. This saves memory at the cost of disallowing adding extra attributes to an object, which in practice isn’t that common so it’s often a good tradeoff. ImplementationPeak memory usage (MB) Model.model_validate_json() 2000 ijson 1200 ijson+@dataclass(slots=True) 450This particular use case, of loading a large number of objects, may not be something Pydantic developers care about, or have the time to prioritize.
Or read this on Hacker News
