Loading Pydantic models from JSON without running out of memory
You have a large JSON file, and you want to load the data into Pydantic.
Unfortunately, this uses a lot of memory, to the point where large JSON files are very difficult to read.
What to do?
Assuming you’re stuck with JSON, in this article we’ll cover:
The high memory usage you get with Pydantic’s default JSON loading.
How to reduce memory usage by switching to another JSON library.
Going further by switching to dataclasses with slots.
The problem: 20× memory multiplier
We’re going to start with...
Read more at pythonspeed.com