Get the latest tech news
HRT's Python fork: Leveraging PEP 690 for faster imports
At HRT, we’ve found that centralizing our codebase facilitates cross-team collaboration and rapid deployment of new projects. Therefore, the majority of our software development takes place in a monorepo, and our Python ecosystem is set up such that internal modules are importable everywhere. Unfortunately, the convenience of this arrangement has led to a conundrum: a vast proliferation of imports.
For users, this has surfaced as scripts starting tens of seconds later, notebooks taking minutes to load, and even the simplest distributed jobs spending a substantial portion of their runtime on imports. When a LazyImport is retrieved from a dict(e.g. a name is referenced, incurring a globals() lookup) the interpreter completes the resolution and evaluation, transparently returning whatever was originally imported. For example, in our codebase, some applications unintentionally needed a particular sys.excepthook to take precedence, while others accidentally required a precise sequence of logging configurations to produce the expected output.
Or read this on Hacker News