Get the latest tech news
Show HN: Keep your PyTorch model in VRAM by hot swapping code
Pytorch script hot swap: Change code without unloading your LLM from VRAM - valine/training-hot-swap
This is a barebones implementation of a method to keep large models in VRAM even after your training script exits. I personally like to build out UI for my training scripts to monitor progress, loss over time, and enable easy evaluation. It takes about 0.32 seconds on my machine from when I run the code to when I can interact with the model, and that's including initializtion time for the GUI.
Or read this on Hacker News
