Get the latest tech news
Bf16 might be freezing your weight updates
If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent…
None
Or read this on Hacker News