Get the latest tech news
EROFS Being Extended To Handle Massive Amounts Of Data For AI Model Training
The EROFS open-source, read-only Linux file-system is set to be extended with the upcoming Linux 6.15 kernel cycle to support massive amounts of data to support AI model training.
In advance of the Linux 6.15 merge window opening following the v6.14 kernel release, EROFS is seeing 48-bit addressing support added to handle larger file-systems with an emphasis on making the file-system more suitable for AI training purposes and other large-scale data archival needs. However, several new use cases now require larger capacity support: - Massive datasets for model training to boost random sampling performance for each epoch; - Object storage clients using EROFS direct passthrough. In addition, it introduces an mtime field to 32-byte compact inodes for basic timestamp support, as well as expands the superblock root NID to an 8-byte rootnid_8b for out-of-place update incremental builds.
Or read this on Phoronix