Get the latest tech news
Everything I've learned so far about running local LLMs
program.com/blog/2024/11/10/ Over the past month I’ve been exploring the rapidly evolving world of Large Language Models (LLM). It’s now accessible enough to run a LLM on a Raspberry Pi smarter than the original ChatGPT (November 2022).
From just two files, EXE and GGUF (model), both designed to load via memory map, you could likely still run the same LLM 25 years from now, in exactly the same way, out-of-the-box on some future Windows OS. So pick some special tokens that don’t appear in inputs, use them to delimit a prefix and suffix, and middle (PSM) — or sometimes ordered suffix-prefix-middle (SPM) — in a large training corpus. Have an argument with Benjamin Franklin (note: this probably violates the Acceptable Use Policy of some models), hang out with a character from your favorite book, or generate a new scene of Falstaff’s blustering antics.
Or read this on Hacker News
