Get the latest tech news
Finding Optimal Tokenizers
In this post, I will present an algorithm that was able to compute an optimal tokenizer in some settings. This result is cool because optimal tokenization is theoretically intractable, but seems to be solvable in practice.
None
Or read this on Hacker News