Get the latest tech news
Calculate the number of language model tokens for a string
Here's an easy way to calculate the number of language model tokens for a string.
You might also need to know the number of tokens to tell if your text fits into the context window of the language model you’re working with, or if you need to chunk it first. In reality, the actual number of tokens strongly depends on the language model you’re using. What’s great about calculating the number of tokens used in a string using a Jupyter Notebook is that it’s running on your machine.
Or read this on Hacker News
