Get the latest tech news
The n-gram Language Model
The n-gram Language Model. Contribute to EurekaLabsAI/ngram development by creating an account on GitHub.
The only difference is that GPT uses a neural network to calculate the probability of the next token, while n-gram uses a simple count-based approach. The Python code also writes out the n-gram probabilities to disk into the dev/ folder, which you can then inspect with the attached Jupyter notebook dev/visualize_probs.ipynb. Instead, it hardcodes n=4, smoothing=0.01, but does the training, sampling, and test perplexity evaluation and achieves the exact same results as the Python version.
Or read this on Hacker News