Get the latest tech news
The secret ingredients of word2vec (2016)
Word2vec is a pervasive tool for learning word embeddings. Its success, however, is mostly due to particular architecture choices. Transferring these choices to traditional distributional methods makes them competitive with popular word embedding methods.
To achieve this, they propose a weighted least squares objective \(J\) that directly aims to minimise the difference between the dot product of the vectors of two words and the logarithm of their number of co-occurrences: Recent papers from Jurafsky's group echo these findings and show that SVD -- not SGNS -- is often the preferred choice when you care about accurate word representations. I hope this blog post was useful in highlighting cool research that sheds light on the link between traditional distributional semantic and in-vogue embedding models.
Or read this on Hacker News