Get the latest tech news
Hill Space: Neural nets that do perfect arithmetic (to 10⁻¹⁶ precision)
The constraint topology that transforms discrete selection from optimization-dependent exploration into systematic mathematical cartography Most neural networks struggle with basic arithmetic. They approximate, they fail on extrapolation, and they're inconsistent.
The constraint topology that transforms discrete selection from optimization-dependent exploration into systematic mathematical cartography When understood and used properly, the constraint W = tanh(Ŵ) ⊙ σ(M̂) (introduced in NALU by Trask et al. 2018) creates a unique parameter topology where optimal weights for discrete operations can be calculated rather than learned. Hill Space—the constraint topology created by W = tanh(Ŵ) ⊙ σ(M̂)—maps any unbounded learned weights to the [-1,1] range, where stable plateaus naturally guide optimization toward discrete selections.
Or read this on Hacker News
