Ideas for building
better software.

How an Intern Released 3 Terabytes Worth of Storage Before BFCM

How AI training scales

We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one potential limit to further growth of AI systems. More broadly, these results show that neural network training need not be considered a mysterious art, but can be rigorized and systematized.

Node.js 10.14.2 (LTS)

Node.js 11.4.0 (Current)

Quantifying generalization in reinforcement learning

We’re releasing CoinRun, a training environment which provides a metric for an agent’s ability to transfer its experience to novel situations and has already helped clarify a longstanding puzzle in reinforcement learning. CoinRun strikes a desirable balance in complexity: the environment is simpler than traditional platformer games like Sonic the Hedgehog but still poses a worthy generalization challenge for state of the art algorithms.

Director of Engineering, Lawrence Mandel Talks Road to Leadership, Growth, and Finding Balance.

Node.js 6.15.1 (LTS)

Node.js 10.14.1 (LTS)