The Al-Dente Neural Network: Part I on Weights & Biases

A couple of years ago, Andrej Karpathy posted a tweet on the most common mistakes people make when training neural nets:

A year after, he followed it up with a comprehensive blog post covering all the steps he takes when building a neural network training pipeline that avoids all the aforementioned mistakes (or at least makes them easy to fix). Given the sheer detail and depth into which Andrej’s blog goes into to elucidate these points, it is impossible to cover all of them in a single report. Over the course of a series of reports, I will try to put some of the steps in that recipe into practice and see how each of them impacts the quality of the network we end up with.

‍

‍