Training Feed Forward Neural Network(FFNN) on GPU — Beginners Guide

Dataset — CIFAR 10

If you are someone who wanted to get started with FFNN (feed forward neural networks)but not quite sure which dataset to pick to begin with, then you are at the right place. We see Neural network implementations in classical machine learning to deep neural networks. Today, neural networks are used for solving many business problems such as sales forecasting, customer research, data validation, and risk management, Let us start by asking couple of fundamental questions —

1. What is a FFNN ?
   A feedforward neural network is an artificial neural network wherein connections between the nodes do not form a cycle. As such, it is different from its descendant: recurrent neural networks. The feedforward neural network was the first and simplest type of artificial neural network devised.
2. What are GPU’s ?
   A GPU (Graphics Processing Unit) is a specialized processor with dedicated memory that conventionally perform floating point operations required for rendering graphics. In other words, it is a single-chip processor used for extensive Graphical and Mathematical computations which frees up CPU cycles for other jobs. GPU’s has more cores and are much faster than CPU.

About Dataset

The CIFAR-10 dataset (Canadian Institute For Advanced Research) is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32×32 color images in 10 different classes. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class.

Table of contents

№1: Introduction

The CIFAR-10 dataset contains 60,000 32×32 color images in 10 different classes. CIFAR-10 is a set of images that can be used to teach a FFNN how to recognize objects. Since the images in CIFAR-10 are low-resolution (32×32), this dataset can allow researchers to quickly try different algorithms to see what works.

List of classes under the CIFAR 10 dataset —

1. Airplanes ✈️
2. Cars 🚗
3. Birds 🐦
4. Cats 😺
5. Deer 🐆
6. Dogs 🐶
7. Frogs 🐸
8. Horses 🐴
9. Ships 🚢
10. Trucks 🚚

№2: Data Pre Processing

Loading required libraries

Since we are using PyTorch to build the neural network. I import all the related library in single go.

Get Data

The dataset in available within the torch vision library. Alternately you can also access the dataset from Kaggle.

№3: Exploring the CIFAR10 dataset

Q: How many images does the training and testing dataset contain?

Q: How many output classes does the dataset contain?

Q: What is the shape of an image tensor from the dataset?

Q: Can you determine the number of images belonging to each class?

№4: Preparing the data for training

Splitting into training and validation sets

Before the training is started, It is important to split the data into training and testing set.

The dataset is split into the training set of 45000 and validation set of 5000.

We can now create data loaders to load the data in batches.

Let’s visualize a batch of data using the make_grid helper function from Torchvision.

Visualizing a batch

Can you label all the images by looking at them? Trying to label a random sample of the data manually is a good way to estimate the difficulty of the problem, and identify errors in labeling, if any.

Configuring the model

I write an accuracy function that calculate the model accuracy by comparing predicted class and the actual class label.

I write an ImageClassificationBase class that contains 4 functions. One function each for training and validation sets which implement the loss and accuracy. The ‘validation_epoch_end’ combines the losses and accuracy for each epoch and ‘epoch_end’ prints the ‘val_loss’ and ‘val_acc’ at the end of each epoch.

Moving the model to GPU

In this section, we will move our model to GPU.

let us first check if the GPU is available in your current system. If it is available then set the default device to GPU else set it as CPU.

Now i load the training, validation and test set to the default device avaliable

Let us also define a couple of helper functions for plotting the losses & accuracies.

№5: Training the model

We can make several attempts at training the model. Each time, try a different architecture and a different set of learning rates. Here are some ideas to try:

• Increase or decrease the number of hidden layers
• Increase of decrease the size of each hidden layer
• Try different activation functions
• Try training for different number of epochs
• Try different learning rates in every epoch What’s the highest validation accuracy you can get to? Can you get to 50% accuracy? What about 60%?

Without training the val_acc is around 10 percent. This is random guess and the possibility of predicting the correct class is 1 out of 10 classes.

Let us start the training now

I plot the captured results via matplotlib

It’s quite clear from the above picture that the model probably won’t cross the accuracy threshold of 48% even after training for a very long time. One possible reason for this is that the learning rate might be too high. The model’s parameters may be “bouncing” around the optimal set of parameters for the lowest loss. we can try reducing the learning rate and training for a few more epochs to see if it helps.

The more likely reason that the model just isn’t powerful enough. Hence to improve the model performance there is a need for a better technique to extract features efficiently from the input images. This can be obtained using a Convolution neural network.

In my next notebook, I will teach how CNN are implemented and can be used to improve the model performance.

№6: Testing with individual images

Loading the testing set from the Torchvision library

Let’s define a helper function predict_image, which returns the predicted label for a single image tensor.

Checking the label and predicted values for few samples

Model performance on test set

№7: Summary

Here is the brief summary of the article and step by step process we followed in training the FFNN on GPU.

1. We briefly learned about the feed-forward neural network.
2. Downloaded the dataset from Torchvision library.
3. We explored the dataset and tried understanding the overall images each class has, total images in training and validation set, etc.
4. Data preparation before training
   i . Splitting into training and validation sets.
   ii. We visualized a batch of the dataset.
   iii. We did the basic model configurations like defining accuracy, evaluation, and fit function. We also defined the ImageClassificationBase class.
   iv. We checked the availability to GPU and moved the dataset to GPU if available else move it to CPU.
5. We trained the model and achieved an accuracy of about 48 % and ran the same model on the test set.
6. We randomly checked the model performance by running it on the few testing samples

№8: Future Work

1. The model performance can future be improved if CNN architecture is implemented.
2. Try implementing the other deep learning framework Tensorflow.
3. Try improving the model performance by updating the number of layers, changing the optimizer and loss function in the FFNN.

№9: References

1. You can access and execute the complete notebook on this link — https://jovian.ai/hargurjeet/cfar-10-dataset-6e9d9
2. https://pytorch.org/
3. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
4. https://jovian.ai/learn/deep-learning-with-pytorch-zero-to-gans

I really hope you guys learned something from this post. Feel free to give a 👏if you like what you learnt. This keeps me motivated.

I will be posting another article within this week on how CNN are implemented and can be used to improve the CIFAR 10 model performance.

Thanks for reading this article. Happy Learning 😃