Regularization: Weight decay, Dropout, Early stopping

Our motivation behind using optimization was to obtain that specific set of weights that incurs the least loss on our training data to achieve the maximum possible accuracy on the test set. But this never works in practice! (yes, you read that right). If we are trying to incur the least loss on the training data, i.e., fit the training data perfectly (called overfitting), our model might not always fit the test data perfectly. It is important to remember that the training and the test set are assumed to be sampled from a common dataset, and our aim is to ensure that our model fits this dataset. To achieve good accuracy, we must ensure that our model generalizes well to fit the test set as much as possible. We apply this in practice using a technique called regularization.

Let's understand this concept with a simple example. The blue points are the training data, and we fit two different models, m1 (polynomial) and m2 (linear). While m1 perfectly fits the training set (overfits!), m2 generalizes the data. Look at the third image, where the yellow points are the test set. Clearly, m2 would perform better than m1. We use regularization to make sure we choose simpler models (generalize more) and prevent the model from overfitting.



Weight decay

In this type of regularization, we prevent overfitting by zeroing out some weights of our model. An additional term is added in computation of loss over the training data as, \begin{align*} L &= \frac{1}{N} \sum_{i = 1}^N L_i(x_i, y_i, W) + \lambda R(W) \end{align*} where $\lambda$ is the hyperparameter that controls the regularization strength. Since we minimize this loss function in our optimization process, the added term decays some weights of our model. $R(W)$ for different types of weight decay can be written as, \begin{align*} \text{L2: } & R(W) = \sum_k \sum_l W_{k,l}^2 \\ \text{L1: } & R(W) = \sum_k \sum_l |W_{k,l}| \\ \text{Elastic Net (L1 + L2) : } & R(W) = \sum_k \sum_l \beta W_{k,l}^2 + |W_{k,l}| \\ \end{align*} L2 regularization shrinks weights proportionally to its size, so bigger weights gets shrunk more (since we are adding the square of them in the loss function). Whereas L1 regularization shrinks all the weights by the same amount, with the smaller weights getting zeroed out.

L2 regularization (also called weight decay) is the most commonly used and can directly be added in our optimizer instance of Pytorch as $\lambda$ = 1e-4 

optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, weight_decay=1e-4)

Dropout

In the forward pass, we randomly set some of the neurons to zero, as shown in the figure below. The probability of dropping is a hyperparameter and is usually set to 0.5.

Since we are randomly zeroing out neurons in each layer of our network in each forward pass, we would obtain outputs that are also random. We need to average out this randomness at test time to have a deterministic output. So, if we are randomly removing half of the nodes in the forward pass (p=0.5), then for testing, we would use the full network but shrink the outputs by half (multiply by dropout probability). A derivation of the same is shown below for a single neuron.

By doing this, we scale the activations so that for each neuron: output at test time = expected output at training time (average out the randomness). Having two different behaviors of our model at training and test time looks cumbersome, so in practice, we use a variant called inverted dropout. In the forward pass, we randomly mask half of the neurons and double the output of the remaining neurons, so we don't have to change anything at the test time.

Introducing randomness in our network helps the model generalize better and prevent overfitting. Furthermore, it helps our optimizer to jump out of local minima by stimulating dead neurons during the training process. Another interpretation of dropouts is training a large ensemble of sub-networks that share weights.

Dropout can directly be implemented in Pytorch when defining our model as, 

model = torch.nn.Sequential(torch.nn.Linear(3072, 100),
	torch.nn.ReLU(),
	torch.nn.Dropout(p=0.5),
    	torch.nn.Linear(100, 10))


Early Stopping

This notion of early stopping helps us to choose the num_steps hyperparameter, i.e., how long should we train our model? When training, we look at three curves: training loss as a function of iterations (should be decaying exponentially) and training and validation accuracy as a function of iterations. These curves give some sense of the health of our network.

Early stopping is a fairly simple regularization technique, where we stop the training process when the validation accuracy decreases. Training further would increase training accuracy (as our model starts to overfit), but it would perform very poorly on our test set (as the validation set is a good representation of the test set).

Comments

Post a Comment

Popular posts from this blog

The move_base ROS node

Image
move_base package provides the move_base node which is a major element of ROS Navigation stack. It moves the robot from its current position to a goal position. When path planning is done, the user gives uses the 2D Nav Goal in RVIZ to specify the goal position and orientation. This position (x,y,z) and orientation(x,y,z,w) are actually published in move_base/goal topic. The move_base node links together a global and local planner to accomplish its global navigation task. The Global Planner It is in charge of calculating the safe path in order to arrive at goal pose. Note that the path is calculated before the robot starts moving so it will not take into account the laser readings while moving i.e. unscanned obstacles that are not in the map. Navfn Planner The Navfn planner is the most commonly used global planner for ROS Navigation. It uses Dijkstra's algorithm in order to calculate the shortest path between the initial pose and the goal pose. Carrot Planner This plann...
Read more

Three Wheeled Omnidirectional Robot : Motion Analysis

Image
A Three-wheeled omnidirectional robot is capable to moving in any given direction without rotating by giving velocities to the three wheels in a certain ratio. All motors are situated at an angle of 120 degrees from each other. f B ,  f L ,  f R  are for back , left and right wheel respectively. a x  and a y  are accelerations in X R  and Y R  axis in Robot frame and w is the angular velocity in Robot frame. Hence putting the values of alpha and making it into a matrix form we get : As we need to obtain the f values, we will invert the matrix. We need  a x  , a y  and w is the world frame, so we apply transformation. The transformation angle is    α 1  = 30 degrees.  Putting this in the previous equation, Multiplying the two matrices, we get the final relation as : For example, if you want to move the robot along X axis(world frame), put  a x W  = 1 , ...
Read more

Overview of ATmega328P

Image
While reading about the ATmega328, I came a across a lot of stuff which I would like to share here. There is a humongous amount of information available in its datasheet. Introduction ATmega328 is a single chip micro-controller created by Atmel. It has a In-System Self-programmable Flash memory of 32 KB, SRAM of 2 KB (2048 bytes) and EEPROM of 1 KB (1024 bytes). It has a 8-bit AVR CPU type which means they can process data in 8-bit chunk as its data bus is 8-bit wide.   It contains 23 general purpose I/O pins and 32 general purpose working registers. It has 32 single-byte registers and 8-bit RISC (reduced instruction set computer) devices and hence the name 328. The P is for ' picoPower '. It has 3 flexible timer/counter with compare modes, internal and external interupts , 6-channel 10-bit A/D converter , programmable watch dog timer with internal oscillator and 5 software selectable power saving modes. Digital communication peripherals include 1-USART, 2-SPI and 1-...
Read more