Images as Tensors

Images are sampled on a 2D grid.

Images as Tensors

In Pytorch, the channel dimension is before the spatial dimensions.

\[C~ \times ~H~ \times ~W\]

Images as Tensors

When training Neural Networks, we use mini-batches.

\[S~ \times ~C~ \times ~H~ \times ~W\]

Hence, we pass 4D Tensors to the network.

MNIST CNN in PyTorch

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, kernel_size=5)
        self.conv2 = nn.Conv2d(20, 50, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(800, 256)
        self.output = nn.Linear(256, 10)

MNIST CNN in PyTorch

...
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 800)
        x = F.relu(self.fc1(x))
        x = self.output(x)
        return x

After 300 iterations over training set: 99.21% validation accuracy.

Learned Kernels

Pre-Trained Networks

The term “Transfer Learning” simply means using a pre-trained network to save on training.

Pre-Trained Networks

The greatest barrier to supervised machine learning is the lack of labelled data.

Researchers have developed neural network architectures for Computer Vision tasks.

What can we use transfer learning for?

VGG16

The VGG group at Oxford university trained VGG-16 and VGG-19 for ImageNet classification.

• Karen Simonyan & Andrew Zisserman, (2014)

VGG16

VGG-16 is a good choice for a first step in transfer learning.

It has a relatively simple architecture:

• Convolutional layers, increasing in depth, decreasing spatially.
• fully-connected layers for classification.
• Max-pooling layers.
• ReLU activation functions.

VGG16

VGG16

This kind of architecture works well for many Computer Vision tasks.

• Small convolutional filters (3x3)
• Max-pooling layers
• ReLU activation functions

Transfer Learning

Two strategies for transfer learning are:

Transfer Learning

There are examples of both fine tuning and feature extraction at the example repository:

https://github.com/uea-teaching/Deep-Learning-for-Computer-Vision

Data Standardisation

Ensure zero-mean and unit standard deviation.

• In numerically diverse data, learning will be dominated by larger values.
• Arguably less important with image data.
• Many pre-trained networks expect standardised data.

Data Standardisation

For regression tasks, we need to standardise the output data too.

Data Standardisation

Extract sample data: pixel values in the case of images.

Compute the mean and standard deviation of the samples.

\[ x' = \frac{x - \mu(x)}{\sigma(x)} \]

Batch Size

Small batch sizes, approximately 1-10.

Batch Size

Large batch sizes, greater than 500-1000.

Batch Size

Typical choice around 64-256, lots of experiments use ~100.

Batch Size

Increasing mini-batch size will improve performance up to the point where all GPU units are in use.

Increasing it further will not improve performance; it will reduce accuracy!

Learning Rate

The amount of change applied to the parameters at each iteration.

DropOut

Over-fitting is a well-known problem in machine learning.

• Dropout reduces over-fitting.

DropOut

During training, randomly choose units to ‘drop out’.

DropOut

Activates a different subset of units for each sample.

DropOut

“I went to my bank. The tellers kept changing and I asked one of them why? He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation between employees to successfully defraud the bank… This made me realise that randomly removing a different subset of neurons on each example would prevent conspiracies and thus reduce over fitting.”

Batch normalisation

Batch normalization (Ioffe, et al. 2015).

Batch normalisation

Apply between internal layers.

• Use BatchNorm2d with a convolutional layer.
• Use BatchNorm1d with a fully-connected layer.

Batch normalisation

Standardise activations per-channel between network layers.

Solves problems caused by exponential growth or shrinkage of layer activations in deep networks.

Dataset augmentation

Reduce over-fitting by enlarging training set.

Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition. Sharif, et al. 2016.

Generative Adversarial Networks

Generative Adversarial Nets. Goodfellow et al. 2014.

Train two networks; one given random parameters to generate an image, another to discriminate between a generated image and one from the training set.

Unsupervised representation Learning with Deep Convolutional Generative Adversarial Nets. Radford, et al. 2015.

A Style-Based Generator Architecture for Generative Adversarial Networks. Karras, et al. 2018