1. Load the MNIST dataset and create a CNN model¶
- load the MNIST dataset from the tensorflow/keras built-in dataset (just like last time)
- use the original train/test split!
- divide each pixel's value by 255 and now do not reshape, leave it as is (2D matrix (28x28) )
- eg for the test set you will have a (10000, 28, 28) shaped vector
train the following network on the training set and generate prediction for the 10.000 test images:
input (28, 28) conv2D, 16 kernels, kernel size = 3, valid padding, relu activation conv2D, 16 kernels, kernel size = 3, valid padding, relu activation maxpooling kernel size = 2*2 conv2D, 32 kernels, kernel size = 3, valid padding, relu activation conv2D, 32 kernels, kernel size = 3, valid padding, relu activation maxpooling kernel size = 2*2 flatten dense, 10 neurons, softmax activation- pay attention to channel format, you will need to expand dims!
- how many parameters do we have for each layer?
- use Adam optimizer with default parameters
- use categorical crossentropy as loss function
- compile the model
- print out a summary of the model
- train the CNN on the training data for 5 epochs with batch size of 32
- use the test data as validation data
calculate the categorical cross-entropy loss and the accuracy! Hint: you should get at least ~98% accuracy
- show the confusion matrix of the predictions (predicted values vs actual labels)
- where does the model make mistakes? Where does it improve compared to fully connected nets?
2. Download the Street View House Numbers (SVHN) Dataset¶
- source: http://ufldl.stanford.edu/housenumbers/
- use the cropped dataset!
- to get the dataset use eg. wget and keep the original splitting, so download train and test matrix files
- preprocess the downloaded data to be able to use it for training and testing, so shapes should be same (except image sizes) as it was in ex 1.
- how many classes do we have in the dataset? how many train and test examples do we have?
- what is the dimension of the images?
- show 5 images from the dataset
- make one-hot encoding for the labels
3. Train the CNN model seen in the 1st exercise for this dataset¶
- create a convolutional neural network
the network should have the following layers:
input (32, 32, 3) conv2D, 16 kernels, kernel size = 3, valid padding, relu actvation conv2D, 16 kernels, kernel size = 3, valid padding, relu actvation maxpooling kernel size = 2*2 conv2D, 32 kernels, kernel size = 3, valid padding, relu actvation conv2D, 32 kernels, kernel size = 3, valid padding, relu actvation maxpooling kernel size = 2*2 flatten dense, 10 neurons, softmax activation how many parameters do we have for each layer?- use Adam optimizer with default parameters
- use categorical crossentropy as loss function
- compile the model
- print out a summary of the model
- train the CNN on the training data for 15 epochs with batch size of 32
- use the test data as validation data
- calculate the categorical cross-entropy loss and the accuracy! Hint: you should get at least ~80-90% accuracy
4. Evaluate performance¶
- plot the training and the validation loss on the same plot!
- plot the training and the validation accuracy on the same plot!
- do we overfit?
- show the confusion matrix of the predictions (predicted values vs actual labels)
- where does the model make mistakes?
5. Train an other CNN¶
- as we can see the previous architecture can be further improved
- come up with an architecture that can achieve more than 91% accuracy on the test set
- print out the summary for this model!
- plot the loss and accuracy curves for this model too!
In [21]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, \
BatchNormalization, Dropout, Flatten
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.datasets import mnist
from tensorflow.keras import utils
from tensorflow.keras import callbacks
from sklearn.metrics import confusion_matrix
import seaborn as sns
from scipy.io import loadmat
1)¶
In [2]:
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# normalize inputs from 0-255 to 0.0-1.0
x_train = np.expand_dims( x_train.astype('float32'), -1)
x_test = np.expand_dims( x_test.astype('float32'), -1 )
x_train = x_train / 255.0
x_test = x_test / 255.0
# one hot encode outputs
print(np.unique(y_train))
y_train = utils.to_categorical(y_train)
y_test = utils.to_categorical(y_test)
num_classes = y_test.shape[1]
x_train.shape, x_test.shape, y_train.shape, y_test.shape, num_classes
Out[2]:
In [3]:
plt.imshow(x_train[42, ..., 0])
Out[3]:
In [4]:
model = Sequential()
model.add( Conv2D(16, (3, 3), input_shape=(28, 28, 1), padding='valid', activation='relu' ) )
model.add( Conv2D(16, (3, 3), activation='relu', padding='valid' ) )
model.add( MaxPooling2D((2, 2)) )
model.add( Conv2D(32, (3, 3), activation='relu', padding='valid' ) )
model.add( Conv2D(32, (3, 3), activation='relu', padding='valid' ) )
model.add( MaxPooling2D((2, 2)) )
model.add (Flatten() )
model.add( Dense(num_classes, activation='softmax') )
# Compile model
epochs = 5
model.compile( loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'] )
print(model.summary())
In [5]:
# Fit the model
history = model.fit( x_train, y_train, validation_data=(x_test, y_test), epochs=5, batch_size=32 )
In [6]:
# summarize history for accuracy
plt.figure( figsize=(6,6), dpi=100 )
plt.plot( np.arange(epochs)+1, history.history['loss'] )
plt.plot( np.arange(epochs)+1, history.history['val_loss'] )
plt.title('Model loss function')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'test'], loc='upper right')
Out[6]:
In [7]:
y_pred = model.predict( x_test, verbose=0)
# Final evaluation of the model
scores = model.evaluate( x_test, y_test, verbose=0 )
print("Accuracy: %.2f%%" % ( scores[1]*100) )
In [8]:
y_test_labels = np.array( [ np.argmax(y_test[i]) for i in np.arange(y_test.shape[0]) ] )
y_pred_labels = np.array( [ np.argmax(y_pred[i]) for i in np.arange(y_pred.shape[0]) ] )
y_test_labels.shape, y_pred_labels.shape
Out[8]:
In [9]:
matrix = confusion_matrix(y_test_labels, y_pred_labels, labels=np.arange(0, 10))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(matrix, annot=True, cmap='Greens', fmt='d', ax=ax)
plt.title('Confusion Matrix for training dataset')
plt.xlabel('Predicted label')
plt.ylabel('True label')
Out[9]:
2)¶
In [49]:
!wget http://ufldl.stanford.edu/housenumbers/train_32x32.mat
!wget http://ufldl.stanford.edu/housenumbers/test_32x32.mat
In [10]:
# Load the data
train_raw = loadmat('train_32x32.mat')
test_raw = loadmat('test_32x32.mat')
# Load images and labels
x_train = np.array(train_raw['X'])
x_test = np.array(test_raw['X'])
y_train = train_raw['y']
y_test = test_raw['y']
# Check the shape of the data
print(x_train.shape)
print(x_test.shape)
# Fix the axes of the images
x_train = np.moveaxis(x_train, -1, 0)
x_test = np.moveaxis(x_test, -1, 0)
print(x_train.shape)
print(x_test.shape)
print(y_train.shape)
print(y_test.shape)
In [11]:
print(np.unique(y_train)), print(np.unique(y_test))
Out[11]:
In [12]:
# normalize inputs from 0-255 to 0.0-1.0
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = x_train / 255.0
x_test = x_test / 255.0
# one hot encode outputs
y_train[ y_train == 10] = 0 # 10 is 0 here
y_test[ y_test == 10] = 0 # 10 is 0 here
y_train = utils.to_categorical(y_train)
y_test = utils.to_categorical(y_test)
num_classes = y_test.shape[1]
x_train.shape, x_test.shape, y_train.shape, y_test.shape, num_classes
Out[12]:
In [13]:
# Plot a random image and its label
plt.imshow(x_train[73239])
plt.show()
print('Label OH vector:', y_train[73239], '\nLabel number:', np.argmax(y_train[73239]))
3)¶
In [29]:
model = Sequential()
model.add( Conv2D(16, (3, 3), input_shape=(32, 32, 3), padding='valid', activation='relu' ) )
model.add( Conv2D(16, (3, 3), activation='relu', padding='valid' ) )
model.add( MaxPooling2D((2, 2)) )
model.add( Conv2D(32, (3, 3), activation='relu', padding='valid' ) )
model.add( Conv2D(32, (3, 3), activation='relu', padding='valid' ) )
model.add( MaxPooling2D((2, 2)) )
model.add (Flatten() )
model.add( Dense(num_classes, activation='softmax') )
# Compile model
epochs = 15
model.compile( loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'] )
print(model.summary())
In [30]:
# Fit the model
history = model.fit( x_train, y_train, validation_data=(x_test, y_test),
epochs=epochs, batch_size=32 )
4)¶
In [31]:
# summarize history for loss
plt.figure( figsize=(6,6), dpi=100 )
plt.plot( np.arange(epochs)+1, history.history['loss'] )
plt.plot( np.arange(epochs)+1, history.history['val_loss'] )
plt.title('Model loss function')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'test'], loc='upper right')
Out[31]:
In [33]:
# summarize history for accuracy
plt.figure( figsize=(6,6), dpi=100 )
plt.plot( np.arange(epochs)+1, history.history['accuracy'] )
plt.plot( np.arange(epochs)+1, history.history['val_accuracy'] )
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['train', 'test'], loc='upper left')
Out[33]:
In [34]:
y_pred = model.predict( x_test, verbose=0)
# Final evaluation of the model
scores = model.evaluate( x_test, y_test, verbose=0 )
print("Accuracy: %.2f%%" % ( scores[1]*100) )
In [35]:
y_test_labels = np.array( [ np.argmax(y_test[i]) for i in np.arange(y_test.shape[0]) ] )
y_pred_labels = np.array( [ np.argmax(y_pred[i]) for i in np.arange(y_pred.shape[0]) ] )
y_test_labels.shape, y_pred_labels.shape
Out[35]:
In [36]:
matrix = confusion_matrix(y_test_labels, y_pred_labels, labels=np.arange(0, 10))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(matrix, annot=True, cmap='Greens', fmt='d', ax=ax)
plt.title('Confusion Matrix for training dataset')
plt.xlabel('Predicted label')
plt.ylabel('True label')
Out[36]:
5)¶
In [23]:
model = Sequential([
Conv2D(32, (3, 3), padding='same',
activation='relu',
input_shape=(32, 32, 3)),
BatchNormalization(),
Conv2D(32, (3, 3), padding='same',
activation='relu'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(64, (3, 3), padding='same',
activation='relu'),
BatchNormalization(),
Conv2D(64, (3, 3), padding='same',
activation='relu'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(128, (3, 3), padding='same',
activation='relu'),
BatchNormalization(),
Conv2D(128, (3, 3), padding='same',
activation='relu'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Flatten(),
Dense(128, activation='relu'),
Dropout(0.4),
Dense(10, activation='softmax')
])
early_stopping = callbacks.EarlyStopping(patience=8)
optimizer = Adam(lr=1e-3, amsgrad=True)
model_checkpoint = callbacks.ModelCheckpoint(
'best_cnn.h5',
save_best_only=True)
model.compile(optimizer=optimizer,
loss='categorical_crossentropy',
metrics=['accuracy'])
print(model.summary())
In [24]:
# Fit the model
history = model.fit( x_train, y_train, validation_data=(x_test, y_test),
epochs=15, batch_size=32,
callbacks=[early_stopping, model_checkpoint])
In [25]:
# Evaluate train and validation accuracies and losses
train_acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
train_loss = history.history['loss']
val_loss = history.history['val_loss']
In [26]:
# Visualize epochs vs. train and validation accuracies and losses
plt.figure(figsize=(20, 10))
plt.subplot(1, 2, 1)
plt.plot(train_acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.legend()
plt.title('Epochs vs. Training and Validation Accuracy')
plt.subplot(1, 2, 2)
plt.plot(train_loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.legend()
plt.title('Epochs vs. Training and Validation Loss')
Out[26]:
In [27]:
# Evaluate model on test data
test_loss, test_acc = model.evaluate(x=x_test, y=y_test, verbose=0)
print('Test accuracy is: {:0.4f} \nTest loss is: {:0.4f}'.
format(test_acc, test_loss))
Sources:¶
- http://ufldl.stanford.edu/housenumbers/
- https://www.tensorflow.org/datasets/catalog/svhn_cropped
- https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/image_classification/svhn.py
- https://www.kaggle.com/dimitriosroussis/svhn-classification-with-cnn-keras-96-acc
- https://medium.com/project-agi/getting-started-with-street-view-house-numbers-svhn-dataset-a96e76962504