Masters Thesis Defense: Javier Fernandez Cruz

Candidate: Javier Fernandez Cruz
Thesis Title: Using Intrinsic Dimensionality to Improve Dropout Regularization
Date & Time: July 8th, 2021 at 10:00 AM

Location: Zoom (online)
Examining Committee:

Dr. Adam Krzyzak (Chair)

Dr. Thomas Fevens (Supervisor)

Dr. Adam Krzyzak (Examiner)

Dr. Eugene Belilovsky (Examiner)

Abstract

The intrinsic dimensionality (ID) of multi-dimensional data collections is one of their most fundamental characteristics. Estimates of ID provide an important notion of the complexity of the data, which, in turn, is crucial to selecting the right approach and designing effective machine learning models. There is a wide range of applications for ID estimation, from widely used dimensionality reduction to adversarial attacks, outlier detection and search indices to more theoretical fields like similarity search, discriminability, graph construction, and extreme value theory. However, the notions provided by ID estimations of the data stop at the threshold when designing machine or deep learning models, providing little to no insight when selecting model hyperparameter configurations.

In this work, we explored the idea of using a relation between the intrinsic and extrinsic dimensionality of an image manifold to provide an intuition for selecting an appropriate dropout rate to regularize neural networks in the context of image classification problems. We studied the characteristics of several ID estimators and applied them to image datasets and introduced a new formula to compute values for the dropout rate dependent on ID estimations.

We empirically studied the effects of using this new rate by analyzing its effects in the training of several state-of-the-art image classification models on benchmark datasets. We showed that using this technique can improve the performance of several well-established models and achieved a new SOTA accuracy result for the MNIST dataset.