Download Notebook

Summary

Harvard University
Spring 2018
Instructors: Rahul Dave
**Due Date: ** Friday, Febrary 16th, 2018 at 10:00am

Instructions:

• Upload your final answers as an iPython notebook containing all of your work to Canvas.

• Structure your notebook and your work to maximize readability.

Problem 1: Optimization via Descent

Suppose you are building a pricing model for laying down telecom cables over a geographical region. Your model takes as input a pair of coordinates, $(x, y)$, and contains two parameters, $\lambda_1, \lambda_2$. Given a coordinate, $(x, y)$, and model parameters, the loss in revenue corresponding to the price model at location $(x, y)$ is described by $L(x, y, \lambda_1, \lambda_2) = 0.000045\lambda_2^2 y - 0.000098\lambda_1^2 x + 0.003926\lambda_1 x\exp\left\{\left(y^2 - x^2\right)\left(\lambda_1^2 + \lambda_2^2\right)\right\}$ Read the data contained in HW3_data.csv. This is a set of coordinates configured on the curve $y^2 - x^2 = -0.1$. Given the data, find parameters $\lambda_1, \lambda_2$ that minimize the net loss over the entire dataset.

Part A

• Visually verify that for $\lambda_1 = 2.05384, \lambda_2 = 0$, the loss function $L$ is minimized for the given data.
• Implement gradient descent for minimizing $L$ for the given data, using the learning rate of 0.001.
• Implement stochastic gradient descent for minimizing $L$ for the given data, using the learning rate of 0.001.

Part B

• Compare the average time it takes to update the parameter estimation in each iteration of the two implementations. Which method is faster? Briefly explain why this result should be expected.
• Compare the number of iterations it takes for each algorithm to obtain an estimate accurate to 1e-3 (you may wish to set a cap for maximum number of iterations). Which method converges to the optimal point in fewer iterations? Briefly explain why this result should be expected.

Part C

Compare the performance of stochastic gradient descent for the following learning rates: 1, 0.1, 0.001, 0.0001. Based on your observations, briefly describe the effect of the choice of learning rate on the performance of the algorithm.

Problem 2. SGD for Multinomial Logistic Regresion on MNIST

The MNIST dataset is one of the classic datasets in Machine Learning and is often one of the first datasets against which new classification algorithms test themselves. It consists of 70,000 images of handwritten digits, each of which is 28x28 pixels. You will be using PyTorch to build a handwritten digit classifier that you will train and test with MNIST.

** The MNIST dataset (including a train/test split which you must use) is part of PyTorch in the torchvision module. The Lab will have details of how to load it. **

Your classifier must implement a multinomial logistic regression model (using softmax). It will take as input an array of pixel values in an image and output the images most likely digit label (i.e. 0-9). You should think of the pixel values as features of the input vector.

1. Plot 10 sample images from the MNIST dataset (to develop intuition for the feature space).
2. Construct a softmax formulation in PyTorch of multinomial logistic regression with Cross Entropy Loss.
3. Train your model using SGD to minimize the cost function. Use a batch size of 64, a learning rate $\eta = 0.01$, and 10 epochs.
4. Plot the cross-entropy loss on the training set as a function of iteration.
5. What are the training and test set accuracies?
6. Plot some (around 5) examples of misclassifications.