{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"hide": true
},
"source": [
"# Learning bounds and the Test set\n",
"\n",
"##### Keywords: empirical risk minimization, Hoeffding's inequality, hypothesis space, training error, out-of-sample error, testing set, training set, test error, complexity parameter"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Contents\n",
"{:.no_toc}\n",
"* \n",
"{: toc}"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false,
"hide": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"//anaconda/envs/py35/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.\n",
" warnings.warn(self.msg_depr % (key, alt_key))\n"
]
}
],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import scipy as sp\n",
"import matplotlib as mpl\n",
"import matplotlib.cm as cm\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"pd.set_option('display.width', 500)\n",
"pd.set_option('display.max_columns', 100)\n",
"pd.set_option('display.notebook_repr_html', True)\n",
"import seaborn as sns\n",
"sns.set_style(\"whitegrid\")\n",
"sns.set_context(\"poster\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false,
"hide": true
},
"outputs": [],
"source": [
"def make_simple_plot():\n",
" fig, axes=plt.subplots(figsize=(12,5), nrows=1, ncols=2);\n",
" axes[0].set_ylabel(\"$y$\")\n",
" axes[0].set_xlabel(\"$x$\")\n",
" axes[1].set_xlabel(\"$x$\")\n",
" axes[1].set_yticklabels([])\n",
" axes[0].set_ylim([-2,2])\n",
" axes[1].set_ylim([-2,2])\n",
" plt.tight_layout();\n",
" return axes\n",
"def make_plot():\n",
" fig, axes=plt.subplots(figsize=(20,8), nrows=1, ncols=2);\n",
" axes[0].set_ylabel(\"$p_R$\")\n",
" axes[0].set_xlabel(\"$x$\")\n",
" axes[1].set_xlabel(\"$x$\")\n",
" axes[1].set_yticklabels([])\n",
" axes[0].set_ylim([0,1])\n",
" axes[1].set_ylim([0,1])\n",
" axes[0].set_xlim([0,1])\n",
" axes[1].set_xlim([0,1])\n",
" plt.tight_layout();\n",
" return axes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Revisiting the model\n",
"\n",
"Let $x$ be the fraction of religious people in a county and $y$ be the probability of voting for Romney as a function of $x$. In other words $y_i$ is data that pollsters have taken which tells us their estimate of people voting for Romney and $x_i$ is the fraction of religious people in county $i$. Because poll samples are finite, there is a margin of error on each data point or county $i$, but we will ignore that for now."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us assume that we have a \"population\" of 200 counties $x$:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" promney | \n",
" rfrac | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0.047790 | \n",
" 0.00 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0.051199 | \n",
" 0.01 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.054799 | \n",
" 0.02 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.058596 | \n",
" 0.03 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0.062597 | \n",
" 0.04 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" f | \n",
" i | \n",
" x | \n",
" y | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0.075881 | \n",
" 7 | \n",
" 0.07 | \n",
" 0.138973 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0.085865 | \n",
" 9 | \n",
" 0.09 | \n",
" 0.050510 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.096800 | \n",
" 11 | \n",
" 0.11 | \n",
" 0.183821 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.184060 | \n",
" 23 | \n",
" 0.23 | \n",
" 0.057621 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0.285470 | \n",
" 33 | \n",
" 0.33 | \n",
" 0.358174 | \n",
"
\n",
" \n",
"
\n",
"