{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" ARTIFICIAL INTELLIGENCE (E016350A)
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Regularization - part II"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from matplotlib import pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from sklearn import linear_model"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(23)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us consider one synthetically made dataset with 100 instances, where the attributes are correlated. We will see how this affects the coefficients of the regression model if we are using different regularizations."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"N = 100\n",
"\n",
"X1 = np.random.uniform(-1, 1, N)\n",
"X2 = np.random.uniform(-1, 1, N)\n",
"\n",
"# there is a linear dependence between the first and the second attribute\n",
"# between the first and the second attribute, there is a small difference given by noise with normal distribution N(0, 1)\n",
"X = np.vstack([X1, 4*X1, X1 + np.random.normal(0, 1, N), X2]).transpose()\n",
"\n",
"# the value of the target variable is a linear combination of X1 and X2\n",
"y = (2 + np.random.normal(size=N))*X1 + (3 + np.random.normal(size=N))*X2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we want to see the behaviour of the model's coefficients, we will use the entire set for training. Also, we will not worry about choosing the optimal values of the hyperparameters."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. Linear regression"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"LinearRegression() In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Ridge(alpha=1) In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Lasso(alpha=0.1) In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. ElasticNet(alpha=0.05) In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"number_of_features = X.shape[1]\n",
"plt.xticks(np.arange(0, number_of_features), ['A1', 'A2', 'A3', 'A4'])\n",
"plt.scatter(np.arange(0, number_of_features), model.coef_, color='red', marker='o', label='no regularization' )\n",
"plt.scatter(np.arange(0, number_of_features), ridge_model.coef_, color='blue', marker='x', label='l2 regularization')\n",
"plt.scatter(np.arange(0, number_of_features), lasso_model.coef_, color='orange', marker='v', label='l1 regularization')\n",
"plt.scatter(np.arange(0, number_of_features), elastic_model.coef_, color='green', marker='^', label='l1 + l2 regularization')\n",
"plt.legend(loc='best')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, the presense of the $\\ell_1$ regularization brings the coefficients the attribute $A_1$ to zero."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}