In the simpler case of parameters estimation, we have some iid (independent and identically distributed) observations and we want to estimate from the parametric family of distributions, which is the most likely distribution. Here, we have some observations \(Y_1, Y_2,..,Y_n\) that are independent but no identically distributed because they are indexed by another random variable \(X_1,X_2,...,X_n\).

Definition of the regression function \(r(x)\)

\(r(x)=E[Y|X=x]\) where conditional expectation is defined here.

\(Y\) is called the response variable.

\(X\) is called the predictor variable, covariate or feature.

Regression analysis

The goal is the obtain \(r(x)\) from data of the form

\[(X_1, Y_1), \dots, (X_n, Y_n) \sim F_{X,Y}\]

If we define \(\epsilon\) as \(\epsilon = Y -r(x)\), then the regression model can be expressed as


with \(E[\epsilon]=0\).

\(r(x)\) corresponds to the deterministic part.