Regression

Supervised learning > regression

• mapping of continuous inputs to discrete or continuous outputs
• concept from statistics: observe data to construct an equation, to be able to make predictions for missing or future data
• regression can fit different orders of polynomial (constant, line, parabola, etc.) or, for vector inputs multiple dimensions (hyperplanes)
• input representation must be numeric and continuous => discrete inputs must be enumerated and ordered

Linear Regression

• linear regression is an attempt to model relationships between a dependent variable $y$ and independent variables ($x_1,x_2,...,x_n$) => want to find equation of type $y={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n$

  • $y$ is the output variable
  • $x_1,x_2,...,x_n$ are the input variables
  • ${\theta }_1,{\theta }_2,...,{\theta }_n$ are parameters or weights of the model
  • the weights which tell how important each corresponding $x$ is to predicting the outcome

• sample data may not perfectly fit a linear model causing error in the model

  • many ways to calculate error e.g. sum of absolute errors, sum of squared errors

  • let $\hat{y}$ be the predicted output, then:

    sum of absolute errors	sum of squared errors
    ${\displaystyle \sum _{i=1}^m|{\hat{y}}_i-y_i|}$	${\displaystyle \frac{1}{2}\sum _{i=1}^m({\hat{y}}_i-y_i{)}^2}$

  • use gradient decent algorithm to find the weighs that minimize the error:

    ${\displaystyle \frac{1}{2}\sum _{i=1}^m(\sum _{j=0}^n{\theta }_j{x}_j^i-y_i{)}^2=0}$
    
    

  • for a constant function the best error is the mean of data points

Polynomial regression

In more general case, for some dataset mapping values $x_i\to y_i$, want to find weight coefficients $c_i$:

$$c_0+c_1{x}_1+c_2{x}_2^2+c_3{x}_3^3+...+c_n{x}_n^n\approx y$$

$$\left[\begin{array}{cccc}1& x_1& x_1^2& x_1^3\\ 1& x_2& x_2^2& x_2^3\\ 1& x_3& x_2^3& x_3^3\\ ⋮\\ 1& x_n& x_n^2& x_n^3\end{array}\right]\times \left[\begin{array}{c}{c}_0\\ c_1\\ c_2\\ ⋮\\ c_n\end{array}\right]\approx \left[\begin{array}{c}{y}_0\\ y_1\\ y_2\\ ⋮\\ y_n\end{array}\right]$$

where ${\displaystyle c=(X^T\cdot X{)}^{-1}{X}^T\cdot Y}$