Building Linear Regression Model from Scratch (No Libraries)

What is Linear Regression?

Linear regression establishes a linear relationship between an independent and dependent variable by fitting a line capturing the underlying data trend. This article presents two approaches: least squares (analytical) and gradient descent (iterative).

Least Squares Method

The least squares approach minimizes error by calculating the difference between actual and predicted y-values. The error is the difference between the actual y value and the predicted y value.

Errors are squared rather than using absolute values for computational efficiency. Squaring keeps the equation differentiable and weights larger errors more heavily, though this can be problematic with outliers.

The method finds the sum of squared errors (E), then uses calculus optimization to solve for parameters m and c by setting partial derivatives to zero.

Gradient Descent Algorithm

Gradient descent is an iterative optimization algorithm minimizing the cost function. The process involves:

• Initialization: Start with initial m and c values (zeros or random)
• Compute Gradient: Calculate partial derivatives of the cost function
• Update Parameters: Adjust values opposite to the gradient direction using learning rate α
• Iteration: Repeat until convergence

Learning Rate Consideration

The learning rate is critical. If it's too large, the algorithm might overshoot the minimum and diverge. If it's too small, the convergence might be very slow.

Python Implementation

The article provides a custom LinearRegression class implementing gradient descent, then compares its performance against scikit-learn's model using mean squared error metrics.