From Solving Inverse Problems to Training Neural Networks

From Solving Inverse Problems to Training Neural Networks

Step 1: Define the Model

The neural network predicts house prices using:

$$ \hat{y} = w \times \text{Size} + b $$

where:

• $ w $ is the weight (to be optimized),
• $ b $ is the bias (to be optimized),
• $ \text{Size} $ is the input feature (house size in square meters),
• $ \hat{y} $ is the predicted price.

Step 2: Initialize Parameters

We start with random initial values:

• $ w = 20 $ (initial guess)
• $ b = 2000 $ (initial guess)
• Learning rate $ \alpha = 0.0000001 $ (small step size to ensure gradual learning)

Step 3: Training Data

We use three training samples:

Iteration 1

Step 3.1: Compute Predictions

Using:

$$ \hat{y} = w \times \text{Size} + b $$

Step 3.2: Compute Errors

$$ \text{Error} = \hat{y} - y $$

Step 3.3: Compute Gradients

Using the formulas:

$$ \frac{d\text{Loss}}{dw} = \frac{2}{N} \sum (\hat{y} - y) \times \text{Size} $$

$$ \frac{d\text{Loss}}{db} = \frac{2}{N} \sum (\hat{y} - y) $$

For weight gradient:

$$ \frac{d\text{Loss}}{dw} = \frac{2}{3} (7000 \times 100 + 17000 \times 200 + 27000 \times 300) $$

$$ = \frac{2}{3} (700000 + 3400000 + 8100000) = \frac{2}{3} \times 12200000 = 8133333 $$

For bias gradient:

$$ \frac{d\text{Loss}}{db} = \frac{2}{3} (7000 + 17000 + 27000) = \frac{2}{3} \times 51000 = 34000 $$

Step 3.4: Update Parameters

$$ w = w - \alpha \times \frac{d\text{Loss}}{dw} = 20 - 0.0000001 \times 8133333 = 20.86667 $$

$$ b = b - \alpha \times \frac{d\text{Loss}}{db} = 2000 - 0.0000001 \times 34000 = 2000.0038 $$

Iteration 2

Step 4.1: Compute Predictions

New predictions using updated parameters:

Step 4.2: Compute Errors

$$ \text{Error} = \hat{y} - y $$

Step 4.3: Compute Gradients

$$ \frac{d\text{Loss}}{dw} = \frac{2}{3} (7866.67 \times 100 + 18733.34 \times 200 + 29600.01 \times 300) $$

$$ = \frac{2}{3} \times 13503333.33 = 9002222.22 $$

$$ \frac{d\text{Loss}}{db} = \frac{2}{3} (7866.67 + 18733.34 + 29600.01) = \frac{2}{3} \times 56200.02 = 37466.68 $$

Step 4.4: Update Parameters

$$ w = 20.86667 - 0.0000001 \times 9002222.22 = 21.72524 $$

$$ b = 2000.0038 - 0.0000001 \times 37466.68 = 2000.007565 $$

Iteration 3

Step 5.1: Compute Predictions

New predictions using updated parameters:

Step 5.2: Compute Errors

$$ \text{Error} = \hat{y} - y $$

Step 5.3: Update Parameters

$$ w = 21.72524 - 0.0000001 \times 9912238.1 = 22.575808 $$

$$ b = 2000.007565 - 0.0000001 \times 40900.96 = 2000.011296 $$

Summary After 3 Iterations

Loss is decreasing, showing learning progress!