ANN From Scratch (NumPy, No DL Libraries)

Building an Artificial Neural Network from the ground up in Python/NumPy — no TensorFlow, PyTorch, or Keras — to actually understand feedforward and backpropagation before moving to high-level frameworks. Help is taken from the paper Learning representations by back-propagating errors

Progress

Stage 1 — Single Neuron / Single-Layer Perceptron

A minimal single-layer network (2 inputs → 1 output) with sigmoid activation. No hidden layer, no training loop — just feedforward on hardcoded weights to verify the core prediction formula (y = x1w1 + x2w2 + b) function work correctly. This is the first formula in the paper as $x_j = \sum_{i} y_i w_{ji}$. Then we apply the activation function $y_j = \frac{1}{1 + e^{-x_j}}$

Stage 2 — Multilayer Feedforward ANN with Backpropagation

A 2-3-1 network (2 inputs → 2 hidden neurons → 1 output), fully hardcoded with known weights/biases from a worked example, used as a debugging oracle. Implements:

• feedforward() — forward pass through hidden and output layers
• backpropogation() — error calculation (output + hidden layer formulas) and weight/bias updates via gradient descent
• epochs() — iterative training loop with convergence tracking

Validated against hand-derived reference values (matched to 4 decimal places on the first iteration), then run for 250+ epochs to confirm the output converges toward the target (0.739 → 0.047 over 250 epochs).

Key formulas implemented:

• Err_output = O(1-O)(target - O)
• Err_hidden = O(1-O)(Err_next * W_next)
• W_new = W_old + learning_rate * O_i * Err_j
• b_new = b_old + learning_rate * Err_j

Stage 3 — Vectorized ANN (Matrix Form)

Rewrote the Stage 2 oracle (2-3-1, same hardcoded weights/target) using NumPy matrix operations instead of scalar-per-weight arithmetic. Same network, same numbers — different implementation, used to validate the matrix formulation before generalizing.

• Input/weights/biases stored as NumPy arrays (W_hidden, W_output, bias_hidden, bias_output) instead of individual named variables (w13, w14, ...)
• linear_transform() — W @ O + b, replacing the scalar weightedSum()
• errorUpdate() — same formula as Stage 2, now operating element-wise on arrays instead of single floats
• weightsUpdate() — uses np.outer(Err, O) for the weight-update outer product, replacing per-weight scalar updates
• biasUpdate() — vector addition, replacing per-bias scalar updates
• Error backpropagated through the weight matrix via W_output.T @ Err_output Validated by running 250 epochs and confirming output converges the same way as the Stage 2 oracle (0.739 → ~0.047), and by checking err_output on epoch 1 matches Stage 2's -0.1425 exactly.

Still hardcoded to a fixed 2-3-1 topology and a single repeated training example — this stage's purpose was proving the matrix math is correct, not generality.

Stage 4 — Generalized N-Layer ANN + XOR problem Soluion

Rebuilding Stage 3's matrix operations into a fully general engine that isn't tied to any fixed topology or example. Target features:

• Arbitrary layer sizes via a single layer_sizes list (e.g. [2, 3, 1] or [6, 2, 1] or [784, 128, 64, 10]) — weights/biases generated in a loop, not hand-declared per layer
• self.weights / self.biases as lists of matrices/vectors (one per layer transition), with feedforward() and backpropogation() looping over them instead of hardcoded per-layer lines
• Random weight initialization (uniform range, configurable) instead of hand-given values
• Trains on a full dataset (multiple input/target pairs looped per epoch), not a single repeated example
• Tracks a proper MSE loss per epoch for convergence monitoring, matching the general form E = 0.5 * Σ(target - output)²
• Numerically stable sigmoid
• Momentum-based weight updates (Δw(t) = -ε·∂E/∂w + α·Δw(t-1)) succesfully implemented to reduce the average loss
• About XOR problem, Error vanishes at 400th Epoch

Why hardcode first?

Stage 2's hardcoded oracle wasn't wasted effort — it's the reference implementation Stage 3 gets checked against. If the vectorized version doesn't reproduce the same numbers on the same fixed input/weights, the bug is in the new code, not the math.

Explanation of General ANN wrt Paper