I Built a Tiny Neural Network From Scratch in Python — No PyTorch

I constructed a compact neural network from fundamentals in Python, without relying on PyTorch or similar libraries. My objective was not to build a framework suitable for production, but to gain a deep understanding of the inner workings of a neural network during the training process. The resulting model would be capable of addressing a binary classification challenge using a series of mathematical operations.

The network would consist of three primary components: the input layer, the hidden layer, and the output layer. For this particular example, the input layer would have three features, the hidden layer would consist of four neurons, and the output layer would contain a single neuron. This single output neuron would represent the probability of the input data belonging to class 1.

To facilitate the learning process, we need to understand the mathematical principles that underpin a neural network. A neuron begins by computing a weighted sum of its inputs, which is then passed through an activation function. In this case, the hidden layer would employ the ReLU activation function, while the output layer would utilize the sigmoid activation function.

The sigmoid function is responsible for converting the weighted sum into a value ranging between 0 and 1, representing the probability of belonging to class 1.

To move forward, we generated a synthetic binary classification dataset consisting of 200 samples, each containing three features. The target variable was derived from a specific mathematical formula that incorporated the three input features. The resulting dataset had a shape of (200, 3) for the input features and (200, 1) for the corresponding target variable.

Next, we implemented the activation functions, ReLU and sigmoid, from scratch using NumPy. The ReLU function returns the input value if it is greater than zero, and zero otherwise. The sigmoid function, on the other hand, maps any real-valued number to a value between 0 and 1. Additionally, we determined the derivative of the ReLU function, which proved to be crucial during the backpropagation phase. The derivative is equal to 1 when the input is greater than zero, and zero otherwise.

With the foundation in place, we proceeded to build the neural network itself. We defined a class called NeuralNetwork, which initializes the weights and biases for the input-to-hidden and hidden-to-output connections. The weights were randomly initialized using NumPy, and the biases were set to zero vectors of appropriate dimensions. The network architecture follows a straightforward pattern: 3 inputs lead to 4 hidden neurons, which in turn produce 1 output neuron.

Forward propagation is the process of passing the input data through the network to generate a prediction. In our implementation, we first calculated the weighted sum of the inputs (z1) and applied the ReLU activation function to obtain the activations (a1) for the hidden layer. Subsequently, we computed the weighted sum of the hidden layer activations (z2) and applied the sigmoid activation function to yield the final output (a2), which serves as our prediction.

To assess the performance of our model, we employed the binary cross-entropy loss function, which quantifies the dissimilarity between the predicted probabilities and the actual target values. This loss function is particularly suitable for binary classification problems, as it penalizes incorrect predictions more severely than correct ones. By minimizing the binary cross-entropy loss during training, our neural network would gradually improve its ability to accurately classify the input data.