Logistic Regression

Logistic Regression is a foundational algorithm in Machine Learning, typically learned after Linear Regression. While Linear Regression draws a straight line to predict continuous values, Logistic Regression is used for binary classification problems, such as predicting whether a student is Placed (1) or Not Placed (0) based on their exam scores.

Linear Regression struggles with binary data because the line extends infinitely, potentially predicting impossible values. Logistic Regression addresses this by squishing the output into an S-shaped curve between 0 and 1, yielding a probability value (like an 85% chance of being placed).

The training process begins with initializing weights and biases. For n features, n weights and a bias term are needed. In our example with two features (CGPA and IQ), three weights are required. Initially, all weights are set to zero, and the bias is set to zero as well. The model then calculates a raw linear score (z) for each student using its current weights: z = bias + (w1 × CGPA) + (w2 × IQ).

With all weights at zero, Student 1 receives a score of z = 0. This score is then passed through the Sigmoid function to convert it into a probability (p): p = 1 / (1 + e^-z). In the case of z = 0, the probability is 0.5, representing a 50% chance of being placed.

The quality of this guess is evaluated using Binary Cross-Entropy (BCE) Loss, a formula that measures the dissimilarity between the actual and predicted probabilities for each data point in the dataset.