Logistic Regression from Scratch: The Gateway to Classification

🏁 Introduction: From Regression to Classification

In my previous post, we mastered predicting continuous values with Linear Regression. But what happens when the answer isn't a number — it's a category? Is this email spam or not? Is this tumor malignant or benign? Does this image contain a cat?

These are classification problems, and Logistic Regression is the simplest, most interpretable tool for solving them. Despite its name, it's a classification algorithm, not a regression one. And once you understand it, you'll see it hiding inside every neural network output layer for binary tasks.

📐 The Problem with Linear Regression for Classification

Suppose we try to use our Linear Regression model to classify. We predict $\hat{y} = \beta_0 + \beta_1 x$ and threshold at 0.5. This immediately breaks down:

• Predictions can be any real number — negative, zero, or greater than 1
• There's no natural interpretation as a probability
• Outliers skew the decision boundary drastically

We need a model whose output is always bounded between 0 and 1, representing a probability.

🔀 The Sigmoid Function: Squashing to [0, 1]

The sigmoid function (also called the logistic function, hence the name) is the key:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

No matter what value $z$ takes, $\sigma(z)$ outputs a number strictly between 0 and 1.

Properties:

• $\sigma(0) = 0.5$ — perfectly uncertain
• As $z \to +\infty$, $\sigma(z) \to 1$
• As $z \to -\infty$, $\sigma(z) \to 0$
• The derivative is elegantly: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$

Our model becomes:

$\hat{y} = \sigma(\beta_0 + \beta_1 x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}$

We interpret $\hat{y}$ as the probability that the input belongs to class 1.

🎯 The Loss Function: Binary Cross-Entropy

We cannot use Mean Squared Error here. With the sigmoid, MSE creates a non-convex loss surface full of local minima that gradient descent struggles to escape.

Instead, we derive the loss from Maximum Likelihood Estimation (MLE). We want to find parameters that maximize the probability of observing our training data. For binary labels $y \in \{0, 1\}$:

$P(y \mid x; \beta) = \hat{y}^y \cdot (1 - \hat{y})^{1-y}$

Taking the negative log of the joint likelihood over $n$ samples gives us the Binary Cross-Entropy Loss:

$J(\beta) = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]$

This loss is convex — it has a single global minimum, making gradient descent reliable.

Intuition:

• When $y_i = 1$: the loss is $-\log(\hat{y}_i)$. If we predict $\hat{y}_i = 0.99$, loss ≈ 0. If we predict $\hat{y}_i = 0.01$, loss is huge.
• When $y_i = 0$: the loss is $-\log(1 - \hat{y}_i)$, penalizing high predictions for negative class.

⚙️ Deriving the Gradient

To apply gradient descent, we need $\frac{\partial J}{\partial \beta}$. Let's derive it for the vector case where $z = X\beta$:

Step 1: $\frac{\partial J}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}$

Step 2: $\frac{\partial \hat{y}}{\partial z} = \hat{y}(1 - \hat{y})$ (sigmoid derivative)

Step 3: $\frac{\partial z}{\partial \beta} = X$

Chaining together, most terms cancel beautifully:

$\frac{\partial J}{\partial \beta} = \frac{1}{n} X^T (\hat{y} - y)$

This is the same form as Linear Regression's gradient — just with a different prediction function. This elegant result is one of the reasons logistic regression is so widely taught.

💻 NumPy Implementation

import numpy as np

class LogisticRegression:
    def __init__(self, learning_rate=0.1, iterations=1000):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = None

    def _sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0

        for _ in range(self.iterations):
            # Forward pass
            z = X @ self.weights + self.bias
            y_hat = self._sigmoid(z)

            # Compute gradients
            dw = (1 / n_samples) * X.T @ (y_hat - y)
            db = (1 / n_samples) * np.sum(y_hat - y)

            # Update parameters
            self.weights -= self.lr * dw
            self.bias -= self.lr * db

    def predict_proba(self, X):
        z = X @ self.weights + self.bias
        return self._sigmoid(z)

    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)

    def accuracy(self, X, y):
        predictions = self.predict(X)
        return np.mean(predictions == y) * 100

📊 The Decision Boundary

The decision boundary is where $\hat{y} = 0.5$, which occurs when $z = 0$:

$\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0$

This defines a linear hyperplane in feature space. For 2 features, it's a line; for 3 features, a plane; and so on. Logistic Regression is a linear classifier — it works perfectly when classes are linearly separable, but struggles with complex, curved boundaries (that's where neural networks come in).

🔢 Multiclass Extension: Softmax Regression

Binary logistic regression extends naturally to multiple classes using the softmax function (which you already know from our neural network from scratch!):

$P(y = k \mid x) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}$

This is also called multinomial logistic regression or softmax regression. In fact, a neural network's output layer is just softmax regression stacked on top of feature-learning layers.

🧠 Connection to Neural Networks

Here's the key insight that unlocks everything:

| Model | Architecture | |-------|-------------| | Logistic Regression | Input → Sigmoid → Output | | Neural Network (1 hidden layer) | Input → Hidden Layer (ReLU) → Sigmoid/Softmax → Output |

A neural network is simply logistic regression where the input features are learned representations from hidden layers. Every concept — sigmoid activation, cross-entropy loss, gradient descent — carries over directly. Logistic regression is the neural network building block in its simplest form.

🧭 Final Thoughts

Logistic Regression gives us:

• A principled, probabilistic framework for classification
• A convex loss function guaranteeing convergence
• An interpretable model (the weights tell us feature importance)
• The conceptual foundation for neural network output layers

In the next post, we're going to take a significant leap: from fully-connected networks to Convolutional Neural Networks (CNNs) — the architecture that made computer vision work. Stay tuned!

📌 Next Steps

• Implement regularization (L1/L2) to prevent overfitting on high-dimensional data
• Explore feature scaling effects on convergence speed
• Try this on a real dataset like the Breast Cancer Wisconsin dataset from scikit-learn

---

Written by Breye.