Python Logistic Regression

Classify outcomes with logistic regression, read predicted probabilities, choose a decision threshold and evaluate with metrics for both classes.

Lesson 94 of 107Advanced13 min read

Logistic regression is a classification model: it estimates the probability that an example belongs to a class, such as "this seed batch will fail germination testing". Despite its name, it isn't used to predict numbers; it predicts categories, most often one of two.

Python
from sklearn.linear_model import LogisticRegression
X = [[1], [2], [3], [7], [8], [9]]       # days in damp storage
y = [0, 0, 0, 1, 1, 1]                   # 1 = batch failed
model = LogisticRegression().fit(X, y)
print(model.predict_proba([[5]])[0][1])  # estimated chance of failure after 5 days

This lesson explains how a linear score becomes a probability, how a threshold turns probabilities into decisions, why scaling belongs in a pipeline, and how to evaluate classifiers beyond accuracy. The examples screen seed batches at a community seed bank. It follows linear regression.

Classification

In classification the target is a category: fail or pass, spam or not spam, which of three plant varieties. Encode a binary target as 0 and 1, and decide which class is the positive class, the one you're trying to detect (here, failing batches):

Python
labels = ["pass", "fail", "pass", "fail"]
y = [1 if label == "fail" else 0 for label in labels]

Logistic regression handles several classes too: scikit-learn fits a multinomial model automatically when y has more than two values.

The name causes confusion because the model is built from a regression-style weighted sum of features. But its output is passed through a function that squashes it into a probability, and it's evaluated as a classifier, by how well it separates the classes, not by numeric error. Using LinearRegression on 0/1 labels instead produces "probabilities" below 0 and above 1, which is one reason logistic regression exists.

Probability mapping

The model computes a linear score, z = intercept + coefficients × features, and converts it with the sigmoid (logistic) function:

probability = 1 / (1 + e⁻ᶻ)

Python
import math
for z in [-4, -1, 0, 1, 4]:
    print(z, round(1 / (1 + math.exp(-z)), 3))

A score of 0 maps to 0.5; large positive scores approach 1 and large negative scores approach 0. Each coefficient therefore shifts the log-odds of the positive class: a positive coefficient means higher feature values make failure more likely.

model.predict_proba(X) returns one column per class, so [:, 1] is the probability of class 1. An estimated probability of 0.8 is not a promise: among many batches given 0.8, about 80% should fail if the model is well calibrated, and 20% won't. Check calibration on held-out data before treating probabilities as literal risks.

Decision thresholds

model.predict() applies a default threshold of 0.5: probability ≥ 0.5 means class 1. That's rarely the right choice for a real decision, because the two kinds of error have different costs:

  • A false negative (a failing batch marked as fine) means gardeners plant seeds that won't grow.
  • A false positive (a good batch flagged) means an unnecessary retest.

If missing a failure is much worse, lower the threshold:

Python
probabilities = model.predict_proba(X_test)[:, 1]
flagged = probabilities >= 0.3      # catch more failures, accept more retests

Lowering the threshold raises recall (the share of real failures caught) and usually lowers precision (the share of flagged batches that really fail). Choose the threshold from the costs of each error, using validation data, and document it. Changing the threshold changes the balance of mistakes, not the model.

Scaling and evaluation

scikit-learn's LogisticRegression applies regularisation by default (C=1.0), which penalises large coefficients. Penalties compare coefficient sizes, so features on very different scales are treated unfairly; standardise them, inside a pipeline so the scaler is fitted on training data only:

Python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, y_train)

Evaluate with more than accuracy. If only 5% of batches fail, a model that predicts "pass" every time is 95% accurate and catches nothing. Use:

  • a confusion matrix: counts of true and false positives and negatives;
  • precision and recall for the positive class, or their combination, F1;
  • ROC AUC, which measures how well probabilities rank positives above negatives across all thresholds.

classification_report(y_test, predictions) prints precision, recall and F1 for every class at once.

Independent practice and a worked check

Task: classify seed batches using a training pipeline, a documented threshold and metrics for both classes.

Python
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

X = [[1, 12], [2, 14], [3, 11], [4, 18], [5, 16], [6, 19], [7, 15], [8, 20],
     [2, 19], [3, 17], [6, 12], [7, 18], [1, 16], [5, 13], [8, 14], [4, 15]]
y = [0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0]

THRESHOLD = 0.4   # missing a failing batch costs more than a retest
clf = make_pipeline(StandardScaler(), LogisticRegression()).fit(X[:12], y[:12])
pred = (clf.predict_proba(X[12:])[:, 1] >= THRESHOLD).astype(int)
print(classification_report(y[12:], pred, target_names=["pass", "fail"], zero_division=0))

The report shows precision and recall for both "pass" and "fail", and the comment records why the threshold is below 0.5. For the worked check, predict_proba provides estimated class probabilities, one column per class.

Reference and next practice

scikit-learn documents LogisticRegression, classification metrics and tuning the decision threshold. Continue with decision trees and KNN. Practise with the Python exercises and Python quiz.

Logistic Regression Summary

  • Classification: Logistic regression commonly estimates probabilities for categorical outcomes.
  • Probability mapping: For binary classification, a linear score is transformed into a value between zero and one.
  • Decision thresholds: A threshold turns a probability into a decision and should reflect the task tradeoffs.
  • Scaling and evaluation: Feature scaling and regularization affect fitting, and held out metrics assess outcomes.

Logistic Regression Examples

Each example shows the code, the output it produces and what each part does.

Example 1

Logistic Regression in a small program

Python
from sklearn.linear_model import LogisticRegression
X = [[1], [2], [3], [7], [8], [9]]
y = [0, 0, 0, 1, 1, 1]
model = LogisticRegression().fit(X, y)
print(model.predict([[2], [8]]).tolist())
Output
[0, 1]
Line-by-line explanation
  1. Lines 2 to 3Short storage times are labelled 0 (pass) and long ones 1 (fail).
  2. Line 4The model learns a boundary between the two groups.
  3. Line 5Two days is predicted to pass and eight days to fail.
Example 2

Thresholds change the balance of errors

Python
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, precision_score, recall_score

# features: days in damp storage, storage temperature (°C); 1 = batch failed
X = [[1, 12], [2, 14], [3, 11], [4, 18], [5, 16], [6, 19], [7, 15], [8, 20],
     [2, 19], [3, 17], [6, 12], [7, 18], [1, 16], [5, 13], [8, 14], [4, 15]]
y = [0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0]

X_train, y_train, X_test, y_test = X[:12], y[:12], X[12:], y[12:]
clf = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
probs = clf.predict_proba(X_test)[:, 1]
print("probabilities:", [round(float(p), 2) for p in probs])

for threshold in [0.5, 0.3]:
    pred = (probs >= threshold).astype(int)
    print(f"threshold {threshold}: confusion {confusion_matrix(y_test, pred).tolist()}, "
          f"precision {precision_score(y_test, pred, zero_division=0):.2f}, "
          f"recall {recall_score(y_test, pred):.2f}")
Output
probabilities: [0.2, 0.31, 0.67, 0.37]
threshold 0.5: confusion [[3, 0], [0, 1]], precision 1.00, recall 1.00
threshold 0.3: confusion [[1, 2], [0, 1]], precision 0.33, recall 1.00
Line-by-line explanation
  1. Line 12The scaler and model are fitted together on the first twelve batches only.
  2. Line 13predict_proba gives each test batch's estimated chance of failing.
  3. Lines 16 to 20The confusion matrix lists [[true negatives, false positives], [false negatives, true positives]]. Here the single failing batch is already caught at 0.5, so lowering the threshold to 0.3 only adds two false positives and cuts precision to 0.33. With four test batches these numbers are illustrative; real evaluations need far more data.

Common Logistic Regression Mistakes

Judging a classifier by accuracy alone

Don't
print("accuracy:", (pred == y_test).mean())   # 0.95 with 5% failures can mean nothing was caught
Do
from sklearn.metrics import classification_report
print(classification_report(y_test, pred, target_names=["pass", "fail"]))

Scaling outside the pipeline

Don't
X_scaled = StandardScaler().fit_transform(X)        # test rows influence the scaling
X_train, X_test = X_scaled[:12], X_scaled[12:]
Do
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, y_train)                            # scaler sees training rows only

Logistic Regression in Real Projects

Logistic regression in practice

Logistic regression is widely used for credit scoring, medical risk scores, churn prediction, fraud screening and spam filtering, partly because its coefficients are explainable: regulators and doctors can see how each factor shifts the odds. It's also the standard baseline for any new classification problem before trying random forests, gradient boosting or neural networks.

Production systems set thresholds from business costs, monitor precision and recall over time, and recalibrate when the share of positive cases changes, since a model trained when failures were rare can misjudge probabilities when they become common.

Logistic Regression FAQ

Is logistic regression classification or regression?

Classification. It models the probability of a category using a regression-style weighted sum passed through the sigmoid function, then assigns a class with a threshold. Use it for yes/no and multi-class outcomes, not numeric targets.

What is the difference between predict and predict_proba?

predict_proba returns the estimated probability of each class. predict returns a class label by applying a 0.5 threshold for binary problems. Use predict_proba when you need to choose your own threshold or rank cases by risk.

How do I choose a classification threshold?

Weigh the cost of false positives against false negatives for your task, then pick the threshold on validation data that gives the best balance, for example the highest precision at a minimum acceptable recall. Document the choice.

What are precision and recall?

Precision is the share of predicted positives that are truly positive. Recall is the share of actual positives the model found. Raising the threshold usually increases precision and lowers recall; lowering it does the opposite.

Do I need to scale features for logistic regression?

It's strongly recommended, because scikit-learn's default regularisation penalises coefficients and features on large scales would be treated differently. Scaling also helps the solver converge. Put the scaler in a pipeline with the model.

Logistic Regression Exercises

Answer each one and check it instantly. If you get stuck, open the hint before the answer.

Multiple choiceeasy

What does predict_proba provide?

Multiple choicemedium

Which review concern is valid for logistic regression?

Logistic Regression Quiz

2 questions to check what you learned.

  1. What does predict_proba provide?

    • Guaranteed outcomes
    • Estimated class probabilities
    • Python ignores the distinction

  2. Which statement correctly describes decision thresholds?

    • A threshold turns a probability into a decision and should reflect the task tradeoffs.
    • Only the example output matters, not the rule
    • Every operation has the same behavior

Related Python topics

93Linear and Multiple RegressionFit simple and multiple linear regression models, read their coefficients and residuals, evaluate them on held-out data and know their limits.95Decision Trees and K-Nearest NeighborsTrain decision trees and k-nearest neighbours, tune depth and neighbour count on validation data, scale features for KNN and compare the two models.97Cross-Validation and EvaluationEstimate model performance reliably with k-fold cross-validation, the right splitter for your data, meaningful metrics and leak-free pipelines.90Introduction to Machine LearningFrame a prediction problem, choose features and a target, compare a model against a simple baseline and evaluate it on data it hasn't seen.
Esc
↑↓ to navigateEnter to open