Python Cross-Validation and Evaluation

With cv=5, scikit-learn:

1. splits the data into five folds;
2. for each fold, fits a fresh copy of the model on the other four and scores it on that fold;
3. returns five scores.

from sklearn.model_selection import cross_validate
results = cross_validate(model, X, y, cv=5, scoring="neg_mean_absolute_error",
                         return_train_score=True)
print(results["test_score"], results["train_score"])

Look at every fold score, not just the mean or the best one. Widely varying scores mean performance depends heavily on which rows the model sees, often a sign of too little data or a few unusual records. A large gap between training and validation scores indicates overfitting.

Cross-validation is for comparing and tuning models using training data. It doesn't replace a final test set: if you use cross-validation scores to choose among many models, the winning score is slightly optimistic, so confirm the final choice once on held-out test data.

The splitter decides which rows end up together, and must match how the data was produced:

from sklearn.model_selection import TimeSeriesSplit
for train_idx, val_idx in TimeSeriesSplit(n_splits=3).split(X):
    print(train_idx.min(), train_idx.max(), "->", val_idx.min(), val_idx.max())

Using plain k-fold on time-ordered data lets models train on future sessions and validate on past ones, producing scores that won't hold in real forecasting. Using it on grouped data lets near-identical rows from the same venue appear in both training and validation. Pass the splitter object as cv=, and for grouped splitters pass groups= too.

The metric should match the decision the model supports:

scikit-learn always maximises scores, so error metrics are returned negated: a score of −3.2 means an MAE of 3.2. Flip the sign when reporting.

A single overall number can hide failure on an important subgroup: a model may predict weekday sessions well and weekend sessions badly. Check errors per subgroup that matters, and always compare with a baseline such as DummyRegressor evaluated with the same folds.

Preprocessing steps that learn from data must be refitted inside each fold. A pipeline does that automatically:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

pipeline = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
scores = cross_val_score(pipeline, X, y, cv=5, scoring="neg_mean_absolute_error")

In each fold, the scaler is fitted on that fold's training rows only, then applied to its validation rows. Scaling the whole dataset first, then cross-validating, lets every validation fold influence the scaling, which leaks information. The same applies to imputing missing values, selecting features and encoding categories.

Pipelines also make hyperparameter search safe: GridSearchCV(pipeline, {"ridge__alpha": [0.1, 1, 10]}, cv=5) tunes the model while keeping preprocessing inside the folds. Parameter names use the step name, two underscores and the parameter.

Task: compare attendance predictors with an appropriate splitter, fold scores, a baseline and an untouched final test set.

from sklearn.dummy import DummyRegressor
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = [[t, w] for t, w in [(12, 0), (14, 1), (9, 0), (16, 1), (18, 0), (11, 1),
                         (20, 0), (15, 1), (22, 0), (13, 1), (19, 0), (24, 1)]]
y = [21, 34, 19, 38, 30, 33, 36, 41, 40, 38, 39, 52]
X_dev, y_dev, X_test, y_test = X[:9], y[:9], X[9:], y[9:]     # last 3 sessions held out

cv = TimeSeriesSplit(n_splits=3)
for name, est in [("baseline", DummyRegressor()), ("ridge", make_pipeline(StandardScaler(), Ridge()))]:
    folds = -cross_val_score(est, X_dev, y_dev, cv=cv, scoring="neg_mean_absolute_error")
    print(name, [round(float(f), 1) for f in folds])

final = make_pipeline(StandardScaler(), Ridge()).fit(X_dev, y_dev)
print("final test MAE:", round(mean_absolute_error(y_test, final.predict(X_test)), 1))

Fold scores guide the choice; the last three sessions are scored once at the end. For the worked check, preprocessing goes inside the cross-validation pipeline to fit it separately on each training fold.

scikit-learn's cross-validation guide and scoring parameter reference cover every splitter and metric. Continue with clustering with k-means. Practise with the Python exercises and Python quiz.