Python Training and Testing Datasets

Each set has one job, and mixing them up quietly inflates results:

X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=0)
# 60% train, 20% validation, 20% test

With small datasets, cross-validation replaces the single validation set by rotating through several training/validation splits.

The rule that matters most: once you look at test results and then change the model, the test set has influenced your choices and has effectively become validation data. Its score will be optimistic. Keep the test set locked away until you've finished deciding, and if you must iterate further, collect new test data.

How you split must reflect how the model will be used:

• Random split: rows are shuffled and divided. Suitable when rows are independent and the future looks like the past.
• Stratified split (stratify=y): keeps class proportions the same in every set, important when one class is rare.
• Time-based split: train on earlier periods, test on later ones. Essential for forecasting, such as tomorrow's bus demand, where a random split would let the model learn from the future.
• Grouped split: keep all rows from the same group, such as the same passenger, school or bus stop, in one set, using GroupShuffleSplit or GroupKFold, so the model is tested on groups it has never seen.

cutoff = int(len(days) * 0.8)
train_days, test_days = days[:cutoff], days[cutoff:]   # data sorted by date

A random split of related rows leaks information: if Monday's and Tuesday's counts for the same stop are nearly identical and one lands in training while the other is in testing, the model gets credit for recognising a near-duplicate rather than predicting.

Anything learned from data, such as the mean and standard deviation for scaling, the vocabulary for text features, the median used to fill gaps or the categories for one-hot encoding, must be learned from the training set only and then applied unchanged to validation and test data:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)          # learn from training rows
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)             # apply, don't refit

Fitting the scaler on all the data before splitting lets test-set statistics influence training. The effect is often small but it's real, and with steps like feature selection or target encoding it can be large.

The reliable way to avoid this is a pipeline: make_pipeline(StandardScaler(), model). Fitting the pipeline fits every step on the training data, and it can be passed straight to cross-validation, which refits it correctly inside each fold.

Random splits depend on a random number generator. Set random_state to an integer so the same rows land in the same sets every time you run the code; otherwise results change between runs and comparisons become unreliable.

train_test_split(X, y, test_size=0.25, random_state=42)   # same split every run

Record everything needed to recreate the split: the method, the random_state, the date cut-off or grouping column, and the data version. Saving the row IDs of each set is even safer when the data may change.

A fixed seed makes a split repeatable, not appropriate. A reproducible random split of time-series data is still wrong for forecasting. And results can vary noticeably between different seeds on small datasets, which is another reason to prefer cross-validation, where you see the spread of scores instead of one lucky or unlucky split.

Task: choose and justify a bus demand split, recording training, validation and final test roles. Write the plan as code comments next to the split:

# 52 weeks of daily passenger counts, sorted by date (one row per day)
weeks = list(range(52 * 7))

# Forecasting next week's demand -> time-based split, never shuffled.
train = weeks[: 40 * 7]               # weeks 1-40: fit models
validation = weeks[40 * 7 : 46 * 7]   # weeks 41-46: compare models and settings
test = weeks[46 * 7 :]                # weeks 47-52: touched once, for the final report

print(len(train), len(validation), len(test))   # 280 42 42
assert max(train) < min(validation) and max(validation) < min(test)

The assertion proves no future day leaks into an earlier set. Scalers and other learned preprocessing are fitted on train only. For the worked check, the observations that should fit a preprocessing transformation are the training observations.

scikit-learn documents train_test_split and its cross-validation splitters, and its common pitfalls page covers leakage. Continue with cross-validation and evaluation. Practise with the Python exercises and Python quiz.