Python Data Distribution and Percentiles

Three questions describe a distribution's shape:

• Centre: where is a typical value? (median)
• Spread: how far apart are typical values? (interquartile range, standard deviation)
• Shape: is it symmetric or skewed, with a long tail on one side? Does it have one peak or several (multimodal)?

Response times are usually right-skewed: most questions are answered in a few minutes, a few take much longer, and nothing takes less than zero. In right-skewed data the mean sits above the median, pulled up by the tail. Two help desks can share the same mean while one answers almost everyone within 5 minutes and the other answers half instantly and half after 20 minutes.

A histogram (see data visualization) shows the shape at a glance; percentiles summarise it in numbers. Report both centre and tail: "median 3 minutes, 90% within 14 minutes" tells a reader much more than "average 6 minutes".

The p-th percentile is the value below which p% of the data falls. Quartiles are the 25th, 50th (median) and 75th percentiles, and the interquartile range (IQR = Q3 − Q1) covers the middle half of the data:

import numpy as np
from statistics import quantiles

times = np.array([1, 2, 2, 3, 3, 4, 5, 7, 12, 25])
q1, median, q3 = np.percentile(times, [25, 50, 75])
print(q1, median, q3, q3 - q1)               # 2.25 3.5 6.5 4.25
print(quantiles(times.tolist(), n=4))        # [2.0, 3.5, 8.25]

With small samples, the 25th percentile usually falls between two data points, and tools must choose how to interpolate. NumPy's default method="linear" and the statistics module's default method="exclusive" use different rules, which is why the two lines above disagree on Q1 and Q3. Neither is wrong. Pick one method, name it in your report, and use the same method when comparing periods. NumPy supports several ("lower", "higher", "nearest", "midpoint", "median_unbiased" and more), and pandas' quantile() uses linear interpolation by default.

An unusually large or small observation can be:

• a data error: a timer left running overnight, a unit mix-up, a duplicated record;
• a real but rare event: a complicated research question that genuinely took an hour.

The 1.5 × IQR rule flags values above Q3 + 1.5 × IQR or below Q1 − 1.5 × IQR as worth checking. It's a screening tool, not a verdict.

q1, q3 = np.percentile(times, [25, 75])
limit = q3 + 1.5 * (q3 - q1)
print(times[times > limit])     # [25]

Investigate before deciding. Deleting errors is right; deleting real long waits would hide exactly the problem a help-desk manager needs to see. When you do exclude values, report how many and why. Percentiles and medians are robust, meaning one extreme value barely moves them, so you can often keep outliers and still describe the typical case honestly.

Distributions aren't fixed. Response times change when staffing changes, when a new booking system launches, or simply between term time and holidays. A percentile calculated last spring may not describe this autumn.

spring = np.array([2, 3, 3, 4, 5, 6, 8])
autumn = np.array([3, 4, 6, 7, 9, 12, 18])
for label, data in [("spring", spring), ("autumn", autumn)]:
    print(label, np.percentile(data, [50, 90]).round(1).tolist())

Compare comparable periods (the same weeks of different terms, not a quiet week against an exam week) and check whether the measurement itself changed, for instance a new definition of when a question counts as "answered".

For machine learning, a shift between the training data's distribution and live data is called drift. A model trained on spring response times may systematically under-predict autumn ones. Monitoring the percentiles of key features over time is a simple, effective early warning.

Task: report help-desk median and a high percentile, naming the calculation method and investigating unusually long observations.

import numpy as np

minutes = np.array([2, 3, 3, 4, 4, 5, 6, 6, 8, 9, 11, 45])
method = "linear"
p50, p90 = np.percentile(minutes, [50, 90], method=method)
q1, q3 = np.percentile(minutes, [25, 75], method=method)
long_ones = minutes[minutes > q3 + 1.5 * (q3 - q1)]

print(f"median {p50:.1f} min, 90th percentile {p90:.1f} min ({method} interpolation, n={len(minutes)})")
print("to investigate:", long_ones.tolist())

This prints median 5.5 min, 90th percentile 10.8 min (linear interpolation, n=12) and flags the 45-minute response. Before removing it, check the ticket: if it was a complex request, keep it and mention it. For the worked check, percentile calculations differ between tools because they can use different interpolation conventions.

NumPy's percentile documentation lists every method, and Python's statistics.quantiles explains its defaults. Continue with linear regression and revisit summary statistics for ML. Practise with the Python exercises and Python quiz.