Python HTTP Requests and REST APIs

The method tells the server what you intend:

requests.get(url, params={"night": "Saturday"}, timeout=10)
requests.post(url, json={"object": "M31", "night": "Saturday"}, timeout=10)
requests.patch(f"{url}/17", json={"status": "observed"}, timeout=10)
requests.delete(f"{url}/17", timeout=10)

These meanings are conventions defined by HTTP; each API documents exactly what its endpoints do, which fields they accept and which status codes they return. Read the documentation rather than assuming, for example, that DELETE is allowed or that POST returns the created object.

Build requests from parts instead of assembling URLs by hand:

• params={...} adds a correctly encoded query string: ?night=Saturday&limit=20.
• json={...} sends a JSON body and sets Content-Type: application/json.
• headers={...} adds headers such as Accept or authentication tokens.

Always pass timeout=. Without it, requests may wait indefinitely for a server that never answers, freezing your program. timeout=10 limits the connection attempt and each wait for data to 10 seconds; timeout=(3, 10) sets them separately.

A timeout is not a total deadline: a server that sends a byte every 9 seconds can keep a download going much longer. If you need an overall time limit, track elapsed time yourself. When a timeout fires, requests raises requests.Timeout; connection failures raise requests.ConnectionError. Both inherit from requests.RequestException, which is the type to catch for "the request didn't complete".

Treat a response in two separate steps: check the status, then decode the body.

response = requests.get(url, timeout=10)
response.raise_for_status()           # raises HTTPError for 4xx and 5xx
try:
    data = response.json()
except requests.JSONDecodeError:
    raise ValueError(f"Expected JSON, got: {response.text[:100]!r}")

A 200 status doesn't guarantee JSON. A misconfigured proxy, a maintenance page or an endpoint that returns CSV can all answer 200 with a body json() can't parse. Conversely, error responses often carry useful JSON describing the problem; read response.json() inside your error handling when the API documents it.

Useful response attributes: status_code, headers (case-insensitive), text (decoded string), content (raw bytes), url (the final URL after redirects) and elapsed. Validate the decoded data's shape before relying on it, as in the JSON lesson.

APIs rarely return thousands of records at once. They return a page and tell you how to get the next one, in one of a few ways:

• a next link in the body ("next": "https://.../observations?page=3") or a Link header;
• a cursor or token to send back ("next_cursor": "abc123");
• page or offset numbers you increment yourself.

def fetch_all(url, session):
    results = []
    while url:
        response = session.get(url, timeout=10)
        response.raise_for_status()
        page = response.json()
        results.extend(page["items"])
        url = page.get("next")          # None on the last page
    return results

Reading only the first page and treating it as complete is a classic data-loss bug: the report looks plausible but silently omits most records. Follow the API's documented scheme, stop when it signals the end, and add a safety limit on the number of pages so a misbehaving API can't loop forever. A requests.Session reuses the connection across pages, which is noticeably faster.

Task: use a real documented schedule endpoint in place of the illustrative URL and handle status, decoding and pagination. Write the client as a function you can point at any compatible API:

import requests

def fetch_schedule(start_url, max_pages=50):
    items, url = [], start_url
    with requests.Session() as session:
        for _ in range(max_pages):
            if not url:
                return items
            response = session.get(url, timeout=10)
            response.raise_for_status()
            try:
                page = response.json()
            except requests.JSONDecodeError:
                raise ValueError(f"{url} did not return JSON")
            items.extend(page.get("items", []))
            url = page.get("next")
    raise RuntimeError(f"Stopped after {max_pages} pages; check the pagination rules")

Point it at the local server from the example above to test it, then at a real API whose pagination uses a next link, adapting the field names to its documentation. For the worked check, a request timeout exists to bound a wait for network activity, so an unresponsive server can't freeze the program.

The requests documentation covers sessions, timeouts and errors, and MDN's HTTP reference explains methods and status codes. Continue with API authentication and errors. Practise with the Python exercises and Python quiz.