Working with Web APIs in Python

Install requests in your virtual environment with pip install requests, then:

import requests

response = requests.get("https://api.example.com/books/42", timeout=10)
print(response.status_code)    # 200
book = response.json()
print(book["title"])

• requests.get(url) sends an HTTP GET request, which asks for data.
• response.status_code is a number describing the result.
• response.json() converts the JSON body into Python dictionaries and lists.
• timeout=10 stops waiting after 10 seconds instead of hanging for ever.

Codes in the 200s mean success, 400s mean a problem with your request, and 500s a problem on the server. response.raise_for_status() raises an HTTPError for any 400 or 500 code, so you can handle failures with try and except instead of checking numbers by hand.

Many APIs take options in the address, after a ?. Let requests build them safely with params:

r = requests.get("https://api.example.com/books",
                 params={"q": "python", "limit": 5}, timeout=10)
# requests visits .../books?q=python&limit=5

To send data, use POST with json=, which converts a dictionary to JSON and sets the right header:

r = requests.post("https://api.example.com/orders",
                  json={"item": "mug", "qty": 2}, timeout=10)

Other methods follow the same pattern: requests.put(), patch() and delete().

Most APIs need a key to identify you, usually sent in a header:

import os
key = os.environ["WEATHER_API_KEY"]
r = requests.get(url, headers={"Authorization": f"Bearer {key}"}, timeout=10)

Never write keys directly in your code: if the file is shared or pushed to GitHub, anyone can use your account. Keep them in environment variables or a .env file that's listed in .gitignore. Read each API's documentation for its exact header name and rate limits.

When you make many requests to the same API, a requests.Session reuses the network connection and remembers headers, which is faster and keeps your code shorter:

import requests

session = requests.Session()
session.headers["Authorization"] = f"Bearer {key}"

page = 1
books = []
while True:
    r = session.get("https://api.example.com/books",
                    params={"page": page}, timeout=10)
    r.raise_for_status()
    data = r.json()
    books.extend(data["results"])
    if not data["next"]:
        break
    page += 1

Most APIs return results in pages, so a loop like this keeps fetching until the response says there is no next page. Field names such as results and next vary between APIs; check the documentation.

If you receive status 429, you are sending requests too quickly. Many APIs include a Retry-After header saying how many seconds to wait. Respect it with time.sleep() before trying again, and don't retry indefinitely.