Python Iterators and Generators

Understand how for loops really work, write generator functions with yield, use generator expressions and process large data one item at a time.

Lesson 40 of 107Advanced18 min read

You've used for loops on lists, strings, dictionaries, ranges and files. They all behave the same way: the loop asks for the next item until there are none left. That shared behaviour has a name, the iterator protocol, and understanding it unlocks one of Python's most useful features: generators.

A generator is a function that produces values one at a time, pausing after each one until the next is needed. Because it never builds the whole collection in memory, a generator can work through a file with ten million lines, or even an endless stream of values, using almost no memory.

In this lesson you'll see what happens inside a for loop, write generator functions with yield, use compact generator expressions, build data pipelines that process big files efficiently, and meet the itertools module.

Iterables and iterators

An iterable is anything you can loop over: a list, string, dict, file or range. Calling iter() on it gives an iterator, an object that hands out items one by one with next():

Python
colours = ["red", "green"]
it = iter(colours)
print(next(it))    # red
print(next(it))    # green
next(it)           # StopIteration

When there's nothing left, the iterator raises StopIteration. A for loop does exactly this behind the scenes and stops quietly when it sees StopIteration. An iterator can only be used once: after it's exhausted, it stays empty.

Generator functions and yield

A function that contains yield is a generator function. Calling it doesn't run its body; it returns a generator object. Each next() runs the code until the next yield, hands out that value and pauses, keeping all its local variables:

Python
def countdown(n):
    while n > 0:
        yield n
        n -= 1

for x in countdown(3):
    print(x)      # 3, 2, 1

When the function reaches its end, the generator is finished. Compare this with building and returning a whole list: the generator produces each value only when the loop asks for it.

Generator expressions

A generator expression looks like a list comprehension but uses round brackets:

Python
squares_list = [n * n for n in range(1_000_000)]   # builds a million items
squares_gen = (n * n for n in range(1_000_000))    # builds nothing yet
total = sum(n * n for n in range(1_000_000))

When a generator expression is the only argument to a function, you can drop the extra brackets, as in the sum() call. Use a list when you need to index, reuse or count the items; use a generator when you only need to loop over them once.

yield from and sending a generator's work elsewhere

When one generator needs to hand out every value from another iterable, yield from does it in one line instead of a loop:

Python
def all_lines(*paths):
    for path in paths:
        with open(path, encoding="utf-8") as file:
            yield from file

for line in all_lines("jan.log", "feb.log"):
    print(line.rstrip())

yield from file is equivalent to for line in file: yield line. It also keeps each file open only while its lines are being read, because the with block finishes when that file is exhausted.

Generators can also tidy code that builds a list only to loop over it once. If a function ends with return results after a loop of results.append(...), replacing each append with yield often makes it simpler and lighter on memory, provided callers only need to iterate. The iterables and iterators lesson goes further, showing how to write your own iterator classes with __iter__ and __next__.

Lazy pipelines and itertools

Because generators are lazy, you can chain them into a pipeline where each item flows through every stage before the next item is read. Only one line is in memory at a time, however big the file.

The itertools module offers ready-made iterator tools:

  • count(start) counts for ever; cycle(items) repeats items for ever.
  • islice(it, n) takes the first n items from any iterator.
  • chain(a, b) joins iterables one after another.
  • groupby(), combinations() and permutations() cover common data tasks.

Python Generators Syntax

Generator patterns

it = iter(iterable)
next(it)
next(it, default)

def gen():
    yield value

(expr for item in iterable if condition)

from itertools import count, islice, chain, cycle
list(islice(count(1), 5))   # [1, 2, 3, 4, 5]

Python Generators Summary

  • An iterable can be looped over; iter() gives an iterator and next() takes the next item.
  • Iterators raise StopIteration when finished, and can only be used once.
  • A function with yield is a generator: it pauses after each value and resumes on the next request.
  • Generator expressions (x for x in ...) produce values lazily without building a list.
  • Chain generators into pipelines to process huge data with little memory; see itertools for more tools.

Python Generators Examples

Each example shows the code, the output it produces and what each part does.

Example 1

A generator that pauses

Python
def make_tickets(prefix):
    number = 1
    while True:
        print("  (creating ticket)")
        yield f"{prefix}-{number:03d}"
        number += 1

tickets = make_tickets("SUP")
print(next(tickets))
print(next(tickets))
Output
  (creating ticket)
SUP-001
  (creating ticket)
SUP-002
Line-by-line explanation
  1. Line 3while True is safe here, because the generator pauses at every yield.
  2. Line 5The value is handed out, and the function waits at this line.
  3. Line 8Calling the function runs nothing yet; the first message appears only at the first next().
Example 2

Processing a big log file lazily

Python
def read_lines(path):
    with open(path) as file:
        for line in file:
            yield line.rstrip("\n")

def errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

lines = read_lines("server.log")
for line in errors(lines):
    print(line)
Output
2025-03-05 10:14:02 ERROR Payment service timeout
2025-03-05 11:40:57 ERROR Disk 91% full
Line-by-line explanation
  1. Lines 1 to 4Yields one line at a time; the file is never loaded into memory in one go.
  2. Lines 6 to 9A second generator filters the first. Each line flows through both before the next is read.
  3. Lines 11 to 13This pipeline works the same on a file of 100 lines or 100 million lines.
Example 3

Generator expressions and itertools

Python
from itertools import islice, cycle

prices = [4.5, 12.0, 3.25, 8.0]
print(sum(p for p in prices if p > 4))

players = cycle(["Ana", "Ben", "Cy"])
turns = list(islice(players, 5))
print(turns)
Output
24.5
['Ana', 'Ben', 'Cy', 'Ana', 'Ben']
Line-by-line explanation
  1. Line 4Adds the prices above 4 without building a temporary list.
  2. Line 6cycle repeats the list for ever, which is perfect for taking turns.
  3. Line 7islice safely takes just the first five values from the endless iterator.

Common Python Generators Mistakes

Using a generator twice

Don't
evens = (n for n in range(10) if n % 2 == 0)
print(sum(evens))    # 20
print(max(evens))    # ValueError: empty
Do
evens = [n for n in range(10) if n % 2 == 0]
print(sum(evens), max(evens))   # 20 8

Returning instead of yielding

Don't
def first_three():
    for n in range(3):
        return n          # stops after the first value
Do
def first_three():
    for n in range(3):
        yield n

Calling list() on an endless generator

Don't
from itertools import count
numbers = list(count(1))     # never finishes
Do
from itertools import count, islice
numbers = list(islice(count(1), 10))

Python Generators in Real Projects

Generators in practice

Data engineers use generator pipelines to stream through huge log and CSV files, web frameworks stream large responses to the browser in chunks, and libraries that read databases fetch rows in batches rather than all at once. Even range(), open() and dict.items() are lazy, which is why looping over them is fast and memory-friendly.

Python Generators FAQ

What is the difference between yield and return?

return sends back one value and ends the function for good. yield sends back a value and pauses, so the function can continue from the same point the next time a value is requested. A function containing yield returns a generator object rather than running its body immediately.

What is the difference between an iterable and an iterator?

An iterable is something you can loop over, such as a list, and it can give you a fresh iterator each time you call iter() on it. An iterator is the object that tracks the current position and returns items with next(). Lists can be looped over many times; an individual iterator is used up after one pass.

When should I use a generator instead of a list?

Use a generator when the data is large, comes from a file or network stream, or might be endless, and you only need to go through it once in order. Use a list when you need its length, random access by index, sorting, or several passes over the same values.

How do I get the first few items from a generator?

Use itertools.islice(gen, 5) for the first five, or call next(gen) repeatedly. next(gen, None) returns None instead of raising StopIteration when the generator is empty, which is handy for "first match or nothing" lookups.

Can I check a generator's length?

Not without consuming it, because a generator doesn't know how many values it will produce until it runs. len(gen) raises TypeError. If you need the count and the values, convert to a list first, or count while you process with enumerate().

Python Generators Exercises

Answer each one and check it instantly. If you get stuck, open the hint before the answer.

Fill in the blankseasy

Complete the generator so it produces the numbers 1, 2 and 3.

def one_two_three():
    for n in [1, 2, 3]:
         n
Predict the outputeasy

What does this print?

it = iter("hi")
print(next(it), next(it))
Predict the outputmedium

What does this print?

def gen():
    yield 1
    yield 2

g = gen()
print(list(g), list(g))
Multiple choicemedium

Which creates a generator expression?

Multiple choicehard

Why is a generator good for a 20 GB log file?

Python Generators Quiz

5 questions to check what you learned.

  1. Which keyword turns a function into a generator?

    • return
    • yield
    • async
    • gen

  2. What does next() raise when an iterator has no more items?

    • IndexError
    • ValueError
    • StopIteration
    • None

  3. What does lazy evaluation mean?

    • Code runs slowly
    • Errors are ignored
    • Values are produced only when they're needed
    • Functions run in the background

  4. What does list(islice(count(5), 3)) give?

    • [0, 1, 2]
    • [5, 5, 5]
    • [3, 4, 5]
    • [5, 6, 7]

  5. What happens to a generator's local variables between next() calls?

    • They're reset
    • They're kept, because the function is paused, not finished
    • They become global
    • They're deleted

Related Python topics

18Loops (for & while)Repeat code with for loops, while loops and range(), and control them with break and continue.29Comprehensions & LambdaBuild new lists, dictionaries and sets in one readable line, and use small lambda functions with sorted, map and filter.31Files (Text, CSV, JSON)Read and write text files safely with open() and with, and work with CSV and JSON data.
Esc
↑↓ to navigateEnter to open