Iterators & Generators
An iterator produces values one at a time, only when asked. Generators are the easiest way to build your own iterators, and they let you work with huge or even endless streams of data without filling up memory.
Iterables and iterators
An iterable is anything you can loop over: lists, strings, tuples, dictionaries, files and ranges. An iterator is the object that actually hands out the values, one by one, and remembers where it is. You get an iterator from an iterable with iter(), and ask for the next value with next().
marks = [72, 85, 90] it = iter(marks) print(next(it)) print(next(it)) print(next(it)) print(next(it, "no more marks")) # default instead of an error
Without the default, a fourth next(it) raises StopIteration. That's how an iterator signals "I'm finished".
What a for loop really does
Every for loop uses this protocol behind the scenes. for mark in marks: is roughly equivalent to:
it = iter(marks)
while True:
try:
mark = next(it)
except StopIteration:
break
print(mark)
Writing an iterator class
A class becomes an iterator if it has __iter__ (returning the iterator, usually self) and __next__ (returning the next value or raising StopIteration).
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for n in Countdown(3):
print(n)
print("Lift off!")
That works, but it's a lot of code for a simple idea. Generators make it much shorter.
Generator functions and yield
A generator function looks like a normal function but uses yield instead of return. Calling it doesn't run the body; it returns a generator object. Each time the next value is requested, the function runs until the next yield, hands over that value and pauses, keeping all its local variables.
def countdown(start):
while start > 0:
yield start
start -= 1
print(list(countdown(5)))
This pausing is easy to see with some print statements:
def steps():
print("Starting")
yield 1
print("Middle")
yield 2
print("End")
for value in steps():
print("Got", value)
The function's code is interleaved with the loop's code: it only runs as far as needed to produce the next value. When the function body finishes, the generator raises StopIteration automatically and the loop ends.
Generator expressions
A generator expression looks like a list comprehension with round brackets. It produces values lazily instead of building the whole list.
squares = (n * n for n in range(1, 6)) print(next(squares), next(squares)) print(list(squares)) # the rest total = sum(n * n for n in range(1, 1001)) print(total)
When a generator expression is the only argument to a function, as in sum(...), you can drop the extra brackets.
Why generators save memory
A list stores every item at once. A generator stores only its current state, no matter how many values it will produce. Summing the squares of the first hundred million numbers with a list comprehension would build a list using several gigabytes of memory; the generator version uses almost none.
This makes generators ideal for reading large files line by line, processing data in a pipeline, or producing values that never end:
from itertools import islice
def fibonacci():
a, b = 0, 1
while True: # an infinite generator
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 10))) # take just the first 10
Generators can also be chained into a pipeline, where each step processes one item at a time:
def read_amounts(lines):
for line in lines:
yield int(line.strip())
def only_large(amounts, limit):
for amount in amounts:
if amount >= limit:
yield amount
lines = ["250\n", "1800\n", "90\n", "5000\n"] # imagine a huge file here
print(list(only_large(read_amounts(lines), 1000)))
yield from
yield from iterable yields every value from another iterable, a tidy shortcut for a loop that yields each item.
def all_items():
yield from ["pen", "book"]
yield from range(2)
print(list(all_items()))
Iterators and generators can only be used once. After you've looped through them, they're empty:
evens = (n for n in range(6) if n % 2 == 0) print(list(evens)) print(list(evens))
If you need the values more than once, store them in a list, or call the generator function again to get a fresh generator.
You can't use len() or indexing such as gen[0] on a generator, because it doesn't know its values in advance. Convert to a list first if you need those, or use itertools.islice to take a slice.
Key points
iter()makes an iterator;next()gets the next value;StopIterationmeans it's finished.- A function containing
yieldis a generator: it pauses after each value and resumes on request. - Generator expressions
(x for x in ...)are lazy versions of list comprehensions. - Generators use very little memory and can even be infinite, but they can only be used once.
Tip: use the ← and → keys to move between pages.
