Lab 11: Iterators and Generators

Due at 11:59pm on Friday, 04/13/2018.

Starter Files

Download lab11.zip. Inside the archive, you will find starter files for the questions in this lab, along with a copy of the Ok autograder.

Submission

By the end of this lab, you should have submitted the lab with python3 ok --submit. You may submit more than once before the deadline; only the final submission will be graded. Check that you have successfully submitted your code on okpy.org.

  • To receive credit for this lab, you must complete Questions 1-4 and submit through OK. Started code for Questions 1-2 can be found in lab11.py.
  • Questions 5-9 are extra practice. They can be found in the lab11_extra.py file. It is recommended that you complete these problems on your own time.

Topics

Iterables and Iterators

Iterables are objects that can be iterated through. One construct that we can use to iterate through an iterable is a for loop:

for elem in iterable:
    # do something

for loops work on any object that is iterable. We previously described it as working with any sequence -- all sequences are iterable, but there are other objects that are also iterable! As it turns out, for loops are actually translated by the interpreter something similar to the following code:

iterator = iter(iterable)
try:
    while True:
        elem = next(iterator)
        # do something
except StopIteration:
    pass

Here's a breakdown of what's happening:

  • First, the built-in iter function is called on the iterable to create a corresponding iterator, an object used to iterate through the iterable by keeping track of which element is next in the sequence.
  • To get the next element in the sequence, the built-in next function is called on this iterator.
  • When next is called but there are no elements left in the iterator, a StopIteration error is raised. In the for loop construct, this exception is caught and execution can continue.

Calling iter on an iterable multiple times returns a new iterator each time with distinct states (otherwise, you'd never be able to iterate through a iterable more than once). You can also call iter on the iterator itself, which will just return the same iterator without changing its state. However, note that you cannot call next directly on an iterable.

Let's see the iter and next functions in action with an iterable we're already familiar with -- a list.

>>> lst = [1, 2, 3, 4]
>>> next(lst)
TypeError: 'list' object is not an iterator
>>> list_iter = iter(lst)
>>> list_iter
<list_iterator object ...>
>>> next(list_iter)
1
>>> next(list_iter)
2
>>> next(iter(list_iter))   # Calling iter on an iterator returns itself
3
>>> list_iter2 = iter(lst)
>>> next(list_iter2)        # Second iterator has new state
1
>>> next(list_iter)         # First iterator is unaffected by second iterator
4
>>> next(list_iter)         # No elements left!
StopIteration
>>> lst                     # Original iterable is unaffected
[1, 2, 3, 4]

Since you can call iter on iterators, this tells us that that they are also iterables. You can use iterators wherever you can use iterables, but note that since iterators keep their state, they're only good to iterate through once:

>>> list_iter = iter([4, 3, 2, 1])
>>> for e in list_iter:
...     print(e)
4
3
2
1
>>> next(list_iter)
StopIteration

Analogy: An iterable is like a book (one can flip through the pages) and an iterator for a book would be a bookmark (saves the position and can locate the next page). Calling iter on a book gives you a new bookmark independent of other bookmarks, but calling iter on a bookmark gives you the bookmark itself, without changing its position at all. Calling next on the bookmark moves it to the next page, but does not change the pages in the book. Calling next on the book wouldn't make sense semantically.

Iterable uses

We know that lists are one type of built-in iterable objects. You may have also encounter the range(start, end) function, which creates an iterable of ascending integers from start (inclusive) to end (exclusive).

>>> for x in range(2, 6):
...     print(x)
...
2
3
4
5

Ranges are useful for many things, including performing some operations for a particular number of iterations or iterating through the indices of a list.

There are also some built-in functions that take in iterables and return useful results:

  • map(f, iterable) - Creates iterator over f(x) for x in iterable
  • filter(f, iterable) - Creates iterator over x for x in iterable if f(x)
  • zip(iter1, iter2) - Creates iterator over co-indexed pairs (x, y) from both input iterables
  • reversed(iterable) - Creates iterator sequence containing elements of iterable in reverse order
  • list(iterable) - Creates a list containing all x in iterable
  • tuple(iterable) - Creates a tuple containing all x in iterable
  • sorted(iterable) - Creates a sorted list containing all x in iterable

Generators

A generator function returns a special type of iterator called a generator. Generator functions have yield statements within the body of the function instead of return statements. Calling a generator function will return a generator object and will not execute the body of the function.

For example, let's consider the following generator function:

def countdown(n):
    print("Beginning countdown!")
    while n >= 0:
        yield n
        n -= 1
    print("Blastoff!")

Calling countdown will return a generator object that counts down from n to 0. Since generators are iterators, we can call iter on the resulting object, which will simply return the same object. Note that the body is not executed at this point; nothing is printed and no numbers are output.

>>> c = countdown(5)
>>> c
<generator object countdown ...>
>>> c is iter(c)
True

So how is the counting done? Again, since generators are a type of iterator, we can also call next on them! The first time next is called, execution begins at the first line of the function body and continues until the yield statement is reached. The result of evaluating the expression in the yield statement is returned. The following interactive session continues from the one above.

>>> next(c)
Beginning countdown!
5

Unlike functions we've seen before in this course, generator functions can remember their state. On any consecutive calls to next, execution picks up from the line after the yield statement that was previously executed. Like the first call to next, execution will continue until the next yield statement is reached.

>>> next(c)
4
>>> next(c)
3

Can you predict what would happen if we continue to call next on c 4 more times?

The next 3 calls to next will continue to yield consecutive descending integers until 0. On the following call, a StopIteration error will be raised because there are no more values to yield (i.e. the end of the function body was reached before hitting a yield statement). 
>>> next(c)
2
>>> next(c)
1
>>> next(c)
0
>>> next(c)
Blastoff!
StopIteration

Separate calls to countdown will create distinct generator objects with their own state. Usually, generators shouldn't restart. If you'd like to reset the sequence, create another generator object by calling the generator function again.

>>> c1, c2 = countdown(5), countdown(5)
>>> c1 is c2
False
>>> next(c1)
5
>>> next(c2)
5

Here is a summary of the above:

  • A generator function has a yield statement and returns a generator object.
  • Calling the iter function on a generator object returns the same object without modifying its current state.
  • The body of a generator function is not evaluated until next is called on a resulting generator object. Calling the next function on a generator object computes and returns the next object in its sequence. If the sequence is exhausted, StopIteration is raised.

  • A generator "remembers" its state for the next next call. Therefore,

    • the first next call works like this:

      1. Enter the function and run until the line with yield.
      2. Return the value in the yield statement, but remember the state of the function for future next calls.

    • And subsequent next calls work like this:

      1. Re-enter the function, start at the line after yield, and run until the next yield statement.
      2. Return the value in the yield statement, but remember the state of the function for future next calls.
  • A generator should not restart unless it's defined that way. But calling the generator function returns a brand new generator object (like calling iter on an iterable object).

Another useful tool for generators is the yield from statement (introduced in Python 3.3). yield from will yield all values from an iterator or iterable.

>>> def gen_list(lst):
...     yield from lst
...
>>> g = gen_list([1, 2, 3, 4])
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
4
>>> next(g)
StopIteration

Required Questions

WWPD

Q1: WWPD: Iterators

Use Ok to test your knowledge with the following "What Would Python Display?" questions:

python3 ok -q iterators -u

Enter Error if you believe an error occurs, StopIteration if a StopIteration exception is raised, and Iterator if the output is a iterator object.

>>> s = [1, 2, 3, 4]
>>> t = iter(s)
>>> next(s)

______
TypeError

>>> next(t)

______
1

>>> next(t)

______
2

>>> iter(s)

______
Iterator

>>> next(iter(s))

______
1

>>> next(iter(t))

______
3

>>> next(iter(s))

______
1

>>> next(iter(t))

______
4

>>> next(t)

______
StopIteration
>>> r = range(6)
>>> r_iter = iter(r)
>>> next(r_iter)

______
0

>>> [x + 1 for x in r]

______
[1, 2, 3, 4, 5, 6]

>>> [x + 1 for x in r_iter]

______
[2, 3, 4, 5, 6]

>>> next(r_iter)

______
StopIteration

>>> list(range(-2, 4))   # Converts an iterable into a list

______
[-2, -1, 0, 1, 2, 3]
>>> map_iter = map(lambda x : x + 10, range(5))
>>> next(map_iter)

______
10

>>> next(map_iter)

______
11

>>> list(map_iter)

______
[12, 13, 14]

>>> for e in filter(lambda x : x % 2 == 0, range(1000, 1008)):
...     print(e)
...

______
1000
1002
1004
1006

>>> [x + y for x, y in zip([1, 2, 3], [4, 5, 6])]

______
[5, 7, 9]

>>> for e in zip([10, 9, 8], range(3)):
...   print(tuple(map(lambda x: x + 2, e)))
...

______
(12, 2)
(11, 3)
(10, 4)

Q2: WWPD: Generators

Use Ok to test your knowledge with the following What would Python Display questions:

python3 ok -q generators -u

Enter Error if you believe an error occurs, Function if the output is a function object, and Generator if the output is a generator object.

def gen():
    print("Starting here")
    i = 0
    while i < 6:
        print("Before yield")
        yield i
        print("After yield")
        i += 1
>>> next(gen)

______
TypeError

>>> gen

______
<function gen ...>

>>> g = gen()
>>> g

______
<generator object gen ...>

>>> g == iter(g)

______
True

>>> next(g)

______
Starting here
Before yield
0

>>> next(g)

______
After yield
Before yield
1

>>> next(g)

______
After yield
Before yield
2

>>> g2 = gen()
>>> next(g2)

______
Starting here
Before yield
0

>>> iter(g2)

______
<generator object gen ...>

>>> next(iter(g))

______
After yield
Before yield
3

>>> next(gen())

______
Starting here
Before yield
0

Coding Practice

Q3: Scale

Implement the generator function scale(s, k), which yields elements of the given iterable s, scaled by k. As an extra challenge, try writing this function using a yield from statement!

def scale(s, k):
    """Yield elements of the iterable s scaled by a number k.

    >>> s = scale([1, 5, 2], 5)
    >>> type(s)
    <class 'generator'>
    >>> list(s)
    [5, 25, 10]

    >>> m = scale(naturals(), 2)
    >>> [next(m) for _ in range(5)]
    [2, 4, 6, 8, 10]
    """

    "*** YOUR CODE HERE ***"

    for elem in s:
        yield elem * k

# Alternate solution
def scale(s, k):
    yield from map(lambda x: x*k, s)

Use Ok to test your code:

python3 ok -q scale

Q4: Trap

Write a generator function that yields the first k values in iterable s, but raises a ValueError exception if any more values are requested. You may assume that s has at least k values.

def trap(s, k):
    """Return a generator that yields the first K values in iterable S,
    but raises a ValueError exception if any more values are requested.

    >>> t = trap([3, 2, 1], 2)
    >>> next(t)
    3
    >>> next(t)
    2
    >>> next(t)
    ValueError
    >>> list(trap(range(5), 5))
    ValueError
    >>> t2 = trap(map(abs, reversed(range(-6, -4))), 2)
    >>> next(t2)
    5
    >>> next(t2)
    6
    >>> next(t2)
    ValueError
    """

    "*** YOUR CODE HERE ***"

    t = iter(s)
    while k > 0:
        yield next(t)
        k -= 1
    raise ValueError("It's a trap!")

Use Ok to test your code:

python3 ok -q trap

Optional Questions

Q5: Hailstone

Write a generator that outputs the hailstone sequence from homework 1.

Here's a quick reminder of how the hailstone sequence is defined:

  1. Pick a positive integer n as the start.
  2. If n is even, divide it by 2.
  3. If n is odd, multiply it by 3 and add 1.
  4. Continue this process until n is 1.
def hailstone(n):
    """
    >>> for num in hailstone(10):
    ...     print(num)
    ...
    10
    5
    16
    8
    4
    2
    1
    """

    "*** YOUR CODE HERE ***"

    while n > 1:
        yield n
        if n % 2 == 0:
            n //= 2
        else:
            n = n * 3 + 1
    yield n

Use Ok to test your code:

python3 ok -q hailstone

Q6: Repeated

Implement a function (not a generator function) that returns the first value in iterable t that appears k times in a row.

def repeated(t, k):
    """Return the first value in iterable T that appears K times in a row.

    >>> s = [3, 2, 1, 2, 1, 4, 4, 5, 5, 5]
    >>> repeated(trap(s, 7), 2)
    4
    >>> repeated(trap(s, 10), 3)
    5
    >>> print(repeated([4, None, None, None], 3))
    None
    """
    assert k > 1

    "*** YOUR CODE HERE ***"

    count = 0
    last_item = None

    for item in t:
        if item == last_item:
            count += 1
        else:
            last_item = item
            count = 1

        if count == k:
            return item

Use Ok to test your code:

python3 ok -q repeated

Q7: Merge

Implement merge(s0, s1), which takes two iterables s0 and s1 whose elements are ordered. merge yields elements from s0 and s1 in sorted order, eliminating repetition. You may assume s0 and s1 themselves do not contain repeats, and that none of the elements of either are None. You may not assume that the iterables are finite; either may produce an infinite stream of results.

You will probably find it helpful to use the two-argument version of the built-in next function: next(s, v) is the same as next(s), except that instead of raising StopIteration when s runs out of elements, it returns v.

See the doctest for examples of behavior.

def merge(s0, s1):
    """Yield the elements of strictly increasing iterables s0 and s1, removing
    repeats. Assume that s0 and s1 have no repeats. s0 or s1 may be infinite
    sequences.

    >>> m = merge([0, 2, 4, 6, 8, 10, 12, 14], [0, 3, 6, 9, 12, 15])
    >>> type(m)
    <class 'generator'>
    >>> list(m)
    [0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
    >>> def big(n):
    ...    k = 0
    ...    while True: yield k; k += n
    >>> m = merge(big(2), big(3))
    >>> [next(m) for _ in range(11)]
    [0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
    """
    i0, i1 = iter(s0), iter(s1)
    e0, e1 = next(i0, None), next(i1, None)

    "*** YOUR CODE HERE ***"

    while True:
        if e0 is None and e1 is None:
            return
        elif e0 is None:
            yield e1
            e1 = next(i1, None)
        elif e1 is None:
            yield e0
            e0 = next(i0, None)
        else:
            yield min(e0, e1)
            if e0 < e1:
                e0 = next(i0, None)
            elif e1 < e0:
                e1 = next(i1, None)
            else:
                e0, e1 = next(i0, None), next(i1, None)

Use Ok to test your code:

python3 ok -q merge

Q8: Remainder Generator

Like functions, generators can also be higher-order. For this problem, we will be writing remainders_generator, which yields a series of generator objects.

remainders_generator takes in an integer m, and yields m different generators. The first generator is a generator of multiples of m, i.e. numbers where the remainder is 0. The second, a generator of natural numbers with remainder 1 when divided by m. The last generator yield natural numbers with remainder m - 1 when divided by m.

def remainders_generator(m):
    """
    Takes in an integer m, and yields m different remainder groups
    of m.

    >>> remainders_mod_four = remainders_generator(4)
    >>> for rem_group in remainders_mod_four:
    ...     for _ in range(3):
    ...         print(next(rem_group))
    0
    4
    8
    1
    5
    9
    2
    6
    10
    3
    7
    11
    """

    "*** YOUR CODE HERE ***"

    def remainder_group(rem):
        start = rem
        while True:
            yield start
            start += m

    for rem in range(m):
        yield remainder_group(rem)

Note that if you have implemented this correctly, each of the generators yielded by remainder_generator will be infinite - you can keep calling next on them forever without running into a StopIteration exception.

Hint: Consider defining an inner generator function. What arguments should it take in? Where should you call it?

Use Ok to test your code:

python3 ok -q remainders_generator

Q9: Zip Generator

For this problem, we will be writing zip_generator, which yields a series of lists, each containing the nth items of each iterable. It should stop when the smallest iterable runs out of elements.

def zip_generator(*iterables):
    """
    Takes in any number of iterables and zips them together.
    Returns a generator that outputs a series of lists, each
    containing the nth items of each iterable.
    >>> z = zip_generator([1, 2, 3], [4, 5, 6], [7, 8])
    >>> for i in z:
    ...     print(i)
    ...
    [1, 4, 7]
    [2, 5, 8]
    """

    "*** YOUR CODE HERE ***"

    iterators = [iter(iterable) for iterable in iterables]
    while True:
        yield [next(iterator) for iterator in iterators]

Use Ok to test your code:

python3 ok -q zip_generator