Lab 7: Iterators and Generators

Due at 11:59pm on 3/10/2017.

Starter Files

Download lab07.zip. Inside the archive, you will find starter files for the questions in this lab, along with a copy of the OK autograder.

Submission

By the end of this lab, you should have submitted the lab with python3 ok --submit. You may submit more than once before the deadline; only the final submission will be graded.

  • To receive credit for this lab, you must complete Questions 1, 2, 3, 4, 5, 6 and 7 in lab07.py and submit through OK.
  • Questions 8, 9, and 10 are extra practice. They can be found in the lab07_extra.py file. It is recommended that you complete these problems on your on time.

Iterables and Iterators

In lecture, we studied several Python object interfaces, or protocols. In this lab, we will study a new protocol, the iterator protocol. Implementing this protocol allows us to use our objects in for loops! Remember the for loop? (We really hope so!)

for elem in something_iterable:
    # do something

for loops work on any object that is iterable. We previously described it as working with any sequence -- all sequences are iterable, but there are other objects that are also iterable! As it turns out, for loops are actually translated by the interpreter into the following code:

the_iterator = iter(something_iterable)
try:
    while True:
        elem = next(the_iterator)
        # do something
except StopIteration:
    pass

That is, it first calls the built-in iter function to create an iterator, saving it in some new, hidden variable (we've called it the_iterator here). It then repeatedly calls the built-in next function on this iterator to get values of elem and stops when that function raises StopIteration.

Question 1: Does it work?

Consider the following iterators. Which ones work and which ones don't? Why?

Use OK to test your knowledge with the following conceptual questions:

python3 ok -q does_it_work -u
class IteratorA:
    def __init__(self):
        self.start = 10

    def __next__(self):
        if self.start > 100:
            raise StopIteration
        self.start += 20
        return self.start

    def __iter__(self):
        return self

Does IteratorA work?

No problem, this is a beautiful iterator.

class IteratorB:
    def __init__(self):
        self.start = 5

    def __iter__(self):
        return self

Does IteratorB work?

Oh no! Where is __next__? This fails to implement the iterator interface because calling __iter__ doesn't return something that has a __next__ method.

class IteratorC:
    def __init__(self):
        self.start = 5

    def __next__(self):
        if self.start == 10:
            raise StopIteration
        self.start += 1
        return self.start

Does IteratorC work?

This also fails to implement the iterator interface. Without the __iter__ method, the for loop will error. The for loop needs to call __iter__ first because some objects might not implement the __next__ method themselves, but calling __iter__ will return an object that does.

class IteratorD:
    def __init__(self):
        self.start = 1

    def __next__(self):
        self.start += 1
        return self.start

    def __iter__(self):
        return self

Does IteratorD work?

This is technically an iterator, because it implements both __iter__ and __next__. Notice that it's an infinite sequence! Sequences like these are the reason iterators are useful. Because iterators delay computation, we can use a finite amount of memory to represent an infinitely long sequence.

Question 2: WWPD: Odds and Evens

So far, the __iter__ method of our iterators only returns self. What if we have called next a few times and then want to start at the beginning? Intuitively, we should create a new iterator that would start at the beginning. However, our current iterator implementations won't allow that.

Consider the following OddNaturalsIterator and EvenNaturalsIterator. Which implementation allows us to start a new iterator at the beginning?

class OddNaturalsIterator:
    def __init__(self):
        self.current = 1

    def __next__(self):
        result = self.current
        self.current += 2
        return result

    def __iter__(self):
        return self

class EvenNaturalsIterator:
    def __init__(self):
        self.current = 0

    def __next__(self):
        result = self.current
        self.current += 2
        return result

    def __iter__(self):
        return EvenNaturalsIterator()

Use OK to test your knowledge with the following conceptual questions:

python3 ok -q odds_evens -u
>>> odds = OddNaturalsIterator()
>>> odd_iter1 = iter(odds)
>>> odd_iter2 = iter(odds)
>>> next(odd_iter1)

______
1

>>> next(odd_iter1)

______
3

>>> next(odd_iter1)

______
5

>>> next(odd_iter2)

______
7

>>> next(odd_iter1)

______
9

>>> next(odd_iter2)

______
11
>>> evens = EvenNaturalsIterator()
>>> even_iter1 = iter(evens)
>>> even_iter2 = iter(evens)
>>> next(even_iter1)

______
0

>>> next(even_iter1)

______
2

>>> next(even_iter1)

______
4

>>> next(even_iter2)

______
0

>>> next(even_iter2)

______
2
>>> class DoubleIterator:
...     def __init__(self):
...         self.current = 2
...     def __next__(self):
...         result = self.current
...         self.current += result
...         return result
...     def __iter__(self):
...         return DoubleIterator()
>>> doubleI = DoubleIterator()
>>> dIter = iter(doubleI)
>>> next(doubleI)

______
2

>>> next(doubleI)

______
4

>>> next(dIter)

______
2

>>> next(dIter)

______
4

>>> next(doubleI)

______
8
>>> class ThreeIterator:
...     def __init__(self):
...         self.current = 10
...     def __next__(self):
...         result = self.current
...         self.current -= 3
...         return result
...     def __iter__(self):
...         return self
>>> threeI = ThreeIterator()
>>> tIter = iter(threeI)
>>> next(threeI)

______
10

>>> next(threeI)

______
7

>>> next(tIter)

______
4

>>> next(tIter)

______
1

>>> next(threeI)

______
-2

Question 3: Str

Write an iterator that takes a string as input and outputs the letters in order when iterated over.

class Str:
    def __init__(self, str):
        self.str = str
    def __iter__(self):
        return iter(self.str)

That works (why?), but just kidding.

class Str:
    """
    >>> s = Str("hello")
    >>> for char in s:
    ...     print(char)
    ...
    h
    e
    l
    l
    o
    >>> for char in s:    # a standard iterator does not restart
    ...     print(char)
    """

    "*** YOUR CODE HERE ***"

    def __init__(self, str):
        self.str = str
        self.i = -1

    def __iter__(self):
        return self

    def __next__(self):
        self.i += 1
        if self.i >= len(self.str):
            raise StopIteration
        return self.str[self.i]

Use OK to test your code:

python3 ok -q Str

Generators

A generator function returns a special type of iterator called a generator object. Generator functions have yield statements within the body of the function. Calling a generator function makes it return a generator object rather than executing the body of the function.

The reason we say a generator object is a special type of iterator is that it has all the properties of an iterator, meaning that:

  • Calling the __iter__ method makes a generator object return itself without modifying its current state.
  • Calling the __next__ method makes a generator object compute and return the next object in its sequence. If the sequence is exhausted, StopIteration is raised.
  • Typically, a generator should not restart unless it's defined that way. But calling the generator function returns a brand new generator object (like calling __iter__ on an iterable object).

However, they do have some fundamental differences:

  • An iterator is a class with __next__ and __iter__ explicitly defined, but a generator can be written as a mere function with a yield in it.
  • __next__ in an iterator uses return, but a generator uses yield.

  • A generator "remembers" its state for the next __next__ call. Therefore,

    • the first __next__ call works like this:

      1. Enter the function, run until the line with yield.
      2. Return the value in the yield statement, but remember the state of the function for future __next__ calls.

    • And subsequent __next__ calls work like this:

      1. Re-enter the function, start at the line after yield, and run until the next yield statement.
      2. Return the value in the yield statement, but remember the state of the function for future __next__ calls.

When a generator runs to the end of the function, it raises StopIteration.

Another useful tool for generators is the yield from statement (introduced in Python 3.3). yield from will yield all values from an iterator or iterable.

Question 4: WWPD: Generators

Use OK to test your knowledge with the following What would Python Display questions:

python3 ok -q generators -u
def generator():
    print("Starting here")
    i = 0
    while i < 6:
        print("Before yield")
        yield i
        print("After yield")
        i += 1
>>> g = generator()
>>> g # what type of object is this?

______
<generator object>

>>> g == iter(g) # equivalent of g.__iter__()

______
True

>>> next(g) # equivalent of g.__next__()

______
Starting here
Before yield
0

>>> next(g)

______
After yield
Before yield
1

>>> next(g)

______
After yield
Before yield
2
def generator():
    print("Starting")
    i = 2
    while i < 6:
        print("foo", i)
        yield i
        i += 1
        print("bar")
        yield i*2
        i += 2
>>> h = generator()
>>> iter(h) == h

______
True

>>> next(h)

______
Starting
foo 2
2

>>> next(h)

______
bar
6

>>> next(h)

______
foo 5
5

Question 5: Scale

Implement the generator function scale(s, k), which yields elements of the given iterable s, scaled by k.

def scale(s, k):
    """Yield elements of the iterable s scaled by a number k.

    >>> s = scale([1, 5, 2], 5)
    >>> type(s)
    <class 'generator'>
    >>> list(s)
    [5, 25, 10]

    >>> m = scale(naturals(), 2)
    >>> [next(m) for _ in range(5)]
    [2, 4, 6, 8, 10]
    """

    "*** YOUR CODE HERE ***"

    for elem in s:
        yield elem * k

Use OK to test your code:

python3 ok -q scale

Question 6: Countdown

Write both a generator function and an iterator (that is not a generator) that count down to 0.

def countdown(n):
    """
    A generator that counts down from N to 0.
    >>> for number in countdown(5):
    ...     print(number)
    ...
    5
    4
    3
    2
    1
    0
    >>> for number in countdown(2):
    ...     print(number)
    ...
    2
    1
    0
    """

    "*** YOUR CODE HERE ***"

    while n >= 0:
        yield n
        n = n - 1
class Countdown:
    """
    An iterator that counts down from N to 0.
    >>> for number in Countdown(5):
    ...     print(number)
    ...
    5
    4
    3
    2
    1
    0
    >>> for number in Countdown(2):
    ...     print(number)
    ...
    2
    1
    0
    """
    def __init__(self, cur):
        self.cur = cur

    def __next__(self):

        "*** YOUR CODE HERE ***"

        if self.cur < 0:
            raise StopIteration
        self.cur -= 1
        return self.cur + 1

    def __iter__(self):
        """So that we can use this iterator as an iterable."""
        return self

Use OK to test your code:

python3 ok -q countdown
python3 ok -q Countdown

Question 7: Hailstone

Write a generator that outputs the hailstone sequence from homework 1.

Here's a quick reminder of how the hailstone sequence is defined:

  1. Pick a positive integer n as the start.
  2. If n is even, divide it by 2.
  3. If n is odd, multiply it by 3 and add 1.
  4. Continue this process until n is 1.
def hailstone(n):
    """
    >>> for num in hailstone(10):
    ...     print(num)
    ...
    10
    5
    16
    8
    4
    2
    1
    """

    "*** YOUR CODE HERE ***"

    i = n
    while i > 1:
        yield i
        if i % 2 == 0:
            i //= 2
        else:
            i = i * 3 + 1
    yield i

Use OK to test your code:

python3 ok -q hailstone

Extra Questions

Question 8: Merge

Implement merge(s0, s1), which takes two iterables s0 and s1 whose elements are ordered. merge yields elements from s0 and s1 in sorted order, eliminating repetition. You may assume s0 and s1 themselves do not contain repeats, and that none of the elements of either are None. You may not assume that the iterables are finite; either may produce an infinite stream of results.

You will probably find it helpful to use the two-argument version of the built-in next function: next(s, v) is the same as next(s), except that instead of raising StopIteration when s runs out of elements, it returns v.

See the doctest for examples of behavior.

def merge(s0, s1):
    """Yield the elements of strictly increasing iterables s0 and s1, removing
    repeats. Assume that s0 and s1 have no repeats. You can also assume that s0
    and s1 represent infinite sequences.

    >>> m = merge([0, 2, 4, 6, 8, 10, 12, 14], [0, 3, 6, 9, 12, 15])
    >>> type(m)
    <class 'generator'>
    >>> list(m)
    [0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
    >>> def big(n):
    ...    k = 0
    ...    while True: yield k; k += n
    >>> m = merge(big(2), big(3))
    >>> [next(m) for _ in range(11)]
    [0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
    """
    i0, i1 = iter(s0), iter(s1)
    e0, e1 = next(i0, None), next(i1, None)

    "*** YOUR CODE HERE ***"

    while True:
        if e0 is None and e1 is None:
            return
        elif e0 is None:
            yield e1
            e1 = next(i1, None)
        elif e1 is None:
            yield e0
            e0 = next(i0, None)
        else:
            yield min(e0, e1)
            if e0 < e1:
                e0 = next(i0, None)
            elif e1 < e0:
                e1 = next(i1, None)
            else:
                e0, e1 = next(i0, None), next(i1, None)

Use OK to test your code:

python3 ok -q merge

Question 9: Remainder Generator

Like functions, generators can also be higher-order. For this problem, we will be writing remainders_generator, which yields a series of generator objects.

remainders_generator takes in an integer m, and yields m different generators. The first generator is a generator of multiples of m, i.e. numbers where the remainder is 0. The second, a generator of natural numbers with remainder 1 when divided by m. The last generator yield natural numbers with remainder m - 1 when divided by m.

def remainders_generator(m):
    """
    Takes in an integer m, and yields m different remainder groups
    of m.

    >>> remainders_mod_four = remainders_generator(4)
    >>> for rem_group in remainders_mod_four:
    ...     for _ in range(3):
    ...         print(next(rem_group))
    0
    4
    8
    1
    5
    9
    2
    6
    10
    3
    7
    11
    """

    "*** YOUR CODE HERE ***"

    def remainder_group(rem):
        start = rem
        while True:
            yield start
            start += m

    for rem in range(m):
        yield remainder_group(rem)

Note that if you have implemented this correctly, each of the generators yielded by remainder_generator will be infinite - you can keep calling next on them forever without running into a StopIteration exception.

Hint: Consider defining an inner generator function. What arguments should it take in? Where should you call it?

Use OK to test your code:

python3 ok -q remainders_generator

Question 10: Zip generator

For this problem, we will be writing zip_generator, which yields a series of lists, each containing the nth items of each iterable. It should stop when the smallest iterable runs out of elements.

def zip(*iterables):
    """
    Takes in any number of iterables and zips them together.
    Returns a generator that outputs a series of lists, each
    containing the nth items of each iterable.
    >>> z = zip([1, 2, 3], [4, 5, 6], [7, 8])
    >>> for i in z:
    ...     print(i)
    ...
    [1, 4, 7]
    [2, 5, 8]
    """

    "*** YOUR CODE HERE ***"

    iterators = [iter(iterable) for iterable in iterables]
    while True:
        yield [next(iterator) for iterator in iterators]

Use OK to test your code:

python3 ok -q zip