# Lab 10: Iterators and Generators

*Due at 11:59pm on Friday, 07/27/2018.*

*Lab Check-in 6 questions here.*

## Starter Files

Download lab10.zip. Inside the archive, you will find starter files for the questions in this lab, along with a copy of the Ok autograder.

## Submission

By the end of this lab, you should have submitted the lab with
`python3 ok --submit`

. You may submit more than once before the
deadline; only the final submission will be graded.
Check that you have successfully submitted your code on
okpy.org.

- To receive credit for this lab, you must complete Questions 1-4 and submit through OK. Starter code for Questions 3-4 can be found in lab10.py.
- Questions 5-9 are
*extra practice*. They can be found in the lab10_extra.py file. It is recommended that you complete these problems on your own time.

# Topics

## Iterables and Iterators

An iterable is any object that can be iterated through, or gone through one element at a time. One construct that we've used to iterate through an iterable is a for loop:

```
for elem in iterable:
# do something
```

`for`

loops work on any object that is *iterable*. We previously described it
as working with any sequence -- all sequences are iterable, but there are other
objects that are also iterable! We define an **iterable** as an object on which
calling the built-in function `iter`

function returns an *iterator*. An
iterator is another type of object that allows us to iterate through an
iterable by keeping track of which element is next in the sequence.

To illustrate this, consider the following block of code, which does the exact same thing as a the for statement above:

```
iterator = iter(iterable)
try:
while True:
elem = next(iterator)
# do something
except StopIteration:
pass
```

Here's a breakdown of what's happening:

- First, the built-in
`iter`

function is called on the iterable to create a corresponding*iterator*. - To get the next element in the sequence, the built-in
`next`

function is called on this iterator. - When
`next`

is called but there are no elements left in the iterator, a`StopIteration`

error is raised. In the for loop construct, this exception is caught and execution can continue.

Calling `iter`

on an iterable multiple times returns a new iterator each time
with distinct states (otherwise, you'd never be able to iterate through a
iterable more than once). You can also call `iter`

on the iterator itself, which
will just return the same iterator without changing its state. However, note
that you cannot call `next`

directly on an iterable.

Let's see the `iter`

and `next`

functions in action with an iterable we're
already familiar with -- a list.

```
>>> lst = [1, 2, 3, 4]
>>> next(lst) # Calling next on an iterable
TypeError: 'list' object is not an iterator
>>> list_iter = iter(lst) # Creates an iterator for the list
>>> list_iter
<list_iterator object ...>
>>> next(list_iter) # Calling next on an iterator
1
>>> next(list_iter) # Calling next on the same iterator
2
>>> next(iter(list_iter)) # Calling iter on an iterator returns itself
3
>>> list_iter2 = iter(lst)
>>> next(list_iter2) # Second iterator has new state
1
>>> next(list_iter) # First iterator is unaffected by second iterator
4
>>> next(list_iter) # No elements left!
StopIteration
>>> lst # Original iterable is unaffected
[1, 2, 3, 4]
```

Since you can call `iter`

on iterators, this tells us that that they are also
iterables! Note that while all iterators are iterables, the converse is not
true - that is, not all iterables are iterators. You can use iterators wherever
you can use iterables, but note that since iterators keep their state, they're
only good to iterate through an iterable once:

```
>>> list_iter = iter([4, 3, 2, 1])
>>> for e in list_iter:
... print(e)
4
3
2
1
>>> for e in list_iter:
... print(e)
```

Analogy: An iterable is like a book (one can flip through the pages) and an iterator for a book would be a bookmark (saves the position and can locate the next page). Calling`iter`

on a book gives you a new bookmark independent of other bookmarks, but calling`iter`

on a bookmark gives you the bookmark itself, without changing its position at all. Calling`next`

on the bookmark moves it to the next page, but does not change the pages in the book. Calling`next`

on the book wouldn't make sense semantically. We can also have multiple bookmarks, all independent of each other.

### Iterable Uses

We know that lists are one type of built-in iterable objects. You may have also
encountered the `range(start, end)`

function, which creates an iterable of
ascending integers from start (inclusive) to end (exclusive).

```
>>> for x in range(2, 6):
... print(x)
...
2
3
4
5
```

Ranges are useful for many things, including performing some operations for a particular number of iterations or iterating through the indices of a list.

There are also some built-in functions that take in iterables and return useful results:

`map(f, iterable)`

- Creates iterator over`f(x)`

for each`x`

in`iterable`

`filter(f, iterable)`

- Creates iterator over`x`

for each`x`

in`iterable`

if`f(x)`

`zip(iter1, iter2)`

- Creates iterator over co-indexed pairs (x, y) from both input iterables`reversed(iterable)`

- Creates iterator over all the elements in the input iterable in reverse order`list(iterable)`

- Creates a list containing all the elements in the input iterable`tuple(iterable)`

- Creates a tuple containing all the elements in the input iterable`sorted(iterable)`

- Creates a sorted list containing all the elements in the input iterable

## Generators

We can create our own custom iterators by writing a *generator function*, which
returns a special type of iterator called a **generator**. Generator functions
have `yield`

statements within the body of the function instead of `return`

statements. Calling a generator function will return a generator object and
will *not* execute the body of the function.

For example, let's consider the following generator function:

```
def countdown(n):
print("Beginning countdown!")
while n >= 0:
yield n
n -= 1
print("Blastoff!")
```

Calling `countdown(k)`

will return a generator object that counts down from `k`

to 0. Since generators are iterators, we can call `iter`

on the resulting
object, which will simply return the same object. Note that the body is not
executed at this point; nothing is printed and no numbers are output.

```
>>> c = countdown(5)
>>> c
<generator object countdown ...>
>>> c is iter(c)
True
```

So how is the counting done? Again, since generators are iterators, we call
`next`

on them to get the next element! The first time `next`

is called,
execution begins at the first line of the function body and continues until the
`yield`

statement is reached. The result of evaluating the expression in the
`yield`

statement is returned. The following interactive session continues
from the one above.

```
>>> next(c)
Beginning countdown!
5
```

Unlike functions we've seen before in this course, generator functions can
remember their state. On any consecutive calls to `next`

, execution picks up
from the line after the `yield`

statement that was previously executed. Like
the first call to `next`

, execution will continue until the next `yield`

statement is reached. Note that because of this, `Beginning countdown!`

doesn't
get printed again.

```
>>> next(c)
4
>>> next(c)
3
```

The next 3 calls to `next`

will continue to yield consecutive descending
integers until 0. On the following call, a `StopIteration`

error will be
raised because there are no more values to yield (i.e. the end of the function
body was reached before hitting a `yield`

statement).

```
>>> next(c)
2
>>> next(c)
1
>>> next(c)
0
>>> next(c)
Blastoff!
StopIteration
```

Separate calls to `countdown`

will create distinct generator objects with their
own state. Usually, generators shouldn't restart. If you'd like to reset the
sequence, create another generator object by calling the generator function
again.

```
>>> c1, c2 = countdown(5), countdown(5)
>>> c1 is c2
False
>>> next(c1)
5
>>> next(c2)
5
```

Here is a summary of the above:

- A
*generator function*has a`yield`

statement and returns a*generator object*. - Calling the
`iter`

function on a generator object returns the same object without modifying its current state. - The body of a generator function is not evaluated until
`next`

is called on a resulting generator object. Calling the`next`

function on a generator object computes and returns the next object in its sequence. If the sequence is exhausted,`StopIteration`

is raised. A generator "remembers" its state for the next

`next`

call. Therefore,the first

`next`

call works like this:- Enter the function and run until the line with
`yield`

. - Return the value in the
`yield`

statement, but remember the state of the function for future`next`

calls.

- Enter the function and run until the line with
And subsequent

`next`

calls work like this:- Re-enter the function, start at
**the line after the**, and run until the next`yield`

statement that was previously executed`yield`

statement. - Return the value in the
`yield`

statement, but remember the state of the function for future`next`

calls.

- Re-enter the function, start at

- Calling a generator function returns a brand new generator object (like
calling
`iter`

on an iterable object). - A generator should not restart unless it's defined that way. To start over from the first element in a generator, just call the generator function again to create a new generator.

Another useful tool for generators is the `yield from`

statement (introduced in
Python 3.3). `yield from`

will yield all values from an iterator or iterable.

```
>>> def gen_list(lst):
... yield from lst
...
>>> g = gen_list([1, 2, 3, 4])
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
4
>>> next(g)
StopIteration
```

# Required Questions

## WWPD

### Q1: WWPD: Iterators

Use Ok to test your knowledge with the following "What Would Python Display?" questions:

`python3 ok -q iterators -u`

Enter

`Error`

if you believe an error occurs,`StopIteration`

if a`StopIteration`

exception is raised, and`Iterator`

if the output is a iterator object.

```
>>> s = [1, 2, 3, 4]
>>> t = iter(s)
>>> next(s)
______TypeError
>>> next(t)
______1
>>> next(t)
______2
>>> iter(s)
______Iterator
>>> next(iter(s))
______1
>>> next(iter(t))
______3
>>> next(iter(s))
______1
>>> next(iter(t))
______4
>>> next(t)
______StopIteration
```

```
>>> r = range(6)
>>> r_iter = iter(r)
>>> next(r_iter)
______0
>>> [x + 1 for x in r]
______[1, 2, 3, 4, 5, 6]
>>> [x + 1 for x in r_iter]
______[2, 3, 4, 5, 6]
>>> next(r_iter)
______StopIteration
>>> list(range(-2, 4)) # Converts an iterable into a list
______[-2, -1, 0, 1, 2, 3]
```

```
>>> map_iter = map(lambda x : x + 10, range(5))
>>> next(map_iter)
______10
>>> next(map_iter)
______11
>>> list(map_iter)
______[12, 13, 14]
>>> for e in filter(lambda x : x % 2 == 0, range(1000, 1008)):
... print(e)
...
______1000
1002
1004
1006
>>> [x + y for x, y in zip([1, 2, 3], [4, 5, 6])]
______[5, 7, 9]
>>> for e in zip([10, 9, 8], range(3)):
... print(tuple(map(lambda x: x + 2, e)))
...
______(12, 2)
(11, 3)
(10, 4)
```

### Q2: WWPD: Generators

Use Ok to test your knowledge with the following What would Python Display questions:

`python3 ok -q generators -u`

Enter

`Error`

if you believe an error occurs,`Function`

if the output is a function object, and`Generator`

if the output is a generator object.

```
def gen():
print("Starting here")
i = 0
while i < 6:
print("Before yield")
yield i
print("After yield")
i += 1
```

```
>>> next(gen)
______TypeError
>>> gen
______<function gen ...>
>>> g = gen()
>>> g
______<generator object gen ...>
>>> g == iter(g)
______True
>>> next(g)
______Starting here
Before yield
0
>>> next(g)
______After yield
Before yield
1
>>> next(g)
______After yield
Before yield
2
>>> g2 = gen()
>>> next(g2)
______Starting here
Before yield
0
>>> iter(g2)
______<generator object gen ...>
>>> next(iter(g))
______After yield
Before yield
3
>>> next(gen())
______Starting here
Before yield
0
```

## Coding Practice

### Q3: Scale

Implement the generator function `scale(s, k)`

, which yields elements of the
given iterable `s`

, scaled by `k`

. As an extra challenge, try writing this
function using a `yield from`

statement!

```
def scale(s, k):
"""Yield elements of the iterable s scaled by a number k.
>>> s = scale([1, 5, 2], 5)
>>> type(s)
<class 'generator'>
>>> list(s)
[5, 25, 10]
>>> m = scale(naturals(), 2)
>>> [next(m) for _ in range(5)]
[2, 4, 6, 8, 10]
"""
"*** YOUR CODE HERE ***"
for elem in s:
yield elem * k
# Alternate solution
def scale(s, k):
yield from map(lambda x: x*k, s)
```

Use Ok to test your code:

`python3 ok -q scale`

### Q4: Trap

Write a generator function that yields the first `k`

values in iterable `s`

,
but raises a `ValueError`

exception if any more values are requested. You may
assume that `s`

has at least `k`

values.

To raise an exception, use a

`raise`

statement:`>>> def foo(): ... raise ValueError("This is an error message.") ... >>> foo() ValueError: This is an error message.`

```
def trap(s, k):
"""Return a generator that yields the first K values in iterable S,
but raises a ValueError exception if any more values are requested.
>>> t = trap([3, 2, 1], 2)
>>> next(t)
3
>>> next(t)
2
>>> next(t)
ValueError
>>> list(trap(range(5), 5))
ValueError
>>> t2 = trap(map(abs, reversed(range(-6, -4))), 2)
>>> next(t2)
5
>>> next(t2)
6
>>> next(t2)
ValueError
"""
"*** YOUR CODE HERE ***"
t = iter(s)
for _ in range(k):
yield next(t)
raise ValueError("It's a trap!")
Video walkthrough: https://youtu.be/TRqNVgQSScI?t=57m2s
```

Use Ok to test your code:

`python3 ok -q trap`

# Optional Questions

### Q5: Hailstone

Write a generator that outputs the hailstone sequence from homework 1.

Here's a quick reminder of how the hailstone sequence is defined:

- Pick a positive integer
`n`

as the start. - If
`n`

is even, divide it by 2. - If
`n`

is odd, multiply it by 3 and add 1. - Continue this process until
`n`

is 1.

For some extra practice, try writing a solution using recursion. Since
`hailstone`

returns a generator, you can `yield from`

a call to `hailstone`

!

```
def hailstone(n):
"""
>>> for num in hailstone(10):
... print(num)
...
10
5
16
8
4
2
1
"""
"*** YOUR CODE HERE ***"
while n > 1:
yield n
if n % 2 == 0:
n //= 2
else:
n = n * 3 + 1
yield n
# Alternate Solution
def hailstone_alt(n):
yield n
if n > 1:
if n % 2 == 0:
yield from hailstone_alt(n // 2)
else:
yield from hailstone_alt(n * 3 + 1)
Video walkthrough: https://youtu.be/fQlIJa2_yqw?t=1h18m52s
```

Use Ok to test your code:

`python3 ok -q hailstone`

### Q6: Repeated

Implement a function (not a generator function) that returns the first value in
iterable `t`

that appears `k`

times in a row.

```
def repeated(t, k):
"""Return the first value in iterable T that appears K times in a row.
>>> repeated([10, 9, 10, 9, 9, 10, 8, 8, 8, 7], 2)
9
>>> repeated([10, 9, 10, 9, 9, 10, 8, 8, 8, 7], 3)
8
>>> s = [3, 2, 1, 2, 1, 4, 4, 5, 5, 5]
>>> repeated(trap(s, 7), 2)
4
>>> repeated(trap(s, 10), 3)
5
>>> print(repeated([4, None, None, None], 3))
None
"""
assert k > 1
"*** YOUR CODE HERE ***"
count = 0
last_item = None
for item in t:
if item == last_item:
count += 1
else:
last_item = item
count = 1
if count == k:
return item
Video walkthrough: https://youtu.be/fQlIJa2_yqw?t=1h5m35s
```

Use Ok to test your code:

`python3 ok -q repeated`

### Q7: Merge

Implement `merge(s0, s1)`

, which takes two iterables `s0`

and `s1`

whose
elements are ordered. `merge`

yields elements from `s0`

and `s1`

in sorted
order, eliminating repetition. You may assume `s0`

and `s1`

themselves do not
contain repeats, and that none of the elements of either are `None`

.
You may **not** assume that the iterables are finite; either may produce an infinite
stream of results.

You will probably find it helpful to use the two-argument version of the built-in
`next`

function: `next(s, v)`

is the same as `next(s)`

, except that instead of
raising `StopIteration`

when `s`

runs out of elements, it returns `v`

.

See the doctest for examples of behavior.

```
def merge(s0, s1):
"""Yield the elements of strictly increasing iterables s0 and s1, removing
repeats. Assume that s0 and s1 have no repeats. s0 or s1 may be infinite
sequences.
>>> m = merge([0, 2, 4, 6, 8, 10, 12, 14], [0, 3, 6, 9, 12, 15])
>>> type(m)
<class 'generator'>
>>> list(m)
[0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
>>> def big(n):
... k = 0
... while True: yield k; k += n
>>> m = merge(big(2), big(3))
>>> [next(m) for _ in range(11)]
[0, 2, 3, 4, 6, 8, 9, 10, 12, 14, 15]
"""
i0, i1 = iter(s0), iter(s1)
e0, e1 = next(i0, None), next(i1, None)
"*** YOUR CODE HERE ***"
while True:
if e0 is None and e1 is None:
raise StopIteration
elif e0 is None or e1 is not None and e1 < e0:
yield e1
e1 = next(i1, None)
elif e1 is None or e0 is not None and e0 < e1:
yield e0
e0 = next(i0, None)
else:
yield e0
e0, e1 = next(i0, None), next(i1, None)
```

Use Ok to test your code:

`python3 ok -q merge`

### Q8: Remainder Generator

Like functions, generators can also be*higher-order*. For this problem, we will be writing

`remainders_generator`

, which yields a series of generator
objects.
`remainders_generator`

takes in an integer `m`

, and yields `m`

different
generators. The first generator is a generator of multiples of `m`

, i.e.
numbers where the remainder is 0. The second is a generator of natural numbers
with remainder 1 when divided by `m`

. The last generator yields natural numbers
with remainder `m - 1`

when divided by `m`

.

Hint: You can call the`naturals`

function to create a generator of infinite natural numbers.

Hint: Consider defining an inner generator function. Each yielded generator varies only in that the elements of each generator have a particular remainder when divided by`m`

. What does that tell you about the argument(s) that the inner function should take in?

```
def remainders_generator(m):
"""
Yields m generators. The ith yielded generator yields natural numbers whose
remainder is i when divided by m.
>>> import types
>>> [isinstance(gen, types.GeneratorType) for gen in remainders_generator(5)]
[True, True, True, True, True]
>>> remainders_four = remainders_generator(4)
>>> for i in range(4):
... print("First 3 natural numbers with remainder {0} when divided by 4:".format(i))
... gen = next(remainders_four)
... for _ in range(3):
... print(next(gen))
First 3 natural numbers with remainder 0 when divided by 4:
4
8
12
First 3 natural numbers with remainder 1 when divided by 4:
1
5
9
First 3 natural numbers with remainder 2 when divided by 4:
2
6
10
First 3 natural numbers with remainder 3 when divided by 4:
3
7
11
"""
"*** YOUR CODE HERE ***"
def gen(i):
for e in naturals():
if e % m == i:
yield e
for i in range(m):
yield gen(i)
```

Note that if you have implemented this correctly, each of the
generators yielded by `remainder_generator`

will be *infinite* - you
can keep calling `next`

on them forever without running into a
`StopIteration`

exception.

Use Ok to test your code:

`python3 ok -q remainders_generator`

### Q9: Zip Generator

For this problem, we will be writing `zip_generator`

, which yields a
series of lists, each containing the nth items of each iterable.
It should stop when the smallest iterable runs out of elements.

```
def zip_generator(*iterables):
"""
Takes in any number of iterables and zips them together.
Returns a generator that outputs a series of lists, each
containing the nth items of each iterable.
>>> z = zip_generator([1, 2, 3], [4, 5, 6], [7, 8])
>>> for i in z:
... print(i)
...
[1, 4, 7]
[2, 5, 8]
"""
"*** YOUR CODE HERE ***"
iterators = [iter(iterable) for iterable in iterables]
while True:
yield [next(iterator) for iterator in iterators]
```

Use Ok to test your code:

`python3 ok -q zip_generator`