Python Generators

A generator is a function that returns an iterator. It uses yield instead of return. Each time the generator fetches a value and gives it back to the calling function, it yields control to the caller while keeping track of its current state. When the caller accesses the next value in the iterator, the generator uses that saved state to deliver it.

Generators, like lists, provide access to an iterable object. They are, however, usually more useful.

Below is an example of a fibonacci numbers generator. It first yields 0, then yields 1, after that it uses a loop to generate the rest of the series yielding each value as is calculated. The state is kept in the variables before_last and last.

def fibonacci_gen():
  before_last = 0
  last = 1
  yield before_last
  yield last
  while True:
    next = before_last + last
    before_last = last
    last = next
    yield next

Lists are eager, This means: a list will fetch as many elements (matching the list criteria) as it can find, at the end of which it will then return them as an iterable object to the user. It is not difficult to create an infinite loop while building a list.

Generators are not eager, they are lazy. They implement what is called “lazy evaluation” which is another way of saying that they create values as they are needed.

So what are the advantages of generators?

• Faster: Due to this yield of control, using generators is usually faster than using a list because the processing of the values can start immediately after the creation of the first value.
• Lazy: Generators will only produce as many values as needed, while still containing the code necessary to produce the rest of the values in the algorithm. This also makes them often faster than lists.
• No infinite loops: Because values are created as needed, there is no danger of producing infinite loops with generators.
• Less memory: Since the generator only produces one value at a time and returns it immediately, it only requires enough memory to hold that one value. A list, on the other hand, requires memory for each of its elements.
• Flow control: Generators give the developer a means to control the sequence in which code is executed. This trait of generators makes them useful beyond just replacing lists.

Using the generator function

Once a generator function has been defined, it can be used by

1. Assigning it return value to a generator object
2. repeatedly calling next() with the generator object as parameter

def fibonacci_gen():
   before_last = 0
   last = 1
   yield before_last
   yield last
   while True:
     next = before_last + last
     before_last = last
     last = next
     yield next

def main():
   fibo = fibonacci_gen()    # get a generator object named fibo
   for i in range(20):
     next_fibo = next(fibo)  # access the values yielded by the generator object

Generator class

A generator can be a class, either by subclassing collections.abc.Generator or by implementing __iter__(self) and __next__(self)

#!/usr/bin/env python3
from collections.abc import Generator

class Fibonacci(Generator):
  def __init__(self):
    self.previous_val = 0
    self.current_val = 0

  def send(self,ignored_arg):
    ret_val = self.current_val

    if self.current_val == 0:
      next_val = 1
    else:
      next_val = self.current_val + self.previous_val

    self.previous_val = self.current_val
    self.current_val = next_val
    return ret_val

  def throw(self,type=None,value=None,tracenack=None):
    raise StopIteration

  def next(self):
    return self.__next__()

class Fibonacci_1:
  def __init__(self):
    self.previous_val = 0
    self.current_val = 0

  def __iter__(self):
    return self

  def __next__(self):
    ret_val = self.current_val

    if self.current_val == 0:
      next_val = 1
    else:
      next_val = self.current_val + self.previous_val

    self.previous_val = self.current_val
    self.current_val = next_val
    return ret_val

def fibonacci_gen():
  previous_val = 0
  current_val = 1
  yield previous_val
  while True:
    ret_val = current_val
    next_val = current_val + previous_val
    previous_val = current_val
    current_val = next_val
    yield ret_val

def main():
  print ("using generator function")
  fibo = fibonacci_gen()    # get a generator object from generator function
  for i in range(20):
    next_fibo = next(fibo)  # access the values yielded by the generator object
    print (next_fibo)

  print ("using subclass of collections.abc.Generator")
  fibo = Fibonacci()        # get a generator object from generator class
  for i in range(20):
    next_fibo = next(fibo)
    print(next_fibo)
  
  print ("using direct implementation of __iter__ and __next__")
  fibo = Fibonacci_1()
  for i in range(20):
    next_fibo = next(fibo)
    print(next_fibo)

if __name__ == "__main__":
   main()