Flattening sequences

Sometimes, we'll have zipped data that needs to be flattened. For example, our input could be a file that has columnar data. It looks like this:

2     3      5      7     11     13     17     19     23     29
31    37     41     43    47     53     59     61     67     71
...

We can easily use (line.split() for line in file) to create a sequence. Each item within that sequence will be a 10-item tuple from the values on a single line.

This creates data in blocks of 10 values. It looks as follows:

>>> blocked = list(line.split() for line in file)
>>> blocked
[['2', '3', '5', '7', '11', '13', '17', '19', '23', '29'], ['31', '37', '41', '43', '47', '53', '59', '61', '67', '71'], ['179', '181', '191', '193', '197', '199', '211', '223', '227', '229']]

This is a start, but it isn't complete. We want to get the numbers into a single, flat sequence. Each item in the input is a 10 tuple; we'd rather not deal with decomposing this one item at a time.

We can use a two-level generator expression, as shown in the following code snippet, for this kind of flattening:

>>> (x for line in blocked for x in line)
<generator object <genexpr> at 0x101cead70>
>>> list(_)
['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', 
 '37', '41', '43', '47', '53', '59', '61', '67', '71',
 ... ]

The first for clause assigns each item—a list of 10 values— from the blocked list to the line variable. The second for clause assigns each individual string from the line variable to the x variable. The final generator is this sequence of values assigned to the x variable.

We can understand this via a simple rewrite as follows:

def flatten(data: Iterable[Iterable[Any]]) -> Iterable[Any]:
    for line in data:
        for x in line:
            yield x

This transformation shows us how the generator expression works. The first for clause (for line in data) steps through each 10-tuple in the data. The second for clause (for x in line) steps through each item in the first for clause.

This expression flattens a sequence-of-sequence structure into a single sequence. More generally, it flattens any iterable that contains an iterable into a single, flat iterable. It will work for list-of-list as well as list-of-set or any other combination of nested iterables.