The same summary written twice. The second version is not slower — it returns a different answer.
stream = (line for line in ['a,1', 'b,2', 'a,3'])
# Two passes over a one-shot iterator
names = {line.split(',')[0] for line in stream} # reads all of it
totals = {n: 0 for n in names}
for line in stream: # nothing left
...
totals # -> {'a': 0, 'b': 0}
# One pass, accumulating as it goes
totals = {}
for line in stream:
name, _, amount = line.partition(',')
totals[name] = totals.get(name, 0) + int(amount)
- Reading an iterator consumes it
A generator or file object has no beginning to go back to — each item is produced on demand and then forgotten. So a second for-loop over the same object iterates zero times and raises nothing at all, which is why this bug survives review: the code looks like it works and the numbers are merely wrong. A list can be read twice; that is the difference, and it is not visible at the call site.
- len() and indexing are the tells
If your function calls len(lines), reads lines[0], or sorts them, it has quietly required a sequence and will raise TypeError on a real file. Everything a stream can support is: iterate it, once, in order. Writing to that limit is what makes the function work on four gigabytes and on a three-item list alike.
- Accumulate into a small thing, not a big one
The result here is one entry per distinct name — bounded by how many names exist, not by how many lines there are. That is what makes streaming worthwhile: the input is unbounded and the state is not. When the accumulator would grow with the input, streaming has bought you nothing and you may as well have the list.
One for-loop over lines, and a dict you add to as you go. Do not call list(), len(), or sorted() on the input at any point — those all need the whole thing.
Strip each line and skip it if it is empty. Split it on the first comma, and treat 'no comma' as an error rather than letting the unpacking fail with a message about values. Add to the running total with dict.get.
line.partition(',') gives head, separator, tail — and an empty separator is how you detect a line with no comma at all. Wrap the int() in a try so the error you raise can name the line; that is the whole reason for catching it.
def stream_totals(lines): totals = {} for line in ____: line = line.strip() if not line: continue name, comma, amount = line.____(',') if not comma: raise ____(f’not a name,amount line: {line!r}') try: value = int(amount) except ValueError as error: raise ValueError(f’bad amount in {line!r}') ____ error name = name.strip() totals[name] = totals.____(name, 0) + value return totals
def stream_totals(lines):
totals = {}
# One pass. No list(), no len(), no indexing — everything this does
# works on a file object as well as on a list.
for line in lines:
line = line.strip()
if not line:
continue
name, comma, amount = line.partition(",")
if not comma:
raise ValueError(f"not a name,amount line: {line!r}")
try:
value = int(amount)
except ValueError as error:
raise ValueError(f"bad amount in {line!r}") from error
name = name.strip()
totals[name] = totals.get(name, 0) + value
return totals
partition always returns three pieces, and when the separator is absent the middle one is empty and the whole string lands in the head. So `if not comma` is a precise test for 'this line has no comma', with no length check and no exception to catch. Compare unpacking a split, which fails with a message about values rather than about the line.
collections.defaultdict(int) would also work and returns 0 for keys nobody added — including, later, keys the caller merely asked about, which quietly creates entries. For a result that leaves this function and gets read elsewhere, a plain dict with an explicit default is the safer shape. Counter would do the same job with the same caveat.
It never asks how many lines there are, never looks ahead, and never goes back — which is exactly the discipline that makes it work on a stream from a network socket or a subprocess. Every convenience you add that needs the whole input silently changes the class of thing the function can be handed.
If the data fits comfortably in memory and you need several different summaries of it, read it once into a list and pass over it as often as you like — streaming is a constraint you accept for a reason, not a virtue. For anything genuinely tabular and large, pandas.read_csv with chunksize, or DuckDB reading the file directly, will beat a hand-written loop and give you the group-by for free. And when you do need several passes over a real stream, itertools.tee exists — but it buffers whatever the slower consumer has not read yet, which can quietly reintroduce the memory problem you were avoiding.
- Return the number of lines skipped as blank alongside the totals, without a second pass.
- Add a limit so the function stops after n valid lines, and check that it leaves the rest of the stream unread rather than consuming it.
- Collect the bad lines instead of raising on the first one, and decide what the totals mean when some of the file could not be read.