What split(',') does to a line with a quoted comma in it, and the shape of the loop that gets it right.
line = 'ada,"London, UK",42'
line.split(',')
# -> ['ada', '"London', ' UK"', '42'] four fields instead of three
# The scanner: the same character means different things in each state
# inside a quoted field -> a comma is data, a quote may end the field
# outside one -> a comma separates, a quote may start one
#
# and the escape: "a""b" is the four characters a"b
- The flag is the state, and it changes what characters mean
There are only two states here — inside a quoted field or not — so one boolean is enough, and every character is interpreted in the light of it. That is what a state machine is: not a class hierarchy, just a variable that survives between characters. Recognising when a parsing problem needs one saves you from a pile of increasingly specific regular expressions.
- Escaping by doubling needs one character of lookahead
When a quote appears inside a quoted field you cannot tell yet whether it closes the field or is the first half of an escaped quote — the next character decides. Peeking at it and skipping two positions when it is another quote is the entire rule, and it is why this loop uses an index rather than `for char in line`.
- Only at the start of a field
A quote is special at position zero of a field and ordinary anywhere else, which is what makes 5" nails a perfectly good unquoted value. Testing whether anything has been collected for the current field is the cheapest way to know where you are — and it is why the field buffer starts empty rather than as a string being appended to.
Walk the characters with a flag saying whether you are inside a quoted field, collecting the current field as you go. Flush the field on a comma, and once more after the loop.
Use an index rather than `for char in line`, because when you meet a quote inside a quoted field you need to look at the next character to decide whether it closes the field or is an escaped quote.
Collect each field into a list of characters and ''.join it when you flush — cheaper than growing a string, and `if not current` is then also the test for 'am I at the start of a field'. After the loop, raise if the flag is still set, then flush the last field unconditionally.
def split_csv_line(line): fields, current = [], [] inside = False index = 0 while index < len(line): char = line[index] if inside: if char == '"': if index + 1 < len(line) and line[index + 1] == '"': current.append('"') index += ____ continue inside = ____ else: current.append(char) else: if char == '"' and not ____: inside = True elif char == ',': fields.append(''.join(current)) current = [] else: current.append(char) index += 1 if ____: raise ValueError(f’unterminated quoted field: {line!r}') fields.append(''.join(current)) return fields
def split_csv_line(line):
fields = []
current = []
inside = False
index = 0
while index < len(line):
char = line[index]
if inside:
if char == '"':
# A doubled quote is one literal quote; only the next
# character can tell us which this is.
if index + 1 < len(line) and line[index + 1] == '"':
current.append('"')
index += 2
continue
inside = False
else:
current.append(char)
else:
if char == '"' and not current:
inside = True
elif char == ",":
fields.append("".join(current))
current = []
else:
current.append(char)
index += 1
if inside:
raise ValueError(f"unterminated quoted field: {line!r}")
# The last field is never followed by a comma, so it is flushed here —
# unconditionally, because an empty last field is still a field.
fields.append("".join(current))
return fields
Guarding it with `if current` would drop the empty field a trailing comma creates and turn an empty line into zero fields — both of which change the number of columns in a row, which is the one thing a CSV reader must never do. Every character between two separators is a field, including none of them.
It tests whether the current field is empty, which — outside a quoted field — is exactly the same question as 'are we at the start of a field'. That is why 5" nails works with no extra code: by the time the quote arrives, current already holds the 5. A flag tracking position within the field would be a second piece of state saying the same thing.
A field ending in a quote at the very end of the line — 'a,""' — has no next character, and reading line[index + 1] would raise IndexError on a perfectly valid input. Any lookahead needs a bounds check, and the order matters: check first, then read.
Use the csv module. It handles this and everything past it — different delimiters and quote characters, backslash escaping, embedded newlines inside a quoted field (which this function cannot even see, because it is handed one line at a time), and the several dialects Excel produces. csv.reader on a file object is one line of code and has been correct for two decades. Write this to understand the scanner; ship the standard library. The one genuine reason to hand-write it is a hot path where you know the data has no quotes at all — and then you are writing split(','), not this.
- Take the delimiter and quote character as arguments, and see how much of the loop stops mentioning specific characters.
- Handle a backslash escape as well as the doubled quote, and work out which one wins when both appear.
- Feed your function and csv.reader the same awkward lines and find the first one where they disagree.