The same sentence through two tokenisers. The first one is what everybody writes; look at what it thinks the words are.
line = "The cat sat. The cat's hat, the CAT." print(line.split()) # ['The', 'cat', 'sat.', 'The', "cat's", 'hat,', 'the', 'CAT.'] # -> 'sat.' and 'hat,' carry punctuation, and 'The'/'the'/'CAT' are # three different words import re print(re.findall(r"[a-z']+", line.lower())) # ['the', 'cat', 'sat', "cat's", 'hat', 'the', 'cat']
- split() divides, findall() selects
str.split cuts the text at separators and keeps everything else, punctuation included. re.findall does the opposite — it keeps only what matches the pattern and discards the rest, so the punctuation never enters the result. When you want 'the parts that look like this' rather than 'the parts between those', findall is the tool.
- Lowercase before counting, not after
Counting first and merging cases afterwards means deciding which spelling wins and re-adding the totals — work that disappears entirely if the text is folded on the way in. Normalise at the boundary, count once. This is the same rule that makes tag lists and email addresses behave.
- The pattern is where the policy lives
[a-z']+ says apostrophes belong inside words and hyphens do not, which is why "don’t" survives and 'well-known' becomes two. Nothing else in the function has an opinion about it. Keeping the whole definition of 'word' in one expression means changing your mind later is a one-line change, not an archaeology exercise.
Three steps that do not overlap: turn the text into a list of words, count them, then take the top few. Do the lowercasing before the counting.
re.findall with a pattern for 'letters and apostrophes' pulls the words out and leaves the punctuation behind. collections.Counter counts a list in one call, and its .most_common(limit) does the ranking.
words = re.findall(r"[a-z']+", text.lower()), then strip the leading and trailing apostrophes off each one with .strip("'") and drop anything that is left empty. Counter(words).most_common(limit) returns exactly the list of pairs you want — and it breaks ties by first-seen order already.
import re from collections import Counter def word_frequency(text, limit): if limit < ____: raise ____(f’limit cannot be negative, got {limit}') found = re.____(r"[a-z']+", text.____()) words = [word.strip("'") for word in found if word.strip("'")] return Counter(words).____(limit)
import re
from collections import Counter
WORD = re.compile(r"[a-z']+")
def word_frequency(text, limit):
if limit < 0:
raise ValueError(f"limit cannot be negative, got {limit}")
words = [
word.strip("'")
for word in WORD.findall(text.lower())
if word.strip("'")
]
return Counter(words).most_common(limit)
Counter remembers the order keys were first added, and most_common sorts by count with a stable sort — so equal counts come back in first-seen order without you arranging it. Writing sorted(counts.items(), key=lambda pair: -pair[1]) gets the same result for the same reason; writing sorted(..., reverse=True) on the whole pair does not, because it then sorts ties by the word.
Counter walks the list once. The tempting alternative — {word: words.count(word) for word in set(words)} — re-scans the entire list for every distinct word, which is O(n × distinct) and turns a novel-sized input from instant into a coffee break. list.count inside a loop over the same list is one of the most reliable ways to write accidentally quadratic code.
The pattern cannot distinguish an apostrophe from a single quotation mark, so 'hello' arrives with both ends attached. Stripping afterwards is simpler than a cleverer pattern, and the `if` that follows it matters as much: a token that was nothing but quote marks strips down to an empty string, and an empty string must not become the commonest word in the document.
Every version below is correct and verified against the same cases by the build. They are here because knowing which one to reach for is the part that takes judgement, and the trade-off is stated rather than left for you to guess.
import re
def word_frequency(text, limit):
if limit < 0:
raise ValueError('limit cannot be negative')
counts = {}
for word in re.findall(r"[a-z']+", text.lower()):
word = word.strip("'")
if word:
counts[word] = counts.get(word, 0) + 1
ranked = sorted(counts.items(), key=lambda pair: -pair[1])
return ranked[:limit]
Correct, and it shows what Counter is doing — the .get(word, 0) + 1 line is the whole trick. Worth writing once so the import stops being magic. In real code, Counter says the same thing in one line and gives you most_common, addition and subtraction for free.
The moment real language is involved, this is a toy. 'run' and 'running' count separately, 'the' and 'and' drown out everything interesting, and any language without spaces between words defeats the pattern entirely. Real text analysis wants stop-word removal, stemming or lemmatisation, and a tokeniser that knows about the language — spaCy and NLTK exist for this. Write this when you want a rough sense of a body of text you already understand; do not build a search engine on it.
- Add a stop_words argument and drop those before counting, then look at how much more interesting the top ten becomes.
- Return the words that appear exactly once — the hapaxes — and notice that most_common cannot help you, because they are at the wrong end.
- Count pairs of adjacent words instead of single words, and see how much of the code survives the change.