Two one-character decisions. Each of them is the difference between working and silently deleting the page.
import re
source = '<b>Hello</b> <i>world</i>'
re.sub(r'<.*>', '', source) # -> '' everything, in one match
re.sub(r'<.*?>', '', source) # -> 'Hello world'
re.sub(r'<[^>]*>', '', source) # -> 'Hello world'
# Decoding order matters as much
text = '&lt;' # the literal five characters <
text.replace('&', '&').replace('<', '<') # -> '<' wrong
text.replace('<', '<').replace('&', '&') # -> '<' right
- .* is greedy and will eat the whole line
A regular expression’s * takes as much as it possibly can and only gives characters back when the rest of the pattern fails. So <.*> starts at the first < and matches through to the *last* >, swallowing all the text in between. Adding a '?' after the quantifier makes it lazy — take as little as possible — and [^>]* says the same thing more directly: characters that are not the closing one.
- & is decoded last because it can create other entities
Every other entity begins with an ampersand, so turning & into & first can manufacture an entity that was not in the original text — &lt; is somebody writing the literal characters <, and decoding in the wrong order turns it into a <. Whenever one decoding step can produce input for another, the order is part of the correctness, not a matter of taste.
- The attribute case is unfixable, so pin it down
Given <a title="a > b">, the first > is inside a quoted attribute and no regular expression that scans for a closing bracket can know that. You could add quote handling and still lose to a comment containing a quote. The professional move is not a cleverer pattern — it is a test that records what actually happens, so the limitation is documented rather than lurking.
Three steps in a fixed order: remove the tags, decode the entities, tidy the whitespace. Doing them in a different order gives different answers — think about what decoding <b> before removing tags would produce.
re.sub with a pattern for 'a <, then anything that is not a >, then a >' removes the tags. Then a series of .replace calls for the entities, with & at the end. Then ' '.join(text.split()) for the whitespace.
The pattern is r'<[^>]*>'. Keep the entities in a list of pairs at module level so the ordering is visible and editable in one place, and loop over it — a chain of .replace calls hides which one runs last, which is the detail that matters most here.
import re TAG = re.compile(r'<[^____]*>') ENTITIES = [ ('<', '<'), ('>', '>'), ('"', '\"'), (''', "'"), ('&____;', '&'), # last, and this is the whole reason for the list ] def strip_tags(source): text = TAG.____('', source) for entity, character in ENTITIES: text = text.____(entity, character) return ' '.join(text.____())
import re
TAG = re.compile(r"<[^>]*>")
# Order is correctness, not style: every other entity starts with an
# ampersand, so & has to be decoded after all of them.
ENTITIES = [
("<", "<"),
(">", ">"),
(""", '"'),
("'", "'"),
("&", "&"),
]
def strip_tags(source):
text = TAG.sub("", source)
for entity, character in ENTITIES:
text = text.replace(entity, character)
return " ".join(text.split())
Both are non-greedy in effect and both pass every case here. The character class says what it means — 'characters that are not the terminator' — and it cannot backtrack, so it is faster on long inputs and cannot be talked into a pathological match. Reach for a negated class whenever you are scanning up to a known delimiter; save lazy quantifiers for when the delimiter is longer than one character.
Decoding first would turn the text <script> into <script> and the tag remover would then delete it — so an author who deliberately showed some markup on their page loses it. The general rule: undo the encodings after you have finished treating the input as encoded, never before.
split() with no argument breaks on runs of any whitespace and drops the empty pieces; joining with a single space then normalises everything, including the leading and trailing edges, in one expression. It also flattens the newlines that indented markup leaves behind — which is what makes the result usable as a one-line summary.
Every version below is correct and verified against the same cases by the build. They are here because knowing which one to reach for is the part that takes judgement, and the trade-off is stated rather than left for you to guess.
import html
import re
TAG = re.compile(r'<[^>]*>')
def strip_tags(source):
return ' '.join(html.unescape(TAG.sub('', source)).split())
The entity half, done properly. html.unescape knows every named and numeric entity and decodes in a single pass, so the &-last ordering problem cannot arise at all. Note that the tag removal still comes first — swap those two calls and encoded markup gets decoded into real tags and then deleted. The tag half is still a regex, so the attribute limitation is unchanged.
Do not use this on untrusted input, and do not use it to sanitise anything. It removes tags but keeps their contents, so the code inside a <script> block comes out as text; it can be defeated by a quoted attribute; and 'strip the dangerous bits with a regex' is the origin story of a great many security incidents. For sanitising, use a library built for it — bleach or nh3 — with an allowlist. For extracting text properly, html.parser is in the standard library and BeautifulSoup’s get_text() handles real-world broken markup. This function is right for tidy, trusted fragments you generated yourself: a description field, a stored snippet, your own CMS output.
- Use html.unescape instead of the entity list and work out what it handles that yours does not — there are well over a thousand named entities.
- Drop the contents of <script> and <style> blocks as well as their tags, and notice that this needs matching pairs, which is where the regex approach starts to strain.
- Insert a space where a block-level tag was, so <p>one</p><p>two</p> reads as two words, and decide how you know which tags are block-level.