01Brief08 / 12

A row-by-row loop over a table

shipping_costs(frame) works and its tests pass. Say the same thing in whole-column operations, with no per-row iteration.

What is wrong with itThe function walks the frame a row at a time with iterrows, pulls three values out of each row, does four lines of arithmetic and appends the answer to a list. Every one of those four lines is something pandas will do to a whole column in one expression. The loop is not just slower — it is where the logic hides, because you have to hold a row in your head to read it.

Function	shipping_costs()
Track	Tables with pandas
Difficulty
Time	~26 min
Concepts	whole-column arithmetic np.where and Series.clip iterrows costs more than it looks
Cases	12
Needs	pandas, numpy about 9 MB, fetched once and then cached by your browser

02The contract

Return a list of floats, one per row, each rounded to 2 places.
The cost is weight_kg * 2.50, plus 5.00 if the destination is not 'domestic'.
Orders marked express pay double the whole thing, surcharge included.
The cost is never below the 3.00 minimum and never above the 60.00 cap.
Row order is preserved.
An empty frame gives an empty list.
The caller’s frame is not modified.

—The barchecked, not suggested

No per-row iteration and no apply. Every line of the loop body has a whole-column form — that is the point of the exercise.
One pass, no loops at all beyond the comprehension that is not there.
Said as column operations this is four or five expressions. If it is longer than twelve lines, some of the loop has survived.

Every case still has to pass. These are checked by reading your code, so a solution that is merely correct will not clear them.

03Why this one

This is the single most common shape of pandas code written by someone who learned Python first: correct, readable at ten rows, and thirty to a hundred times slower than the same logic said once per column. Rewriting it is mostly a matter of knowing the four or five operators that replace the body of a loop — arithmetic, a comparison, a conditional, a floor — and that iterrows has a second cost nobody mentions: it builds a Series per row, so every value passes through dtype coercion on the way out.

—The concept behind it

Lists on pythonmastery.io →

That is the reading half. Read it there if the idea is new, then come back here — explanation is what makes something make sense, and practice is what makes it stay.

—Worth doing first

How the answers split, as a percentage

Not a gate — start this one whenever you like. These cover ideas it leans on, and a tick means you have already done them.

04Study first: the loop body, line by line, as column operations

Every line below is the same logic said twice — once per row, then once per column. The column version is what the refactor is made of.

import pandas as pd
import numpy as np
frame = pd.DataFrame({'kg': [2.0, 5.0], 'far': [False, True]})

# Arithmetic: no loop needed, the operator already means 'every row'.
frame['kg'] * 2.5                      # -> 5.0, 12.5

# A conditional per row -> np.where(condition, if_true, if_false)
np.where(frame['far'], 5.0, 0.0)       # -> array([0., 5.])

# A comparison gives a boolean column, which multiplies as 1 and 0:
frame['far'] * 5.0                     # -> 0.0, 5.0  (same answer)

# min and max per row -> clip, which takes both at once
pd.Series([1.0, 99.0]).clip(3.0, 60.0)  # -> 3.0, 60.0

# Rounding a column, then leaving pandas behind:
pd.Series([1.005, 2.675]).round(2).tolist()   # -> [1.0, 2.68]

# What iterrows costs, beyond the loop: each row is a new Series, and a
# Series holds one dtype — so a row of mixed columns becomes object, and
# every number in it is boxed on the way out.
for _, row in frame.iterrows():
    print(type(row['kg']).__name__)     # -> float, float ... via object

np.where is the per-row if/else, said once
np.where(condition, a, b) evaluates the condition for every row and picks from a or b elementwise, where a and b can themselves be columns. That covers almost every if/else inside a row loop. The one thing to watch is that both branches are computed for every row — so np.where(x != 0, y / x, 0) still divides by zero and still warns, even though the result is discarded.
A boolean column is already 1 and 0
frame['express'] * 2 is 2 where True and 0 where False, and (1 + frame['express']) is the multiplier this exercise needs — 2 for express, 1 otherwise. Arithmetic on booleans is not a trick; it is the same thing sum() relies on when it counts a condition. Whether it reads better than np.where depends on the expression, and it is worth having both.
clip is min and max in one call, and it is the readable one
clip(lower, upper) applies a floor and a ceiling to a whole column, which is what max(3.0, min(60.0, cost)) says per row — and clip says it in an order a reader does not have to unpick. The nested min/max form is a small puzzle every time you meet it, because the floor is on the outside. Worth knowing which way round they go, too: Series.clip(60, 3) — bounds reversed — leaves the column completely unchanged, while np.clip(array, 60, 3) applies them in order and flattens everything to 3.

your_code.py

05One way to write it

import pandas as pd
import numpy as np

BASE_PER_KG = 2.50
OVERSEAS_SURCHARGE = 5.00
MINIMUM = 3.00
CAP = 60.00


def shipping_costs(frame):
    base = frame["weight_kg"] * BASE_PER_KG
    # The two ifs, as columns: np.where picks branch-per-row, elementwise.
    far = np.where(frame["destination"] != "domestic", OVERSEAS_SURCHARGE, 0.0)
    multiplier = np.where(frame["express"], 2, 1)

    # Brackets before the doubling: express pays double the surcharge too.
    costs = (base + far) * multiplier

    # clip is the floor and the ceiling in one, applied last.
    return costs.clip(MINIMUM, CAP).round(2).tolist()

The loop’s shape survives; only its scope changes

Read the two versions side by side and they are line-for-line the same calculation in the same order — base, surcharge, multiplier, bounds, round. That is what a good vectorisation looks like: not a clever one-liner, but the same steps with each one applied to a column instead of a value. When the rewrite ends up shaped differently from the original, that is usually a sign the logic changed with it.

np.where returns an array, not a Series, and that is fine here

It drops the index, which matters if you are about to combine it with a differently-indexed object — but adding an array to a Series lines up by position and keeps the Series' index, which is what happens above. When you do want the index kept explicitly, Series.where(cond, other) is the pandas version: same idea, opposite argument order, and it keeps only where the condition is *true*.

tolist() converts; .to_numpy() and .values do not

list(series) and .tolist() both hand back plain Python floats — pandas boxes its scalars on the way out of a Series. What leaks numpy’s float64 is dropping to the array first: list(series.to_numpy()), or iterating .values, gives you the stored values untouched. Say .tolist() and the question never arises; it is the same boundary conversion as int() on a count, spelled for a column.

—Other ways to write itall of these pass

Every version below is correct and verified against the same cases by the build. They are here because knowing which one to reach for is the part that takes judgement, and the trade-off is stated rather than left for you to guess.

boolean arithmetic instead of np.wherefine

import pandas as pd

BASE_PER_KG = 2.50
OVERSEAS_SURCHARGE = 5.00
MINIMUM = 3.00
CAP = 60.00


def shipping_costs(frame):
    base = frame['weight_kg'] * BASE_PER_KG
    surcharge = (frame['destination'] != 'domestic') * OVERSEAS_SURCHARGE
    multiplier = 1 + frame['express']

    costs = (base + surcharge) * multiplier
    return costs.clip(MINIMUM, CAP).round(2).tolist()

A boolean column is 1 and 0, so the surcharge is a multiplication and the multiplier is 1 plus the flag. It is shorter and it needs no numpy import — against which, np.where names the two branches where this makes the reader work out that (1 + express) means 'double if express'. Worth knowing, worth using sparingly.

06When not to write this at all

Vectorising is not always the right answer. When the per-row logic genuinely cannot be expressed in column operations — a call to an external service, a recursive rule where row n depends on your own row n-1, a per-row branch over ten cases with different lookups — a loop is honest and a chain of nested np.where is not. In that case prefer itertuples() over iterrows(): it does not build a Series per row and it is several times faster for the same loop. And at ten rows, none of this is about speed: the reason to prefer the column form here is that the whole calculation is visible in six lines instead of hidden inside an iteration.

07Take it further

Add a per-destination rate from a dict and use map() on the column instead of np.where.
Return the surcharge and the multiplier alongside the cost, so a customer can be shown the breakdown — without putting any of them into the caller’s frame.
Time both versions on a hundred thousand rows, and then on ten, and see where the crossover is.