Every comparison with nan is False, including the one you would expect to be True. And it spreads.
import numpy as np
nan = float('nan')
nan == nan # -> False
nan != nan # -> True the only value for which this holds
nan < 1, nan > 1 # -> (False, False) not even consistently ordered
values = np.array([1.0, 2.0, nan])
values.mean() # -> nan one missing value poisons the lot
values == nan # -> [False, False, False] cannot find it this way
np.isnan(values) # -> [False, False, True] this is how
np.nanmean(values) # -> 1.5 the nan-ignoring twin
- nan != nan, and that is the correct answer
Two missing measurements are not evidence that the two things were equal, so the IEEE standard says every comparison involving nan is false. The consequence is that `value == nan` never finds anything and `nan in array` is unreliable — np.isnan is the only way to ask. It is also the one place `x != x` is a meaningful expression rather than a typo.
- It propagates, which is a feature and a nuisance
Any arithmetic touching nan produces nan, so a single missing value makes a whole sum, mean or deviation missing. That is honest — the answer really is unknown — and it is rarely what you want, which is why numpy ships nansum, nanmean, nanstd, nanmax and the rest. Reaching for those is a decision to treat missing as absent rather than as unknown, and it is worth making deliberately.
- All-nan has no mean, and numpy warns rather than raising
np.nanmean of an array with nothing but nan returns nan and emits a RuntimeWarning — it does not raise. So the empty-ish case has to be handled by counting first, not by inspecting the result afterwards. Anything that relies on a warning being noticed will eventually be run with warnings filtered off.
Find the missing values with a function rather than a comparison — nothing equals nan, including nan. Then decide whether there is anything left to average before averaging it.
np.isnan(values) gives a boolean array; its sum is how many are missing, because True counts as 1. The count itself is values.size, not len(values), which only measures the first axis.
np.nanmean ignores the missing values, but on an all-missing array it returns nan and warns rather than raising — so compare the missing count against the total first and return None when they match. That check also covers the empty array, where both are 0.
import numpy as np def describe(values): count = int(values.____) missing = int(np.____(values).sum()) if missing == ____: mean = None else: mean = float(np.____(values)) return {'count': count, 'missing': missing, 'mean': mean}
import numpy as np
def describe(values):
# .size is every element; len() would only measure the first axis.
count = int(values.size)
# isnan, because nothing — including nan — compares equal to nan.
missing = int(np.isnan(values).sum())
# nanmean returns nan and merely warns when there is nothing to
# average, so the empty case is decided by counting instead.
if missing == count:
mean = None
else:
mean = float(np.nanmean(values))
return {
"count": count,
"missing": missing,
"mean": mean,
}
An all-nan array has missing equal to count, and an empty array has both equal to zero — so one comparison handles 'nothing to average' in both of its forms. Checking `count == 0` separately would work and would be two conditions describing one situation.
nan is a float, so it survives being put in a JSON payload as the non-standard `NaN` token, compares false against everything including itself, and makes `if mean:` behave unpredictably. None is unambiguous and every consumer already knows what it means. The exception is when you are handing the value straight back to numpy, which would rather have the nan.
np.isnan gives an array of booleans, and summing booleans counts the True ones — the same trick as sum(1 for x in xs if ...) with none of the loop. It generalises: (values > 10).sum() counts the large ones, and (values > 10).any() and .all() ask the other two questions.
pandas treats missing data as a first-class concern and is the better tool the moment you have columns rather than an array: .isna(), .dropna(), .fillna() and a mean that skips missing values by default. That default is the opposite of numpy’s and catches people moving between them — pandas' Series.mean() ignores nan, numpy’s ndarray.mean() does not. Also worth knowing: nan is only available in float arrays, so an integer column with missing values has to become float or use pandas' nullable Int64. And treating missing as absent is itself a modelling decision — sometimes the fact that a value is missing is the most predictive feature you have, and dropping it throws that away.
- Add the deviation of the present values, and check which nan-aware function you need and what its default divisor is.
- Report which positions are missing using np.where on the mask, so a caller can go and look at those rows.
- Try `values == np.nan` and `np.nan in values` on an array containing nan, and confirm for yourself that neither finds it.