NaN handling#

Real time series have gaps: a failed sensor, a market halt, a missing tick. Every screamer function declares how it treats a NaN on its input, and there are three policies:

ignore: the NaN is skipped and the output is NaN at that point, but the internal state is untouched, so the next value recovers. Used by the summary statistics (rolling, EW, bands, oscillators, filters).
propagate: the NaN flows through the formula, so it lingers as long as it sits in the lookback, then clears. Used by positional functions (Diff, Lag, Return, and similar).
nan-aware: the function's purpose is to remove NaN. FillNa replaces each gap with a constant; Ffill carries the last finite value forward. Neither looks ahead.

Which function uses which policy, and the exact rules, are in NaN and warmup. A single series with one gap is enough to see all three.

import numpy as np
import pandas as pd
from screamer import RollingMean, Diff, FillNa, Ffill, Dropna

x = np.array([1.0, 2.0, 3.0, np.nan, 5.0, 6.0])   # one gap, at t = 3

The three policies on one gap#

Running one representative function per policy on the same series lines the behaviours up side by side. Watch the gap at t = 3, and how far the NaN spreads after it.

table = pd.DataFrame({
    "input": x,
    "ignore: RollingMean(3)": RollingMean(3)(x).round(3),
    "propagate: Diff(1)": Diff(1)(x),
    "nan-aware: FillNa(0)": FillNa(0.0)(x),
    "nan-aware: Ffill": Ffill()(x),
})
table.index.name = "t"
table

	input	ignore: RollingMean(3)	propagate: Diff(1)	nan-aware: FillNa(0)	nan-aware: Ffill
t
0	1.0	NaN	NaN	1.0	1.0
1	2.0	NaN	1.0	2.0	2.0
2	3.0	2.000	1.0	3.0	3.0
3	NaN	NaN	NaN	0.0	3.0
4	5.0	3.333	NaN	5.0	5.0
5	6.0	4.667	1.0	6.0	6.0

Each policy responds differently at the gap (t = 3) and immediately after:

ignore (RollingMean) is NaN only at t = 3. The window skipped the gap, so t = 4 is finite again.
propagate (Diff) is NaN at t = 3 and t = 4. Diff(1) subtracts the previous value from each current one, so the gap spoils two outputs before it slides out at t = 5. Surfacing the NaN is safer than computing a difference across the gap without flagging it.
nan-aware (FillNa, Ffill) has no NaN at all: the gap is replaced by a constant or by the last finite value.

Dropping gaps across streams: `Dropna`#

The three policies act within a single series. The multi-stream layer adds Dropna, which removes events whose value is NaN entirely, producing a shorter, gap-free stream. It changes the number of events (fewer out than in) and returns results as a (values, index) pair. The full alignment model is in Streams, values, and alignment.

idx = np.arange(x.size)
values, kept_idx = Dropna()(x, index=idx)

print("input       :", x)
print("kept values :", values)
print("kept index  :", kept_idx)
print(f"length {x.size} -> {values.size} (one gap removed)")

input       : [ 1.  2.  3. nan  5.  6.]
kept values : [1. 2. 3. 5. 6.]
kept index  : [0 1 2 4 5]
length 6 -> 5 (one gap removed)

NaN handling#

The three policies on one gap#

Dropping gaps across streams: Dropna#

Dropping gaps across streams: `Dropna`#