Choosing a trade-off

Life is full of trade-offs. There are many ways to think about them, but here I want to explore one that helps me make decisions with incomplete information. I compare what an unnecessary action might cost with what I could lose by waiting.

I run into this with notifications. Turning them off gives me longer stretches of concentration, although I may be slow to notice a colleague who needs me. Letting everything through has its own cost. I might welcome an interruption when someone is blocked, and resent the same alert when the request could easily have waited.

An unnecessary interruption is a false alarm; an urgent request I overlook is a miss. In statistics, these are called Type I and Type II errors. Even a reasonable rule can produce both. Considering them together helps me see what I am accepting when I change a rule, and whether finding out more could give me a better set of choices.

Choose a situation

A test result comes before a diagnosis.

When you decide to test someone, their disease status is still uncertain. First you choose who to test. The test measures something, then applies a rule for calling the result positive or negative.

For a test that reports only positive or negative, that cutoff is already part of its design. Here, you can change an invented cutoff to see what that design choice does.

What these examples assume

The dots are a fixed sample from the model. Curves show the model’s expected distribution, with height indicating how concentrated the scores are. Dots settle into stacks above the axis while the model curve appears; their horizontal scores stay fixed. Curve areas reflect group sizes; counts from the sample can differ slightly from expected counts.

The need for action is fixed before the decision. The coloured groups become available only when we check the outcome. They are not information we had when deciding.

Selection preserves each underlying group’s score distribution here. It changes the weighted curves and the usefulness of positive results, while the conditional curves and ROC stay the same. Real selection can also change the score distributions.

Score units are arbitrary. The dot chart includes every sampled score, including long tails. Changing the population mix or evidence changes the simulated example; selecting a named group restores its preset mix.

In diagnostic-test research, a separate reference assessment establishes disease status.

David Heeger’s signal detection notes explain measurement and criterion. Room to be wrong explores learning before acting.

Who gets tested?

People with unknown status

Each dot is one person. Everyone is neutral because we have not established who has the disease. The higher-risk group is identified from information available before testing.

Show distribution

Start with one score

The curve shows how the scores are spread out, from low on the left to high on the right. A taller part means more cases have scores around that value.

We start with 500 cases in each group.

In this example, of messages are urgent.

Adjust the evidenceShow / hide

The curves describe the scores in each group. What we know about a request, invitation, draft, or test result affects how we score it. More varied cases can produce wider curves.

Peak · Width · Tail

The peak marks the most common score. A wider curve describes a more varied group; a longer tail allows more cases far from the peak.

The upper charts show how much the scores overlap, first comparing the shapes and then accounting for the size of each group. Below them, we can see the balance between false alarms and misses at different cutoffs, and how their combined cost changes as we move the line.

Fig. 001 / Within each group

Evidence in each group

Both curves have the same area. This view hides how many cases are in each group.

Evidence clarity ↔

—False positives

Fig. 003 / All cutoff choices

Possible trade-offs

Up: more needs caught. Left: fewer false positives.

Fig. 004 / Total cost

Cost of this rule

Total = both costs added. ◆ = best cutoff.

An unnecessary alert costs units. Missing an urgent message costs units.

We act when the score is above .

Invented examples · Drag any chart handle · Arrow keys also work · Counts are expected values per 1,000 cases

More information can separate the distributions

Moving the cutoff leaves the distributions where they are. Distinguishing cases with the same score requires more information about them. That information can reveal differences between the groups and reduce their overlap, allowing a rule to catch more cases that need attention without prompting as many unnecessary actions.

With two measurements, each case has a position on a plane. Two cases that looked identical along the original axis may now sit far apart. A rule can combine those measurements into a new score, or use their position on the map directly.

From one score to two

The original scores have not changed. A useful second measurement can distinguish cases that looked alike, which is easier to see from above. In the angled view, the surfaces rise where more cases share a similar pair of scores.

Finding out more takes effort. Collecting that information is worth considering when it could change the decision enough to justify the effort. The expected benefit is called the value of information.

What we can learn from an alert

Try changing how common the need is: of messages are urgent.

Two people can agree on the evidence and still choose different cutoffs because they place different weight on the consequences.

A rule needs to be judged across many cases, including the needs it misses. An action that proves unnecessary can still have been a sensible choice with the information available at the time.

Putting a rule into practice

A useful goal accounts for both the problems caught and the extra work created. Those costs fall on particular people. Involving them in the decision can reveal trade-offs that disappear in an overall total.

Leaving the rule unchanged is a choice too. Describing its benefits and costs makes that choice easier to discuss, and gives a basis for revising it as the results become clear.

False alarms and misses

Alpha (α) is the false-positive rate: how often you act when there is no need. Beta (β) is the false-negative rate: how often you miss a real need. Each rate uses a different group, so they do not add up to 100%.

The share of real needs you catch is called sensitivity, recall, or power. It equals 1 − β. Specificity is the share of cases without a need that you leave alone: 1 − α.

Precision asks a different question: of the actions you take, how many respond to a real need? That answer depends on how common the need is. A high detection rate can go with low precision.

Choosing a rule

One approach sets a limit on false alarms, then catches as many real needs as possible within that limit. Another compares the expected costs of false alarms and misses. Changing those costs can change the best rule, even when the evidence stays the same.

In hypothesis testing, a Type I error rejects a true default claim. A Type II error fails to reject a false one. Failing to reject a claim does not prove it. The chance of missing an effect also depends on its size.

How common something is changes the mix of results. How we value the consequences determines which balance we prefer.

E · Model and formulas

Each group has its own preset, peak position, width, and tail balance. The peak is the mode, not necessarily the average.

Width scales both sides around the main peak. Tail balance stretches one side and compresses the other. We normalize the density so its total area stays one. Tail balance is an editing control, not a statistical skewness value.

The gap control is the distance from the blue main peak to the red main peak. A negative gap puts the red peak to the left.

The isometric chart adds a normally distributed second score, independent of the first within each group. Both surfaces have equal volume to compare their shapes. The two-cutoff rule acts only when both scores exceed their thresholds. Because the scores are independent within each group, the probability of passing both is the product of their pass probabilities. Its counts use the population mix and costs above; both surfaces keep equal volume for comparing shapes. The original-score charts continue to show the one-cutoff rule.

The formulas below describe the original equal-width bell curves only. Other shapes use their own cumulative probabilities.

α = 1 - Φ (c + \frac{d^{'}}{2})

β = Φ (c - \frac{d^{'}}{2})

PPV = \frac{p (1 - β)}{p (1 - β) + (1 - p) α}

\begin{array}{l} Cost within selection = N \times \\ [(1 - p) α C_{FP} + p β C_{FN}] \end{array}

c_{best} = \frac{\ln (\frac{(1 - p) C_{FP}}{p C_{FN}})}{d^{'}}

Here, N is the selected group size. Add the number of excluded needs multiplied by the missed-need cost to obtain total cost. $p$ is prevalence, $c$ is the cutoff, and $Φ$ is the standard normal cumulative distribution function.

The last formula assumes d′ is positive. For the original equal-width curves with no separation, the best rule classifies everyone the same way, according to prevalence and costs.

Curves can extend beyond the visible score range. Counts use the full distributions. Edited curves use a numerical search for the best single cutoff, except for equal-width, untilted normal curves.

With different widths or two clusters, a higher score need not always mean a greater chance of a real need. A more complex rule can sometimes do better than one cutoff.

Outcome	Current cutoff	Both cutoffs


Total cost