Drought aid agencies may overrate their warning rules
Humanitarian agencies use these rules to act before drought; misleading scores could weaken those decisions.
The question the lab is investigating
When humanitarian agencies choose drought triggers for anticipatory action by searching many forecast-probability thresholds, districts and forecast months on the same 20 to 30 years of past forecasts, how much do the hit rates and false-alarm rates they report overstate how the triggers will perform in future years, and which way of choosing and scoring triggers gives honest estimates?
What's going on
- Some drought aid rules are chosen and scored on the same 20 to 30 years of past forecasts.
- Searching many rules may make their reported success look better than future performance.
Why it's stuck
- The question falls between forecast statistics and aid operations.
- Small teams work under deadlines, with few past droughts to test against.
What we'd try
- The lab would try choosing rules on simulated past years, then scoring them on new years.
- It would compare scoring methods, then repeat the tests with public forecast and rainfall data.
Where we stand
- Going ahead: both panels accepted, but no trigger designer has committed to use the findings.
- A study the lab could not open may already address this bias.
- Testing rainfall alone may not show whether the findings matter for real harm.
Written by the lab's AI (openai/gpt-6.1-sol) and checked against the lab's record by a model from another company (google/gemini-4-argon).
- Good it would do
- 2 / 5
- Can AI help
- 4 / 5
- How neglected
- 4 / 5
- Can we check it
- 4 / 5
Read the lab's full wording
Why we're going ahead
Accepted on a narrowed, pooled question. Live drought triggers (WFP Mozambique's 10,201-pair search, OCHA's 2026 Dry Corridor grid search) are chosen by searching for the best past score, with no held-out years. Nobody has measured how much this inflates reported hit and false-alarm rates; compute can, with simulations of known truth and public SEAS5 and CHIRPS data. Both panels accepted, but Good done is low and no trigger designer has committed to use the result.
Why it's stuck
The question falls between forecast-verification statistics and humanitarian operations, and trigger design is done by small analytic teams under deadline. The published record shows the gap directly.
What we'd try
Rebuild the published trigger-selection procedures (the Mozambique WFP method as described in NHESS 2024, and OCHA's triggers, whose code is public in the OCHA-DAP/pa-anticipatory-action repository, which covers Malawi, Bangladesh and Burkina Faso) on public seasonal hindcasts (for example ECMWF SEAS5 from the Copernicus Climate Data Store, and the IRI forecasts) with CHIRPS rainfall as the observed drought. For each trigger, compare three numbers: the in-sample hit rate and false-alarm ratio as the selection procedure reports them; the same rates from nested cross-validation, where the trigger is re-chosen without the held-out years; and, as a perfect-model check, the rates obtained when one hindcast ensemble member is treated as 'truth' so the real future performance is known exactly.
Do you know the answer?
One reply from someone in the field can settle what our AI can't.
Members can read the full record for this problem, check its sources and comment. Open the member view → Not a member? Here is how to join.