Can you ship a private feature when its local model is unavailable?
A fictional field notes app offers to suggest checklist items while offline.
“When possible, consider offering a non-AI fallback.” (opens the source)
The question
People use the app to make checklists from notes saved on their phone. The model can suggest items. People must always be able to read their notes and make a checklist manually.
Note text must stay on the device. The pilot must deliver a suggestion within 1.5 seconds on at least 80% of all first openings of the checklist screen.
| Count | |
|---|---|
| Pilot first openings | 1000 |
| Pilot openings on supported devices | 900 |
| Warm benchmark runs with a ready model | 100 |
| Warm suggestions within 1.5 seconds | 90 |
The pilot has one offline opening per person. The support check covers the device, system version and region. Model readiness in the pilot is still unknown.
“What should ship, and what would justify a wider release?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Set the boundary
What would you do here? Open the move you’d make.
Keep a manual local path and count all first openings
The strong move. This keeps the checklist usable and measures the release criterion.
Upload the note when the local model cannot run
Tempting, but it misses something. Sending the note breaks the product’s local-only rule.
Hide the manual checklist while waiting for the model
Tempting, but it misses something. People must be able to make a checklist even when the model is unavailable.
See how a strong candidate says it
“Keep the note on the phone. Let people open the manual checklist every time. Count timely suggestions across all first openings, including the openings where the model cannot run.”
Why it works: The local-only rule applies throughout. The release rate uses all first openings.
Step 2: Read the benchmark
What would you do here? Open the move you’d make.
Ask for readiness and timing from the first-opening pilot
The strong move. The release gate depends on what people get at their first opening.
Approve the release using the 90% benchmark result
Tempting, but it misses something. The warm benchmark measures only runs that start with a ready model.
Count all 900 supported openings as timely
Tempting, but it misses something. The support check establishes platform compatibility. Readiness and response time still need measurement.
See how a strong candidate says it
“The warm model met the time limit in 90 of 100 runs. If all 900 supported first openings were ready and kept that rate, we would get 810 timely suggestions. I’d check readiness and timing on the actual first openings.”
Why it works: The warm benchmark starts with a ready model. The projection also assumes that every supported first opening has a ready model.
Step 3: Check readiness
Of the 900 supported openings, 630 had a ready model, 180 had an incomplete download, and 90 had the setting off. The model offered a suggestion within 1.5 seconds in 567 of the 630 ready openings.
What would you do here? Open the move you’d make.
Report 56.7% across all first openings
The strong move. The denominator includes people whose device is unsupported and people whose model is unavailable.
Report 90% across the pilot
Tempting, but it misses something. The 90% figure uses only the 630 ready openings.
Speed up ready runs until the pilot reaches 80%
Tempting, but it misses something. With 630 ready openings, perfect timing can reach only 63% of all first openings.
See how a strong candidate says it
“The pilot got 567 timely suggestions in 1,000 first openings, or 56.7%. Of openings with a ready model, 90% got a timely suggestion. Even if every ready opening became timely, the overall rate would reach only 63%.”
Why it works: The full pilot misses the 80% release gate. With readiness fixed, faster suggestions can help at most 630 first openings.
Step 4: Decide what ships
What would you do here? Open the move you’d make.
Keep the manual checklist and limit suggestions to the pilot
The strong move. This keeps the task usable and preserves the release criterion for the optional feature.
Count manual completions as timely AI suggestions
Tempting, but it misses something. Manual completion measures the core task. It does not measure a timely suggestion.
Turn the model on for people who chose to leave it off
Tempting, but it misses something. The feature is optional. The next pilot must measure readiness while respecting people’s settings.
See how a strong candidate says it
“Keep the manual checklist available for everyone. Keep suggestions in the pilot where the local model is ready and the setting is on. We reached 567 timely openings against a target of 800. For the next pilot, measure first-opening readiness, timely suggestions and manual checklist completion separately.”
Why it works: The manual checklist serves the core task. A wider suggestion release still needs the measured first-opening rate.
The answer, in short
- Decision
- Keep the manual checklist and limit suggestions to the pilot.
- Because
- The first-opening suggestion rate is 56.7%, below the 80% release gate.
- What would change it
- A new first-opening pilot meets the gate while keeping notes local and the manual path available.
Common traps
- Check readiness at first opening.
- Measure manual completion separately.
Check your answer
- I kept note text on the device.
- I explained what the warm benchmark measured.
- I used all first openings for the release rate.
- I said what ships and what evidence would change it.
Last reviewed Nov 11, 2026