Score report: Route customer message to the right queue

Route customer message to the right queue

ASSIST ONLY

Use it to draft, keep a human on every case.

Pass rate 77.2% [71.6%, 82.0%] is well short of the 95% target and no confident subset reaches it; useful as a draft-generator, not as an unattended step.

Cases evaluated: 250
Passed: 193
Pass rate: 77.2% [71.6%, 82.0%]
Target: 95%
Calibration error: 0.42 (ranks cases, scale unreliable)

The pass rate carries a 95% confidence interval. With 250 cases the true rate could plausibly sit anywhere in that range, which is why the interval and not the headline number drives the decision.

Coverage vs precision
Confidence floor	Handled	Precision	To review
0.03	100%	77.2% [71.6%, 82.0%]	0
0.10	85%	81.7% [76.0%, 86.3%]	37
0.18	70%	85.1% [79.1%, 89.7%]	75
0.26	57%	88.0% [81.7%, 92.4%]	108
0.39	40%	90.0% [82.6%, 94.5%]	150
0.51	27%	94.1% [85.8%, 97.7%]	182
0.66	14%	97.2% [85.8%, 99.5%]	214
0.96	0%	100.0% [20.7%, 100.0%]	249

Calibration
Stated confidence	Cases	Mean confidence	Actual accuracy
0.0-0.2	83	0.11	61%
0.2-0.4	74	0.30	78%
0.4-0.6	50	0.51	84%
0.6-0.8	30	0.70	97%
0.8-1.0	13	0.87	100%

Caveats

Calibration error 0.42 exceeds 0.15: the confidence score ranks cases usefully but its scale is not a probability. Treat any threshold below as an opaque cut point, and recalibrate before reading it as a percentage.

Failures worth reading (57 total)

msg-0001 (confidence 0.07): expected 'failed_transfer', got 'balance_not_updated_after_bank_transfer'
msg-0010 (confidence 0.12): expected 'wrong_exchange_rate_for_cash_withdrawal', got 'declined_cash_withdrawal'
msg-0022 (confidence 0.05): expected 'getting_spare_card', got 'card_linking'
msg-0023 (confidence 0.15): expected 'pending_card_payment', got 'transfer_timing'
msg-0024 (confidence 0.11): expected 'pending_transfer', got 'balance_not_updated_after_bank_transfer'