Article
Language models fail in a peculiar way: they rarely go quiet. They return an answer that sounds certain, reads cleanly, and looks entirely reasonable. An adult might double-check it; a child who is still learning the topic will not, because they have nothing to check it against.
Engineering has a principle called fail-closed: when something breaks, stop and say so rather than keep running in a broken state. Translated into an education product it reads: when the AI cannot do something reliably, it should say “I am not sure about this step, it needs a human check” instead of inventing something to fill the gap.
It is an unrewarding choice product-wise. Admitting uncertainty makes a demo less smooth and can make the AI look less impressive. The alternative is a child filing a wrong answer away as knowledge, with nobody noticing.
So “say so when unsure” is written into our design principles, and “teacher confirmation before AI output reaches students” is written into the teacher workflow. That AI makes mistakes is a premise, not an accident, and the product has to be designed from that premise.