← cd ~
~/42-firenze/call-me-maybe
42 firenze · milestone 3 · python · llm
Call-Me-Maybe
Function calling with a 0.6B-parameter language model. The model turns a request in natural language into a JSON call with typed arguments. On its own a model this small doesn't reliably follow a schema. Here, at every generation step, every token that would break the JSON or the schema is masked out, so the output is valid by construction.
user
output
candidates
chosen by the modelmasked: breaks the grammarinjected by the grammar, no forward pass
same model, no constraints · red = text after the object, the output is no longer valid json
A real trace recorded from the project's decoder running Qwen3-0.6B: the top-8 raw logits at each step, and which of them the grammar mask allowed.
how it works
- Grammar as DFA: numbers, strings (escapes and surrogate pairs included), booleans and function names each have their own automaton. A token is legal only if the automaton can consume all of it.
- Precomputed masks: boolean masks over the whole vocabulary are cached per DFA state or accumulated prefix. Each step is then one mask lookup plus a vectorized argmax, never a scan of the vocabulary.
- Skipping the model: once a name or literal is unambiguous, the rest is injected verbatim, and a step with only one legal token needs no forward pass.
- Shared prefixes: for names like
fn_addvsfn_add_numbers, the logits decide between closing the string and continuing it. - Guaranteed termination: near the step budget, the string grammar only accepts tokens that close the value, so every output file stays schema-valid.