Get Reliable JSON Output from an LLM

Why JSON is hard (and how to make it easy)

The moment another program — not a human — needs to read an LLM's output, you need it in a strict, predictable format, and JSON is the universal choice. The problem: by default, models love to wrap their answer in friendly chatter ("Sure! Here's the JSON:") or markdown fences, which break your parser. Getting clean, reliable JSON is a solved problem once you combine three techniques.

1. Ask precisely

Be explicit about the schema and forbid anything extra:

Return ONLY valid JSON matching:
{"title": string, "tags": string[], "difficulty": "easy"|"hard"}
No markdown, no commentary.

Naming the exact fields, their types, and the allowed values — plus "ONLY" and "no commentary" — removes most of the chatter that breaks parsing.

2. Use native structured output

Asking nicely helps, but modern APIs can guarantee valid JSON with a structured-output or JSON mode. Always prefer it when available — it's far more reliable than prompting alone:

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    response_format={"type": "json_object"},
)

Many APIs go further and let you pass an actual schema (JSON Schema or a Pydantic model), so the output is guaranteed to match your structure, not just "some JSON."

3. Always validate

Never trust raw model output. Parse it and validate against a schema with a tool like Pydantic (Python) or Zod (JavaScript):

• If parsing succeeds, you have a typed, safe object to work with.
• If it fails, retry once with the parser's error message appended to the prompt ("Your last response failed validation with: ... Return corrected JSON only"). This self-correction loop catches almost everything.

Common pitfalls

• Trailing prose — forgetting "ONLY" lets the model add a sentence before or after the JSON. Structured-output mode eliminates it entirely.
• Hallucinated fields — the model invents keys you didn't ask for. A strict schema rejects them.
• No fallback — always handle the parse-failure case; a single malformed response shouldn't crash your app.

The takeaway

Reliable JSON comes from layering defenses: ask precisely, use native JSON or structured-output mode, and validate-then-retry. With those three, malformed responses go from a daily headache to a rare, automatically-handled event.

Try it now: Take a prompt that returns messy output, switch on your API's JSON mode, and add a Pydantic (or Zod) validation step. You'll stop hand-cleaning responses for good.