# Bug: Claude article-generation JSON parse failures

**Severity:** Medium — causes ~10–15% of Claude article generations to land as posts titled "Error generating article" containing the raw JSON text as the body. Recoverable by deleting the post and retrying, but wasteful and surprising.

**First observed:** during astronomy-site character refinement on 2026-04-20 / 2026-04-21.

## Where

- File: `api/app/services/chat.py`
- Parse site: line 286 — `data = json.loads(cleaned_json)`
- Error handler: line 294 — logs `Failed to parse Claude response: {e}` and falls through to an `ArticleResponse` whose `title="Error generating article"` and whose `body` is the raw response text
- Existing cleanup helper: `extract_json_from_response` at line 41

## Observed failure modes

From `/data/tsai/api/logs/api.log` during the session:

| Character | Error message | Likely cause |
|---|---|---|
| Vera K. | `Invalid \escape: line 3 column 9401 (char 9500)` | Claude emitted a `\<char>` inside a JSON string where `<char>` is not a valid escape (e.g. `\$`, `\m`, Markdown-style `\*`) |
| Harlo S. | `Unterminated string starting at: line 3 column 11 (char 140)` | Unescaped `"` character inside a string value prematurely closed it |
| Georg R. (×3) | `Unterminated string starting at: line 3 column 11 (char 105)` | Same as Harlo — unescaped internal quote |

Retrying the generation succeeds (often on the next attempt) because Claude's output varies.

## What `extract_json_from_response` currently handles

Reading lines 41–~90:

- Strips ` ```json ... ``` ` markdown code fences
- Extracts the `{...}` region if there's prose before/after
- Escapes literal `\n` and `\r` inside string values
- Strips invalid control characters (`\x00`–`\x1f` except `\n`, `\r`, `\t`)

## What it does NOT handle

1. **Unescaped `"` inside string values.** E.g. body contains `he said "hello"` as plain text. This ends the string value early, and the parser then sees garbage.
2. **Invalid backslash escapes.** E.g. `\$`, `\*` (Markdown escape), `\%`, `\(`. Python's `json.loads` rejects any backslash sequence that isn't one of `\" \\ \/ \b \f \n \r \t \uXXXX`.

Both failure modes are things the model produces when writing prose-heavy body content, so the rate scales with article length — which matches the observation that these happen most often on longer pieces (2000+ words).

## Suggested fix

Extend `extract_json_from_response` with two additional passes, applied only inside string values (using the same `in_string` state-machine already in place):

### 1. Fix invalid backslash escapes

Inside a string value, if we see `\` followed by a character that isn't in `"\/bfnrtu`, either:

- **Option A (safest):** double the backslash so the literal `\x` becomes `\\x`, preserving author intent.
- **Option B (simpler):** drop the backslash so `\$` becomes `$`.

Either is preferable to a parse failure. Option A preserves round-tripping if any downstream code re-stringifies.

### 2. Balance unescaped internal quotes

This is trickier because the state machine as written uses `"` to toggle `in_string`. A heuristic:

- Track brace depth outside strings (`{}` / `[]`). If we hit a `"` that would end a string and the remaining content up to the next `,` or `}` doesn't parse as a valid JSON value boundary (e.g. next non-whitespace char is a letter), treat the `"` as a literal and escape it.
- Simpler heuristic: if we're in a `"body"` or `"title"` string and we see `"` followed by alphanumeric, it's an internal quote — escape it.

The simpler heuristic misses edge cases but is probably a 90% fix.

### 3. Fallback strategy

If the cleaned-JSON parse still fails, try one more pass with a looser parser:

- `json5` (pip package) — permissive JSON with unquoted keys, single quotes, trailing commas, etc.
- Or `demjson3` — repair-oriented JSON parser.

Add this as a second-chance path inside the existing `try/except`:

```python
try:
    data = json.loads(cleaned_json)
except json.JSONDecodeError:
    try:
        import json5  # type: ignore
        data = json5.loads(cleaned_json)
        logging.warning("Claude JSON parsed only under json5 (lenient) mode")
    except Exception:
        raise  # fall through to existing error handler
```

## Alternative — use Anthropic's native tool-use / structured output

Claude API supports tool-use-based structured output that bypasses free-form JSON entirely. If the ChatAnthropic wrapper in use (via LangChain) supports binding a Pydantic schema as a tool, we could force-route article generation through that path and eliminate this class of bug at the source. Worth evaluating — but more invasive than the parse-layer fix above.

## Test plan

Reproducing the failure is probabilistic, but we can create a regression test directly:

```python
# tests/unit/test_chat_extract_json.py
from app.services.chat import extract_json_from_response
import json

def test_invalid_backslash_escape():
    raw = '{"title": "X", "body": "He paid \\$100 for it"}'  # invalid \$
    cleaned = extract_json_from_response(raw)
    data = json.loads(cleaned)
    assert "$100" in data["body"]

def test_unescaped_internal_quote():
    raw = '{"title": "X", "body": "She said "hello" to him"}'
    cleaned = extract_json_from_response(raw)
    data = json.loads(cleaned)
    assert "hello" in data["body"]

def test_already_valid_json_passes_through():
    raw = '{"title": "X", "body": "normal prose"}'
    cleaned = extract_json_from_response(raw)
    assert json.loads(cleaned) == {"title": "X", "body": "normal prose"}
```

## Priority

Medium — the `/refine-character` skill tolerates it via delete-and-retry, but normal scheduled article generation has no retry path, so these "Error generating article" posts will appear on production sites silently if no one is reviewing output. Fixing is straightforward (one function, clear test cases).