Skip to content

fix: parse first complete unfenced JSON value - #42015

Open
fritz-fujii wants to merge 1 commit into
langgenius:mainfrom
fritz-fujii:fix-42006-json-markdown-first-value
Open

fix: parse first complete unfenced JSON value#42015
fritz-fujii wants to merge 1 commit into
langgenius:mainfrom
fritz-fujii:fix-42006-json-markdown-first-value

Conversation

@fritz-fujii

@fritz-fujii fritz-fujii commented Sep 8, 2026

Copy link
Copy Markdown

Summary

Fix parse_json_markdown() so multiple unfenced JSON values do not get concatenated into a single json.loads() call.

The parser now consumes the first complete unfenced JSON value while preserving the existing fenced-content and scalar behavior.

Problem

Given:

{"a": 1}
{"a": 2}

the current implementation selects the first opening brace and the last closing brace, causing both values to be passed to json.loads() and raising JSONDecodeError: Extra data.

This surfaces in the Question Classifier node when a model emits more than one JSON value in a row.

Fix

Use json.JSONDecoder().raw_decode() for the unfenced JSON path so parsing stops after the first complete JSON value.

Existing fenced JSON behavior is intentionally preserved — in particular, test_parse_and_check_json_markdown_multiple_blocks_fails (multiple fenced blocks still fail) is left unchanged.

Tests

Added regression coverage for multiple unfenced JSON values and verified the existing JSON-in-Markdown parser tests continue to pass.

New cases: first value is an object / an array / contains nested objects / contains brackets inside string values / is followed by prose, a malformed first value still raises, and the reported parse_and_check_json_markdown failure mode now resolves.

tests/unit_tests/libs/test_json_in_md_parser.py: 20 passed (13 existing + 7 new).

Checklist

  • This change requires a documentation update, included: Dify Document — not needed, internal parser behavior only
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly. — N/A
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

Note on the last item: the full make targets need uv, which was not available in my environment. I ran the equivalent checks scoped to the two changed files instead: ruff check (all checks passed), ruff format --check (already formatted), and mypy --check-untyped-defs (no issues). Happy to run anything else the full pipeline flags.

Fixes #42006

From Claude Code

When JSON is not wrapped in code fences there is no end marker, so
anchoring on the last "}" / "]" can span a *later* value. Models
sometimes emit more than one JSON value in a row, and the whole slice
was handed to json.loads() at once, failing with "Extra data".

Decode only the first complete value with JSONDecoder.raw_decode() on
the unfenced path. Fenced content keeps its existing behavior.

Fixes langgenius#42006

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KmXVqNv6rrbj26ConME7LW
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 63.92% 63.91% -0.01%
Strict coverage 63.52% 63.51% -0.01%
Typed symbols 45,606 45,593 -13
Untyped symbols 25,904 25,911 +7
Modules 3384 3384 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Question Classifier can fail with JSONDecodeError "Extra data" when model returns multiple JSON objects

1 participant