Skip to content

fix(rag,extractors): resolve markdown table corruption and Content-Type parsing across document extractors - #42002

Open
vjymisal0 wants to merge 1 commit into
langgenius:mainfrom
vjymisal0:fix/rag-extractors-markdown-tables-and-content-type
Open

fix(rag,extractors): resolve markdown table corruption and Content-Type parsing across document extractors#42002
vjymisal0 wants to merge 1 commit into
langgenius:mainfrom
vjymisal0:fix/rag-extractors-markdown-tables-and-content-type

Conversation

@vjymisal0

Copy link
Copy Markdown

Summary

Fixes multiple table formatting and header parsing issues across RAG document extractors (#41992, #41990, #41986, #41955):

  1. Notion Extractor (Notion extractor corrupts Markdown tables for cells with mixed formatting or empty data cells #41992): Concatenate all rich-text segments within each Notion table cell into a single column string, preserve empty cells ([]) without column shifting, and escape pipe characters.
  2. Word Extractor (Word extractor produces broken Markdown tables when a cell contains a pipe character #41990): Escape pipe characters (|) in Word table cells to prevent table delimiter breakage in generated Markdown tables.
  3. Markdown Extractor (Markdown extractor corrupts code blocks containing angle brackets (<...> sequences are silently deleted) #41986): Preserve angle brackets (<...>) in fenced code blocks while cleaning HTML tags outside code blocks.
  4. Extract Processor (load_from_url misroutes files when Content-Type has parameters (e.g. "application/pdf; charset=binary") #41955): Strip Content-Type parameters (e.g. ; charset=binary) before deriving file extension suffix.

Files Changed (8 files)

  • api/core/rag/extractor/notion_extractor.py
  • api/core/rag/extractor/word_extractor.py
  • api/core/rag/extractor/markdown_extractor.py
  • api/core/rag/extractor/extract_processor.py
  • api/tests/unit_tests/core/rag/extractor/test_notion_extractor.py
  • api/tests/unit_tests/core/rag/extractor/test_word_extractor.py
  • api/tests/unit_tests/core/rag/extractor/test_markdown_extractor.py
  • api/tests/unit_tests/core/rag/extractor/test_extract_processor.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant