Jin Hai 360f5c1179 Move token related functions to common (#10942) ...

### What problem does this PR solve?

As title

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>

2025-11-03 08:50:05 +08:00

..

resume

Fix: resolve regex library warnings (#7782)

2025-05-22 10:06:28 +08:00

__init__.py

Feat: advanced markdown parsing (#9607)

2025-08-21 09:36:18 +08:00

docling_parser.py

Feat: add Docling parser (#10759)

2025-10-23 19:44:25 +08:00

docx_parser.py

Refactor parser code (#9042)

2025-07-25 12:04:07 +08:00

excel_parser.py

Fix: parsing excel with chartsheet & Clamp begin to a minimum of 0 to prevent negative indexing (#10819)

2025-10-28 09:40:37 +08:00

figure_parser.py

Fix:wrong param in manual chunk (#10710)

2025-10-21 20:10:54 +08:00

html_parser.py

Fix: set default chunk_token_num in html_parser (#10118)

2025-09-17 09:36:31 +08:00

json_parser.py

Feat: parsing supports jsonl or ldjson format (#9087)

2025-07-30 09:48:20 +08:00

markdown_parser.py

Fix:enhance delimiters in markdown parser (#10896)

2025-10-30 17:36:51 +08:00

mineru_parser.py

Fix:KeyError: 'table_body' of mineru parser (#10773)

2025-10-31 10:07:56 +08:00

pdf_parser.py

Move 'get_project_base_directory' to common directory (#10940)

2025-11-02 21:05:28 +08:00

ppt_parser.py

fix "TypeError: '<' not supported between instances of 'Emu' and 'Non… (#9209)

2025-08-04 16:07:03 +08:00

tcadp_parser.py

Feat: add TCADP Parser (#10775)

2025-10-27 15:14:58 +08:00

txt_parser.py

Move token related functions to common (#10942)

2025-11-03 08:50:05 +08:00

utils.py

Update comments (#4569)

2025-01-21 20:52:28 +08:00