Import rag_tokenizer from Infinity (#11647)

### What problem does this PR solve?

- Original rag/nlp/rag_tokenizer.py is put to Infinity and infinity-sdk
via https://github.com/infiniflow/infinity/pull/3117 .
Import rag_tokenizer from infinity and inherit from
rag_tokenizer.RagTokenizer in new rag/nlp/rag_tokenizer.py.

- Bump infinity to 0.6.8

### Type of change
- [x] Refactoring
This commit is contained in:
qinling0210
2025-12-02 14:59:37 +08:00
committed by GitHub
parent e3987e21b9
commit 2ffe6f7439
7 changed files with 3712 additions and 4082 deletions

View File

@ -96,7 +96,7 @@ ragflow:
infinity:
image:
repository: infiniflow/infinity
tag: v0.6.7
tag: v0.6.8
pullPolicy: IfNotPresent
pullSecrets: []
storage: