Import rag_tokenizer from Infinity (#11647)

### What problem does this PR solve?

- Original rag/nlp/rag_tokenizer.py is put to Infinity and infinity-sdk
via https://github.com/infiniflow/infinity/pull/3117 .
Import rag_tokenizer from infinity and inherit from
rag_tokenizer.RagTokenizer in new rag/nlp/rag_tokenizer.py.

- Bump infinity to 0.6.8

### Type of change
- [x] Refactoring
This commit is contained in:
qinling0210
2025-12-02 14:59:37 +08:00
committed by GitHub
parent e3987e21b9
commit 2ffe6f7439
7 changed files with 3712 additions and 4082 deletions

View File

@ -1,5 +1,5 @@
[general]
version = "0.6.7"
version = "0.6.8"
time_zone = "utc-8"
[network]