360f5c1179
Move token related functions to common ( #10942 )
...
### What problem does this PR solve?
As title
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2025-11-03 08:50:05 +08:00
44f2d6f5da
Move 'get_project_base_directory' to common directory ( #10940 )
...
### What problem does this PR solve?
As title
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2025-11-02 21:05:28 +08:00
6447b737ab
Move singleton to common directory ( #10935 )
...
### What problem does this PR solve?
As title
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2025-11-02 12:24:08 +08:00
0ecccd27eb
Refactor:improve the logic for rerank models to cal the total token count ( #10882 )
...
### What problem does this PR solve?
improve the logic for rerank models to cal the total token count
### Type of change
- [x] Refactoring
2025-10-31 09:46:16 +08:00
d86d7061ea
Refactor: Improve how to get total token count for AnthropicCV ( #10658 )
...
### What problem does this PR solve?
Improve how to get total token count for AnthropicCV
### Type of change
- [x] Refactoring
2025-10-29 09:41:15 +08:00
766d900a41
Refactor: rename rmSpace to remove_redundant_spaces ( #10796 )
...
### What problem does this PR solve?
- rename rmSpace to remove_redundant_spaces
- move clean_markdown_block to common module
- add unit tests for remove_redundant_spaces and clean_markdown_block
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2025-10-28 09:46:32 +08:00
4585edc20e
Refactor: improve cv model logics ( #10414 )
...
1. improve how to get total token count
Improve how to get total token count
### Type of change
- [x] Refactoring
2025-10-09 09:47:36 +08:00
94dbd4aac9
Refactor: use the same implement for total token count from res ( #10197 )
...
### What problem does this PR solve?
use the same implement for total token count from res
### Type of change
- [x] Refactoring
2025-09-22 17:17:06 +08:00
49086964b8
Fix: type violations. ( #6262 )
...
### What problem does this PR solve?
#6238
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2025-03-19 12:12:34 +08:00
6e8d0e3177
Fix: rank feat issue. ( #6225 )
...
### What problem does this PR solve?
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2025-03-18 16:07:29 +08:00
5cf610af40
Feat: add vision LLM PDF parser ( #6173 )
...
### What problem does this PR solve?
Add vision LLM PDF parser
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com >
2025-03-18 14:52:20 +08:00
1333d3c02a
Fix: float transfer exception. ( #6197 )
...
### What problem does this PR solve?
#6177
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2025-03-18 11:13:44 +08:00
c5da3cdd97
Tagging ( #4426 )
...
### What problem does this PR solve?
#4367
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2025-01-09 17:07:21 +08:00
0d68a6cd1b
Fix errors detected by Ruff ( #3918 )
...
### What problem does this PR solve?
Fix errors detected by Ruff
### Type of change
- [x] Refactoring
2024-12-08 14:21:12 +08:00
996c94a8e7
Move clk100k_base tokenizer to docker image ( #3411 )
...
### What problem does this PR solve?
Move the tiktoken of cl100k_base into docker image
issue: #3338
### Type of change
- [x] Refactoring
Signed-off-by: jinhai <haijin.chn@gmail.com >
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com >
2024-11-15 10:18:40 +08:00
34abcf7704
style: fix typo and format code ( #2618 )
...
### What problem does this PR solve?
- Fix typo
- Remove unused import
- Format code
### Type of change
- [x] Other (please describe): typo and format
2024-09-27 13:17:25 +08:00
9d4bb5767c
make highlight friendly to English ( #2417 )
...
### What problem does this PR solve?
#2415
### Type of change
- [x] Performance Improvement
2024-09-13 17:03:51 +08:00
59ba34e167
fix: Fix return type annotation in truncate function ( #2044 )
...
### What problem does this PR solve?
Fix return type annotation in truncate function
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-22 09:29:02 +08:00
fdd5b1b8cf
Fix token list , stats in api app.py ( #1896 )
...
### What problem does this PR solve?
#1842
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-09 19:03:01 +08:00
9ed0e50f6b
Update info ( #1005 )
...
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2024-05-31 09:53:04 +08:00
4153a36683
truncate text to fitin embedding model ( #692 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-05-09 11:35:08 +08:00
9d60a84958
refactor code ( #583 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-28 13:19:54 +08:00
f89c6c9d59
rm useless warning message ( #201 )
...
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
Issue link:#[[Link the issue
here](https://github.com/infiniflow/ragflow/issues/200 )]
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-02 13:50:05 +08:00
de09b0e1a4
resolve table issues ( #125 )
2024-03-15 14:59:28 +08:00
4a858d33b6
add conversation API ( #35 )
2024-01-18 19:28:37 +08:00
3198faf2d2
add alot of api ( #23 )
...
* clean rust version project
* clean rust version project
* build python version rag-flow
* add alot of api
2024-01-15 19:47:25 +08:00
30791976d5
build python version rag-flow ( #21 )
...
* clean rust version project
* clean rust version project
* build python version rag-flow
2024-01-15 08:46:22 +08:00
