pengjunjie/ragflow
1
0
Fork 1
mirror of https://github.com/infiniflow/ragflow.git synced 2025-12-08 20:42:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix token list , stats in api app.py (#1896)

Browse Source 
### What problem does this PR solve?

#1842 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
This commit is contained in:
H
2024-08-09 19:03:01 +08:00
committed by GitHub
parent 827042f72b
commit fdd5b1b8cf
3 changed files with 14 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
rag/utils/__init__.py
View File

@ -74,10 +74,15 @@ def findMaxTm(fnm):
encoder = tiktoken.encoding_for_model("gpt-3.5-turbo")
def num_tokens_from_string(string: str) -> int:
"""Returns the number of tokens in a text string."""
num_tokens = len(encoder.encode(string))
return num_tokens
try:
num_tokens = len(encoder.encode(string))
return num_tokens
except Exception as e:
pass
return 0
def truncate(string: str, max_len: int) -> int: