Update pyproject.toml to 0.15.1 (#4222)

### What problem does this PR solve?

Missing update information

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>

.github/workflows/release.yml

@@ -119,6 +119,6 @@ jobs:
         if: startsWith(github.ref, 'refs/tags/v')
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          packages-dir: dist/
+          packages-dir: sdk/python/dist/
           password: ${{ secrets.PYPI_API_TOKEN }}
           verbose: true

README.md

@@ -168,7 +168,7 @@ releases! 🌟
 
 3. Start up the server using the pre-built Docker images:
 
-   > The command below downloads the `v0.15.0-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download an RAGFlow edition different from `v0.14.1-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1` for the full edition `v0.14.1`.
+   > The command below downloads the `v0.15.0-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download an RAGFlow edition different from `v0.15.0-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0` for the full edition `v0.15.0`.
 
    ```bash
    $ cd ragflow

README_id.md

@@ -161,7 +161,7 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
 
 3. Bangun image Docker pre-built dan jalankan server:
 
-   > Perintah di bawah ini mengunduh edisi v0.15.0-slim dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.14.1-slim, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server. Misalnya, atur RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1 untuk edisi lengkap v0.14.1.
+   > Perintah di bawah ini mengunduh edisi v0.15.0-slim dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.15.0-slim, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server. Misalnya, atur RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0 untuk edisi lengkap v0.15.0.
 
    ```bash
    $ cd ragflow

README_ja.md

@@ -142,7 +142,7 @@
 
 3. ビルド済みの Docker イメージをビルドし、サーバーを起動する:
 
-   > 以下のコマンドは、RAGFlow Dockerイメージの v0.15.0-slim エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.15.0-slim とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。例えば、完全版 v0.14.1 をダウンロードするには、RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1 と設定します。
+   > 以下のコマンドは、RAGFlow Dockerイメージの v0.15.0-slim エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.15.0-slim とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。例えば、完全版 v0.15.0 をダウンロードするには、RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0 と設定します。
 
    ```bash
    $ cd ragflow

README_ko.md

@@ -147,7 +147,7 @@
 
 3. 미리 빌드된 Docker 이미지를 생성하고 서버를 시작하세요:
 
-   > 아래 명령어는 RAGFlow Docker 이미지의 v0.15.0-slim 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.15.0-slim과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오. 예를 들어, 전체 버전인 v0.14.1을 다운로드하려면 RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1로 설정합니다.
+   > 아래 명령어는 RAGFlow Docker 이미지의 v0.15.0-slim 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.15.0-slim과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오. 예를 들어, 전체 버전인 v0.15.0을 다운로드하려면 RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0로 설정합니다.
 
    ```bash
    $ cd ragflow

README_zh.md

@@ -143,7 +143,7 @@
 
 3. 进入 **docker** 文件夹，利用提前编译好的 Docker 镜像启动服务器：
 
-   > 运行以下命令会自动下载 RAGFlow slim Docker 镜像 `v0.15.0-slim`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.15.0-slim` 的 Docker 镜像，请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。比如，你可以通过设置 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1` 来下载 RAGFlow 镜像的 `v0.14.1` 完整发行版。
+   > 运行以下命令会自动下载 RAGFlow slim Docker 镜像 `v0.15.0-slim`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.15.0-slim` 的 Docker 镜像，请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。比如，你可以通过设置 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0` 来下载 RAGFlow 镜像的 `v0.15.0` 完整发行版。
 
    ```bash
    $ cd ragflow

agent/component/base.py

@@ -477,6 +477,13 @@ class ComponentBase(ABC):
                             assert False, f"Can't find parameter '{key}' for {cpn_id}"
                         continue
 
+                    if q["component_id"].lower().find("answer") == 0:
+                        for r, c in self._canvas.history[::-1]:
+                            if r == "user":
+                                self._param.inputs.append(pd.DataFrame([{"content": c, "component_id": q["component_id"]}]))
+                                break
+                        continue
+
                     outs.append(self._canvas.get_component(q["component_id"])["obj"].output(allow_partial=False)[1])
                     self._param.inputs.append({"component_id": q["component_id"],
                                                "content": "\n".join(

api/apps/sdk/chat.py

@@ -267,9 +267,10 @@ def delete(tenant_id):
 def list_chat(tenant_id):
     id = request.args.get("id")
     name = request.args.get("name")
-    chat = DialogService.query(id=id,name=name,status=StatusEnum.VALID.value,tenant_id=tenant_id)
-    if not chat:
-        return get_error_data_result(message="The chat doesn't exist")
+    if id or name:
+        chat = DialogService.query(id=id, name=name, status=StatusEnum.VALID.value, tenant_id=tenant_id)
+        if not chat:
+            return get_error_data_result(message="The chat doesn't exist")
     page_number = int(request.args.get("page", 1))
     items_per_page = int(request.args.get("page_size", 30))
     orderby = request.args.get("orderby", "create_time")

api/apps/sdk/doc.py

@@ -42,9 +42,30 @@ from rag.nlp import search
 from rag.utils import rmSpace
 from rag.utils.storage_factory import STORAGE_IMPL
 
+from pydantic import BaseModel, Field, validator
+
 MAXIMUM_OF_UPLOADING_FILES = 256
 
 
+class Chunk(BaseModel):
+    id: str = ""
+    content: str = ""
+    document_id: str = ""
+    docnm_kwd: str = ""
+    important_keywords: list = Field(default_factory=list)
+    questions: list = Field(default_factory=list)
+    question_tks: str = ""
+    image_id: str = ""
+    available: bool = True
+    positions: list[list[int]] = Field(default_factory=list)
+
+    @validator('positions')
+    def validate_positions(cls, value):
+        for sublist in value:
+            if len(sublist) != 5:
+                raise ValueError("Each sublist in positions must have a length of 5")
+        return value
+
 @manager.route("/datasets/<dataset_id>/documents", methods=["POST"])  # noqa: F821
 @token_required
 def upload(dataset_id, tenant_id):
@@ -848,20 +869,6 @@ def list_chunks(tenant_id, dataset_id, document_id):
                 "available_int": sres.field[id].get("available_int", 1),
                 "positions": sres.field[id].get("position_int", []),
             }
-            if len(d["positions"]) % 5 == 0:
-                poss = []
-                for i in range(0, len(d["positions"]), 5):
-                    poss.append(
-                        [
-                            float(d["positions"][i]),
-                            float(d["positions"][i + 1]),
-                            float(d["positions"][i + 2]),
-                            float(d["positions"][i + 3]),
-                            float(d["positions"][i + 4]),
-                        ]
-                    )
-                d["positions"] = poss
-
             origin_chunks.append(d)
             if req.get("id"):
                 if req.get("id") == id:
@@ -892,6 +899,7 @@ def list_chunks(tenant_id, dataset_id, document_id):
         if renamed_chunk["available"] == 1:
             renamed_chunk["available"] = True
         res["chunks"].append(renamed_chunk)
+        _ = Chunk(**renamed_chunk) # validate the chunk
     return get_result(data=res)
 
 
@@ -1031,6 +1039,7 @@ def add_chunk(tenant_id, dataset_id, document_id):
         if key in key_mapping:
             new_key = key_mapping.get(key, key)
             renamed_chunk[new_key] = value
+    _ = Chunk(**renamed_chunk)  # validate the chunk
     return get_result(data={"chunk": renamed_chunk})
     # return get_result(data={"chunk_id": chunk_id})
 

api/apps/sdk/session.py

@@ -47,7 +47,8 @@ def create(tenant_id, chat_id):
         "id": get_uuid(),
         "dialog_id": req["dialog_id"],
         "name": req.get("name", "New session"),
-        "message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}]
+        "message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}],
+        "user_id": req.get("user_id", "")
     }
     if not conv.get("name"):
         return get_error_data_result(message="`name` can not be empty.")
@@ -65,23 +66,40 @@ def create(tenant_id, chat_id):
 @manager.route('/agents/<agent_id>/sessions', methods=['POST'])  # noqa: F821
 @token_required
 def create_agent_session(tenant_id, agent_id):
+    req = request.json
     e, cvs = UserCanvasService.get_by_id(agent_id)
     if not e:
         return get_error_data_result("Agent not found.")
 
+    if not UserCanvasService.query(user_id=tenant_id, id=agent_id):
+        return get_error_data_result("You cannot access the agent.")
+
     if not isinstance(cvs.dsl, str):
         cvs.dsl = json.dumps(cvs.dsl, ensure_ascii=False)
 
     canvas = Canvas(cvs.dsl, tenant_id)
-    if canvas.get_preset_param():
-        return get_error_data_result("The agent can't create a session directly")
+    canvas.reset()
+    query = canvas.get_preset_param()
+    if query:
+        for ele in query:
+            if not ele["optional"]:
+                if not req.get(ele["key"]):
+                    return get_error_data_result(f"`{ele['key']}` is required")
+                ele["value"] = req[ele["key"]]
+            if ele["optional"]:
+                if req.get(ele["key"]):
+                    ele["value"] = req[ele['key']]
+                else:
+                    if "value" in ele:
+                        ele.pop("value")
+    cvs.dsl = json.loads(str(canvas))
     conv = {
         "id": get_uuid(),
         "dialog_id": cvs.id,
-        "user_id": tenant_id,
+        "user_id": req.get("user_id", "") if isinstance(req, dict) else "",
         "message": [{"role": "assistant", "content": canvas.get_prologue()}],
         "source": "agent",
-        "dsl": json.loads(cvs.dsl)
+        "dsl": cvs.dsl
     }
     API4ConversationService.save(**conv)
     conv["agent_id"] = conv.pop("dialog_id")
@@ -115,11 +133,11 @@ def update(tenant_id, chat_id, session_id):
 def chat_completion(tenant_id, chat_id):
     req = request.json
     if not req or not req.get("session_id"):
-        req = {"question":""}
-    if not DialogService.query(tenant_id=tenant_id,id=chat_id,status=StatusEnum.VALID.value):
+        req = {"question": ""}
+    if not DialogService.query(tenant_id=tenant_id, id=chat_id, status=StatusEnum.VALID.value):
         return get_error_data_result(f"You don't own the chat {chat_id}")
     if req.get("session_id"):
-        if not ConversationService.query(id=req["session_id"],dialog_id=chat_id):
+        if not ConversationService.query(id=req["session_id"], dialog_id=chat_id):
             return get_error_data_result(f"You don't own the session {req['session_id']}")
     if req.get("stream", True):
         resp = Response(rag_completion(tenant_id, chat_id, **req), mimetype="text/event-stream")
@@ -140,28 +158,34 @@ def chat_completion(tenant_id, chat_id):
 @manager.route('/agents/<agent_id>/completions', methods=['POST'])  # noqa: F821
 @token_required
 def agent_completions(tenant_id, agent_id):
-        req = request.json
-        cvs = UserCanvasService.query(user_id=tenant_id, id=agent_id)
-        if not cvs:
-            return get_error_data_result(f"You don't own the agent {agent_id}")
-        if req.get("session_id"):
-            conv = API4ConversationService.query(id=req["session_id"], dialog_id=agent_id)
-            if not conv:
-                return get_error_data_result(f"You don't own the session {req['session_id']}")
-        else:
-            req["question"]=""
-        if req.get("stream", True):
-            resp = Response(agent_completion(tenant_id, agent_id, **req), mimetype="text/event-stream")
-            resp.headers.add_header("Cache-control", "no-cache")
-            resp.headers.add_header("Connection", "keep-alive")
-            resp.headers.add_header("X-Accel-Buffering", "no")
-            resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
-            return resp
-        try:
-            for answer in agent_completion(tenant_id, agent_id, **req):
-                return get_result(data=answer)
-        except Exception as e:
-            return get_error_data_result(str(e))
+    req = request.json
+    cvs = UserCanvasService.query(user_id=tenant_id, id=agent_id)
+    if not cvs:
+        return get_error_data_result(f"You don't own the agent {agent_id}")
+    if req.get("session_id"):
+        dsl = cvs[0].dsl
+        if not isinstance(dsl, str):
+            dsl = json.dumps(dsl)
+        canvas = Canvas(dsl, tenant_id)
+        if canvas.get_preset_param():
+            req["question"] = ""
+        conv = API4ConversationService.query(id=req["session_id"], dialog_id=agent_id)
+        if not conv:
+            return get_error_data_result(f"You don't own the session {req['session_id']}")
+    else:
+        req["question"] = ""
+    if req.get("stream", True):
+        resp = Response(agent_completion(tenant_id, agent_id, **req), mimetype="text/event-stream")
+        resp.headers.add_header("Cache-control", "no-cache")
+        resp.headers.add_header("Connection", "keep-alive")
+        resp.headers.add_header("X-Accel-Buffering", "no")
+        resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
+        return resp
+    try:
+        for answer in agent_completion(tenant_id, agent_id, **req):
+            return get_result(data=answer)
+    except Exception as e:
+        return get_error_data_result(str(e))
 
 
 @manager.route('/chats/<chat_id>/sessions', methods=['GET'])  # noqa: F821
@@ -174,11 +198,12 @@ def list_session(tenant_id, chat_id):
     page_number = int(request.args.get("page", 1))
     items_per_page = int(request.args.get("page_size", 30))
     orderby = request.args.get("orderby", "create_time")
+    user_id = request.args.get("user_id")
     if request.args.get("desc") == "False" or request.args.get("desc") == "false":
         desc = False
     else:
         desc = True
-    convs = ConversationService.get_list(chat_id, page_number, items_per_page, orderby, desc, id, name)
+    convs = ConversationService.get_list(chat_id, page_number, items_per_page, orderby, desc, id, name, user_id)
     if not convs:
         return get_result(data=[])
     for conv in convs:
@@ -199,17 +224,15 @@ def list_session(tenant_id, chat_id):
                         chunks = conv["reference"][chunk_num]["chunks"]
                         for chunk in chunks:
                             new_chunk = {
-                                "id": chunk["chunk_id"],
-                                "content": chunk["content_with_weight"],
-                                "document_id": chunk["doc_id"],
-                                "document_name": chunk["docnm_kwd"],
-                                "dataset_id": chunk["kb_id"],
-                                "image_id": chunk.get("image_id", ""),
-                                "similarity": chunk["similarity"],
-                                "vector_similarity": chunk["vector_similarity"],
-                                "term_similarity": chunk["term_similarity"],
-                                "positions": chunk["positions"],
+                                "id": chunk.get("chunk_id", chunk.get("id")),
+                                "content": chunk.get("content_with_weight", chunk.get("content")),
+                                "document_id": chunk.get("doc_id", chunk.get("document_id")),
+                                "document_name": chunk.get("docnm_kwd", chunk.get("document_name")),
+                                "dataset_id": chunk.get("kb_id", chunk.get("dataset_id")),
+                                "image_id": chunk.get("image_id", chunk.get("img_id")),
+                                "positions": chunk.get("positions", chunk.get("position_int")),
                             }
+
                             chunk_list.append(new_chunk)
                     chunk_num += 1
                     messages[message_num]["reference"] = chunk_list
@@ -224,8 +247,6 @@ def list_agent_session(tenant_id, agent_id):
     if not UserCanvasService.query(user_id=tenant_id, id=agent_id):
         return get_error_data_result(message=f"You don't own the agent {agent_id}.")
     id = request.args.get("id")
-    if not API4ConversationService.query(id=id, user_id=tenant_id):
-        return get_error_data_result(f"You don't own the session {id}")
     page_number = int(request.args.get("page", 1))
     items_per_page = int(request.args.get("page_size", 30))
     orderby = request.args.get("orderby", "update_time")
@@ -254,16 +275,13 @@ def list_agent_session(tenant_id, agent_id):
                         chunks = conv["reference"][chunk_num]["chunks"]
                         for chunk in chunks:
                             new_chunk = {
-                                "id": chunk["chunk_id"],
-                                "content": chunk["content"],
-                                "document_id": chunk["doc_id"],
-                                "document_name": chunk["docnm_kwd"],
-                                "dataset_id": chunk["kb_id"],
-                                "image_id": chunk.get("image_id", ""),
-                                "similarity": chunk["similarity"],
-                                "vector_similarity": chunk["vector_similarity"],
-                                "term_similarity": chunk["term_similarity"],
-                                "positions": chunk["positions"],
+                                "id": chunk.get("chunk_id", chunk.get("id")),
+                                "content": chunk.get("content_with_weight", chunk.get("content")),
+                                "document_id": chunk.get("doc_id", chunk.get("document_id")),
+                                "document_name": chunk.get("docnm_kwd", chunk.get("document_name")),
+                                "dataset_id": chunk.get("kb_id", chunk.get("dataset_id")),
+                                "image_id": chunk.get("image_id", chunk.get("img_id")),
+                                "positions": chunk.get("positions", chunk.get("position_int")),
                             }
                             chunk_list.append(new_chunk)
                     chunk_num += 1
@@ -429,5 +447,3 @@ def agent_bot_completions(agent_id):
 
     for answer in agent_completion(objs[0].tenant_id, agent_id, **req):
         return get_result(data=answer)
-
-

api/db/db_models.py

@@ -31,7 +31,6 @@ from peewee import (
 )
 from playhouse.pool import PooledMySQLDatabase, PooledPostgresqlDatabase
 
-
 from api.db import SerializedType, ParserType
 from api import settings
 from api import utils
@@ -926,6 +925,7 @@ class Conversation(DataBaseModel):
     name = CharField(max_length=255, null=True, help_text="converastion name", index=True)
     message = JSONField(null=True)
     reference = JSONField(null=True, default=[])
+    user_id = CharField(max_length=255, null=True, help_text="user_id", index=True)
 
     class Meta:
         db_table = "conversation"
@@ -1070,13 +1070,13 @@ def migrate_db():
             pass
         try:
             migrate(
-                migrator.add_column("tenant_llm","max_tokens",IntegerField(default=8192,index=True))
+                migrator.add_column("tenant_llm", "max_tokens", IntegerField(default=8192, index=True))
             )
         except Exception:
             pass
         try:
             migrate(
-                migrator.add_column("api_4_conversation","dsl",JSONField(null=True, default={}))
+                migrator.add_column("api_4_conversation", "dsl", JSONField(null=True, default={}))
             )
         except Exception:
             pass
@@ -1105,3 +1105,10 @@ def migrate_db():
             )
         except Exception:
             pass
+        try:
+            migrate(
+                migrator.add_column("conversation", "user_id",
+                                    CharField(max_length=255, null=True, help_text="user_id", index=True))
+            )
+        except Exception:
+            pass

api/db/services/api_service.py

@@ -41,11 +41,14 @@ class API4ConversationService(CommonService):
 
     @classmethod
     @DB.connection_context()
-    def get_list(cls,dialog_id, tenant_id,
-                 page_number, items_per_page, orderby, desc, id):
-        sessions = cls.model.select().where(cls.model.dialog_id ==dialog_id)
+    def get_list(cls, dialog_id, tenant_id,
+                 page_number, items_per_page,
+                 orderby, desc, id, user_id=None):
+        sessions = cls.model.select().where(cls.model.dialog_id == dialog_id)
         if id:
             sessions = sessions.where(cls.model.id == id)
+        if user_id:
+            sessions = sessions.where(cls.model.user_id == user_id)
         if desc:
             sessions = sessions.order_by(cls.model.getter_by(orderby).desc())
         else:

api/db/services/canvas_service.py

@@ -74,21 +74,32 @@ def completion(tenant_id, agent_id, question, session_id=None, stream=True, **kw
                     else:
                         if "value" in ele:
                             ele.pop("value")
-            cvs.dsl = json.loads(str(canvas))
-            temp_dsl = cvs.dsl
-            UserCanvasService.update_by_id(agent_id, cvs.to_dict())
-        else:
-            temp_dsl = json.loads(cvs.dsl)
-        session_id = get_uuid()
+        cvs.dsl = json.loads(str(canvas))
+        session_id=get_uuid()
         conv = {
             "id": session_id,
             "dialog_id": cvs.id,
-            "user_id": kwargs.get("user_id", ""),
+            "user_id": kwargs.get("usr_id", "") if isinstance(kwargs, dict) else "",
+            "message": [{"role": "assistant", "content": canvas.get_prologue()}],
             "source": "agent",
-            "dsl": temp_dsl
+            "dsl": cvs.dsl
         }
         API4ConversationService.save(**conv)
-        conv = API4Conversation(**conv)
+        if query:
+            yield "data:" + json.dumps({"code": 0,
+                                        "message": "",
+                                        "data": {
+                                            "session_id": session_id,
+                                            "answer": canvas.get_prologue(),
+                                            "reference": [],
+                                            "param": canvas.get_preset_param()
+                                        }
+                                        },
+                                       ensure_ascii=False) + "\n\n"
+            yield "data:" + json.dumps({"code": 0, "message": "", "data": True}, ensure_ascii=False) + "\n\n"
+            return
+        else:
+            conv = API4Conversation(**conv)
     else:
         e, conv = API4ConversationService.get_by_id(session_id)
         assert e, "Session not found!"

api/db/services/conversation_service.py

@@ -28,12 +28,14 @@ class ConversationService(CommonService):
 
     @classmethod
     @DB.connection_context()
-    def get_list(cls,dialog_id,page_number, items_per_page, orderby, desc, id , name):
-        sessions = cls.model.select().where(cls.model.dialog_id ==dialog_id)
+    def get_list(cls, dialog_id, page_number, items_per_page, orderby, desc, id, name, user_id=None):
+        sessions = cls.model.select().where(cls.model.dialog_id == dialog_id)
         if id:
             sessions = sessions.where(cls.model.id == id)
         if name:
             sessions = sessions.where(cls.model.name == name)
+        if user_id:
+            sessions = sessions.where(cls.model.user_id == user_id)
         if desc:
             sessions = sessions.order_by(cls.model.getter_by(orderby).desc())
         else:
@@ -52,15 +54,16 @@ def structure_answer(conv, ans, message_id, session_id):
 
     def get_value(d, k1, k2):
         return d.get(k1, d.get(k2))
+
     chunk_list = [{
-            "id": get_value(chunk, "chunk_id", "id"),
-            "content": get_value(chunk, "content", "content_with_weight"),
-            "document_id": get_value(chunk, "doc_id", "document_id"),
-            "document_name": get_value(chunk, "docnm_kwd", "document_name"),
-            "dataset_id": get_value(chunk, "kb_id", "dataset_id"),
-            "image_id": get_value(chunk, "image_id", "img_id"),
-            "positions": get_value(chunk, "positions", "position_int"),
-        } for chunk in reference.get("chunks", [])]
+        "id": get_value(chunk, "chunk_id", "id"),
+        "content": get_value(chunk, "content", "content_with_weight"),
+        "document_id": get_value(chunk, "doc_id", "document_id"),
+        "document_name": get_value(chunk, "docnm_kwd", "document_name"),
+        "dataset_id": get_value(chunk, "kb_id", "dataset_id"),
+        "image_id": get_value(chunk, "image_id", "img_id"),
+        "positions": get_value(chunk, "positions", "position_int"),
+    } for chunk in reference.get("chunks", [])]
 
     reference["chunks"] = chunk_list
     ans["id"] = message_id
@@ -88,10 +91,11 @@ def completion(tenant_id, chat_id, question, name="New session", session_id=None
     if not session_id:
         session_id = get_uuid()
         conv = {
-            "id":session_id ,
+            "id": session_id,
             "dialog_id": chat_id,
             "name": name,
-            "message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}]
+            "message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}],
+            "user_id": kwargs.get("user_id", "")
         }
         ConversationService.save(**conv)
         yield "data:" + json.dumps({"code": 0, "message": "",
@@ -226,4 +230,3 @@ def iframe_completion(dialog_id, question, session_id=None, stream=True, **kwarg
             API4ConversationService.append_message(conv.id, conv.to_dict())
             break
         yield answer
-

api/db/services/dialog_service.py

@@ -23,7 +23,7 @@ from copy import deepcopy
 from timeit import default_timer as timer
 import datetime
 from datetime import timedelta
-from api.db import LLMType, ParserType,StatusEnum
+from api.db import LLMType, ParserType, StatusEnum
 from api.db.db_models import Dialog, DB
 from api.db.services.common_service import CommonService
 from api.db.services.knowledgebase_service import KnowledgebaseService
@@ -41,14 +41,14 @@ class DialogService(CommonService):
     @classmethod
     @DB.connection_context()
     def get_list(cls, tenant_id,
-                 page_number, items_per_page, orderby, desc, id , name):
+                 page_number, items_per_page, orderby, desc, id, name):
         chats = cls.model.select()
         if id:
             chats = chats.where(cls.model.id == id)
         if name:
             chats = chats.where(cls.model.name == name)
         chats = chats.where(
-              (cls.model.tenant_id == tenant_id)
+            (cls.model.tenant_id == tenant_id)
             & (cls.model.status == StatusEnum.VALID.value)
         )
         if desc:
@@ -124,7 +124,7 @@ def kb_prompt(kbinfos, max_tokens):
     for i, ck in enumerate(kbinfos["chunks"]):
         if i >= chunks_num:
             break
-        doc2chunks["docnm_kwd"].append(ck["content_with_weight"])
+        doc2chunks[ck["docnm_kwd"]].append(ck["content_with_weight"])
 
     knowledges = []
     for nm, chunks in doc2chunks.items():
@@ -137,25 +137,37 @@ def kb_prompt(kbinfos, max_tokens):
 
 def chat(dialog, messages, stream=True, **kwargs):
     assert messages[-1]["role"] == "user", "The last content of this conversation is not from user."
-    st = timer()
-    llm_id, fid = TenantLLMService.split_model_name_and_factory(dialog.llm_id)
-    llm = LLMService.query(llm_name=llm_id) if not fid else LLMService.query(llm_name=llm_id, fid=fid)
+
+    chat_start_ts = timer()
+
+    # Get llm model name and model provider name
+    llm_id, model_provider = TenantLLMService.split_model_name_and_factory(dialog.llm_id)
+
+    # Get llm model instance by model and provide name
+    llm = LLMService.query(llm_name=llm_id) if not model_provider else LLMService.query(llm_name=llm_id, fid=model_provider)
+
     if not llm:
-        llm = TenantLLMService.query(tenant_id=dialog.tenant_id, llm_name=llm_id) if not fid else \
-            TenantLLMService.query(tenant_id=dialog.tenant_id, llm_name=llm_id, llm_factory=fid)
+        # Model name is provided by tenant, but not system built-in
+        llm = TenantLLMService.query(tenant_id=dialog.tenant_id, llm_name=llm_id) if not model_provider else \
+            TenantLLMService.query(tenant_id=dialog.tenant_id, llm_name=llm_id, llm_factory=model_provider)
         if not llm:
             raise LookupError("LLM(%s) not found" % dialog.llm_id)
         max_tokens = 8192
     else:
         max_tokens = llm[0].max_tokens
+
+    check_llm_ts = timer()
+
     kbs = KnowledgebaseService.get_by_ids(dialog.kb_ids)
-    embd_nms = list(set([kb.embd_id for kb in kbs]))
-    if len(embd_nms) != 1:
+    embedding_list = list(set([kb.embd_id for kb in kbs]))
+    if len(embedding_list) != 1:
         yield {"answer": "**ERROR**: Knowledge bases use different embedding models.", "reference": []}
         return {"answer": "**ERROR**: Knowledge bases use different embedding models.", "reference": []}
 
-    is_kg = all([kb.parser_id == ParserType.KG for kb in kbs])
-    retr = settings.retrievaler if not is_kg else settings.kg_retrievaler
+    embedding_model_name = embedding_list[0]
+
+    is_knowledge_graph = all([kb.parser_id == ParserType.KG for kb in kbs])
+    retriever = settings.retrievaler if not is_knowledge_graph else settings.kg_retrievaler
 
     questions = [m["content"] for m in messages if m["role"] == "user"][-3:]
     attachments = kwargs["doc_ids"].split(",") if "doc_ids" in kwargs else None
@@ -165,15 +177,21 @@ def chat(dialog, messages, stream=True, **kwargs):
             if "doc_ids" in m:
                 attachments.extend(m["doc_ids"])
 
-    embd_mdl = LLMBundle(dialog.tenant_id, LLMType.EMBEDDING, embd_nms[0])
+    create_retriever_ts = timer()
+
+    embd_mdl = LLMBundle(dialog.tenant_id, LLMType.EMBEDDING, embedding_model_name)
     if not embd_mdl:
-        raise LookupError("Embedding model(%s) not found" % embd_nms[0])
+        raise LookupError("Embedding model(%s) not found" % embedding_model_name)
+
+    bind_embedding_ts = timer()
 
     if llm_id2llm_type(dialog.llm_id) == "image2text":
         chat_mdl = LLMBundle(dialog.tenant_id, LLMType.IMAGE2TEXT, dialog.llm_id)
     else:
         chat_mdl = LLMBundle(dialog.tenant_id, LLMType.CHAT, dialog.llm_id)
 
+    bind_llm_ts = timer()
+
     prompt_config = dialog.prompt_config
     field_map = KnowledgebaseService.get_field_map(dialog.kb_ids)
     tts_mdl = None
@@ -200,32 +218,35 @@ def chat(dialog, messages, stream=True, **kwargs):
         questions = [full_question(dialog.tenant_id, dialog.llm_id, messages)]
     else:
         questions = questions[-1:]
-    refineQ_tm = timer()
-    keyword_tm = timer()
+
+    refine_question_ts = timer()
 
     rerank_mdl = None
     if dialog.rerank_id:
         rerank_mdl = LLMBundle(dialog.tenant_id, LLMType.RERANK, dialog.rerank_id)
 
-    for _ in range(len(questions) // 2):
-        questions.append(questions[-1])
+    bind_reranker_ts = timer()
+    generate_keyword_ts = bind_reranker_ts
+
     if "knowledge" not in [p["key"] for p in prompt_config["parameters"]]:
         kbinfos = {"total": 0, "chunks": [], "doc_aggs": []}
     else:
         if prompt_config.get("keyword", False):
             questions[-1] += keyword_extraction(chat_mdl, questions[-1])
-            keyword_tm = timer()
+            generate_keyword_ts = timer()
 
         tenant_ids = list(set([kb.tenant_id for kb in kbs]))
-        kbinfos = retr.retrieval(" ".join(questions), embd_mdl, tenant_ids, dialog.kb_ids, 1, dialog.top_n,
-                                        dialog.similarity_threshold,
-                                        dialog.vector_similarity_weight,
-                                        doc_ids=attachments,
-                                        top=dialog.top_k, aggs=False, rerank_mdl=rerank_mdl)
+        kbinfos = retriever.retrieval(" ".join(questions), embd_mdl, tenant_ids, dialog.kb_ids, 1, dialog.top_n,
+                                      dialog.similarity_threshold,
+                                      dialog.vector_similarity_weight,
+                                      doc_ids=attachments,
+                                      top=dialog.top_k, aggs=False, rerank_mdl=rerank_mdl)
+
+    retrieval_ts = timer()
+
     knowledges = kb_prompt(kbinfos, max_tokens)
     logging.debug(
         "{}->{}".format(" ".join(questions), "\n->".join(knowledges)))
-    retrieval_tm = timer()
 
     if not knowledges and prompt_config.get("empty_response"):
         empty_res = prompt_config["empty_response"]
@@ -249,17 +270,20 @@ def chat(dialog, messages, stream=True, **kwargs):
             max_tokens - used_token_count)
 
     def decorate_answer(answer):
-        nonlocal prompt_config, knowledges, kwargs, kbinfos, prompt, retrieval_tm
+        nonlocal prompt_config, knowledges, kwargs, kbinfos, prompt, retrieval_ts
+
+        finish_chat_ts = timer()
+
         refs = []
         if knowledges and (prompt_config.get("quote", True) and kwargs.get("quote", True)):
-            answer, idx = retr.insert_citations(answer,
-                                                       [ck["content_ltks"]
-                                                        for ck in kbinfos["chunks"]],
-                                                       [ck["vector"]
-                                                        for ck in kbinfos["chunks"]],
-                                                       embd_mdl,
-                                                       tkweight=1 - dialog.vector_similarity_weight,
-                                                       vtweight=dialog.vector_similarity_weight)
+            answer, idx = retriever.insert_citations(answer,
+                                                     [ck["content_ltks"]
+                                                      for ck in kbinfos["chunks"]],
+                                                     [ck["vector"]
+                                                      for ck in kbinfos["chunks"]],
+                                                     embd_mdl,
+                                                     tkweight=1 - dialog.vector_similarity_weight,
+                                                     vtweight=dialog.vector_similarity_weight)
             idx = set([kbinfos["chunks"][int(i)]["doc_id"] for i in idx])
             recall_docs = [
                 d for d in kbinfos["doc_aggs"] if d["doc_id"] in idx]
@@ -274,10 +298,20 @@ def chat(dialog, messages, stream=True, **kwargs):
 
         if answer.lower().find("invalid key") >= 0 or answer.lower().find("invalid api") >= 0:
             answer += " Please set LLM API-Key in 'User Setting -> Model providers -> API-Key'"
-        done_tm = timer()
-        prompt += "\n\n### Elapsed\n  - Refine Question: %.1f ms\n  - Keywords: %.1f ms\n  - Retrieval: %.1f ms\n  - LLM: %.1f ms" % (
-            (refineQ_tm - st) * 1000, (keyword_tm - refineQ_tm) * 1000, (retrieval_tm - keyword_tm) * 1000,
-            (done_tm - retrieval_tm) * 1000)
+        finish_chat_ts = timer()
+
+        total_time_cost = (finish_chat_ts - chat_start_ts) * 1000
+        check_llm_time_cost = (check_llm_ts - chat_start_ts) * 1000
+        create_retriever_time_cost = (create_retriever_ts - check_llm_ts) * 1000
+        bind_embedding_time_cost = (bind_embedding_ts - create_retriever_ts) * 1000
+        bind_llm_time_cost = (bind_llm_ts - bind_embedding_ts) * 1000
+        refine_question_time_cost = (refine_question_ts - bind_llm_ts) * 1000
+        bind_reranker_time_cost = (bind_reranker_ts - refine_question_ts) * 1000
+        generate_keyword_time_cost = (generate_keyword_ts - bind_reranker_ts) * 1000
+        retrieval_time_cost = (retrieval_ts - generate_keyword_ts) * 1000
+        generate_result_time_cost = (finish_chat_ts - retrieval_ts) * 1000
+
+        prompt = f"{prompt}\n\n - Total: {total_time_cost:.1f}ms\n  - Check LLM: {check_llm_time_cost:.1f}ms\n  - Create retriever: {create_retriever_time_cost:.1f}ms\n  - Bind embedding: {bind_embedding_time_cost:.1f}ms\n  - Bind LLM: {bind_llm_time_cost:.1f}ms\n  - Tune question: {refine_question_time_cost:.1f}ms\n  - Bind reranker: {bind_reranker_time_cost:.1f}ms\n  - Generate keyword: {generate_keyword_time_cost:.1f}ms\n  - Retrieval: {retrieval_time_cost:.1f}ms\n  - Generate answer: {generate_result_time_cost:.1f}ms"
         return {"answer": answer, "reference": refs, "prompt": prompt}
 
     if stream:
@@ -304,15 +338,15 @@ def chat(dialog, messages, stream=True, **kwargs):
 
 
 def use_sql(question, field_map, tenant_id, chat_mdl, quota=True):
-    sys_prompt = "你是一个DBA。你需要这对以下表的字段结构，根据用户的问题列表，写出最后一个问题对应的SQL。"
-    user_promt = """
-表名：{}；
-数据库表字段说明如下：
+    sys_prompt = "You are a Database Administrator. You need to check the fields of the following tables based on the user's list of questions and write the SQL corresponding to the last question."
+    user_prompt = """
+Table name: {};
+Table of database fields are as follows:
 {}
 
-问题如下：
+Question are as follows:
 {}
-请写出SQL, 且只要SQL，不要有其他说明及文字。
+Please write the SQL, only SQL, without any other explanations or text.
 """.format(
         index_name(tenant_id),
         "\n".join([f"{k}: {v}" for k, v in field_map.items()]),
@@ -321,10 +355,10 @@ def use_sql(question, field_map, tenant_id, chat_mdl, quota=True):
     tried_times = 0
 
     def get_table():
-        nonlocal sys_prompt, user_promt, question, tried_times
-        sql = chat_mdl.chat(sys_prompt, [{"role": "user", "content": user_promt}], {
+        nonlocal sys_prompt, user_prompt, question, tried_times
+        sql = chat_mdl.chat(sys_prompt, [{"role": "user", "content": user_prompt}], {
             "temperature": 0.06})
-        logging.debug(f"{question} ==> {user_promt} get SQL: {sql}")
+        logging.debug(f"{question} ==> {user_prompt} get SQL: {sql}")
         sql = re.sub(r"[\r\n]+", " ", sql.lower())
         sql = re.sub(r".*select ", "select ", sql.lower())
         sql = re.sub(r" +", " ", sql)
@@ -352,21 +386,23 @@ def use_sql(question, field_map, tenant_id, chat_mdl, quota=True):
     if tbl is None:
         return None
     if tbl.get("error") and tried_times <= 2:
-        user_promt = """
-        表名：{}；
-        数据库表字段说明如下：
+        user_prompt = """
+        Table name: {};
+        Table of database fields are as follows:
+        {}
+        
+        Question are as follows:
+        {}
+        Please write the SQL, only SQL, without any other explanations or text.
+        
+
+        The SQL error you provided last time is as follows:
         {}
 
-        问题如下：
+        Error issued by database as follows:
         {}
 
-        你上一次给出的错误SQL如下：
-        {}
-
-        后台报错如下：
-        {}
-
-        请纠正SQL中的错误再写一遍，且只要SQL，不要有其他说明及文字。
+        Please correct the error and write SQL again, only SQL, without any other explanations or text.
         """.format(
             index_name(tenant_id),
             "\n".join([f"{k}: {v}" for k, v in field_map.items()]),
@@ -381,21 +417,21 @@ def use_sql(question, field_map, tenant_id, chat_mdl, quota=True):
 
     docid_idx = set([ii for ii, c in enumerate(
         tbl["columns"]) if c["name"] == "doc_id"])
-    docnm_idx = set([ii for ii, c in enumerate(
+    doc_name_idx = set([ii for ii, c in enumerate(
         tbl["columns"]) if c["name"] == "docnm_kwd"])
-    clmn_idx = [ii for ii in range(
-        len(tbl["columns"])) if ii not in (docid_idx | docnm_idx)]
+    column_idx = [ii for ii in range(
+        len(tbl["columns"])) if ii not in (docid_idx | doc_name_idx)]
 
-    # compose markdown table
-    clmns = "|" + "|".join([re.sub(r"(/.*|（[^（）]+）)", "", field_map.get(tbl["columns"][i]["name"],
-                                                                        tbl["columns"][i]["name"])) for i in
-                            clmn_idx]) + ("|Source|" if docid_idx and docid_idx else "|")
+    # compose Markdown table
+    columns = "|" + "|".join([re.sub(r"(/.*|（[^（）]+）)", "", field_map.get(tbl["columns"][i]["name"],
+                                                                          tbl["columns"][i]["name"])) for i in
+                              column_idx]) + ("|Source|" if docid_idx and docid_idx else "|")
 
-    line = "|" + "|".join(["------" for _ in range(len(clmn_idx))]) + \
+    line = "|" + "|".join(["------" for _ in range(len(column_idx))]) + \
            ("|------|" if docid_idx and docid_idx else "")
 
     rows = ["|" +
-            "|".join([rmSpace(str(r[i])) for i in clmn_idx]).replace("None", " ") +
+            "|".join([rmSpace(str(r[i])) for i in column_idx]).replace("None", " ") +
             "|" for r in tbl["rows"]]
     rows = [r for r in rows if re.sub(r"[ |]+", "", r)]
     if quota:
@@ -404,24 +440,24 @@ def use_sql(question, field_map, tenant_id, chat_mdl, quota=True):
         rows = "\n".join([r + f" ##{ii}$$ |" for ii, r in enumerate(rows)])
     rows = re.sub(r"T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+Z)?\|", "|", rows)
 
-    if not docid_idx or not docnm_idx:
+    if not docid_idx or not doc_name_idx:
         logging.warning("SQL missing field: " + sql)
         return {
-            "answer": "\n".join([clmns, line, rows]),
+            "answer": "\n".join([columns, line, rows]),
             "reference": {"chunks": [], "doc_aggs": []},
             "prompt": sys_prompt
         }
 
     docid_idx = list(docid_idx)[0]
-    docnm_idx = list(docnm_idx)[0]
+    doc_name_idx = list(doc_name_idx)[0]
     doc_aggs = {}
     for r in tbl["rows"]:
         if r[docid_idx] not in doc_aggs:
-            doc_aggs[r[docid_idx]] = {"doc_name": r[docnm_idx], "count": 0}
+            doc_aggs[r[docid_idx]] = {"doc_name": r[doc_name_idx], "count": 0}
         doc_aggs[r[docid_idx]]["count"] += 1
     return {
-        "answer": "\n".join([clmns, line, rows]),
-        "reference": {"chunks": [{"doc_id": r[docid_idx], "docnm_kwd": r[docnm_idx]} for r in tbl["rows"]],
+        "answer": "\n".join([columns, line, rows]),
+        "reference": {"chunks": [{"doc_id": r[docid_idx], "docnm_kwd": r[doc_name_idx]} for r in tbl["rows"]],
                       "doc_aggs": [{"doc_id": did, "doc_name": d["doc_name"], "count": d["count"]} for did, d in
                                    doc_aggs.items()]},
         "prompt": sys_prompt
@@ -492,7 +528,7 @@ Requirements:
     kwd = chat_mdl.chat(prompt, msg[1:], {"temperature": 0.2})
     if isinstance(kwd, tuple):
         kwd = kwd[0]
-    if kwd.find("**ERROR**") >=0:
+    if kwd.find("**ERROR**") >= 0:
         return ""
     return kwd
 
@@ -605,16 +641,16 @@ def tts(tts_mdl, text):
 
 def ask(question, kb_ids, tenant_id):
     kbs = KnowledgebaseService.get_by_ids(kb_ids)
-    embd_nms = list(set([kb.embd_id for kb in kbs]))
+    embedding_list = list(set([kb.embd_id for kb in kbs]))
 
-    is_kg = all([kb.parser_id == ParserType.KG for kb in kbs])
-    retr = settings.retrievaler if not is_kg else settings.kg_retrievaler
+    is_knowledge_graph = all([kb.parser_id == ParserType.KG for kb in kbs])
+    retriever = settings.retrievaler if not is_knowledge_graph else settings.kg_retrievaler
 
-    embd_mdl = LLMBundle(tenant_id, LLMType.EMBEDDING, embd_nms[0])
+    embd_mdl = LLMBundle(tenant_id, LLMType.EMBEDDING, embedding_list[0])
     chat_mdl = LLMBundle(tenant_id, LLMType.CHAT)
     max_tokens = chat_mdl.max_length
     tenant_ids = list(set([kb.tenant_id for kb in kbs]))
-    kbinfos = retr.retrieval(question, embd_mdl, tenant_ids, kb_ids, 1, 12, 0.1, 0.3, aggs=False)
+    kbinfos = retriever.retrieval(question, embd_mdl, tenant_ids, kb_ids, 1, 12, 0.1, 0.3, aggs=False)
     knowledges = kb_prompt(kbinfos, max_tokens)
     prompt = """
     Role: You're a smart assistant. Your name is Miss R.
@@ -636,14 +672,14 @@ def ask(question, kb_ids, tenant_id):
 
     def decorate_answer(answer):
         nonlocal knowledges, kbinfos, prompt
-        answer, idx = retr.insert_citations(answer,
-                                            [ck["content_ltks"]
-                                             for ck in kbinfos["chunks"]],
-                                            [ck["vector"]
-                                             for ck in kbinfos["chunks"]],
-                                            embd_mdl,
-                                            tkweight=0.7,
-                                            vtweight=0.3)
+        answer, idx = retriever.insert_citations(answer,
+                                                 [ck["content_ltks"]
+                                                  for ck in kbinfos["chunks"]],
+                                                 [ck["vector"]
+                                                  for ck in kbinfos["chunks"]],
+                                                 embd_mdl,
+                                                 tkweight=0.7,
+                                                 vtweight=0.3)
         idx = set([kbinfos["chunks"][int(i)]["doc_id"] for i in idx])
         recall_docs = [
             d for d in kbinfos["doc_aggs"] if d["doc_id"] in idx]
@@ -664,4 +700,3 @@ def ask(question, kb_ids, tenant_id):
         answer = ans
         yield {"answer": answer, "reference": {}}
     yield decorate_answer(answer)
-

api/db/services/file_service.py

@@ -345,6 +345,8 @@ class FileService(CommonService):
                 MAX_FILE_NUM_PER_USER = int(os.environ.get('MAX_FILE_NUM_PER_USER', 0))
                 if MAX_FILE_NUM_PER_USER > 0 and DocumentService.get_doc_count(kb.tenant_id) >= MAX_FILE_NUM_PER_USER:
                     raise RuntimeError("Exceed the maximum file number of a free user!")
+                if len(file.filename) >= 128:
+                    raise RuntimeError("Exceed the maximum length of file name!")
 
                 filename = duplicate_name(
                     DocumentService.query,

api/db/services/llm_service.py

@@ -72,10 +72,12 @@ class TenantLLMService(CommonService):
             return model_name, None
         if len(arr) > 2:
             return "@".join(arr[0:-1]), arr[-1]
+
+        # model name must be xxx@yyy
         try:
-            fact = json.load(open(os.path.join(get_project_base_directory(), "conf/llm_factories.json"), "r"))["factory_llm_infos"]
-            fact = set([f["name"] for f in fact])
-            if arr[-1] not in fact:
+            model_factories = json.load(open(os.path.join(get_project_base_directory(), "conf/llm_factories.json"), "r"))["factory_llm_infos"]
+            model_providers = set([f["name"] for f in model_factories])
+            if arr[-1] not in model_providers:
                 return model_name, None
             return arr[0], arr[-1]
         except Exception as e:
@@ -113,11 +115,11 @@ class TenantLLMService(CommonService):
             if llm_type in [LLMType.EMBEDDING, LLMType.RERANK]:
                 llm = LLMService.query(llm_name=mdlnm) if not fid else LLMService.query(llm_name=mdlnm, fid=fid)
                 if llm and llm[0].fid in ["Youdao", "FastEmbed", "BAAI"]:
-                    model_config = {"llm_factory": llm[0].fid, "api_key":"", "llm_name": mdlnm, "api_base": ""}
+                    model_config = {"llm_factory": llm[0].fid, "api_key": "", "llm_name": mdlnm, "api_base": ""}
             if not model_config:
                 if mdlnm == "flag-embedding":
                     model_config = {"llm_factory": "Tongyi-Qianwen", "api_key": "",
-                                "llm_name": llm_name, "api_base": ""}
+                                    "llm_name": llm_name, "api_base": ""}
                 else:
                     if not mdlnm:
                         raise LookupError(f"Type of {llm_type} model is not set.")
@@ -200,8 +202,8 @@ class TenantLLMService(CommonService):
                 return num
             else:
                 tenant_llm = tenant_llms[0]
-                num = cls.model.update(used_tokens=tenant_llm.used_tokens + used_tokens)\
-                    .where(cls.model.tenant_id == tenant_id, cls.model.llm_factory == tenant_llm.llm_factory, cls.model.llm_name == llm_name)\
+                num = cls.model.update(used_tokens=tenant_llm.used_tokens + used_tokens) \
+                    .where(cls.model.tenant_id == tenant_id, cls.model.llm_factory == tenant_llm.llm_factory, cls.model.llm_name == llm_name) \
                     .execute()
         except Exception:
             logging.exception("TenantLLMService.increase_usage got exception")
@@ -231,7 +233,7 @@ class LLMBundle(object):
         for lm in LLMService.query(llm_name=llm_name):
             self.max_length = lm.max_tokens
             break
-    
+
     def encode(self, texts: list):
         embeddings, used_tokens = self.mdl.encode(texts)
         if not TenantLLMService.increase_usage(
@@ -274,11 +276,11 @@ class LLMBundle(object):
 
     def tts(self, text):
         for chunk in self.mdl.tts(text):
-            if isinstance(chunk,int):
+            if isinstance(chunk, int):
                 if not TenantLLMService.increase_usage(
-                    self.tenant_id, self.llm_type, chunk, self.llm_name):
-                        logging.error(
-                            "LLMBundle.tts can't update token usage for {}/TTS".format(self.tenant_id))
+                        self.tenant_id, self.llm_type, chunk, self.llm_name):
+                    logging.error(
+                        "LLMBundle.tts can't update token usage for {}/TTS".format(self.tenant_id))
                 return
             yield chunk
 
@@ -287,7 +289,8 @@ class LLMBundle(object):
         if isinstance(txt, int) and not TenantLLMService.increase_usage(
                 self.tenant_id, self.llm_type, used_tokens, self.llm_name):
             logging.error(
-                "LLMBundle.chat can't update token usage for {}/CHAT llm_name: {}, used_tokens: {}".format(self.tenant_id, self.llm_name, used_tokens))
+                "LLMBundle.chat can't update token usage for {}/CHAT llm_name: {}, used_tokens: {}".format(self.tenant_id, self.llm_name,
+                                                                                                           used_tokens))
         return txt
 
     def chat_streamly(self, system, history, gen_conf):
@@ -296,6 +299,7 @@ class LLMBundle(object):
                 if not TenantLLMService.increase_usage(
                         self.tenant_id, self.llm_type, txt, self.llm_name):
                     logging.error(
-                        "LLMBundle.chat_streamly can't update token usage for {}/CHAT llm_name: {}, content: {}".format(self.tenant_id, self.llm_name, txt))
+                        "LLMBundle.chat_streamly can't update token usage for {}/CHAT llm_name: {}, content: {}".format(self.tenant_id, self.llm_name,
+                                                                                                                        txt))
                 return
             yield txt

api/db/services/task_service.py

@@ -17,6 +17,7 @@ import os
 import random
 import xxhash
 import bisect
+from datetime import datetime
 
 from api.db.db_utils import bulk_insert_into_db
 from deepdoc.parser import PdfParser
@@ -84,7 +85,7 @@ class TaskService(CommonService):
         if not docs:
             return None
 
-        msg = "\nTask has been received."
+        msg = f"\n{datetime.now().strftime('%H:%M:%S')} Task has been received."
         prog = random.random() / 10.0
         if docs[0]["retry_count"] >= 3:
             msg = "\nERROR: Task is abandoned after 3 times attempts."
@@ -204,7 +205,7 @@ def queue_tasks(doc: dict, bucket: str, name: str):
     def new_task():
         return {"id": get_uuid(), "doc_id": doc["id"], "progress": 0.0, "from_page": 0, "to_page": 100000000}
 
-    tsks = []
+    parse_task_array = []
 
     if doc["type"] == FileType.PDF.value:
         file_bin = STORAGE_IMPL.get(bucket, name)
@@ -224,7 +225,7 @@ def queue_tasks(doc: dict, bucket: str, name: str):
                 task = new_task()
                 task["from_page"] = p
                 task["to_page"] = min(p + page_size, e)
-                tsks.append(task)
+                parse_task_array.append(task)
 
     elif doc["parser_id"] == "table":
         file_bin = STORAGE_IMPL.get(bucket, name)
@@ -233,12 +234,12 @@ def queue_tasks(doc: dict, bucket: str, name: str):
             task = new_task()
             task["from_page"] = i
             task["to_page"] = min(i + 3000, rn)
-            tsks.append(task)
+            parse_task_array.append(task)
     else:
-        tsks.append(new_task())
+        parse_task_array.append(new_task())
 
     chunking_config = DocumentService.get_chunking_config(doc["id"])
-    for task in tsks:
+    for task in parse_task_array:
         hasher = xxhash.xxh64()
         for field in sorted(chunking_config.keys()):
             hasher.update(str(chunking_config[field]).encode("utf-8"))
@@ -251,7 +252,7 @@ def queue_tasks(doc: dict, bucket: str, name: str):
     prev_tasks = TaskService.get_tasks(doc["id"])
     ck_num = 0
     if prev_tasks:
-        for task in tsks:
+        for task in parse_task_array:
             ck_num += reuse_prev_task_chunks(task, prev_tasks, chunking_config)
         TaskService.filter_delete([Task.doc_id == doc["id"]])
         chunk_ids = []
@@ -263,13 +264,13 @@ def queue_tasks(doc: dict, bucket: str, name: str):
                                          chunking_config["kb_id"])
     DocumentService.update_by_id(doc["id"], {"chunk_num": ck_num})
 
-    bulk_insert_into_db(Task, tsks, True)
+    bulk_insert_into_db(Task, parse_task_array, True)
     DocumentService.begin2parse(doc["id"])
 
-    tsks = [task for task in tsks if task["progress"] < 1.0]
-    for t in tsks:
+    unfinished_task_array = [task for task in parse_task_array if task["progress"] < 1.0]
+    for unfinished_task in unfinished_task_array:
         assert REDIS_CONN.queue_product(
-            SVR_QUEUE_NAME, message=t
+            SVR_QUEUE_NAME, message=unfinished_task
         ), "Can't access Redis. Please check the Redis' status."
 
 

api/utils/__init__.py

@@ -24,6 +24,7 @@ import time
 import uuid
 import requests
 import logging
+import copy
 from enum import Enum, IntEnum
 import importlib
 from Cryptodome.PublicKey import RSA
@@ -65,6 +66,10 @@ CONFIGS = read_config()
 def show_configs():
     msg = f"Current configs, from {conf_realpath(SERVICE_CONF)}:"
     for k, v in CONFIGS.items():
+        if isinstance(v, dict):
+            if "password" in v:
+                v = copy.deepcopy(v)
+                v["password"] = "*" * 8
         msg += f"\n\t{k}: {v}"
     logging.info(msg)
 

api/utils/api_utils.py

@@ -283,7 +283,10 @@ def construct_error_response(e):
 def token_required(func):
     @wraps(func)
     def decorated_function(*args, **kwargs):
-        authorization_list=flask_request.headers.get('Authorization').split()
+        authorization_str=flask_request.headers.get('Authorization')
+        if not authorization_str:
+            return get_json_result(data=False,message="`Authorization` can't be empty")
+        authorization_list=authorization_str.split()
         if len(authorization_list) < 2:
             return get_json_result(data=False,message="Please check your authorization format.")
         token = authorization_list[1]

conf/llm_factories.json

@@ -952,6 +952,12 @@
                     "max_tokens": 30720,
                     "model_type": "chat"
                 },
+                {
+                    "llm_name": "gemini-2.0-flash-exp",
+                    "tags": "LLM,CHAT,1024K",
+                    "max_tokens": 1048576,
+                    "model_type": "chat"
+                },
                 {
                     "llm_name": "gemini-1.0-pro-vision-latest",
                     "tags": "LLM,IMAGE2TEXT,12K",
@@ -1008,6 +1014,18 @@
                     "max_tokens": 131072,
                     "model_type": "chat"
                 },
+                {
+                    "llm_name": "llama-3.3-70b-versatile",
+                    "tags": "LLM,CHAT,128k",
+                    "max_tokens": 128000,
+                    "model_type": "chat"
+                },
+                {
+                    "llm_name": "llama-3.3-70b-specdec",
+                    "tags": "LLM,CHAT,8k",
+                    "max_tokens": 8192,
+                    "model_type": "chat"
+                },
                 {
                     "llm_name": "mixtral-8x7b-32768",
                     "tags": "LLM,CHAT,5k",

deepdoc/parser/excel_parser.py

@@ -90,7 +90,7 @@ class RAGFlowExcelParser:
             for sheetname in wb.sheetnames:
                 ws = wb[sheetname]
                 total += len(list(ws.rows))
-                return total
+            return total
 
         if fnm.split(".")[-1].lower() in ["csv", "txt"]:
             encoding = find_codec(binary)

deepdoc/parser/ppt_parser.py

@@ -10,7 +10,7 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-
+import logging
 from io import BytesIO
 from pptx import Presentation
 
@@ -53,9 +53,12 @@ class RAGFlowPptParser(object):
             texts = []
             for shape in sorted(
                     slide.shapes, key=lambda x: ((x.top if x.top is not None else 0) // 10, x.left)):
-                txt = self.__extract(shape)
-                if txt:
-                    texts.append(txt)
+                try:
+                    txt = self.__extract(shape)
+                    if txt:
+                        texts.append(txt)
+                except Exception as e:
+                    logging.exception(e)
             txts.append("\n".join(texts))
 
         return txts

deepdoc/vision/recognizer.py

@@ -26,6 +26,7 @@ from api.utils.file_utils import get_project_base_directory
 from .operators import *  # noqa: F403
 from .operators import preprocess
 
+
 class Recognizer(object):
     def __init__(self, label_list, task_name, model_dir=None):
         """
@@ -144,11 +145,11 @@ class Recognizer(object):
             return 0
         x0_ = max(b["x0"], x0)
         x1_ = min(b["x1"], x1)
-        assert x0_ <= x1_, "Fuckedup! T:{},B:{},X0:{},X1:{} ==> {}".format(
+        assert x0_ <= x1_, "Bbox mismatch! T:{},B:{},X0:{},X1:{} ==> {}".format(
             tp, btm, x0, x1, b)
         tp_ = max(b["top"], tp)
         btm_ = min(b["bottom"], btm)
-        assert tp_ <= btm_, "Fuckedup! T:{},B:{},X0:{},X1:{} => {}".format(
+        assert tp_ <= btm_, "Bbox mismatch! T:{},B:{},X0:{},X1:{} => {}".format(
             tp, btm, x0, x1, b)
         ov = (btm_ - tp_) * (x1_ - x0_) if x1 - \
                                            x0 != 0 and btm - tp != 0 else 0

docker/.env

@@ -52,7 +52,7 @@ MYSQL_DBNAME=rag_flow
 # allowing EXTERNAL access to the MySQL database running inside the Docker container. 
 MYSQL_PORT=5455
 
-# The hostname where the MySQL service is exposed
+# The hostname where the MinIO service is exposed
 MINIO_HOST=minio
 # The port used to expose the MinIO console interface to the host machine, 
 # allowing EXTERNAL access to the web-based console running inside the Docker container. 

docker/docker-compose-base.yml

@@ -39,7 +39,7 @@ services:
     container_name: ragflow-infinity
     profiles:
       - infinity
-    image: infiniflow/infinity:v0.5.0
+    image: infiniflow/infinity:v0.5.2
     volumes:
       - infinity_data:/var/infinity
       - ./infinity_conf.toml:/infinity_conf.toml

docs/quickstart.mdx

@@ -185,7 +185,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
 3. Use the pre-built Docker images and start up the server:
 
    :::tip NOTE
-   The command below downloads the `v0.15.0-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download an RAGFlow edition different from `v0.14.1-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.14.1` for the full edition `v0.14.1`.
+   The command below downloads the `v0.15.0-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download an RAGFlow edition different from `v0.15.0-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.15.0` for the full edition `v0.15.0`.
    :::
 
    ```bash

graphrag/leiden.py

@@ -141,3 +141,4 @@ def add_community_info2graph(graph: nx.Graph, nodes: list[str], community_title)
         if "communities" not in graph.nodes[n]:
             graph.nodes[n]["communities"] = []
         graph.nodes[n]["communities"].append(community_title)
+        graph.nodes[n]["communities"] = list(set(graph.nodes[n]["communities"]))

graphrag/utils.py

@@ -101,7 +101,7 @@ def get_embed_cache(llmnm, txt):
     bin = REDIS_CONN.get(k)
     if not bin:
         return
-    return np.array(json.loads(bin.decode("utf-8")))
+    return np.array(json.loads(bin))
 
 
 def set_embed_cache(llmnm, txt, arr):

poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.5 and should not be changed by hand.
 
 [[package]]
 name = "accelerate"
@@ -44,87 +44,87 @@ files = [
 
 [[package]]
 name = "aiohttp"
-version = "3.11.10"
+version = "3.11.11"
 description = "Async http client/server framework (asyncio)"
 optional = false
 python-versions = ">=3.9"
 files = [
-    {file = "aiohttp-3.11.10-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cbad88a61fa743c5d283ad501b01c153820734118b65aee2bd7dbb735475ce0d"},
-    {file = "aiohttp-3.11.10-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:80886dac673ceaef499de2f393fc80bb4481a129e6cb29e624a12e3296cc088f"},
-    {file = "aiohttp-3.11.10-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:61b9bae80ed1f338c42f57c16918853dc51775fb5cb61da70d590de14d8b5fb4"},
-    {file = "aiohttp-3.11.10-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9e2e576caec5c6a6b93f41626c9c02fc87cd91538b81a3670b2e04452a63def6"},
-    {file = "aiohttp-3.11.10-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:02c13415b5732fb6ee7ff64583a5e6ed1c57aa68f17d2bda79c04888dfdc2769"},
-    {file = "aiohttp-3.11.10-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4cfce37f31f20800a6a6620ce2cdd6737b82e42e06e6e9bd1b36f546feb3c44f"},
-    {file = "aiohttp-3.11.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3bbbfff4c679c64e6e23cb213f57cc2c9165c9a65d63717108a644eb5a7398df"},
-    {file = "aiohttp-3.11.10-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:49c7dbbc1a559ae14fc48387a115b7d4bbc84b4a2c3b9299c31696953c2a5219"},
-    {file = "aiohttp-3.11.10-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:68386d78743e6570f054fe7949d6cb37ef2b672b4d3405ce91fafa996f7d9b4d"},
-    {file = "aiohttp-3.11.10-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:9ef405356ba989fb57f84cac66f7b0260772836191ccefbb987f414bcd2979d9"},
-    {file = "aiohttp-3.11.10-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:5d6958671b296febe7f5f859bea581a21c1d05430d1bbdcf2b393599b1cdce77"},
-    {file = "aiohttp-3.11.10-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:99b7920e7165be5a9e9a3a7f1b680f06f68ff0d0328ff4079e5163990d046767"},
-    {file = "aiohttp-3.11.10-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:0dc49f42422163efb7e6f1df2636fe3db72713f6cd94688e339dbe33fe06d61d"},
-    {file = "aiohttp-3.11.10-cp310-cp310-win32.whl", hash = "sha256:40d1c7a7f750b5648642586ba7206999650208dbe5afbcc5284bcec6579c9b91"},
-    {file = "aiohttp-3.11.10-cp310-cp310-win_amd64.whl", hash = "sha256:68ff6f48b51bd78ea92b31079817aff539f6c8fc80b6b8d6ca347d7c02384e33"},
-    {file = "aiohttp-3.11.10-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:77c4aa15a89847b9891abf97f3d4048f3c2d667e00f8a623c89ad2dccee6771b"},
-    {file = "aiohttp-3.11.10-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:909af95a72cedbefe5596f0bdf3055740f96c1a4baa0dd11fd74ca4de0b4e3f1"},
-    {file = "aiohttp-3.11.10-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:386fbe79863eb564e9f3615b959e28b222259da0c48fd1be5929ac838bc65683"},
-    {file = "aiohttp-3.11.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3de34936eb1a647aa919655ff8d38b618e9f6b7f250cc19a57a4bf7fd2062b6d"},
-    {file = "aiohttp-3.11.10-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0c9527819b29cd2b9f52033e7fb9ff08073df49b4799c89cb5754624ecd98299"},
-    {file = "aiohttp-3.11.10-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:65a96e3e03300b41f261bbfd40dfdbf1c301e87eab7cd61c054b1f2e7c89b9e8"},
-    {file = "aiohttp-3.11.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98f5635f7b74bcd4f6f72fcd85bea2154b323a9f05226a80bc7398d0c90763b0"},
-    {file = "aiohttp-3.11.10-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:03b6002e20938fc6ee0918c81d9e776bebccc84690e2b03ed132331cca065ee5"},
-    {file = "aiohttp-3.11.10-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6362cc6c23c08d18ddbf0e8c4d5159b5df74fea1a5278ff4f2c79aed3f4e9f46"},
-    {file = "aiohttp-3.11.10-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:3691ed7726fef54e928fe26344d930c0c8575bc968c3e239c2e1a04bd8cf7838"},
-    {file = "aiohttp-3.11.10-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:31d5093d3acd02b31c649d3a69bb072d539d4c7659b87caa4f6d2bcf57c2fa2b"},
-    {file = "aiohttp-3.11.10-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:8b3cf2dc0f0690a33f2d2b2cb15db87a65f1c609f53c37e226f84edb08d10f52"},
-    {file = "aiohttp-3.11.10-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:fbbaea811a2bba171197b08eea288b9402faa2bab2ba0858eecdd0a4105753a3"},
-    {file = "aiohttp-3.11.10-cp311-cp311-win32.whl", hash = "sha256:4b2c7ac59c5698a7a8207ba72d9e9c15b0fc484a560be0788b31312c2c5504e4"},
-    {file = "aiohttp-3.11.10-cp311-cp311-win_amd64.whl", hash = "sha256:974d3a2cce5fcfa32f06b13ccc8f20c6ad9c51802bb7f829eae8a1845c4019ec"},
-    {file = "aiohttp-3.11.10-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b78f053a7ecfc35f0451d961dacdc671f4bcbc2f58241a7c820e9d82559844cf"},
-    {file = "aiohttp-3.11.10-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ab7485222db0959a87fbe8125e233b5a6f01f4400785b36e8a7878170d8c3138"},
-    {file = "aiohttp-3.11.10-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:cf14627232dfa8730453752e9cdc210966490992234d77ff90bc8dc0dce361d5"},
-    {file = "aiohttp-3.11.10-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:076bc454a7e6fd646bc82ea7f98296be0b1219b5e3ef8a488afbdd8e81fbac50"},
-    {file = "aiohttp-3.11.10-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:482cafb7dc886bebeb6c9ba7925e03591a62ab34298ee70d3dd47ba966370d2c"},
-    {file = "aiohttp-3.11.10-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bf3d1a519a324af764a46da4115bdbd566b3c73fb793ffb97f9111dbc684fc4d"},
-    {file = "aiohttp-3.11.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:24213ba85a419103e641e55c27dc7ff03536c4873470c2478cce3311ba1eee7b"},
-    {file = "aiohttp-3.11.10-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b99acd4730ad1b196bfb03ee0803e4adac371ae8efa7e1cbc820200fc5ded109"},
-    {file = "aiohttp-3.11.10-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:14cdb5a9570be5a04eec2ace174a48ae85833c2aadc86de68f55541f66ce42ab"},
-    {file = "aiohttp-3.11.10-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:7e97d622cb083e86f18317282084bc9fbf261801b0192c34fe4b1febd9f7ae69"},
-    {file = "aiohttp-3.11.10-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:012f176945af138abc10c4a48743327a92b4ca9adc7a0e078077cdb5dbab7be0"},
-    {file = "aiohttp-3.11.10-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:44224d815853962f48fe124748227773acd9686eba6dc102578defd6fc99e8d9"},
-    {file = "aiohttp-3.11.10-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c87bf31b7fdab94ae3adbe4a48e711bfc5f89d21cf4c197e75561def39e223bc"},
-    {file = "aiohttp-3.11.10-cp312-cp312-win32.whl", hash = "sha256:06a8e2ee1cbac16fe61e51e0b0c269400e781b13bcfc33f5425912391a542985"},
-    {file = "aiohttp-3.11.10-cp312-cp312-win_amd64.whl", hash = "sha256:be2b516f56ea883a3e14dda17059716593526e10fb6303189aaf5503937db408"},
-    {file = "aiohttp-3.11.10-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8cc5203b817b748adccb07f36390feb730b1bc5f56683445bfe924fc270b8816"},
-    {file = "aiohttp-3.11.10-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5ef359ebc6949e3a34c65ce20230fae70920714367c63afd80ea0c2702902ccf"},
-    {file = "aiohttp-3.11.10-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9bca390cb247dbfaec3c664326e034ef23882c3f3bfa5fbf0b56cad0320aaca5"},
-    {file = "aiohttp-3.11.10-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:811f23b3351ca532af598405db1093f018edf81368e689d1b508c57dcc6b6a32"},
-    {file = "aiohttp-3.11.10-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ddf5f7d877615f6a1e75971bfa5ac88609af3b74796ff3e06879e8422729fd01"},
-    {file = "aiohttp-3.11.10-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6ab29b8a0beb6f8eaf1e5049252cfe74adbaafd39ba91e10f18caeb0e99ffb34"},
-    {file = "aiohttp-3.11.10-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c49a76c1038c2dd116fa443eba26bbb8e6c37e924e2513574856de3b6516be99"},
-    {file = "aiohttp-3.11.10-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7f3dc0e330575f5b134918976a645e79adf333c0a1439dcf6899a80776c9ab39"},
-    {file = "aiohttp-3.11.10-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:efb15a17a12497685304b2d976cb4939e55137df7b09fa53f1b6a023f01fcb4e"},
-    {file = "aiohttp-3.11.10-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:db1d0b28fcb7f1d35600150c3e4b490775251dea70f894bf15c678fdd84eda6a"},
-    {file = "aiohttp-3.11.10-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:15fccaf62a4889527539ecb86834084ecf6e9ea70588efde86e8bc775e0e7542"},
-    {file = "aiohttp-3.11.10-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:593c114a2221444f30749cc5e5f4012488f56bd14de2af44fe23e1e9894a9c60"},
-    {file = "aiohttp-3.11.10-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7852bbcb4d0d2f0c4d583f40c3bc750ee033265d80598d0f9cb6f372baa6b836"},
-    {file = "aiohttp-3.11.10-cp313-cp313-win32.whl", hash = "sha256:65e55ca7debae8faaffee0ebb4b47a51b4075f01e9b641c31e554fd376595c6c"},
-    {file = "aiohttp-3.11.10-cp313-cp313-win_amd64.whl", hash = "sha256:beb39a6d60a709ae3fb3516a1581777e7e8b76933bb88c8f4420d875bb0267c6"},
-    {file = "aiohttp-3.11.10-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:0580f2e12de2138f34debcd5d88894786453a76e98febaf3e8fe5db62d01c9bf"},
-    {file = "aiohttp-3.11.10-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:a55d2ad345684e7c3dd2c20d2f9572e9e1d5446d57200ff630e6ede7612e307f"},
-    {file = "aiohttp-3.11.10-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:04814571cb72d65a6899db6099e377ed00710bf2e3eafd2985166f2918beaf59"},
-    {file = "aiohttp-3.11.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e44a9a3c053b90c6f09b1bb4edd880959f5328cf63052503f892c41ea786d99f"},
-    {file = "aiohttp-3.11.10-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:502a1464ccbc800b4b1995b302efaf426e8763fadf185e933c2931df7db9a199"},
-    {file = "aiohttp-3.11.10-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:613e5169f8ae77b1933e42e418a95931fb4867b2991fc311430b15901ed67079"},
-    {file = "aiohttp-3.11.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4cca22a61b7fe45da8fc73c3443150c3608750bbe27641fc7558ec5117b27fdf"},
-    {file = "aiohttp-3.11.10-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:86a5dfcc39309470bd7b68c591d84056d195428d5d2e0b5ccadfbaf25b026ebc"},
-    {file = "aiohttp-3.11.10-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:77ae58586930ee6b2b6f696c82cf8e78c8016ec4795c53e36718365f6959dc82"},
-    {file = "aiohttp-3.11.10-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:78153314f26d5abef3239b4a9af20c229c6f3ecb97d4c1c01b22c4f87669820c"},
-    {file = "aiohttp-3.11.10-cp39-cp39-musllinux_1_2_ppc64le.whl", hash = "sha256:98283b94cc0e11c73acaf1c9698dea80c830ca476492c0fe2622bd931f34b487"},
-    {file = "aiohttp-3.11.10-cp39-cp39-musllinux_1_2_s390x.whl", hash = "sha256:53bf2097e05c2accc166c142a2090e4c6fd86581bde3fd9b2d3f9e93dda66ac1"},
-    {file = "aiohttp-3.11.10-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:c5532f0441fc09c119e1dca18fbc0687e64fbeb45aa4d6a87211ceaee50a74c4"},
-    {file = "aiohttp-3.11.10-cp39-cp39-win32.whl", hash = "sha256:47ad15a65fb41c570cd0ad9a9ff8012489e68176e7207ec7b82a0940dddfd8be"},
-    {file = "aiohttp-3.11.10-cp39-cp39-win_amd64.whl", hash = "sha256:c6b9e6d7e41656d78e37ce754813fa44b455c3d0d0dced2a047def7dc5570b74"},
-    {file = "aiohttp-3.11.10.tar.gz", hash = "sha256:b1fc6b45010a8d0ff9e88f9f2418c6fd408c99c211257334aff41597ebece42e"},
+    {file = "aiohttp-3.11.11-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:a60804bff28662cbcf340a4d61598891f12eea3a66af48ecfdc975ceec21e3c8"},
+    {file = "aiohttp-3.11.11-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4b4fa1cb5f270fb3eab079536b764ad740bb749ce69a94d4ec30ceee1b5940d5"},
+    {file = "aiohttp-3.11.11-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:731468f555656767cda219ab42e033355fe48c85fbe3ba83a349631541715ba2"},
+    {file = "aiohttp-3.11.11-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cb23d8bb86282b342481cad4370ea0853a39e4a32a0042bb52ca6bdde132df43"},
+    {file = "aiohttp-3.11.11-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f047569d655f81cb70ea5be942ee5d4421b6219c3f05d131f64088c73bb0917f"},
+    {file = "aiohttp-3.11.11-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dd7659baae9ccf94ae5fe8bfaa2c7bc2e94d24611528395ce88d009107e00c6d"},
+    {file = "aiohttp-3.11.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:af01e42ad87ae24932138f154105e88da13ce7d202a6de93fafdafb2883a00ef"},
+    {file = "aiohttp-3.11.11-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5854be2f3e5a729800bac57a8d76af464e160f19676ab6aea74bde18ad19d438"},
+    {file = "aiohttp-3.11.11-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6526e5fb4e14f4bbf30411216780c9967c20c5a55f2f51d3abd6de68320cc2f3"},
+    {file = "aiohttp-3.11.11-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:85992ee30a31835fc482468637b3e5bd085fa8fe9392ba0bdcbdc1ef5e9e3c55"},
+    {file = "aiohttp-3.11.11-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:88a12ad8ccf325a8a5ed80e6d7c3bdc247d66175afedbe104ee2aaca72960d8e"},
+    {file = "aiohttp-3.11.11-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:0a6d3fbf2232e3a08c41eca81ae4f1dff3d8f1a30bae415ebe0af2d2458b8a33"},
+    {file = "aiohttp-3.11.11-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:84a585799c58b795573c7fa9b84c455adf3e1d72f19a2bf498b54a95ae0d194c"},
+    {file = "aiohttp-3.11.11-cp310-cp310-win32.whl", hash = "sha256:bfde76a8f430cf5c5584553adf9926534352251d379dcb266ad2b93c54a29745"},
+    {file = "aiohttp-3.11.11-cp310-cp310-win_amd64.whl", hash = "sha256:0fd82b8e9c383af11d2b26f27a478640b6b83d669440c0a71481f7c865a51da9"},
+    {file = "aiohttp-3.11.11-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ba74ec819177af1ef7f59063c6d35a214a8fde6f987f7661f4f0eecc468a8f76"},
+    {file = "aiohttp-3.11.11-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4af57160800b7a815f3fe0eba9b46bf28aafc195555f1824555fa2cfab6c1538"},
+    {file = "aiohttp-3.11.11-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ffa336210cf9cd8ed117011085817d00abe4c08f99968deef0013ea283547204"},
+    {file = "aiohttp-3.11.11-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:81b8fe282183e4a3c7a1b72f5ade1094ed1c6345a8f153506d114af5bf8accd9"},
+    {file = "aiohttp-3.11.11-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3af41686ccec6a0f2bdc66686dc0f403c41ac2089f80e2214a0f82d001052c03"},
+    {file = "aiohttp-3.11.11-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:70d1f9dde0e5dd9e292a6d4d00058737052b01f3532f69c0c65818dac26dc287"},
+    {file = "aiohttp-3.11.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:249cc6912405917344192b9f9ea5cd5b139d49e0d2f5c7f70bdfaf6b4dbf3a2e"},
+    {file = "aiohttp-3.11.11-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0eb98d90b6690827dcc84c246811feeb4e1eea683c0eac6caed7549be9c84665"},
+    {file = "aiohttp-3.11.11-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ec82bf1fda6cecce7f7b915f9196601a1bd1a3079796b76d16ae4cce6d0ef89b"},
+    {file = "aiohttp-3.11.11-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:9fd46ce0845cfe28f108888b3ab17abff84ff695e01e73657eec3f96d72eef34"},
+    {file = "aiohttp-3.11.11-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:bd176afcf8f5d2aed50c3647d4925d0db0579d96f75a31e77cbaf67d8a87742d"},
+    {file = "aiohttp-3.11.11-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:ec2aa89305006fba9ffb98970db6c8221541be7bee4c1d027421d6f6df7d1ce2"},
+    {file = "aiohttp-3.11.11-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:92cde43018a2e17d48bb09c79e4d4cb0e236de5063ce897a5e40ac7cb4878773"},
+    {file = "aiohttp-3.11.11-cp311-cp311-win32.whl", hash = "sha256:aba807f9569455cba566882c8938f1a549f205ee43c27b126e5450dc9f83cc62"},
+    {file = "aiohttp-3.11.11-cp311-cp311-win_amd64.whl", hash = "sha256:ae545f31489548c87b0cced5755cfe5a5308d00407000e72c4fa30b19c3220ac"},
+    {file = "aiohttp-3.11.11-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:e595c591a48bbc295ebf47cb91aebf9bd32f3ff76749ecf282ea7f9f6bb73886"},
+    {file = "aiohttp-3.11.11-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3ea1b59dc06396b0b424740a10a0a63974c725b1c64736ff788a3689d36c02d2"},
+    {file = "aiohttp-3.11.11-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8811f3f098a78ffa16e0ea36dffd577eb031aea797cbdba81be039a4169e242c"},
+    {file = "aiohttp-3.11.11-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bd7227b87a355ce1f4bf83bfae4399b1f5bb42e0259cb9405824bd03d2f4336a"},
+    {file = "aiohttp-3.11.11-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d40f9da8cabbf295d3a9dae1295c69975b86d941bc20f0a087f0477fa0a66231"},
+    {file = "aiohttp-3.11.11-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ffb3dc385f6bb1568aa974fe65da84723210e5d9707e360e9ecb51f59406cd2e"},
+    {file = "aiohttp-3.11.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a8f5f7515f3552d899c61202d99dcb17d6e3b0de777900405611cd747cecd1b8"},
+    {file = "aiohttp-3.11.11-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3499c7ffbfd9c6a3d8d6a2b01c26639da7e43d47c7b4f788016226b1e711caa8"},
+    {file = "aiohttp-3.11.11-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8e2bf8029dbf0810c7bfbc3e594b51c4cc9101fbffb583a3923aea184724203c"},
+    {file = "aiohttp-3.11.11-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:b6212a60e5c482ef90f2d788835387070a88d52cf6241d3916733c9176d39eab"},
+    {file = "aiohttp-3.11.11-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:d119fafe7b634dbfa25a8c597718e69a930e4847f0b88e172744be24515140da"},
+    {file = "aiohttp-3.11.11-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:6fba278063559acc730abf49845d0e9a9e1ba74f85f0ee6efd5803f08b285853"},
+    {file = "aiohttp-3.11.11-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:92fc484e34b733704ad77210c7957679c5c3877bd1e6b6d74b185e9320cc716e"},
+    {file = "aiohttp-3.11.11-cp312-cp312-win32.whl", hash = "sha256:9f5b3c1ed63c8fa937a920b6c1bec78b74ee09593b3f5b979ab2ae5ef60d7600"},
+    {file = "aiohttp-3.11.11-cp312-cp312-win_amd64.whl", hash = "sha256:1e69966ea6ef0c14ee53ef7a3d68b564cc408121ea56c0caa2dc918c1b2f553d"},
+    {file = "aiohttp-3.11.11-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:541d823548ab69d13d23730a06f97460f4238ad2e5ed966aaf850d7c369782d9"},
+    {file = "aiohttp-3.11.11-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:929f3ed33743a49ab127c58c3e0a827de0664bfcda566108989a14068f820194"},
+    {file = "aiohttp-3.11.11-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0882c2820fd0132240edbb4a51eb8ceb6eef8181db9ad5291ab3332e0d71df5f"},
+    {file = "aiohttp-3.11.11-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b63de12e44935d5aca7ed7ed98a255a11e5cb47f83a9fded7a5e41c40277d104"},
+    {file = "aiohttp-3.11.11-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:aa54f8ef31d23c506910c21163f22b124facb573bff73930735cf9fe38bf7dff"},
+    {file = "aiohttp-3.11.11-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a344d5dc18074e3872777b62f5f7d584ae4344cd6006c17ba12103759d407af3"},
+    {file = "aiohttp-3.11.11-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b7fb429ab1aafa1f48578eb315ca45bd46e9c37de11fe45c7f5f4138091e2f1"},
+    {file = "aiohttp-3.11.11-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c341c7d868750e31961d6d8e60ff040fb9d3d3a46d77fd85e1ab8e76c3e9a5c4"},
+    {file = "aiohttp-3.11.11-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ed9ee95614a71e87f1a70bc81603f6c6760128b140bc4030abe6abaa988f1c3d"},
+    {file = "aiohttp-3.11.11-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:de8d38f1c2810fa2a4f1d995a2e9c70bb8737b18da04ac2afbf3971f65781d87"},
+    {file = "aiohttp-3.11.11-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:a9b7371665d4f00deb8f32208c7c5e652059b0fda41cf6dbcac6114a041f1cc2"},
+    {file = "aiohttp-3.11.11-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:620598717fce1b3bd14dd09947ea53e1ad510317c85dda2c9c65b622edc96b12"},
+    {file = "aiohttp-3.11.11-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:bf8d9bfee991d8acc72d060d53860f356e07a50f0e0d09a8dfedea1c554dd0d5"},
+    {file = "aiohttp-3.11.11-cp313-cp313-win32.whl", hash = "sha256:9d73ee3725b7a737ad86c2eac5c57a4a97793d9f442599bea5ec67ac9f4bdc3d"},
+    {file = "aiohttp-3.11.11-cp313-cp313-win_amd64.whl", hash = "sha256:c7a06301c2fb096bdb0bd25fe2011531c1453b9f2c163c8031600ec73af1cc99"},
+    {file = "aiohttp-3.11.11-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:3e23419d832d969f659c208557de4a123e30a10d26e1e14b73431d3c13444c2e"},
+    {file = "aiohttp-3.11.11-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:21fef42317cf02e05d3b09c028712e1d73a9606f02467fd803f7c1f39cc59add"},
+    {file = "aiohttp-3.11.11-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:1f21bb8d0235fc10c09ce1d11ffbd40fc50d3f08a89e4cf3a0c503dc2562247a"},
+    {file = "aiohttp-3.11.11-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1642eceeaa5ab6c9b6dfeaaa626ae314d808188ab23ae196a34c9d97efb68350"},
+    {file = "aiohttp-3.11.11-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2170816e34e10f2fd120f603e951630f8a112e1be3b60963a1f159f5699059a6"},
+    {file = "aiohttp-3.11.11-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8be8508d110d93061197fd2d6a74f7401f73b6d12f8822bbcd6d74f2b55d71b1"},
+    {file = "aiohttp-3.11.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4eed954b161e6b9b65f6be446ed448ed3921763cc432053ceb606f89d793927e"},
+    {file = "aiohttp-3.11.11-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d6c9af134da4bc9b3bd3e6a70072509f295d10ee60c697826225b60b9959acdd"},
+    {file = "aiohttp-3.11.11-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:44167fc6a763d534a6908bdb2592269b4bf30a03239bcb1654781adf5e49caf1"},
+    {file = "aiohttp-3.11.11-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:479b8c6ebd12aedfe64563b85920525d05d394b85f166b7873c8bde6da612f9c"},
+    {file = "aiohttp-3.11.11-cp39-cp39-musllinux_1_2_ppc64le.whl", hash = "sha256:10b4ff0ad793d98605958089fabfa350e8e62bd5d40aa65cdc69d6785859f94e"},
+    {file = "aiohttp-3.11.11-cp39-cp39-musllinux_1_2_s390x.whl", hash = "sha256:b540bd67cfb54e6f0865ceccd9979687210d7ed1a1cc8c01f8e67e2f1e883d28"},
+    {file = "aiohttp-3.11.11-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:1dac54e8ce2ed83b1f6b1a54005c87dfed139cf3f777fdc8afc76e7841101226"},
+    {file = "aiohttp-3.11.11-cp39-cp39-win32.whl", hash = "sha256:568c1236b2fde93b7720f95a890741854c1200fba4a3471ff48b2934d2d93fd3"},
+    {file = "aiohttp-3.11.11-cp39-cp39-win_amd64.whl", hash = "sha256:943a8b052e54dfd6439fd7989f67fc6a7f2138d0a2cf0a7de5f18aa4fe7eb3b1"},
+    {file = "aiohttp-3.11.11.tar.gz", hash = "sha256:bb49c7f1e6ebf3821a42d81d494f538107610c3a705987f53068546b0e90303e"},
 ]
 
 [package.dependencies]
@@ -196,13 +196,13 @@ files = [
 
 [[package]]
 name = "akshare"
-version = "1.15.49"
+version = "1.15.53"
 description = "AKShare is an elegant and simple financial data interface library for Python, built for human beings!"
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "akshare-1.15.49-py3-none-any.whl", hash = "sha256:b1bbcdc0e8fbdcd751210e93ba93ff4cf4eec1bbd7707fcb455b9c9724c5c2d8"},
-    {file = "akshare-1.15.49.tar.gz", hash = "sha256:ae0e400f8c61b2258363c92ec9aacf00f232b9017c8b0a9ada02a6ed3a809a4b"},
+    {file = "akshare-1.15.53-py3-none-any.whl", hash = "sha256:f19c951d20f967c7fa95f95ae64d8534b71bffac7c69442e9046e0e668d64591"},
+    {file = "akshare-1.15.53.tar.gz", hash = "sha256:8f55fe3a8d1fe1eb7d8489e1c865c95df924850b358a90760c10a9ee53dea928"},
 ]
 
 [package.dependencies]
@@ -565,13 +565,13 @@ dev = ["freezegun (>=1.0,<2.0)", "pytest (>=6.0)", "pytest-cov"]
 
 [[package]]
 name = "bce-python-sdk"
-version = "0.9.23"
+version = "0.9.25"
 description = "BCE SDK for python"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, <4"
 files = [
-    {file = "bce_python_sdk-0.9.23-py3-none-any.whl", hash = "sha256:8debe21a040e00060f6044877d594765ed7b18bc765c6bf16b878bca864140a3"},
-    {file = "bce_python_sdk-0.9.23.tar.gz", hash = "sha256:19739fed5cd0725356fc5ffa2acbdd8fb23f2a81edb91db21a03174551d0cf41"},
+    {file = "bce_python_sdk-0.9.25-py3-none-any.whl", hash = "sha256:cd1ab4c887e163adba6bfb3cd40465a365e5f4255705a015b0cdbe768e649877"},
+    {file = "bce_python_sdk-0.9.25.tar.gz", hash = "sha256:93a0623fbb1bf3a58b4f2d7bdbd799a3b342a538f0c72950c77168e431470e86"},
 ]
 
 [package.dependencies]
@@ -636,12 +636,12 @@ lxml = ["lxml"]
 
 [[package]]
 name = "bibtexparser"
-version = "1.4.2"
+version = "1.4.3"
 description = "Bibtex parser for python 3"
 optional = false
 python-versions = "*"
 files = [
-    {file = "bibtexparser-1.4.2.tar.gz", hash = "sha256:dd5d54a1ec6d27b6485ce2f6b9aa514b183fb2b8d4bf5f19333a906eedaf8eaa"},
+    {file = "bibtexparser-1.4.3.tar.gz", hash = "sha256:a9c7ded64bc137720e4df0b1b7f12734edc1361185f1c9097048ff7c35af2b8f"},
 ]
 
 [package.dependencies]
@@ -1213,13 +1213,13 @@ files = [
 
 [[package]]
 name = "click"
-version = "8.1.7"
+version = "8.1.8"
 description = "Composable command line interface toolkit"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "click-8.1.7-py3-none-any.whl", hash = "sha256:ae74fb96c20a0277a1d615f1e4d73c8414f5a98db8b799a7931d1582f3390c28"},
-    {file = "click-8.1.7.tar.gz", hash = "sha256:ca9853ad459e787e2192211578cc907e7594e294c7ccc834310722b41b9ca6de"},
+    {file = "click-8.1.8-py3-none-any.whl", hash = "sha256:63c132bbbed01578a06712a2d1f497bb62d9c1c0d329b7903a866228027263b2"},
+    {file = "click-8.1.8.tar.gz", hash = "sha256:ed53c9d8990d83c2a27deae68e4ee337473f6330c040a31d4225c9574d16096a"},
 ]
 
 [package.dependencies]
@@ -1993,42 +1993,42 @@ files = [
 
 [[package]]
 name = "fastavro"
-version = "1.9.7"
+version = "1.10.0"
 description = "Fast read/write of AVRO files"
 optional = false
-python-versions = ">=3.8"
+python-versions = ">=3.9"
 files = [
-    {file = "fastavro-1.9.7-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cc811fb4f7b5ae95f969cda910241ceacf82e53014c7c7224df6f6e0ca97f52f"},
-    {file = "fastavro-1.9.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fb8749e419a85f251bf1ac87d463311874972554d25d4a0b19f6bdc56036d7cf"},
-    {file = "fastavro-1.9.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b2f9bafa167cb4d1c3dd17565cb5bf3d8c0759e42620280d1760f1e778e07fc"},
-    {file = "fastavro-1.9.7-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:e87d04b235b29f7774d226b120da2ca4e60b9e6fdf6747daef7f13f218b3517a"},
-    {file = "fastavro-1.9.7-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:b525c363e267ed11810aaad8fbdbd1c3bd8837d05f7360977d72a65ab8c6e1fa"},
-    {file = "fastavro-1.9.7-cp310-cp310-win_amd64.whl", hash = "sha256:6312fa99deecc319820216b5e1b1bd2d7ebb7d6f221373c74acfddaee64e8e60"},
-    {file = "fastavro-1.9.7-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ec8499dc276c2d2ef0a68c0f1ad11782b2b956a921790a36bf4c18df2b8d4020"},
-    {file = "fastavro-1.9.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:76d9d96f98052615ab465c63ba8b76ed59baf2e3341b7b169058db104cbe2aa0"},
-    {file = "fastavro-1.9.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:919f3549e07a8a8645a2146f23905955c35264ac809f6c2ac18142bc5b9b6022"},
-    {file = "fastavro-1.9.7-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9de1fa832a4d9016724cd6facab8034dc90d820b71a5d57c7e9830ffe90f31e4"},
-    {file = "fastavro-1.9.7-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1d09227d1f48f13281bd5ceac958650805aef9a4ef4f95810128c1f9be1df736"},
-    {file = "fastavro-1.9.7-cp311-cp311-win_amd64.whl", hash = "sha256:2db993ae6cdc63e25eadf9f93c9e8036f9b097a3e61d19dca42536dcc5c4d8b3"},
-    {file = "fastavro-1.9.7-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:4e1289b731214a7315884c74b2ec058b6e84380ce9b18b8af5d387e64b18fc44"},
-    {file = "fastavro-1.9.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:eac69666270a76a3a1d0444f39752061195e79e146271a568777048ffbd91a27"},
-    {file = "fastavro-1.9.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9be089be8c00f68e343bbc64ca6d9a13e5e5b0ba8aa52bcb231a762484fb270e"},
-    {file = "fastavro-1.9.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d576eccfd60a18ffa028259500df67d338b93562c6700e10ef68bbd88e499731"},
-    {file = "fastavro-1.9.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ee9bf23c157bd7dcc91ea2c700fa3bd924d9ec198bb428ff0b47fa37fe160659"},
-    {file = "fastavro-1.9.7-cp312-cp312-win_amd64.whl", hash = "sha256:b6b2ccdc78f6afc18c52e403ee68c00478da12142815c1bd8a00973138a166d0"},
-    {file = "fastavro-1.9.7-cp38-cp38-macosx_11_0_universal2.whl", hash = "sha256:7313def3aea3dacface0a8b83f6d66e49a311149aa925c89184a06c1ef99785d"},
-    {file = "fastavro-1.9.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:536f5644737ad21d18af97d909dba099b9e7118c237be7e4bd087c7abde7e4f0"},
-    {file = "fastavro-1.9.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2af559f30383b79cf7d020a6b644c42ffaed3595f775fe8f3d7f80b1c43dfdc5"},
-    {file = "fastavro-1.9.7-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:edc28ab305e3c424de5ac5eb87b48d1e07eddb6aa08ef5948fcda33cc4d995ce"},
-    {file = "fastavro-1.9.7-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:ec2e96bdabd58427fe683329b3d79f42c7b4f4ff6b3644664a345a655ac2c0a1"},
-    {file = "fastavro-1.9.7-cp38-cp38-win_amd64.whl", hash = "sha256:3b683693c8a85ede496ebebe115be5d7870c150986e34a0442a20d88d7771224"},
-    {file = "fastavro-1.9.7-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:58f76a5c9a312fbd37b84e49d08eb23094d36e10d43bc5df5187bc04af463feb"},
-    {file = "fastavro-1.9.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:56304401d2f4f69f5b498bdd1552c13ef9a644d522d5de0dc1d789cf82f47f73"},
-    {file = "fastavro-1.9.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2fcce036c6aa06269fc6a0428050fcb6255189997f5e1a728fc461e8b9d3e26b"},
-    {file = "fastavro-1.9.7-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:17de68aae8c2525f5631d80f2b447a53395cdc49134f51b0329a5497277fc2d2"},
-    {file = "fastavro-1.9.7-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:7c911366c625d0a997eafe0aa83ffbc6fd00d8fd4543cb39a97c6f3b8120ea87"},
-    {file = "fastavro-1.9.7-cp39-cp39-win_amd64.whl", hash = "sha256:912283ed48578a103f523817fdf0c19b1755cea9b4a6387b73c79ecb8f8f84fc"},
-    {file = "fastavro-1.9.7.tar.gz", hash = "sha256:13e11c6cb28626da85290933027cd419ce3f9ab8e45410ef24ce6b89d20a1f6c"},
+    {file = "fastavro-1.10.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:1a9fe0672d2caf0fe54e3be659b13de3cad25a267f2073d6f4b9f8862acc31eb"},
+    {file = "fastavro-1.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:86dd0410770e0c99363788f0584523709d85e57bb457372ec5c285a482c17fe6"},
+    {file = "fastavro-1.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:190e80dc7d77d03a6a8597a026146b32a0bbe45e3487ab4904dc8c1bebecb26d"},
+    {file = "fastavro-1.10.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:bf570d63be9155c3fdc415f60a49c171548334b70fff0679a184b69c29b6bc61"},
+    {file = "fastavro-1.10.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:e07abb6798e95dccecaec316265e35a018b523d1f3944ad396d0a93cb95e0a08"},
+    {file = "fastavro-1.10.0-cp310-cp310-win_amd64.whl", hash = "sha256:37203097ed11d0b8fd3c004904748777d730cafd26e278167ea602eebdef8eb2"},
+    {file = "fastavro-1.10.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:d183c075f527ab695a27ae75f210d4a86bce660cda2f85ae84d5606efc15ef50"},
+    {file = "fastavro-1.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d7a95a2c0639bffd7c079b59e9a796bfc3a9acd78acff7088f7c54ade24e4a77"},
+    {file = "fastavro-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0a678153b5da1b024a32ec3f611b2e7afd24deac588cb51dd1b0019935191a6d"},
+    {file = "fastavro-1.10.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:67a597a5cfea4dddcf8b49eaf8c2b5ffee7fda15b578849185bc690ec0cd0d8f"},
+    {file = "fastavro-1.10.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1fd689724760b17f69565d8a4e7785ed79becd451d1c99263c40cb2d6491f1d4"},
+    {file = "fastavro-1.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:4f949d463f9ac4221128a51e4e34e2562f401e5925adcadfd28637a73df6c2d8"},
+    {file = "fastavro-1.10.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:cfe57cb0d72f304bd0dcc5a3208ca6a7363a9ae76f3073307d095c9d053b29d4"},
+    {file = "fastavro-1.10.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:74e517440c824cb65fb29d3e3903a9406f4d7c75490cef47e55c4c82cdc66270"},
+    {file = "fastavro-1.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:203c17d44cadde76e8eecb30f2d1b4f33eb478877552d71f049265dc6f2ecd10"},
+    {file = "fastavro-1.10.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6575be7f2b5f94023b5a4e766b0251924945ad55e9a96672dc523656d17fe251"},
+    {file = "fastavro-1.10.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fe471deb675ed2f01ee2aac958fbf8ebb13ea00fa4ce7f87e57710a0bc592208"},
+    {file = "fastavro-1.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:567ff515f2a5d26d9674b31c95477f3e6022ec206124c62169bc2ffaf0889089"},
+    {file = "fastavro-1.10.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:82263af0adfddb39c85f9517d736e1e940fe506dfcc35bc9ab9f85e0fa9236d8"},
+    {file = "fastavro-1.10.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:566c193109ff0ff84f1072a165b7106c4f96050078a4e6ac7391f81ca1ef3efa"},
+    {file = "fastavro-1.10.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e400d2e55d068404d9fea7c5021f8b999c6f9d9afa1d1f3652ec92c105ffcbdd"},
+    {file = "fastavro-1.10.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9b8227497f71565270f9249fc9af32a93644ca683a0167cfe66d203845c3a038"},
+    {file = "fastavro-1.10.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8e62d04c65461b30ac6d314e4197ad666371e97ae8cb2c16f971d802f6c7f514"},
+    {file = "fastavro-1.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:86baf8c9740ab570d0d4d18517da71626fe9be4d1142bea684db52bd5adb078f"},
+    {file = "fastavro-1.10.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:5bccbb6f8e9e5b834cca964f0e6ebc27ebe65319d3940b0b397751a470f45612"},
+    {file = "fastavro-1.10.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b0132f6b0b53f61a0a508a577f64beb5de1a5e068a9b4c0e1df6e3b66568eec4"},
+    {file = "fastavro-1.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca37a363b711202c6071a6d4787e68e15fa3ab108261058c4aae853c582339af"},
+    {file = "fastavro-1.10.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:cf38cecdd67ca9bd92e6e9ba34a30db6343e7a3bedf171753ee78f8bd9f8a670"},
+    {file = "fastavro-1.10.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:f4dd10e0ed42982122d20cdf1a88aa50ee09e5a9cd9b39abdffb1aa4f5b76435"},
+    {file = "fastavro-1.10.0-cp39-cp39-win_amd64.whl", hash = "sha256:aaef147dc14dd2d7823246178fd06fc5e477460e070dc6d9e07dd8193a6bc93c"},
+    {file = "fastavro-1.10.0.tar.gz", hash = "sha256:47bf41ac6d52cdfe4a3da88c75a802321321b37b663a900d12765101a5d6886f"},
 ]
 
 [package.extras]
@@ -2282,13 +2282,13 @@ sqlalchemy = ["flask-sqlalchemy (>=3.0.5)"]
 
 [[package]]
 name = "flatbuffers"
-version = "24.3.25"
+version = "24.12.23"
 description = "The FlatBuffers serialization format for Python"
 optional = false
 python-versions = "*"
 files = [
-    {file = "flatbuffers-24.3.25-py2.py3-none-any.whl", hash = "sha256:8dbdec58f935f3765e4f7f3cf635ac3a77f83568138d6a2311f524ec96364812"},
-    {file = "flatbuffers-24.3.25.tar.gz", hash = "sha256:de2ec5b203f21441716617f38443e0a8ebf3d25bf0d9c0bb0ce68fa00ad546a4"},
+    {file = "flatbuffers-24.12.23-py2.py3-none-any.whl", hash = "sha256:c418e0d48890f4142b92fd3e343e73a48f194e1f80075ddcc5793779b3585444"},
+    {file = "flatbuffers-24.12.23.tar.gz", hash = "sha256:2910b0bc6ae9b6db78dd2b18d0b7a0709ba240fb5585f286a3a2b30785c22dac"},
 ]
 
 [[package]]
@@ -2643,13 +2643,13 @@ grpcio-gcp = ["grpcio-gcp (>=0.2.2,<1.0.dev0)"]
 
 [[package]]
 name = "google-api-python-client"
-version = "2.155.0"
+version = "2.156.0"
 description = "Google API Client Library for Python"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "google_api_python_client-2.155.0-py2.py3-none-any.whl", hash = "sha256:83fe9b5aa4160899079d7c93a37be306546a17e6686e2549bcc9584f1a229747"},
-    {file = "google_api_python_client-2.155.0.tar.gz", hash = "sha256:25529f89f0d13abcf3c05c089c423fb2858ac16e0b3727543393468d0d7af67c"},
+    {file = "google_api_python_client-2.156.0-py2.py3-none-any.whl", hash = "sha256:6352185c505e1f311f11b0b96c1b636dcb0fec82cd04b80ac5a671ac4dcab339"},
+    {file = "google_api_python_client-2.156.0.tar.gz", hash = "sha256:b809c111ded61716a9c1c7936e6899053f13bae3defcdfda904bd2ca68065b9c"},
 ]
 
 [package.dependencies]
@@ -3678,12 +3678,12 @@ vision = ["colpali-engine (>=0.3.1,<0.4.0)", "pillow", "timm", "torchvision"]
 
 [[package]]
 name = "infinity-sdk"
-version = "0.5.0"
+version = "0.5.2"
 description = "infinity"
 optional = false
 python-versions = ">=3.10"
 files = [
-    {file = "infinity_sdk-0.5.0-py3-none-any.whl", hash = "sha256:ac88dfe5984b45bb11cadd46567552e16dd5a7fb336e164c2e3faf63c20394ca"},
+    {file = "infinity_sdk-0.5.2-py3-none-any.whl", hash = "sha256:fb45ea3e7eaf834ba05fc233c23b075fce5b89f330500d53a92c417270193800"},
 ]
 
 [package.dependencies]
@@ -3794,13 +3794,13 @@ files = [
 
 [[package]]
 name = "jinja2"
-version = "3.1.4"
+version = "3.1.5"
 description = "A very fast and expressive template engine."
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "jinja2-3.1.4-py3-none-any.whl", hash = "sha256:bc5dd2abb727a5319567b7a813e6a2e7318c39f4f487cfe6c89c6f9c7d25197d"},
-    {file = "jinja2-3.1.4.tar.gz", hash = "sha256:4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369"},
+    {file = "jinja2-3.1.5-py3-none-any.whl", hash = "sha256:aba0f4dc9ed8013c424088f68a5c226f7d6097ed89b246d7749c2ec4175c6adb"},
+    {file = "jinja2-3.1.5.tar.gz", hash = "sha256:8fefff8dc3034e27bb80d67c671eb8a9bc424c0ef4c0826edbff304cceff43bb"},
 ]
 
 [package.dependencies]
@@ -6268,32 +6268,32 @@ files = [
 
 [[package]]
 name = "psutil"
-version = "6.1.0"
+version = "6.1.1"
 description = "Cross-platform lib for process and system monitoring in Python."
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 files = [
-    {file = "psutil-6.1.0-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:ff34df86226c0227c52f38b919213157588a678d049688eded74c76c8ba4a5d0"},
-    {file = "psutil-6.1.0-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:c0e0c00aa18ca2d3b2b991643b799a15fc8f0563d2ebb6040f64ce8dc027b942"},
-    {file = "psutil-6.1.0-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:000d1d1ebd634b4efb383f4034437384e44a6d455260aaee2eca1e9c1b55f047"},
-    {file = "psutil-6.1.0-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:5cd2bcdc75b452ba2e10f0e8ecc0b57b827dd5d7aaffbc6821b2a9a242823a76"},
-    {file = "psutil-6.1.0-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:045f00a43c737f960d273a83973b2511430d61f283a44c96bf13a6e829ba8fdc"},
-    {file = "psutil-6.1.0-cp27-none-win32.whl", hash = "sha256:9118f27452b70bb1d9ab3198c1f626c2499384935aaf55388211ad982611407e"},
-    {file = "psutil-6.1.0-cp27-none-win_amd64.whl", hash = "sha256:a8506f6119cff7015678e2bce904a4da21025cc70ad283a53b099e7620061d85"},
-    {file = "psutil-6.1.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:6e2dcd475ce8b80522e51d923d10c7871e45f20918e027ab682f94f1c6351688"},
-    {file = "psutil-6.1.0-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:0895b8414afafc526712c498bd9de2b063deaac4021a3b3c34566283464aff8e"},
-    {file = "psutil-6.1.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9dcbfce5d89f1d1f2546a2090f4fcf87c7f669d1d90aacb7d7582addece9fb38"},
-    {file = "psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:498c6979f9c6637ebc3a73b3f87f9eb1ec24e1ce53a7c5173b8508981614a90b"},
-    {file = "psutil-6.1.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d905186d647b16755a800e7263d43df08b790d709d575105d419f8b6ef65423a"},
-    {file = "psutil-6.1.0-cp36-cp36m-win32.whl", hash = "sha256:6d3fbbc8d23fcdcb500d2c9f94e07b1342df8ed71b948a2649b5cb060a7c94ca"},
-    {file = "psutil-6.1.0-cp36-cp36m-win_amd64.whl", hash = "sha256:1209036fbd0421afde505a4879dee3b2fd7b1e14fee81c0069807adcbbcca747"},
-    {file = "psutil-6.1.0-cp37-abi3-win32.whl", hash = "sha256:1ad45a1f5d0b608253b11508f80940985d1d0c8f6111b5cb637533a0e6ddc13e"},
-    {file = "psutil-6.1.0-cp37-abi3-win_amd64.whl", hash = "sha256:a8fb3752b491d246034fa4d279ff076501588ce8cbcdbb62c32fd7a377d996be"},
-    {file = "psutil-6.1.0.tar.gz", hash = "sha256:353815f59a7f64cdaca1c0307ee13558a0512f6db064e92fe833784f08539c7a"},
+    {file = "psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:9ccc4316f24409159897799b83004cb1e24f9819b0dcf9c0b68bdcb6cefee6a8"},
+    {file = "psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:ca9609c77ea3b8481ab005da74ed894035936223422dc591d6772b147421f777"},
+    {file = "psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:8df0178ba8a9e5bc84fed9cfa61d54601b371fbec5c8eebad27575f1e105c0d4"},
+    {file = "psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:1924e659d6c19c647e763e78670a05dbb7feaf44a0e9c94bf9e14dfc6ba50468"},
+    {file = "psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:018aeae2af92d943fdf1da6b58665124897cfc94faa2ca92098838f83e1b1bca"},
+    {file = "psutil-6.1.1-cp27-none-win32.whl", hash = "sha256:6d4281f5bbca041e2292be3380ec56a9413b790579b8e593b1784499d0005dac"},
+    {file = "psutil-6.1.1-cp27-none-win_amd64.whl", hash = "sha256:c777eb75bb33c47377c9af68f30e9f11bc78e0f07fbf907be4a5d70b2fe5f030"},
+    {file = "psutil-6.1.1-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:fc0ed7fe2231a444fc219b9c42d0376e0a9a1a72f16c5cfa0f68d19f1a0663e8"},
+    {file = "psutil-6.1.1-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:0bdd4eab935276290ad3cb718e9809412895ca6b5b334f5a9111ee6d9aff9377"},
+    {file = "psutil-6.1.1-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b6e06c20c05fe95a3d7302d74e7097756d4ba1247975ad6905441ae1b5b66003"},
+    {file = "psutil-6.1.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:97f7cb9921fbec4904f522d972f0c0e1f4fabbdd4e0287813b21215074a0f160"},
+    {file = "psutil-6.1.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:33431e84fee02bc84ea36d9e2c4a6d395d479c9dd9bba2376c1f6ee8f3a4e0b3"},
+    {file = "psutil-6.1.1-cp36-cp36m-win32.whl", hash = "sha256:384636b1a64b47814437d1173be1427a7c83681b17a450bfc309a1953e329603"},
+    {file = "psutil-6.1.1-cp36-cp36m-win_amd64.whl", hash = "sha256:8be07491f6ebe1a693f17d4f11e69d0dc1811fa082736500f649f79df7735303"},
+    {file = "psutil-6.1.1-cp37-abi3-win32.whl", hash = "sha256:eaa912e0b11848c4d9279a93d7e2783df352b082f40111e078388701fd479e53"},
+    {file = "psutil-6.1.1-cp37-abi3-win_amd64.whl", hash = "sha256:f35cfccb065fff93529d2afb4a2e89e363fe63ca1e4a5da22b603a85833c2649"},
+    {file = "psutil-6.1.1.tar.gz", hash = "sha256:cf8496728c18f2d0b45198f06895be52f36611711746b7f30c464b422b50e2f5"},
 ]
 
 [package.extras]
-dev = ["black", "check-manifest", "coverage", "packaging", "pylint", "pyperf", "pypinfo", "pytest-cov", "requests", "rstcheck", "ruff", "sphinx", "sphinx-rtd-theme", "toml-sort", "twine", "virtualenv", "wheel"]
+dev = ["abi3audit", "black", "check-manifest", "coverage", "packaging", "pylint", "pyperf", "pypinfo", "pytest-cov", "requests", "rstcheck", "ruff", "sphinx", "sphinx-rtd-theme", "toml-sort", "twine", "virtualenv", "vulture", "wheel"]
 test = ["enum34", "futures", "ipaddress", "mock (==1.0.1)", "pytest (==4.6.11)", "pytest-xdist", "setuptools", "unittest2"]
 
 [[package]]
@@ -7024,24 +7024,24 @@ image = ["Pillow"]
 
 [[package]]
 name = "pypdfium2"
-version = "4.30.0"
+version = "4.30.1"
 description = "Python bindings to PDFium"
 optional = false
-python-versions = ">=3.6"
+python-versions = ">= 3.6"
 files = [
-    {file = "pypdfium2-4.30.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:b33ceded0b6ff5b2b93bc1fe0ad4b71aa6b7e7bd5875f1ca0cdfb6ba6ac01aab"},
-    {file = "pypdfium2-4.30.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:4e55689f4b06e2d2406203e771f78789bd4f190731b5d57383d05cf611d829de"},
-    {file = "pypdfium2-4.30.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e6e50f5ce7f65a40a33d7c9edc39f23140c57e37144c2d6d9e9262a2a854854"},
-    {file = "pypdfium2-4.30.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3d0dd3ecaffd0b6dbda3da663220e705cb563918249bda26058c6036752ba3a2"},
-    {file = "pypdfium2-4.30.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cc3bf29b0db8c76cdfaac1ec1cde8edf211a7de7390fbf8934ad2aa9b4d6dfad"},
-    {file = "pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f1f78d2189e0ddf9ac2b7a9b9bd4f0c66f54d1389ff6c17e9fd9dc034d06eb3f"},
-    {file = "pypdfium2-4.30.0-py3-none-musllinux_1_1_aarch64.whl", hash = "sha256:5eda3641a2da7a7a0b2f4dbd71d706401a656fea521b6b6faa0675b15d31a163"},
-    {file = "pypdfium2-4.30.0-py3-none-musllinux_1_1_i686.whl", hash = "sha256:0dfa61421b5eb68e1188b0b2231e7ba35735aef2d867d86e48ee6cab6975195e"},
-    {file = "pypdfium2-4.30.0-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:f33bd79e7a09d5f7acca3b0b69ff6c8a488869a7fab48fdf400fec6e20b9c8be"},
-    {file = "pypdfium2-4.30.0-py3-none-win32.whl", hash = "sha256:ee2410f15d576d976c2ab2558c93d392a25fb9f6635e8dd0a8a3a5241b275e0e"},
-    {file = "pypdfium2-4.30.0-py3-none-win_amd64.whl", hash = "sha256:90dbb2ac07be53219f56be09961eb95cf2473f834d01a42d901d13ccfad64b4c"},
-    {file = "pypdfium2-4.30.0-py3-none-win_arm64.whl", hash = "sha256:119b2969a6d6b1e8d55e99caaf05290294f2d0fe49c12a3f17102d01c441bd29"},
-    {file = "pypdfium2-4.30.0.tar.gz", hash = "sha256:48b5b7e5566665bc1015b9d69c1ebabe21f6aee468b509531c3c8318eeee2e16"},
+    {file = "pypdfium2-4.30.1-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:e07c47633732cc18d890bb7e965ad28a9c5a932e548acb928596f86be2e5ae37"},
+    {file = "pypdfium2-4.30.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:5ea2d44e96d361123b67b00f527017aa9c847c871b5714e013c01c3eb36a79fe"},
+    {file = "pypdfium2-4.30.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1de7a3a36803171b3f66911131046d65a732f9e7834438191cb58235e6163c4e"},
+    {file = "pypdfium2-4.30.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b8a4231efb13170354f568c722d6540b8d5b476b08825586d48ef70c40d16e03"},
+    {file = "pypdfium2-4.30.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6f434a4934e8244aa95343ffcf24e9ad9f120dbb4785f631bb40a88c39292493"},
+    {file = "pypdfium2-4.30.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f454032a0bc7681900170f67d8711b3942824531e765f91c2f5ce7937f999794"},
+    {file = "pypdfium2-4.30.1-py3-none-musllinux_1_1_aarch64.whl", hash = "sha256:bbf9130a72370ee9d602e39949b902db669a2a1c24746a91e5586eb829055d9f"},
+    {file = "pypdfium2-4.30.1-py3-none-musllinux_1_1_i686.whl", hash = "sha256:5cb52884b1583b96e94fd78542c63bb42e06df5e8f9e52f8f31f5ad5a1e53367"},
+    {file = "pypdfium2-4.30.1-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:1a9e372bd4867ff223cc8c338e33fe11055dad12f22885950fc27646cc8d9122"},
+    {file = "pypdfium2-4.30.1-py3-none-win32.whl", hash = "sha256:421f1cf205e213e07c1f2934905779547f4f4a2ff2f59dde29da3d511d3fc806"},
+    {file = "pypdfium2-4.30.1-py3-none-win_amd64.whl", hash = "sha256:598a7f20264ab5113853cba6d86c4566e4356cad037d7d1f849c8c9021007e05"},
+    {file = "pypdfium2-4.30.1-py3-none-win_arm64.whl", hash = "sha256:c2b6d63f6d425d9416c08d2511822b54b8e3ac38e639fc41164b1d75584b3a8c"},
+    {file = "pypdfium2-4.30.1.tar.gz", hash = "sha256:5f5c7c6d03598e107d974f66b220a49436aceb191da34cda5f692be098a814ce"},
 ]
 
 [[package]]
@@ -9294,13 +9294,13 @@ tbb = ["tbb (>=2019.0)"]
 
 [[package]]
 name = "unlzw3"
-version = "0.2.2"
+version = "0.2.3"
 description = "Pure Python decompression module for .Z files compressed using Unix compress utility"
 optional = false
-python-versions = ">=3.7"
+python-versions = ">=3.9"
 files = [
-    {file = "unlzw3-0.2.2-py3-none-any.whl", hash = "sha256:f2a40fbd2a3c9648745227e5d269a851dbed7802999db4654d358a40a5d61c2f"},
-    {file = "unlzw3-0.2.2.tar.gz", hash = "sha256:d037a9b6823d1a455d6de1e0258af8c0f5dbf40aba3b19cc514448e78da77062"},
+    {file = "unlzw3-0.2.3-py3-none-any.whl", hash = "sha256:7760fb4f3afa1225623944c061991d89a061f7fb78665dbc4cddfdb562bb4a8b"},
+    {file = "unlzw3-0.2.3.tar.gz", hash = "sha256:ede5d928c792fff9da406f20334f9739693327f448f383ae1df1774627197bbb"},
 ]
 
 [package.extras]
@@ -9320,13 +9320,13 @@ files = [
 
 [[package]]
 name = "urllib3"
-version = "2.2.3"
+version = "2.3.0"
 description = "HTTP library with thread-safe connection pooling, file post, and more."
 optional = false
-python-versions = ">=3.8"
+python-versions = ">=3.9"
 files = [
-    {file = "urllib3-2.2.3-py3-none-any.whl", hash = "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac"},
-    {file = "urllib3-2.2.3.tar.gz", hash = "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9"},
+    {file = "urllib3-2.3.0-py3-none-any.whl", hash = "sha256:1cee9ad369867bfdbbb48b7dd50374c0967a0bb7710050facf0dd6911440e3df"},
+    {file = "urllib3-2.3.0.tar.gz", hash = "sha256:f8c5449b3cf0861679ce7e0503c7b44b5ec981bec0d1d3795a07f1ba96f0204d"},
 ]
 
 [package.dependencies]
@@ -10150,4 +10150,4 @@ cffi = ["cffi (>=1.11)"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.10,<3.13"
-content-hash = "4c592934d67923be94f92704ac2cd646f8f7a7f59c4a94707677f4be29c5a7b1"
+content-hash = "fea9a86cc56c010942cd40789eb68be585721e777f069bf3158b0df98fb521df"

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "ragflow"
-version = "0.15.0"
+version = "0.15.1"
 description = "[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data."
 authors = ["Your Name <you@example.com>"]
 license = "https://github.com/infiniflow/ragflow/blob/main/LICENSE"
@@ -48,7 +48,7 @@ hanziconv = "0.3.2"
 html-text = "0.6.2"
 httpx = "0.27.0"
 huggingface-hub = "^0.25.0"
-infinity-sdk = "0.5.0"
+infinity-sdk = "0.5.2"
 infinity-emb = "^0.0.66"
 itsdangerous = "2.1.2"
 markdown = "3.6"

rag/app/table.py

@@ -66,6 +66,8 @@ class Excel(ExcelParser):
                     continue
                 data.append(row)
                 done += 1
+            if np.array(data).size == 0:
+                continue
             res.append(pd.DataFrame(np.array(data), columns=headers))
 
         callback(0.3, ("Extract records: {}~{}".format(from_page + 1, min(to_page, from_page + rn)) + (

rag/llm/embedding_model.py

@@ -47,6 +47,7 @@ class Base(ABC):
 
 class DefaultEmbedding(Base):
     _model = None
+    _model_name = ""
     _model_lock = threading.Lock()
     def __init__(self, key, model_name, **kwargs):
         """
@@ -60,15 +61,16 @@ class DefaultEmbedding(Base):
         ^_-
 
         """
-        if not settings.LIGHTEN and not DefaultEmbedding._model:
+        if not settings.LIGHTEN:
             with DefaultEmbedding._model_lock:
                 from FlagEmbedding import FlagModel
                 import torch
-                if not DefaultEmbedding._model:
+                if not DefaultEmbedding._model or model_name != DefaultEmbedding._model_name:
                     try:
                         DefaultEmbedding._model = FlagModel(os.path.join(get_home_cache_dir(), re.sub(r"^[a-zA-Z0-9]+/", "", model_name)),
                                                             query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：",
                                                             use_fp16=torch.cuda.is_available())
+                        DefaultEmbedding._model_name = model_name
                     except Exception:
                         model_dir = snapshot_download(repo_id="BAAI/bge-large-zh-v1.5",
                                                       local_dir=os.path.join(get_home_cache_dir(), re.sub(r"^[a-zA-Z0-9]+/", "", model_name)),
@@ -77,6 +79,7 @@ class DefaultEmbedding(Base):
                                                             query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：",
                                                             use_fp16=torch.cuda.is_available())
         self._model = DefaultEmbedding._model
+        self._model_name = DefaultEmbedding._model_name
 
     def encode(self, texts: list):
         batch_size = 16
@@ -248,9 +251,8 @@ class OllamaEmbed(Base):
         return np.array(res["embedding"]), 128
 
 
-class FastEmbed(Base):
-    _model = None
-
+class FastEmbed(DefaultEmbedding):
+    
     def __init__(
             self,
             key: str | None = None,
@@ -259,9 +261,21 @@ class FastEmbed(Base):
             threads: int | None = None,
             **kwargs,
     ):
-        if not settings.LIGHTEN and not FastEmbed._model:
-            from fastembed import TextEmbedding
-            self._model = TextEmbedding(model_name, cache_dir, threads, **kwargs)
+        if not settings.LIGHTEN:
+            with FastEmbed._model_lock:
+                from fastembed import TextEmbedding
+                if not DefaultEmbedding._model or model_name != DefaultEmbedding._model_name:
+                    try:
+                        DefaultEmbedding._model = TextEmbedding(model_name, cache_dir, threads, **kwargs)
+                        DefaultEmbedding._model_name = model_name
+                    except Exception:
+                        cache_dir = snapshot_download(repo_id="BAAI/bge-small-en-v1.5",
+                                                      local_dir=os.path.join(get_home_cache_dir(),
+                                                                             re.sub(r"^[a-zA-Z0-9]+/", "", model_name)),
+                                                      local_dir_use_symlinks=False)
+                        DefaultEmbedding._model = TextEmbedding(model_name, cache_dir, threads, **kwargs)
+        self._model = DefaultEmbedding._model
+        self._model_name = model_name
 
     def encode(self, texts: list):
         # Using the internal tokenizer to encode the texts and get the total

rag/llm/rerank_model.py

@@ -422,10 +422,12 @@ class VoyageRerank(Base):
         self.model_name = model_name
 
     def similarity(self, query: str, texts: list):
+        rank = np.zeros(len(texts), dtype=float)
+        if not texts:
+            return rank, 0
         res = self.client.rerank(
             query=query, documents=texts, model=self.model_name, top_k=len(texts)
         )
-        rank = np.zeros(len(texts), dtype=float)
         for r in res.results:
             rank[r.index] = r.relevance_score
         return rank, res.total_tokens

rag/nlp/search.py

@@ -70,7 +70,7 @@ class Dealer:
         pg = int(req.get("page", 1)) - 1
         topk = int(req.get("topk", 1024))
         ps = int(req.get("size", topk))
-        offset, limit = pg * ps, (pg + 1) * ps
+        offset, limit = pg * ps, ps
 
         src = req.get("fields", ["docnm_kwd", "content_ltks", "kb_id", "img_id", "title_tks", "important_kwd", "position_int",
                                  "doc_id", "page_num_int", "top_int", "create_timestamp_flt", "knowledge_graph_kwd", "question_kwd", "question_tks",
@@ -313,7 +313,7 @@ class Dealer:
         ranks["total"] = sres.total
 
         if page <= RERANK_PAGE_LIMIT:
-            if rerank_mdl:
+            if rerank_mdl and sres.total > 0:
                 sim, tsim, vsim = self.rerank_by_model(rerank_mdl,
                     sres, question, 1 - vector_similarity_weight, vector_similarity_weight)
             else:
@@ -380,6 +380,13 @@ class Dealer:
 
     def chunk_list(self, doc_id: str, tenant_id: str, kb_ids: list[str], max_count=1024, fields=["docnm_kwd", "content_with_weight", "img_id"]):
         condition = {"doc_id": doc_id}
-        res = self.dataStore.search(fields, [], condition, [], OrderByExpr(), 0, max_count, index_name(tenant_id), kb_ids)
-        dict_chunks = self.dataStore.getFields(res, fields)
-        return dict_chunks.values()
+        res = []
+        bs = 128
+        for p in range(0, max_count, bs):
+            es_res = self.dataStore.search(fields, [], condition, [], OrderByExpr(), p, bs, index_name(tenant_id), kb_ids)
+            dict_chunks = self.dataStore.getFields(es_res, fields)
+            if dict_chunks:
+                res.extend(dict_chunks.values())
+            if len(dict_chunks.values()) < bs:
+                break
+        return res

rag/svr/task_executor.py

@@ -83,7 +83,7 @@ FACTORY = {
 
 CONSUMER_NAME = "task_consumer_" + CONSUMER_NO
 PAYLOAD: Payload | None = None
-BOOT_AT = datetime.now().isoformat()
+BOOT_AT = datetime.now().astimezone().isoformat(timespec="milliseconds")
 PENDING_TASKS = 0
 LAG_TASKS = 0
 
@@ -116,6 +116,8 @@ def set_progress(task_id, from_page=0, to_page=-1, prog=None, msg="Processing...
     if to_page > 0:
         if msg:
             msg = f"Page({from_page + 1}~{to_page + 1}): " + msg
+    if msg:
+        msg = datetime.now().strftime("%H:%M:%S") + " " + msg
     d = {"progress_msg": msg}
     if prog is not None:
         d["progress"] = prog
@@ -550,7 +552,7 @@ def report_status():
             with mt_lock:
                 heartbeat = json.dumps({
                     "name": CONSUMER_NAME,
-                    "now": now.isoformat(),
+                    "now": now.astimezone().isoformat(timespec="milliseconds"),
                     "boot_at": BOOT_AT,
                     "pending": PENDING_TASKS,
                     "lag": LAG_TASKS,

rag/utils/es_conn.py

@@ -196,7 +196,7 @@ class ESConnection(DocStoreConnection):
             s = s.sort(*orders)
 
         if limit > 0:
-            s = s[offset:limit]
+            s = s[offset:offset+limit]
         q = s.to_dict()
         logger.debug(f"ESConnection.search {str(indexNames)} query: " + json.dumps(q))
 

rag/utils/infinity_conn.py

@@ -3,6 +3,7 @@ import os
 import re
 import json
 import time
+import copy
 import infinity
 from infinity.common import ConflictType, InfinityException, SortType
 from infinity.index import IndexInfo, IndexType
@@ -57,8 +58,11 @@ def concat_dataframes(df_list: list[pl.DataFrame], selectFields: list[str]) -> p
     if df_list:
         return pl.concat(df_list)
     schema = dict()
-    for fieldnm in selectFields:
-        schema[fieldnm] = str
+    for field_name in selectFields:
+        if field_name == 'score()': # Workaround: fix schema is changed to score()
+            schema['SCORE'] = str
+        else:
+            schema[field_name] = str
     return pl.DataFrame(schema=schema)
 
 @singleton
@@ -138,7 +142,6 @@ class InfinityConnection(DocStoreConnection):
     def health(self) -> dict:
         """
         Return the health status of the database.
-        TODO: Infinity-sdk provides health() to wrap `show global variables` and `show tables`
         """
         inf_conn = self.connPool.get_conn()
         res = inf_conn.show_current_node()
@@ -247,9 +250,12 @@ class InfinityConnection(DocStoreConnection):
         db_instance = inf_conn.get_database(self.dbName)
         df_list = list()
         table_list = list()
-        for essential_field in ["id", "score()", "pagerank_fea"]:
+        for essential_field in ["id"]:
             if essential_field not in selectFields:
                 selectFields.append(essential_field)
+        if matchExprs:
+            for essential_field in ["score()", "pagerank_fea"]:
+                selectFields.append(essential_field)
 
         # Prepare expressions common to all tables
         filter_cond = None
@@ -337,7 +343,8 @@ class InfinityConnection(DocStoreConnection):
                 df_list.append(kb_res)
         self.connPool.release_conn(inf_conn)
         res = concat_dataframes(df_list, selectFields)
-        res = res.sort(pl.col("SCORE") + pl.col("pagerank_fea"), descending=True, maintain_order=True)
+        if matchExprs:
+            res = res.sort(pl.col("SCORE") + pl.col("pagerank_fea"), descending=True, maintain_order=True)
         res = res.limit(limit)
         logger.debug(f"INFINITY search final result: {str(res)}")
         return res, total_hits_count
@@ -386,7 +393,8 @@ class InfinityConnection(DocStoreConnection):
             self.createIdx(indexName, knowledgebaseId, vector_size)
             table_instance = db_instance.get_table(table_name)
 
-        for d in documents:
+        docs = copy.deepcopy(documents)
+        for d in docs:
             assert "_id" not in d
             assert "id" in d
             for k, v in d.items():
@@ -400,17 +408,17 @@ class InfinityConnection(DocStoreConnection):
                     assert isinstance(v, list)
                     arr = [num for row in v for num in row]
                     d[k] = "_".join(f"{num:08x}" for num in arr)
-                elif k in ["page_num_int", "top_int", "position_int"]:
+                elif k in ["page_num_int", "top_int"]:
                     assert isinstance(v, list)
                     d[k] = "_".join(f"{num:08x}" for num in v)
-        ids = ["'{}'".format(d["id"]) for d in documents]
+        ids = ["'{}'".format(d["id"]) for d in docs]
         str_ids = ", ".join(ids)
         str_filter = f"id IN ({str_ids})"
         table_instance.delete(str_filter)
         # for doc in documents:
         #     logger.info(f"insert position_int: {doc['position_int']}")
         # logger.info(f"InfinityConnection.insert {json.dumps(documents)}")
-        table_instance.insert(documents)
+        table_instance.insert(docs)
         self.connPool.release_conn(inf_conn)
         logger.debug(f"INFINITY inserted into {table_name} {str_ids}.")
         return []
@@ -504,7 +512,7 @@ class InfinityConnection(DocStoreConnection):
                     assert isinstance(v, str)
                     if v:
                         arr = [int(hex_val, 16) for hex_val in v.split('_')]
-                        v = [arr[i:i + 4] for i in range(0, len(arr), 4)]
+                        v = [arr[i:i + 5] for i in range(0, len(arr), 5)]
                     else:
                         v = []
                 elif fieldnm in ["page_num_int", "top_int"]:

rag/utils/s3_conn.py

@@ -1,20 +1,21 @@
 import logging
 import boto3
-import os
 from botocore.exceptions import ClientError
 from botocore.client import Config
 import time
 from io import BytesIO
 from rag.utils import singleton
+from rag import settings
 
 @singleton
 class RAGFlowS3(object):
     def __init__(self):
         self.conn = None
-        self.endpoint = os.getenv('ENDPOINT', None)
-        self.access_key = os.getenv('ACCESS_KEY', None)
-        self.secret_key = os.getenv('SECRET_KEY', None)
-        self.region = os.getenv('REGION', None)
+        self.s3_config = settings.S3
+        self.endpoint = self.s3_config.get('endpoint', None)
+        self.access_key = self.s3_config.get('access_key', None)
+        self.secret_key = self.s3_config.get('secret_key', None)
+        self.region = self.s3_config.get('region', None)
         self.__open__()
 
     def __open__(self):

sdk/python/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "ragflow-sdk"
-version = "0.15.0"
+version = "0.15.1"
 description = "Python client sdk of [RAGFlow](https://github.com/infiniflow/ragflow). RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding."
 authors = ["Zhichang Yu <yuzhichang@gmail.com>"]
 license = "Apache License, Version 2.0"

sdk/python/ragflow_sdk/modules/agent.py

@@ -1,7 +1,6 @@
 from .base import Base
-from .session import Session,Message
+from .session import Session
 import requests
-import json
 
 
 class Agent(Base):
@@ -52,8 +51,8 @@ class Agent(Base):
             super().__init__(rag,res_dict)
 
     @staticmethod
-    def create_session(id,rag) -> Session:
-        res = requests.post(f"{rag.api_url}/agents/{id}/sessions",headers={"Authorization": f"Bearer {rag.user_key}"},json={})
+    def create_session(id,rag,**kwargs) -> Session:
+        res = requests.post(f"{rag.api_url}/agents/{id}/sessions",headers={"Authorization": f"Bearer {rag.user_key}"},json=kwargs)
         res = res.json()
         if res.get("code") == 0:
             return Session(rag,res.get("data"))
@@ -74,30 +73,3 @@ class Agent(Base):
                 result_list.append(temp_agent)
             return result_list
         raise Exception(res.get("message"))
-
-    @staticmethod
-    def ask(agent_id,rag,stream=True,**kwargs):
-        url = f"{rag.api_url}/agents/{agent_id}/completions"
-        headers = {"Authorization": f"Bearer {rag.user_key}"}
-        res = requests.post(url=url, headers=headers, json=kwargs,stream=stream)
-        for line in res.iter_lines():
-            line = line.decode("utf-8")
-            if line.startswith("{"):
-                json_data = json.loads(line)
-                raise Exception(json_data["message"])
-            if line.startswith("data:"):
-                json_data = json.loads(line[5:])
-                if json_data["data"] is not True:
-                    if json_data["data"].get("running_status"):
-                        continue
-                    answer = json_data["data"]["answer"]
-                    reference = json_data["data"]["reference"]
-                    temp_dict = {
-                        "content": answer,
-                        "role": "assistant"
-                    }
-                    if "chunks" in reference:
-                        chunks = reference["chunks"]
-                        temp_dict["reference"] = chunks
-                    message = Message(rag, temp_dict)
-                    yield message

sdk/python/ragflow_sdk/modules/session.py

@@ -17,38 +17,40 @@ class Session(Base):
                 self.__session_type = "agent"
         super().__init__(rag, res_dict)
 
-    def ask(self, question,stream=True):
+    def ask(self, question="",stream=True,**kwargs):
         if self.__session_type == "agent":
             res=self._ask_agent(question,stream)
         elif self.__session_type == "chat":
-            res=self._ask_chat(question,stream)
+            res=self._ask_chat(question,stream,**kwargs)
         for line in res.iter_lines():
             line = line.decode("utf-8")
             if line.startswith("{"):
                 json_data = json.loads(line)
                 raise Exception(json_data["message"])
-            if line.startswith("data:"):
-                json_data = json.loads(line[5:])
-                if json_data["data"] is not True:
-                    if json_data["data"].get("running_status"):
-                        continue
-                    answer = json_data["data"]["answer"]
-                    reference = json_data["data"]["reference"]
-                    temp_dict = {
-                        "content": answer,
-                        "role": "assistant"
-                    }
-                    if "chunks" in reference:
-                        chunks = reference["chunks"]
-                        temp_dict["reference"] = chunks
-                    message = Message(self.rag, temp_dict)
-                    yield message
+            if not line.startswith("data:"):
+                continue
+            json_data = json.loads(line[5:])
+            if json_data["data"] is True or json_data["data"].get("running_status"):
+                continue
+            answer = json_data["data"]["answer"]
+            reference = json_data["data"].get("reference", {})
+            temp_dict = {
+                "content": answer,
+                "role": "assistant"
+            }
+            if reference and "chunks" in reference:
+                chunks = reference["chunks"]
+                temp_dict["reference"] = chunks
+            message = Message(self.rag, temp_dict)
+            yield message
 
-
-    def _ask_chat(self, question: str, stream: bool):
+    def _ask_chat(self, question: str, stream: bool,**kwargs):
+        json_data={"question": question, "stream": True,"session_id":self.id}
+        json_data.update(kwargs)
         res = self.post(f"/chats/{self.chat_id}/completions",
-                        {"question": question, "stream": True,"session_id":self.id}, stream=stream)
+                        json_data, stream=stream)
         return res
+
     def _ask_agent(self,question:str,stream:bool):
         res = self.post(f"/agents/{self.agent_id}/completions",
                         {"question": question, "stream": True,"session_id":self.id}, stream=stream)
@@ -68,4 +70,4 @@ class Message(Base):
         self.role = "assistant"
         self.prompt = None
         self.id = None
-        super().__init__(rag, res_dict)
+        super().__init__(rag, res_dict)

web/src/assets/svg/jin10.svg

@@ -1,22 +1,100 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="32px" height="32px" viewBox="0 0 32 32" enable-background="new 0 0 32 32" xml:space="preserve">  <image id="image0" width="32" height="32" x="0" y="0"
-    xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAMAAABEpIrGAAAAIGNIUk0AAHomAACAhAAA+gAAAIDo
-AAB1MAAA6mAAADqYAAAXcJy6UTwAAAEdUExURQAAAFiD5FuD5FuC5lqC5lqC5lqC5lqC5lqC5lqB
-5lqC5lqC5lqC5lqC5lyB41qC5lqC5VqC5lqC5lqC5VqC5lqC5mGH532c64yo7niZ68PS9vX3/f3+
-/////16F57HE8/z9/rHD88XT9v7+/+Ho+q/D85aw77DD88TT9vH1/Zew712F5vz8/niZ6sLR9o6p
-7miN6PT3/WCH5/7//7zM9WyP6Xyc642o7pav7/D0/Yqm7fr7/puz8Hqa61+G597m+q3B8trj+e/z
-/Xub6/P2/aC38GSK6F2E5n+e7GeG14uQrnSKyN6pT/mxMPKvOIWPtdKlXfixMXWKx9ji+YWPtNKl
-XLadfcPR9u/z/GCG5+OrSWOF24yo7Xyb62SF2qIeFM0AAAAVdFJOUwAAACWKxePy+gJr9PNqAv0l
-+Ilq+Vbt7JYAAAABYktHRB3rA3GRAAAAB3RJTUUH6AkYCCg6OzwI7gAAAYhJREFUOMuFU/tDwVAU
-vh5DIsI1zEFtjCGPUN4qFJqi0kP1//8Z3ck2w+b75Z5z7ncf55zvIIRMFitlszuwBg67jbJaTIjg
-yHmMdeByn5DzTg/WhcdtQhYXNoDLi6zYEFZEGRMoZNt0g3QoRAc3IzZkV51whIkCRJlIWI3ZkZJ/
-LJ6ANRJnMTl6ipT9cwCWS6ZSPMcCpBWGQogDZISsZGWFDAC/TQgnIEPLQToDufwWIQKsoH5NYOFC
-SwgywJH76UJx5WY5KJU1BDoKSbIULplKVfJ5iNY0hCuAa7IUGYB6gxgpgNA+Aq6QKjRb7Y6wTVg/
-gav1VaW6vZvbu32fxLjRBOgPhiRyr5NmC/oPo/FEyXinUG0YjB7F6ZNuqTvd4VgUxecdQiwtNYtP
-CT08mYribC4T1HbzuVUGLxi/vs3EhdzuDcHk30tEMB+fxJwvvpZrwWgkV659h37+81v+riVHYUNQ
-h2TvQ95Dg2M2HD2/WRpe3Ttc/hNpvM1eafwD2r0AGX+fl5z/Az9LZaYvSaPqAAAAJXRFWHRkYXRl
-OmNyZWF0ZQAyMDI0LTA5LTI0VDA4OjQwOjU4KzAwOjAwH6ZUgwAAACV0RVh0ZGF0ZTptb2RpZnkA
-MjAyNC0wOS0yNFQwODo0MDo1OCswMDowMG777D8AAAAodEVYdGRhdGU6dGltZXN0YW1wADIwMjQt
-MDktMjRUMDg6NDA6NTgrMDA6MDA57s3gAAAAAElFTkSuQmCC" />
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 width="100%" viewBox="0 0 225 225" enable-background="new 0 0 225 225" xml:space="preserve">
+<path fill="#4E77E1" opacity="1.000000" stroke="none" 
+	d="
+M126.000000,226.000000 
+	C84.000000,226.000000 42.500004,226.000000 1.000005,226.000000 
+	C1.000004,151.000000 1.000004,76.000015 1.000002,1.000016 
+	C75.999992,1.000010 150.999985,1.000010 225.999969,1.000005 
+	C225.999985,75.999985 225.999985,150.999969 226.000000,225.999969 
+	C192.833328,226.000000 159.666672,226.000000 126.000000,226.000000 
+M126.451942,195.953262 
+	C130.111862,195.104523 133.771774,194.255798 136.877396,193.535599 
+	C134.804596,185.717453 132.910065,178.571671 131.019653,171.441498 
+	C88.774292,183.966232 54.862122,153.211212 51.077278,120.135582 
+	C47.432079,88.280296 70.430069,51.832520 113.189255,52.050751 
+	C127.914680,52.125908 140.978256,57.197155 151.874985,67.052521 
+	C171.891449,85.156082 176.767548,107.700661 170.048309,132.396515 
+	C177.698883,134.775635 184.710709,136.956116 191.865250,139.180969 
+	C196.303452,126.220093 197.254868,112.574364 194.890625,98.718338 
+	C187.718872,56.687061 148.770157,22.798378 100.686371,29.494141 
+	C51.300331,36.371246 20.288342,80.451393 29.162020,129.355713 
+	C32.443077,147.438171 41.506237,162.500519 54.837860,175.155014 
+	C68.121231,187.763702 84.540428,194.016388 101.891159,197.384567 
+	C109.416885,198.845474 117.659668,196.612610 126.451942,195.953262 
+M126.769096,161.530502 
+	C128.346542,163.576706 129.923981,165.622894 132.019836,168.341553 
+	C135.908264,148.561630 147.028412,136.956192 166.648529,133.501755 
+	C162.191055,129.496307 157.879639,126.613861 152.068741,125.976753 
+	C149.571976,125.702995 146.736313,124.190811 144.967133,122.359650 
+	C137.319489,114.444107 128.581635,108.167656 118.242104,104.523796 
+	C111.828415,102.263481 105.044189,101.054550 97.718941,99.194038 
+	C98.285645,115.676163 104.039955,128.549362 114.112762,139.226562 
+	C119.788513,145.242874 125.469147,150.751068 125.481583,159.708054 
+	C125.482162,160.124649 125.997719,160.540512 126.769096,161.530502 
+M183.951691,168.486954 
+	C183.679153,167.184875 183.441574,165.874313 183.128418,164.582077 
+	C181.090988,156.174911 173.332703,150.429123 165.132767,151.236694 
+	C156.368698,152.099823 149.725143,158.958099 149.215408,167.668503 
+	C148.733948,175.895798 154.725494,183.192413 163.139450,185.291412 
+	C170.199677,187.052689 177.230988,189.050415 184.080154,191.490662 
+	C188.021027,192.894745 191.574341,195.386597 194.378754,196.892868 
+	C191.058304,188.033127 187.549881,178.671844 183.951691,168.486954 
+z"/>
+<path fill="#FDFEFE" opacity="1.000000" stroke="none" 
+	d="
+M126.014343,196.005386 
+	C117.659668,196.612610 109.416885,198.845474 101.891159,197.384567 
+	C84.540428,194.016388 68.121231,187.763702 54.837860,175.155014 
+	C41.506237,162.500519 32.443077,147.438171 29.162020,129.355713 
+	C20.288342,80.451393 51.300331,36.371246 100.686371,29.494141 
+	C148.770157,22.798378 187.718872,56.687061 194.890625,98.718338 
+	C197.254868,112.574364 196.303452,126.220093 191.865250,139.180969 
+	C184.710709,136.956116 177.698883,134.775635 170.048309,132.396515 
+	C176.767548,107.700661 171.891449,85.156082 151.874985,67.052521 
+	C140.978256,57.197155 127.914680,52.125908 113.189255,52.050751 
+	C70.430069,51.832520 47.432079,88.280296 51.077278,120.135582 
+	C54.862122,153.211212 88.774292,183.966232 131.019653,171.441498 
+	C132.910065,178.571671 134.804596,185.717453 136.877396,193.535599 
+	C133.771774,194.255798 130.111862,195.104523 126.014343,196.005386 
+z"/>
+<path fill="#FCFDFE" opacity="1.000000" stroke="none" 
+	d="
+M126.521263,161.243622 
+	C125.997719,160.540512 125.482162,160.124649 125.481583,159.708054 
+	C125.469147,150.751068 119.788513,145.242874 114.112762,139.226562 
+	C104.039955,128.549362 98.285645,115.676163 97.718941,99.194038 
+	C105.044189,101.054550 111.828415,102.263481 118.242104,104.523796 
+	C128.581635,108.167656 137.319489,114.444107 144.967133,122.359650 
+	C146.736313,124.190811 149.571976,125.702995 152.068741,125.976753 
+	C157.879639,126.613861 162.191055,129.496307 166.648529,133.501755 
+	C147.028412,136.956192 135.908264,148.561630 132.019836,168.341553 
+	C129.923981,165.622894 128.346542,163.576706 126.521263,161.243622 
+M118.577805,132.147232 
+	C122.717079,133.828918 126.714119,132.836227 128.478516,129.128738 
+	C129.756729,126.442894 129.689621,122.369713 128.640198,119.492226 
+	C126.915466,114.763100 121.885849,116.091606 118.496376,116.719200 
+	C116.469025,117.094604 114.263878,120.366257 113.488678,122.771225 
+	C112.230156,126.675621 114.443306,129.761383 118.577805,132.147232 
+z"/>
+<path fill="#F8B031" opacity="1.000000" stroke="none" 
+	d="
+M183.996582,168.898773 
+	C187.549881,178.671844 191.058304,188.033127 194.378754,196.892868 
+	C191.574341,195.386597 188.021027,192.894745 184.080154,191.490662 
+	C177.230988,189.050415 170.199677,187.052689 163.139450,185.291412 
+	C154.725494,183.192413 148.733948,175.895798 149.215408,167.668503 
+	C149.725143,158.958099 156.368698,152.099823 165.132767,151.236694 
+	C173.332703,150.429123 181.090988,156.174911 183.128418,164.582077 
+	C183.441574,165.874313 183.679153,167.184875 183.996582,168.898773 
+z"/>
+<path fill="#5079E1" opacity="1.000000" stroke="none" 
+	d="
+M118.221985,132.004547 
+	C114.443306,129.761383 112.230156,126.675621 113.488678,122.771225 
+	C114.263878,120.366257 116.469025,117.094604 118.496376,116.719200 
+	C121.885849,116.091606 126.915466,114.763100 128.640198,119.492226 
+	C129.689621,122.369713 129.756729,126.442894 128.478516,129.128738 
+	C126.714119,132.836227 122.717079,133.828918 118.221985,132.004547 
+z"/>
 </svg>

web/src/hooks/chat-hooks.ts

@@ -340,6 +340,7 @@ export const useUpdateNextConversation = () => {
       });
       if (data.code === 0) {
         queryClient.invalidateQueries({ queryKey: ['fetchConversationList'] });
+        message.success(i18n.t(`message.modified`));
       }
       return data;
     },

web/src/hooks/knowledge-hooks.ts

@@ -248,4 +248,14 @@ export const useSelectTestingResult = (): ITestingResult => {
     total: 0,
   }) as ITestingResult;
 };
+
+export const useSelectIsTestingSuccess = () => {
+  const status = useMutationState({
+    filters: { mutationKey: ['testChunk'] },
+    select: (mutation) => {
+      return mutation.state.status;
+    },
+  });
+  return status.at(-1) === 'success';
+};
 //#endregion

web/src/locales/vi.ts

@@ -36,6 +36,7 @@ export default {
       vietnamese: 'Tiếng  việt',
       spanish: 'Tiếng Tây Ban Nha',
       japanese: 'Tiếng Nhật',
+      embedIntoSite: 'Nhúng vào trang web',
     },
     login: {
       login: 'Đăng nhập',

web/src/pages/add-knowledge/components/knowledge-chunk/components/knowledge-graph/util.ts

@@ -1,4 +1,5 @@
 import { isEmpty } from 'lodash';
+import { v4 as uuid } from 'uuid';
 
 class KeyGenerator {
   idx = 0;
@@ -64,16 +65,22 @@ export const isDataExist = (data: any) => {
 
 export const buildNodesAndCombos = (nodes: any[]) => {
   const combos: any[] = [];
-  const nextNodes = nodes.map((x) => {
+  nodes.forEach((x) => {
     const combo = Array.isArray(x?.communities) ? x.communities[0] : undefined;
-    if (combo && combos.every((y) => y.id !== combo)) {
+    if (combo && combos.every((y) => y.data.label !== combo)) {
       combos.push({
-        id: combo,
+        id: uuid(),
+        data: {
+          label: combo,
+        },
       });
     }
+  });
+
+  const nextNodes = nodes.map((x) => {
     return {
       ...x,
-      combo,
+      combo: combos.find((y) => y.data.label === x.id)?.id,
     };
   });
 

web/src/pages/add-knowledge/components/knowledge-testing/testing-result/index.tsx

@@ -5,6 +5,7 @@ import { ITestingChunk } from '@/interfaces/database/knowledge';
 import {
   Card,
   Collapse,
+  Empty,
   Flex,
   Pagination,
   PaginationProps,
@@ -14,7 +15,10 @@ import {
 import camelCase from 'lodash/camelCase';
 import SelectFiles from './select-files';
 
-import { useSelectTestingResult } from '@/hooks/knowledge-hooks';
+import {
+  useSelectIsTestingSuccess,
+  useSelectTestingResult,
+} from '@/hooks/knowledge-hooks';
 import { useGetPaginationWithRouter } from '@/hooks/logic-hooks';
 import { useCallback, useState } from 'react';
 import styles from './index.less';
@@ -50,6 +54,7 @@ const TestingResult = ({ handleTesting }: IProps) => {
   const { documents, chunks, total } = useSelectTestingResult();
   const { t } = useTranslate('knowledgeDetails');
   const { pagination, setPagination } = useGetPaginationWithRouter();
+  const isSuccess = useSelectIsTestingSuccess();
 
   const onChange: PaginationProps['onChange'] = (pageNumber, pageSize) => {
     pagination.onChange?.(pageNumber, pageSize);
@@ -105,26 +110,30 @@ const TestingResult = ({ handleTesting }: IProps) => {
         flex={1}
         className={styles.selectFilesCollapse}
       >
-        {chunks?.map((x) => (
-          <Card key={x.chunk_id} title={<ChunkTitle item={x}></ChunkTitle>}>
-            <Flex gap={'middle'}>
-              {x.img_id && (
-                <Popover
-                  placement="left"
-                  content={
-                    <Image
-                      id={x.img_id}
-                      className={styles.imagePreview}
-                    ></Image>
-                  }
-                >
-                  <Image id={x.img_id} className={styles.image}></Image>
-                </Popover>
-              )}
-              <div>{x.content_with_weight}</div>
-            </Flex>
-          </Card>
-        ))}
+        {isSuccess && chunks.length > 0 ? (
+          chunks?.map((x) => (
+            <Card key={x.chunk_id} title={<ChunkTitle item={x}></ChunkTitle>}>
+              <Flex gap={'middle'}>
+                {x.img_id && (
+                  <Popover
+                    placement="left"
+                    content={
+                      <Image
+                        id={x.img_id}
+                        className={styles.imagePreview}
+                      ></Image>
+                    }
+                  >
+                    <Image id={x.img_id} className={styles.image}></Image>
+                  </Popover>
+                )}
+                <div>{x.content_with_weight}</div>
+              </Flex>
+            </Card>
+          ))
+        ) : isSuccess && chunks.length === 0 ? (
+          <Empty></Empty>
+        ) : null}
       </Flex>
       <Pagination
         {...pagination}

web/src/pages/chat/hooks.ts

@@ -535,7 +535,6 @@ export const useRenameConversation = () => {
   const onConversationRenameOk = useCallback(
     async (name: string) => {
       const ret = await updateConversation({
-        ...conversation,
         conversation_id: conversation.id,
         name,
         is_new: false,

web/src/pages/force-graph/constant.ts

@@ -1,241 +0,0 @@
-const nodes = [
-  {
-    type: '"ORGANIZATION"',
-    description:
-      '"厦门象屿是一家公司，其营业收入和市场占有率在2018年至2022年间有所变化。"',
-    source_id: '0',
-    id: '"厦门象屿"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"2018年是一个时间点，标志着厦门象屿营业收入和市场占有率的记录开始。"',
-    source_id: '0',
-    entity_type: '"EVENT"',
-    id: '"2018"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"2019年是一个时间点，厦门象屿的营业收入和市场占有率在此期间有所变化。"',
-    source_id: '0',
-    entity_type: '"EVENT"',
-    id: '"2019"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"2020年是一个时间点，厦门象屿的营业收入和市场占有率在此期间有所变化。"',
-    source_id: '0',
-    entity_type: '"EVENT"',
-    id: '"2020"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"2021年是一个时间点，厦门象屿的营业收入和市场占有率在此期间有所变化。"',
-    source_id: '0',
-    entity_type: '"EVENT"',
-    id: '"2021"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"2022年是一个时间点，厦门象屿的营业收入和市场占有率在此期间有所变化。"',
-    source_id: '0',
-    entity_type: '"EVENT"',
-    id: '"2022"',
-  },
-  {
-    type: '"ORGANIZATION"',
-    description:
-      '"厦门象屿股份有限公司是一家公司，中文简称为厦门象屿，外文名称为Xiamen Xiangyu Co.,Ltd.，外文名称缩写为Xiangyu，法定代表人为邓启东。"',
-    source_id: '1',
-    id: '"厦门象屿股份有限公司"',
-  },
-  {
-    type: '"PERSON"',
-    description: '"邓启东是厦门象屿股份有限公司的法定代表人。"',
-    source_id: '1',
-    entity_type: '"PERSON"',
-    id: '"邓启东"',
-  },
-  {
-    type: '"GEO"',
-    description: '"厦门是一个地理位置，与厦门象屿股份有限公司相关。"',
-    source_id: '1',
-    entity_type: '"GEO"',
-    id: '"厦门"',
-  },
-  {
-    type: '"PERSON"',
-    description:
-      '"廖杰 is the Board Secretary, responsible for handling board-related matters and communications."',
-    source_id: '2',
-    id: '"廖杰"',
-  },
-  {
-    type: '"PERSON"',
-    description:
-      '"史经洋 is the Securities Affairs Representative, responsible for handling securities-related matters and communications."',
-    source_id: '2',
-    entity_type: '"PERSON"',
-    id: '"史经洋"',
-  },
-  {
-    type: '"GEO"',
-    description:
-      '"A geographic location in Xiamen, specifically in the Free Trade Zone, where the company\'s office is situated."',
-    source_id: '2',
-    entity_type: '"GEO"',
-    id: '"厦门市湖里区自由贸易试验区厦门片区"',
-  },
-  {
-    type: '"GEO"',
-    description:
-      '"The building where the company\'s office is located, situated at Xiangyu Road, Xiamen."',
-    source_id: '2',
-    entity_type: '"GEO"',
-    id: '"象屿集团大厦"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"Refers to the year 2021, used for comparing financial metrics with the year 2022."',
-    source_id: '3',
-    id: '"2021年"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"Refers to the year 2022, used for presenting current financial metrics and comparing them with the year 2021."',
-    source_id: '3',
-    entity_type: '"EVENT"',
-    id: '"2022年"',
-  },
-  {
-    type: '"EVENT"',
-    description:
-      '"Indicates the focus on key financial metrics in the table, such as weighted averages and percentages."',
-    source_id: '3',
-    entity_type: '"EVENT"',
-    id: '"主要财务指标"',
-  },
-].map(({ type, ...x }) => ({ ...x }));
-
-const edges = [
-  {
-    weight: 2.0,
-    description: '"厦门象屿在2018年的营业收入和市场占有率被记录。"',
-    source_id: '0',
-    source: '"厦门象屿"',
-    target: '"2018"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿在2019年的营业收入和市场占有率有所变化。"',
-    source_id: '0',
-    source: '"厦门象屿"',
-    target: '"2019"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿在2020年的营业收入和市场占有率有所变化。"',
-    source_id: '0',
-    source: '"厦门象屿"',
-    target: '"2020"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿在2021年的营业收入和市场占有率有所变化。"',
-    source_id: '0',
-    source: '"厦门象屿"',
-    target: '"2021"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿在2022年的营业收入和市场占有率有所变化。"',
-    source_id: '0',
-    source: '"厦门象屿"',
-    target: '"2022"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿股份有限公司的法定代表人是邓启东。"',
-    source_id: '1',
-    source: '"厦门象屿股份有限公司"',
-    target: '"邓启东"',
-  },
-  {
-    weight: 2.0,
-    description: '"厦门象屿股份有限公司位于厦门。"',
-    source_id: '1',
-    source: '"厦门象屿股份有限公司"',
-    target: '"厦门"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"廖杰\'s office is located in the Xiangyu Group Building, indicating his workplace."',
-    source_id: '2',
-    source: '"廖杰"',
-    target: '"象屿集团大厦"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"廖杰 works in the Xiamen Free Trade Zone, a specific area within Xiamen."',
-    source_id: '2',
-    source: '"廖杰"',
-    target: '"厦门市湖里区自由贸易试验区厦门片区"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"史经洋\'s office is also located in the Xiangyu Group Building, indicating his workplace."',
-    source_id: '2',
-    source: '"史经洋"',
-    target: '"象屿集团大厦"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"史经洋 works in the Xiamen Free Trade Zone, a specific area within Xiamen."',
-    source_id: '2',
-    source: '"史经洋"',
-    target: '"厦门市湖里区自由贸易试验区厦门片区"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"The years 2021 and 2022 are related as they are used for comparing financial metrics, showing changes and adjustments over time."',
-    source_id: '3',
-    source: '"2021年"',
-    target: '"2022年"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"The \'主要财务指标\' is related to the year 2021 as it provides the basis for financial comparisons and adjustments."',
-    source_id: '3',
-    source: '"2021年"',
-    target: '"主要财务指标"',
-  },
-  {
-    weight: 2.0,
-    description:
-      '"The \'主要财务指标\' is related to the year 2022 as it presents the current financial metrics and their changes compared to 2021."',
-    source_id: '3',
-    source: '"2022年"',
-    target: '"主要财务指标"',
-  },
-];
-
-export const graphData = {
-  directed: false,
-  multigraph: false,
-  graph: {},
-  nodes,
-  edges,
-  combos: [],
-};

web/src/pages/force-graph/index.less

@@ -1,4 +0,0 @@
-.container {
-  width: 100%;
-  height: 100%;
-}

web/src/pages/force-graph/index.tsx

@@ -1,3 +0,0 @@
-import InputWithUpload from './input-upload';
-
-export default InputWithUpload;

web/src/pages/force-graph/input-upload.tsx

@@ -1,59 +0,0 @@
-import { Authorization } from '@/constants/authorization';
-import { getAuthorization } from '@/utils/authorization-util';
-import { PlusOutlined } from '@ant-design/icons';
-import type { UploadFile, UploadProps } from 'antd';
-import { Image, Input, Upload } from 'antd';
-import { useState } from 'react';
-
-const InputWithUpload = () => {
-  const [previewOpen, setPreviewOpen] = useState(false);
-  const [previewImage, setPreviewImage] = useState('');
-  const [fileList, setFileList] = useState<UploadFile[]>([]);
-
-  const handleChange: UploadProps['onChange'] = ({ fileList: newFileList }) =>
-    setFileList(newFileList);
-
-  const uploadButton = (
-    <button style={{ border: 0, background: 'none' }} type="button">
-      <PlusOutlined />
-      <div style={{ marginTop: 8 }}>Upload</div>
-    </button>
-  );
-  return (
-    <>
-      <Input placeholder="Basic usage"></Input>
-      <Upload
-        action="/v1/document/upload_and_parse"
-        listType="picture-card"
-        fileList={fileList}
-        onChange={handleChange}
-        multiple
-        headers={{ [Authorization]: getAuthorization() }}
-        data={{ conversation_id: '9e9f7d2453e511efb18efa163e197198' }}
-        method="post"
-      >
-        {fileList.length >= 8 ? null : uploadButton}
-      </Upload>
-      {previewImage && (
-        <Image
-          wrapperStyle={{ display: 'none' }}
-          preview={{
-            visible: previewOpen,
-            onVisibleChange: (visible) => setPreviewOpen(visible),
-            afterOpenChange: (visible) => !visible && setPreviewImage(''),
-          }}
-          src={previewImage}
-        />
-      )}
-    </>
-  );
-};
-
-export default () => {
-  return (
-    <section style={{ height: 500, width: 400 }}>
-      <div style={{ height: 200 }}></div>
-      <InputWithUpload></InputWithUpload>
-    </section>
-  );
-};

web/src/pages/force-graph/next.tsx

@@ -1,37 +0,0 @@
-import { useEffect, useRef } from 'react';
-import { ForceGraph2D } from 'react-force-graph';
-import { graphData } from './constant';
-
-const Next = () => {
-  const graphRef = useRef<ForceGraph2D>();
-
-  useEffect(() => {
-    graphRef.current.d3Force('cluster');
-  }, []);
-
-  return (
-    <div>
-      <ForceGraph2D
-        ref={graphRef}
-        graphData={graphData}
-        nodeAutoColorBy={'type'}
-        nodeLabel={(node) => {
-          return node.id;
-        }}
-        // nodeVal={(node) => {
-        //   return <div>xxx</div>;
-        // }}
-        // nodeVal={(node) => 100 / (node.level + 1)}
-        linkAutoColorBy={'type'}
-        nodeCanvasObjectMode={() => 'after'}
-        nodeCanvasObject={(node, ctx) => {
-          console.info(ctx);
-          return ctx.canvas;
-        }}
-        // nodeVal={'id'}
-      />
-    </div>
-  );
-};
-
-export default Next;

web/src/pages/force-graph/util.ts

@@ -1,55 +0,0 @@
-class KeyGenerator {
-  idx = 0;
-  chars: string[] = [];
-  constructor() {
-    const chars = Array(26)
-      .fill(1)
-      .map((x, idx) => String.fromCharCode(97 + idx)); // 26 char
-    this.chars = chars;
-  }
-  generateKey() {
-    const key = this.chars[this.idx];
-    this.idx++;
-    return key;
-  }
-}
-
-// Classify nodes based on edge relationships
-export class Converter {
-  keyGenerator;
-  dict: Record<string, string> = {}; // key is node id, value is combo
-  constructor() {
-    this.keyGenerator = new KeyGenerator();
-  }
-  buildDict(edges: { source: string; target: string }[]) {
-    edges.forEach((x) => {
-      if (this.dict[x.source] && !this.dict[x.target]) {
-        this.dict[x.target] = this.dict[x.source];
-      } else if (!this.dict[x.source] && this.dict[x.target]) {
-        this.dict[x.source] = this.dict[x.target];
-      } else if (!this.dict[x.source] && !this.dict[x.target]) {
-        this.dict[x.source] = this.dict[x.target] =
-          this.keyGenerator.generateKey();
-      }
-    });
-    return this.dict;
-  }
-  buildNodesAndCombos(nodes: any[], edges: any[]) {
-    this.buildDict(edges);
-    const nextNodes = nodes.map((x) => ({ ...x, combo: this.dict[x.id] }));
-
-    const combos = Object.values(this.dict).reduce<any[]>((pre, cur) => {
-      if (pre.every((x) => x.id !== cur)) {
-        pre.push({
-          id: cur,
-          data: {
-            label: `Combo ${cur}`,
-          },
-        });
-      }
-      return pre;
-    }, []);
-
-    return { nodes: nextNodes, combos };
-  }
-}

web/src/routes.ts

@@ -129,11 +129,6 @@ const routes = [
     component: '@/pages/document-viewer',
     layout: false,
   },
-  {
-    path: 'force',
-    component: '@/pages/force-graph',
-    layout: false,
-  },
   {
     path: '/*',
     component: '@/pages/404',