Fix: search list permission (#9767)

### What problem does this PR solve?

Search list permission.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

.github/ISSUE_TEMPLATE/agent_scenario_request.yml

@@ -0,0 +1,46 @@
+name: "❤️‍🔥ᴬᴳᴱᴺᵀ Agent scenario request"
+description: Propose a agent scenario request for RAGFlow.
+title: "[Agent Scenario Request]: "
+labels: ["❤️‍🔥ᴬᴳᴱᴺᵀ agent scenario"]
+body:
+  - type: checkboxes
+    attributes:
+      label: Self Checks
+      description: "Please check the following in order to be responded in time :)"
+      options:
+        - label: I have searched for existing issues [search for existing issues](https://github.com/infiniflow/ragflow/issues), including closed ones.
+          required: true
+        - label: I confirm that I am using English to submit this report ([Language Policy](https://github.com/infiniflow/ragflow/issues/5910)).
+          required: true
+        - label: Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) ([Language Policy](https://github.com/infiniflow/ragflow/issues/5910)).
+          required: true
+        - label: "Please do not modify this template :) and fill in all the required fields."
+          required: true
+  - type: textarea
+    attributes:
+      label: Is your feature request related to a scenario?
+      description: |
+        A clear and concise description of what the scenario is. Ex. I'm always frustrated when [...]
+      render: Markdown
+    validations:
+      required: false
+  - type: textarea
+    attributes:
+      label: Describe the feature you'd like
+      description: A clear and concise description of what you want to happen.
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Documentation, adoption, use case
+      description: If you can, explain some scenarios how users might use this, situations it would be helpful in. Any API designs, mockups, or diagrams are also helpful.
+      render: Markdown
+    validations:
+      required: false
+  - type: textarea
+    attributes:
+      label: Additional information
+      description: |
+        Add any other context or screenshots about the feature request here.
+    validations:
+      required: false

README.md

@@ -22,7 +22,7 @@
         <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -190,7 +190,7 @@ releases! 🌟
 > All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64.
 > If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system.
 
-   > The command below downloads the `v0.20.1-slim` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.20.1-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1` for the full edition `v0.20.1`.
+   > The command below downloads the `v0.20.4-slim` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.20.4-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4` for the full edition `v0.20.4`.
 
    ```bash
    $ cd ragflow/docker
@@ -203,8 +203,8 @@ releases! 🌟
 
    | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
    |-------------------|-----------------|-----------------------|--------------------------|
-   | v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-   | v0.20.1-slim      | &approx;2       | ❌                   | Stable release            |
+   | v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+   | v0.20.4-slim      | &approx;2       | ❌                   | Stable release            |
    | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
    | nightly-slim      | &approx;2       | ❌                   | _Unstable_ nightly build  |
 

README_id.md

@@ -22,7 +22,7 @@
         <img alt="Lencana Daring" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Rilis%20Terbaru" alt="Rilis Terbaru">
@@ -181,7 +181,7 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
 > Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64.
 > Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image).
 
-> Perintah di bawah ini mengunduh edisi v0.20.1-slim dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.20.1-slim, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server. Misalnya, atur RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1 untuk edisi lengkap v0.20.1.
+> Perintah di bawah ini mengunduh edisi v0.20.4-slim dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.20.4-slim, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server. Misalnya, atur RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4 untuk edisi lengkap v0.20.4.
 
 ```bash
 $ cd ragflow/docker
@@ -194,8 +194,8 @@ $ docker compose -f docker-compose.yml up -d
 
 | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
 | ----------------- | --------------- | --------------------- | ------------------------ |
-| v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-| v0.20.1-slim      | &approx;2       | ❌                    | Stable release           |
+| v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+| v0.20.4-slim      | &approx;2       | ❌                    | Stable release           |
 | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
 | nightly-slim      | &approx;2       | ❌                    | _Unstable_ nightly build |
 

README_ja.md

@@ -22,7 +22,7 @@
         <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -160,7 +160,7 @@
 > 現在、公式に提供されているすべての Docker イメージは x86 アーキテクチャ向けにビルドされており、ARM64 用の Docker イメージは提供されていません。
 > ARM64 アーキテクチャのオペレーティングシステムを使用している場合は、[このドキュメント](https://ragflow.io/docs/dev/build_docker_image)を参照して Docker イメージを自分でビルドしてください。
 
-   > 以下のコマンドは、RAGFlow Docker イメージの v0.20.1-slim エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.20.1-slim とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。例えば、完全版 v0.20.1 をダウンロードするには、RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1 と設定します。
+   > 以下のコマンドは、RAGFlow Docker イメージの v0.20.4-slim エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.20.4-slim とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。例えば、完全版 v0.20.4 をダウンロードするには、RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4 と設定します。
 
    ```bash
    $ cd ragflow/docker
@@ -173,8 +173,8 @@
 
    | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
    | ----------------- | --------------- | --------------------- | ------------------------ |
-   | v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-   | v0.20.1-slim      | &approx;2       | ❌                    | Stable release           |
+   | v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+   | v0.20.4-slim      | &approx;2       | ❌                    | Stable release           |
    | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
    | nightly-slim      | &approx;2       | ❌                     | _Unstable_ nightly build |
 

README_ko.md

@@ -22,7 +22,7 @@
         <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -160,7 +160,7 @@
 > 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다.
 > ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image).
 
-   > 아래 명령어는 RAGFlow Docker 이미지의 v0.20.1-slim 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.20.1-slim과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오. 예를 들어, 전체 버전인 v0.20.1을 다운로드하려면 RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1로 설정합니다.
+   > 아래 명령어는 RAGFlow Docker 이미지의 v0.20.4-slim 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.20.4-slim과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오. 예를 들어, 전체 버전인 v0.20.4을 다운로드하려면 RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4로 설정합니다.
 
    ```bash
    $ cd ragflow/docker
@@ -173,8 +173,8 @@
 
    | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
    | ----------------- | --------------- | --------------------- | ------------------------ |
-   | v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-   | v0.20.1-slim      | &approx;2       | ❌                    | Stable release           |
+   | v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+   | v0.20.4-slim      | &approx;2       | ❌                    | Stable release           |
    | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
    | nightly-slim      | &approx;2       | ❌                     | _Unstable_ nightly build |
 

README_pt_br.md

@@ -22,7 +22,7 @@
         <img alt="Badge Estático" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Última%20Relese" alt="Última Versão">
@@ -180,7 +180,7 @@ Experimente nossa demo em [https://demo.ragflow.io](https://demo.ragflow.io).
 > Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64.
 > Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema.
 
-    > O comando abaixo baixa a edição `v0.20.1-slim` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.20.1-slim`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor. Por exemplo: defina `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1` para a edição completa `v0.20.1`.
+    > O comando abaixo baixa a edição `v0.20.4-slim` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.20.4-slim`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor. Por exemplo: defina `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4` para a edição completa `v0.20.4`.
 
     ```bash
     $ cd ragflow/docker
@@ -193,8 +193,8 @@ Experimente nossa demo em [https://demo.ragflow.io](https://demo.ragflow.io).
 
     | Tag da imagem RAGFlow | Tamanho da imagem (GB) | Possui modelos de incorporação? | Estável?                 |
     | --------------------- | ---------------------- | ------------------------------- | ------------------------ |
-    | v0.20.1               | ~9                     | :heavy_check_mark:              | Lançamento estável       |
-    | v0.20.1-slim          | ~2                     | ❌                              | Lançamento estável       |
+    | v0.20.4               | ~9                     | :heavy_check_mark:              | Lançamento estável       |
+    | v0.20.4-slim          | ~2                     | ❌                              | Lançamento estável       |
     | nightly               | ~9                     | :heavy_check_mark:              | _Instável_ build noturno |
     | nightly-slim          | ~2                     | ❌                               | _Instável_ build noturno |
 

README_tzh.md

@@ -22,7 +22,7 @@
         <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -183,7 +183,7 @@
 > 所有 Docker 映像檔都是為 x86 平台建置的。目前，我們不提供 ARM64 平台的 Docker 映像檔。
 > 如果您使用的是 ARM64 平台，請使用 [這份指南](https://ragflow.io/docs/dev/build_docker_image) 來建置適合您系統的 Docker 映像檔。
 
-   > 執行以下指令會自動下載 RAGFlow slim Docker 映像 `v0.20.1-slim`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.20.1-slim` 的 Docker 映像，請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。例如，你可以透過設定 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1` 來下載 RAGFlow 鏡像的 `v0.20.1` 完整發行版。
+   > 執行以下指令會自動下載 RAGFlow slim Docker 映像 `v0.20.4-slim`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.20.4-slim` 的 Docker 映像，請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。例如，你可以透過設定 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4` 來下載 RAGFlow 鏡像的 `v0.20.4` 完整發行版。
 
    ```bash
    $ cd ragflow/docker
@@ -196,8 +196,8 @@
 
    | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
    | ----------------- | --------------- | --------------------- | ------------------------ |
-   | v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-   | v0.20.1-slim      | &approx;2       | ❌                    | Stable release           |
+   | v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+   | v0.20.4-slim      | &approx;2       | ❌                    | Stable release           |
    | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
    | nightly-slim      | &approx;2       | ❌                     | _Unstable_ nightly build |
 

README_zh.md

@@ -22,7 +22,7 @@
         <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
     </a>
     <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
-        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.1">
+        <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.20.4">
     </a>
     <a href="https://github.com/infiniflow/ragflow/releases/latest">
         <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -183,7 +183,7 @@
 > 请注意，目前官方提供的所有 Docker 镜像均基于 x86 架构构建，并不提供基于 ARM64 的 Docker 镜像。
 > 如果你的操作系统是 ARM64 架构，请参考[这篇文档](https://ragflow.io/docs/dev/build_docker_image)自行构建 Docker 镜像。
 
-   > 运行以下命令会自动下载 RAGFlow slim Docker 镜像 `v0.20.1-slim`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.20.1-slim` 的 Docker 镜像，请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。比如，你可以通过设置 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1` 来下载 RAGFlow 镜像的 `v0.20.1` 完整发行版。
+   > 运行以下命令会自动下载 RAGFlow slim Docker 镜像 `v0.20.4-slim`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.20.4-slim` 的 Docker 镜像，请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。比如，你可以通过设置 `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4` 来下载 RAGFlow 镜像的 `v0.20.4` 完整发行版。
 
    ```bash
    $ cd ragflow/docker
@@ -196,8 +196,8 @@
 
    | RAGFlow image tag | Image size (GB) | Has embedding models? | Stable?                  |
    | ----------------- | --------------- | --------------------- | ------------------------ |
-   | v0.20.1           | &approx;9       | :heavy_check_mark:    | Stable release           |
-   | v0.20.1-slim      | &approx;2       | ❌                    | Stable release           |
+   | v0.20.4           | &approx;9       | :heavy_check_mark:    | Stable release           |
+   | v0.20.4-slim      | &approx;2       | ❌                    | Stable release           |
    | nightly           | &approx;9       | :heavy_check_mark:    | _Unstable_ nightly build |
    | nightly-slim      | &approx;2       | ❌                     | _Unstable_ nightly build |
 

agent/canvas.py

@@ -131,7 +131,16 @@ class Canvas:
 
         self.path = self.dsl["path"]
         self.history = self.dsl["history"]
-        self.globals = self.dsl["globals"]
+        if "globals" in self.dsl:
+            self.globals = self.dsl["globals"]
+        else:
+            self.globals = {
+            "sys.query": "",
+            "sys.user_id": "",
+            "sys.conversation_turns": 0,
+            "sys.files": []
+        }
+            
         self.retrieval = self.dsl["retrieval"]
         self.memory = self.dsl.get("memory", [])
 
@@ -417,7 +426,7 @@ class Canvas:
         convs = []
         if window_size <= 0:
             return convs
-        for role, obj in self.history[window_size * -1:]:
+        for role, obj in self.history[window_size * -2:]:
             if isinstance(obj, dict):
                 convs.append({"role": role, "content": obj.get("content", "")})
             else:
@@ -460,6 +469,9 @@ class Canvas:
     def get_prologue(self):
         return self.components["begin"]["obj"]._param.prologue
 
+    def get_mode(self):
+        return self.components["begin"]["obj"]._param.mode
+
     def set_global_param(self, **kwargs):
         self.globals.update(kwargs)
 
@@ -484,7 +496,7 @@ class Canvas:
             threads.append(exe.submit(FileService.parse, file["name"], FileService.get_blob(file["created_by"], file["id"]), True, file["created_by"]))
         return [th.result() for th in threads]
 
-    def tool_use_callback(self, agent_id: str, func_name: str, params: dict, result: Any):
+    def tool_use_callback(self, agent_id: str, func_name: str, params: dict, result: Any, elapsed_time=None):
         agent_ids = agent_id.split("-->")
         agent_name = self.get_component_name(agent_ids[0])
         path = agent_name if len(agent_ids) < 2 else agent_name+"-->"+"-->".join(agent_ids[1:])
@@ -493,16 +505,16 @@ class Canvas:
             if bin:
                 obj = json.loads(bin.encode("utf-8"))
                 if obj[-1]["component_id"] == agent_ids[0]:
-                    obj[-1]["trace"].append({"path": path, "tool_name": func_name, "arguments": params, "result": result})
+                    obj[-1]["trace"].append({"path": path, "tool_name": func_name, "arguments": params, "result": result, "elapsed_time": elapsed_time})
                 else:
                     obj.append({
                     "component_id": agent_ids[0],
-                    "trace": [{"path": path, "tool_name": func_name, "arguments": params, "result": result}]
+                    "trace": [{"path": path, "tool_name": func_name, "arguments": params, "result": result, "elapsed_time": elapsed_time}]
                 })
             else:
                 obj = [{
                     "component_id": agent_ids[0],
-                    "trace": [{"path": path, "tool_name": func_name, "arguments": params, "result": result}]
+                    "trace": [{"path": path, "tool_name": func_name, "arguments": params, "result": result, "elapsed_time": elapsed_time}]
                 }]
             REDIS_CONN.set_obj(f"{self.task_id}-{self.message_id}-logs", obj, 60*10)
         except Exception as e:

agent/component/agent_with_tools.py

@@ -22,9 +22,10 @@ from functools import partial
 from typing import Any
 
 import json_repair
-
+from timeit import default_timer as timer
 from agent.tools.base import LLMToolPluginCallSession, ToolParamBase, ToolBase, ToolMeta
-from api.db.services.llm_service import LLMBundle, TenantLLMService
+from api.db.services.llm_service import LLMBundle
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.db.services.mcp_server_service import MCPServerService
 from api.utils.api_utils import timeout
 from rag.prompts import message_fit_in
@@ -214,8 +215,9 @@ class Agent(LLM, ToolBase):
         hist = deepcopy(history)
         last_calling = ""
         if len(hist) > 3:
+            st = timer()
             user_request = full_question(messages=history, chat_mdl=self.chat_mdl)
-            self.callback("Multi-turn conversation optimization", {}, user_request)
+            self.callback("Multi-turn conversation optimization", {}, user_request, elapsed_time=timer()-st)
         else:
             user_request = history[-1]["content"]
 
@@ -243,7 +245,7 @@ class Agent(LLM, ToolBase):
 
         def complete():
             nonlocal hist
-            need2cite = self._canvas.get_reference()["chunks"] and self._id.find("-->") < 0
+            need2cite = self._param.cite and self._canvas.get_reference()["chunks"] and self._id.find("-->") < 0
             cited = False
             if hist[0]["role"] == "system" and need2cite:
                 if len(hist) < 7:
@@ -262,12 +264,13 @@ class Agent(LLM, ToolBase):
             if not need2cite or cited:
                 return
 
+            st = timer()
             txt = ""
             for delta_ans in self._gen_citations(entire_txt):
                 yield delta_ans, 0
                 txt += delta_ans
 
-            self.callback("gen_citations", {}, txt)
+            self.callback("gen_citations", {}, txt, elapsed_time=timer()-st)
 
         def append_user_content(hist, content):
             if hist[-1]["role"] == "user":
@@ -275,8 +278,9 @@ class Agent(LLM, ToolBase):
             else:
                 hist.append({"role": "user", "content": content})
 
+        st = timer()
         task_desc = analyze_task(self.chat_mdl, prompt, user_request, tool_metas)
-        self.callback("analyze_task", {}, task_desc)
+        self.callback("analyze_task", {}, task_desc, elapsed_time=timer()-st)
         for _ in range(self._param.max_rounds + 1):
             response, tk = next_step(self.chat_mdl, hist, tool_metas, task_desc)
             # self.callback("next_step", {}, str(response)[:256]+"...")
@@ -302,9 +306,10 @@ class Agent(LLM, ToolBase):
 
                         thr.append(executor.submit(use_tool, name, args))
 
+                    st = timer()
                     reflection = reflect(self.chat_mdl, hist, [th.result() for th in thr])
                     append_user_content(hist, reflection)
-                    self.callback("reflection", {}, str(reflection))
+                    self.callback("reflection", {}, str(reflection), elapsed_time=timer()-st)
 
             except Exception as e:
                 logging.exception(msg=f"Wrong JSON argument format in LLM ReAct response: {e}")

agent/component/base.py

@@ -36,7 +36,7 @@ _IS_RAW_CONF = "_is_raw_conf"
 
 class ComponentParamBase(ABC):
     def __init__(self):
-        self.message_history_window_size = 22
+        self.message_history_window_size = 13
         self.inputs = {}
         self.outputs = {}
         self.description = ""
@@ -479,7 +479,7 @@ class ComponentBase(ABC):
 
     def get_input_elements_from_text(self, txt: str) -> dict[str, dict[str, str]]:
         res = {}
-        for r in re.finditer(self.variable_ref_patt, txt, flags=re.IGNORECASE):
+        for r in re.finditer(self.variable_ref_patt, txt, flags=re.IGNORECASE|re.DOTALL):
             exp = r.group(1)
             cpn_id, var_nm = exp.split("@") if exp.find("@")>0 else ("", exp)
             res[exp] = {
@@ -529,8 +529,12 @@ class ComponentBase(ABC):
     @staticmethod
     def string_format(content: str, kv: dict[str, str]) -> str:
         for n, v in kv.items():
+            def repl(_match, val=v):
+                return str(val) if val is not None else ""
             content = re.sub(
-                r"\{%s\}" % re.escape(n), v, content
+                r"\{%s\}" % re.escape(n),
+                repl,
+                content
             )
         return content
 

agent/component/llm.py

@@ -18,13 +18,11 @@ import logging
 import os
 import re
 from typing import Any, Generator
-
 import json_repair
-from copy import deepcopy
 from functools import partial
-
 from api.db import LLMType
-from api.db.services.llm_service import LLMBundle, TenantLLMService
+from api.db.services.llm_service import LLMBundle
+from api.db.services.tenant_llm_service import TenantLLMService
 from agent.component.base import ComponentBase, ComponentParamBase
 from api.utils.api_utils import timeout
 from rag.prompts import message_fit_in, citation_prompt
@@ -129,7 +127,7 @@ class LLM(ComponentBase):
 
         args = {}
         vars = self.get_input_elements() if not self._param.debug_inputs else self._param.debug_inputs
-        prompt = self._param.sys_prompt
+        sys_prompt = self._param.sys_prompt
         for k, o in vars.items():
             args[k] = o["value"]
             if not isinstance(args[k], str):
@@ -140,14 +138,18 @@ class LLM(ComponentBase):
             self.set_input_value(k, args[k])
 
         msg = self._canvas.get_history(self._param.message_history_window_size)[:-1]
-        msg.extend(deepcopy(self._param.prompts))
-        prompt = self.string_format(prompt, args)
+        for p in self._param.prompts:
+            if msg and msg[-1]["role"] == p["role"]:
+                continue
+            msg.append(p)
+
+        sys_prompt = self.string_format(sys_prompt, args)
         for m in msg:
             m["content"] = self.string_format(m["content"], args)
-        if self._canvas.get_reference()["chunks"]:
-            prompt += citation_prompt()
+        if self._param.cite and self._canvas.get_reference()["chunks"]:
+            sys_prompt += citation_prompt()
 
-        return prompt, msg
+        return sys_prompt, msg
 
     def _generate(self, msg:list[dict], **kwargs) -> str:
         if not self.imgs:

agent/component/message.py

@@ -54,6 +54,8 @@ class Message(ComponentBase):
             if k in kwargs:
                 continue
             v = v["value"]
+            if not v:
+                v = ""
             ans = ""
             if isinstance(v, partial):
                 for t in v():
@@ -94,6 +96,8 @@ class Message(ComponentBase):
                 continue
 
             v = self._canvas.get_variable_value(exp)
+            if not v:
+                v = ""
             if isinstance(v, partial):
                 cnt = ""
                 for t in v():

agent/templates/knowledge_base_report.json

@@ -0,0 +1,327 @@
+{
+    "id": 20,
+    "title": "Report Agent Using Knowledge Base",
+    "description": "A report generation assistant using local knowledge base, with advanced capabilities in task planning, reasoning, and reflective analysis. Recommended for academic research paper Q&A",
+    "canvas_type": "Agent",
+    "dsl": {
+        "components": {
+            "Agent:NewPumasLick": {
+                "downstream": [
+                    "Message:OrangeYearsShine"
+                ],
+                "obj": {
+                    "component_name": "Agent",
+                    "params": {
+                        "delay_after_error": 1,
+                        "description": "",
+                        "exception_comment": "",
+                        "exception_default_value": "",
+                        "exception_goto": [],
+                        "exception_method": null,
+                        "frequencyPenaltyEnabled": false,
+                        "frequency_penalty": 0.5,
+                        "llm_id": "qwen3-235b-a22b-instruct-2507@Tongyi-Qianwen",
+                        "maxTokensEnabled": true,
+                        "max_retries": 3,
+                        "max_rounds": 3,
+                        "max_tokens": 128000,
+                        "mcp": [],
+                        "message_history_window_size": 12,
+                        "outputs": {
+                            "content": {
+                                "type": "string",
+                                "value": ""
+                            }
+                        },
+                        "parameter": "Precise",
+                        "presencePenaltyEnabled": false,
+                        "presence_penalty": 0.5,
+                        "prompts": [
+                            {
+                                "content": "# User Query\n {sys.query}",
+                                "role": "user"
+                            }
+                        ],
+                        "sys_prompt": "## Role & Task\nYou are a **\u201cKnowledge Base Retrieval Q\\&A Agent\u201d** whose goal is to break down the user\u2019s question into retrievable subtasks, and then produce a multi-source-verified, structured, and actionable research report using the internal knowledge base.\n## Execution Framework (Detailed Steps & Key Points)\n1. **Assessment & Decomposition**\n   * Actions:\n     * Automatically extract: main topic, subtopics, entities (people/organizations/products/technologies), time window, geographic/business scope.\n     * Output as a list: N facts/data points that must be collected (*N* ranges from 5\u201320 depending on question complexity).\n2. **Query Type Determination (Rule-Based)**\n   * Example rules:\n     * If the question involves a single issue but requests \u201cmethod comparison/multiple explanations\u201d \u2192 use **depth-first**.\n     * If the question can naturally be split into \u22653 independent sub-questions \u2192 use **breadth-first**.\n     * If the question can be answered by a single fact/specification/definition \u2192 use **simple query**.\n3. **Research Plan Formulation**\n   * Depth-first: define 3\u20135 perspectives (methodology/stakeholders/time dimension/technical route, etc.), assign search keywords, target document types, and output format for each perspective.\n   * Breadth-first: list subtasks, prioritize them, and assign search terms.\n   * Simple query: directly provide the search sentence and required fields.\n4. **Retrieval Execution**\n   * After retrieval: perform coverage check (does it contain the key facts?) and quality check (source diversity, authority, latest update time).\n   * If standards are not met, automatically loop: rewrite queries (synonyms/cross-domain terms) and retry \u22643 times, or flag as requiring external search.\n5. **Integration & Reasoning**\n   * Build the answer using a **fact\u2013evidence\u2013reasoning** chain. For each conclusion, attach 1\u20132 strongest pieces of evidence.\n---\n## Quality Gate Checklist (Verify at Each Stage)\n* **Stage 1 (Decomposition)**:\n  * [ ] Key concepts and expected outputs identified\n  * [ ] Required facts/data points listed\n* **Stage 2 (Retrieval)**:\n  * [ ] Meets quality standards (see above)\n  * [ ] If not met: execute query iteration\n* **Stage 3 (Generation)**:\n  * [ ] Each conclusion has at least one direct evidence source\n  * [ ] State assumptions/uncertainties\n  * [ ] Provide next-step suggestions or experiment/retrieval plans\n  * [ ] Final length and depth match user expectations (comply with word count/format if specified)\n---\n## Core Principles\n1. **Strict reliance on the knowledge base**: answers must be **fully bounded** by the content retrieved from the knowledge base.\n2. **No fabrication**: do not generate, infer, or create information that is not explicitly present in the knowledge base.\n3. **Accuracy first**: prefer incompleteness over inaccurate content.\n4. **Output format**:\n   * Hierarchically clear modular structure\n   * Logical grouping according to the MECE principle\n   * Professionally presented formatting\n   * Step-by-step cognitive guidance\n   * Reasonable use of headings and dividers for clarity\n   * *Italicize* key parameters\n   * **Bold** critical information\n5. **LaTeX formula requirements**:\n   * Inline formulas: start and end with `$`\n   * Block formulas: start and end with `$$`, each `$$` on its own line\n   * Block formula content must comply with LaTeX math syntax\n   * Verify formula correctness\n---\n## Additional Notes (Interaction & Failure Strategy)\n* If the knowledge base does not cover critical facts: explicitly inform the user (with sample wording)\n* For time-sensitive issues: enforce time filtering in the search request, and indicate the latest retrieval date in the answer.\n* Language requirement: answer in the user\u2019s preferred language\n",
+                        "temperature": "0.1",
+                        "temperatureEnabled": true,
+                        "tools": [
+                            {
+                                "component_name": "Retrieval",
+                                "name": "Retrieval",
+                                "params": {
+                                    "cross_languages": [],
+                                    "description": "",
+                                    "empty_response": "",
+                                    "kb_ids": [],
+                                    "keywords_similarity_weight": 0.7,
+                                    "outputs": {
+                                        "formalized_content": {
+                                            "type": "string",
+                                            "value": ""
+                                        }
+                                    },
+                                    "rerank_id": "",
+                                    "similarity_threshold": 0.2,
+                                    "top_k": 1024,
+                                    "top_n": 8,
+                                    "use_kg": false
+                                }
+                            }
+                        ],
+                        "topPEnabled": false,
+                        "top_p": 0.75,
+                        "user_prompt": "",
+                        "visual_files_var": ""
+                    }
+                },
+                "upstream": [
+                    "begin"
+                ]
+            },
+            "Message:OrangeYearsShine": {
+                "downstream": [],
+                "obj": {
+                    "component_name": "Message",
+                    "params": {
+                        "content": [
+                            "{Agent:NewPumasLick@content}"
+                        ]
+                    }
+                },
+                "upstream": [
+                    "Agent:NewPumasLick"
+                ]
+            },
+            "begin": {
+                "downstream": [
+                    "Agent:NewPumasLick"
+                ],
+                "obj": {
+                    "component_name": "Begin",
+                    "params": {
+                        "enablePrologue": true,
+                        "inputs": {},
+                        "mode": "conversational",
+                        "prologue": "\u4f60\u597d\uff01 \u6211\u662f\u4f60\u7684\u52a9\u7406\uff0c\u6709\u4ec0\u4e48\u53ef\u4ee5\u5e2e\u5230\u4f60\u7684\u5417\uff1f"
+                    }
+                },
+                "upstream": []
+            }
+        },
+        "globals": {
+            "sys.conversation_turns": 0,
+            "sys.files": [],
+            "sys.query": "",
+            "sys.user_id": ""
+        },
+        "graph": {
+            "edges": [
+                {
+                    "data": {
+                        "isHovered": false
+                    },
+                    "id": "xy-edge__beginstart-Agent:NewPumasLickend",
+                    "source": "begin",
+                    "sourceHandle": "start",
+                    "target": "Agent:NewPumasLick",
+                    "targetHandle": "end"
+                },
+                {
+                    "data": {
+                        "isHovered": false
+                    },
+                    "id": "xy-edge__Agent:NewPumasLickstart-Message:OrangeYearsShineend",
+                    "markerEnd": "logo",
+                    "source": "Agent:NewPumasLick",
+                    "sourceHandle": "start",
+                    "style": {
+                        "stroke": "rgba(91, 93, 106, 1)",
+                        "strokeWidth": 1
+                    },
+                    "target": "Message:OrangeYearsShine",
+                    "targetHandle": "end",
+                    "type": "buttonEdge",
+                    "zIndex": 1001
+                },
+                {
+                    "data": {
+                        "isHovered": false
+                    },
+                    "id": "xy-edge__Agent:NewPumasLicktool-Tool:AllBirdsNailend",
+                    "selected": false,
+                    "source": "Agent:NewPumasLick",
+                    "sourceHandle": "tool",
+                    "target": "Tool:AllBirdsNail",
+                    "targetHandle": "end"
+                }
+            ],
+            "nodes": [
+                {
+                    "data": {
+                        "form": {
+                            "enablePrologue": true,
+                            "inputs": {},
+                            "mode": "conversational",
+                            "prologue": "\u4f60\u597d\uff01 \u6211\u662f\u4f60\u7684\u52a9\u7406\uff0c\u6709\u4ec0\u4e48\u53ef\u4ee5\u5e2e\u5230\u4f60\u7684\u5417\uff1f"
+                        },
+                        "label": "Begin",
+                        "name": "begin"
+                    },
+                    "dragging": false,
+                    "id": "begin",
+                    "measured": {
+                        "height": 48,
+                        "width": 200
+                    },
+                    "position": {
+                        "x": -9.569875358221438,
+                        "y": 205.84018385864917
+                    },
+                    "selected": false,
+                    "sourcePosition": "left",
+                    "targetPosition": "right",
+                    "type": "beginNode"
+                },
+                {
+                    "data": {
+                        "form": {
+                            "content": [
+                                "{Agent:NewPumasLick@content}"
+                            ]
+                        },
+                        "label": "Message",
+                        "name": "Response"
+                    },
+                    "dragging": false,
+                    "id": "Message:OrangeYearsShine",
+                    "measured": {
+                        "height": 56,
+                        "width": 200
+                    },
+                    "position": {
+                        "x": 734.4061285881053,
+                        "y": 199.9706031723009
+                    },
+                    "selected": false,
+                    "sourcePosition": "right",
+                    "targetPosition": "left",
+                    "type": "messageNode"
+                },
+                {
+                    "data": {
+                        "form": {
+                            "delay_after_error": 1,
+                            "description": "",
+                            "exception_comment": "",
+                            "exception_default_value": "",
+                            "exception_goto": [],
+                            "exception_method": null,
+                            "frequencyPenaltyEnabled": false,
+                            "frequency_penalty": 0.5,
+                            "llm_id": "qwen3-235b-a22b-instruct-2507@Tongyi-Qianwen",
+                            "maxTokensEnabled": true,
+                            "max_retries": 3,
+                            "max_rounds": 3,
+                            "max_tokens": 128000,
+                            "mcp": [],
+                            "message_history_window_size": 12,
+                            "outputs": {
+                                "content": {
+                                    "type": "string",
+                                    "value": ""
+                                }
+                            },
+                            "parameter": "Precise",
+                            "presencePenaltyEnabled": false,
+                            "presence_penalty": 0.5,
+                            "prompts": [
+                                {
+                                    "content": "# User Query\n {sys.query}",
+                                    "role": "user"
+                                }
+                            ],
+                            "sys_prompt": "## Role & Task\nYou are a **\u201cKnowledge Base Retrieval Q\\&A Agent\u201d** whose goal is to break down the user\u2019s question into retrievable subtasks, and then produce a multi-source-verified, structured, and actionable research report using the internal knowledge base.\n## Execution Framework (Detailed Steps & Key Points)\n1. **Assessment & Decomposition**\n   * Actions:\n     * Automatically extract: main topic, subtopics, entities (people/organizations/products/technologies), time window, geographic/business scope.\n     * Output as a list: N facts/data points that must be collected (*N* ranges from 5\u201320 depending on question complexity).\n2. **Query Type Determination (Rule-Based)**\n   * Example rules:\n     * If the question involves a single issue but requests \u201cmethod comparison/multiple explanations\u201d \u2192 use **depth-first**.\n     * If the question can naturally be split into \u22653 independent sub-questions \u2192 use **breadth-first**.\n     * If the question can be answered by a single fact/specification/definition \u2192 use **simple query**.\n3. **Research Plan Formulation**\n   * Depth-first: define 3\u20135 perspectives (methodology/stakeholders/time dimension/technical route, etc.), assign search keywords, target document types, and output format for each perspective.\n   * Breadth-first: list subtasks, prioritize them, and assign search terms.\n   * Simple query: directly provide the search sentence and required fields.\n4. **Retrieval Execution**\n   * After retrieval: perform coverage check (does it contain the key facts?) and quality check (source diversity, authority, latest update time).\n   * If standards are not met, automatically loop: rewrite queries (synonyms/cross-domain terms) and retry \u22643 times, or flag as requiring external search.\n5. **Integration & Reasoning**\n   * Build the answer using a **fact\u2013evidence\u2013reasoning** chain. For each conclusion, attach 1\u20132 strongest pieces of evidence.\n---\n## Quality Gate Checklist (Verify at Each Stage)\n* **Stage 1 (Decomposition)**:\n  * [ ] Key concepts and expected outputs identified\n  * [ ] Required facts/data points listed\n* **Stage 2 (Retrieval)**:\n  * [ ] Meets quality standards (see above)\n  * [ ] If not met: execute query iteration\n* **Stage 3 (Generation)**:\n  * [ ] Each conclusion has at least one direct evidence source\n  * [ ] State assumptions/uncertainties\n  * [ ] Provide next-step suggestions or experiment/retrieval plans\n  * [ ] Final length and depth match user expectations (comply with word count/format if specified)\n---\n## Core Principles\n1. **Strict reliance on the knowledge base**: answers must be **fully bounded** by the content retrieved from the knowledge base.\n2. **No fabrication**: do not generate, infer, or create information that is not explicitly present in the knowledge base.\n3. **Accuracy first**: prefer incompleteness over inaccurate content.\n4. **Output format**:\n   * Hierarchically clear modular structure\n   * Logical grouping according to the MECE principle\n   * Professionally presented formatting\n   * Step-by-step cognitive guidance\n   * Reasonable use of headings and dividers for clarity\n   * *Italicize* key parameters\n   * **Bold** critical information\n5. **LaTeX formula requirements**:\n   * Inline formulas: start and end with `$`\n   * Block formulas: start and end with `$$`, each `$$` on its own line\n   * Block formula content must comply with LaTeX math syntax\n   * Verify formula correctness\n---\n## Additional Notes (Interaction & Failure Strategy)\n* If the knowledge base does not cover critical facts: explicitly inform the user (with sample wording)\n* For time-sensitive issues: enforce time filtering in the search request, and indicate the latest retrieval date in the answer.\n* Language requirement: answer in the user\u2019s preferred language\n",
+                            "temperature": "0.1",
+                            "temperatureEnabled": true,
+                            "tools": [
+                                {
+                                    "component_name": "Retrieval",
+                                    "name": "Retrieval",
+                                    "params": {
+                                        "cross_languages": [],
+                                        "description": "",
+                                        "empty_response": "",
+                                        "kb_ids": [],
+                                        "keywords_similarity_weight": 0.7,
+                                        "outputs": {
+                                            "formalized_content": {
+                                                "type": "string",
+                                                "value": ""
+                                            }
+                                        },
+                                        "rerank_id": "",
+                                        "similarity_threshold": 0.2,
+                                        "top_k": 1024,
+                                        "top_n": 8,
+                                        "use_kg": false
+                                    }
+                                }
+                            ],
+                            "topPEnabled": false,
+                            "top_p": 0.75,
+                            "user_prompt": "",
+                            "visual_files_var": ""
+                        },
+                        "label": "Agent",
+                        "name": "Knowledge Base Agent"
+                    },
+                    "dragging": false,
+                    "id": "Agent:NewPumasLick",
+                    "measured": {
+                        "height": 84,
+                        "width": 200
+                    },
+                    "position": {
+                        "x": 347.00048227952215,
+                        "y": 186.49109364794631
+                    },
+                    "selected": false,
+                    "sourcePosition": "right",
+                    "targetPosition": "left",
+                    "type": "agentNode"
+                },
+                {
+                    "data": {
+                        "form": {
+                            "description": "This is an agent for a specific task.",
+                            "user_prompt": "This is the order you need to send to the agent."
+                        },
+                        "label": "Tool",
+                        "name": "flow.tool_10"
+                    },
+                    "dragging": false,
+                    "id": "Tool:AllBirdsNail",
+                    "measured": {
+                        "height": 48,
+                        "width": 200
+                    },
+                    "position": {
+                        "x": 220.24819746977118,
+                        "y": 403.31576836482583
+                    },
+                    "selected": false,
+                    "sourcePosition": "right",
+                    "targetPosition": "left",
+                    "type": "toolNode"
+                }
+            ]
+        },
+        "history": [],
+        "memory": [],
+        "messages": [],
+        "path": [],
+        "retrieval": []
+    },
+    "avatar": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAH0klEQVR4nO2ZC1BU1wGG/3uRp/IygG+DGK0GOjE1U6cxI4tT03Y0E+kENbaJbKpj60wzgNMwnTjuEtu0miGasY+0krI202kMVEnVxtoOLG00oVa0LajVBDcSEI0REFBgkZv/3GWXfdzdvctuHs7kmzmec9//d+45914XCXc4Xwjk1+59VJGGF7C5QAFSWBvgyWmWLl7IKiny6QNL173B5YjB84bOyrpKA4B1DLySdQpLKAiZGtZ7a/KMVoQJz6UfEZyhTWwaEBmssiLvCueu6BJg8EwFqGTTAC+uvNWC9w82sRWcux/JwaSHstjywcogRt4RG0KExwWG4QsVYCebKSwe3L5lR9OOWjyzfg2WL/0a1/jncO3b2FHxGnKeWYqo+Giu8UEMrWJKWBACPMY/DG+63txhvnKshUu+DF2/hayMDFRsL+VScDb++AVc6OjAuInxXPJl2tfnIikrzUyJMi7qQmLRhOEr2fOFbX/7P6STF7BqoWevfdij4NWGQfx+57OYO2sG1wSnsek8Nm15EU8sikF6ouelXz9ph7JwDqYt+5IIZaGEkauDIrH4wPBmhjexCSEws+VdVG1M4NIoj+2xYzBuJtavWcEl/VS8dggx/ZdQvcGzQwp+cxOXsu5RBQQMVkYJM4LA/Txh+ELFMWFVPARS5kFiabZdx8Olh7l17BzdvhzZmROhdJ3j6D/nIyBgOCMlLAgA9xmF4TMV4BSbrgnrLiBl5rOsRCRRbDUsBzQFiJjY91PCBj9w+yiP1lXWsTLAjc9YQGB9I8+Yx1oTiUWFvW9QgDo2PdASaDp/EQ8/sRnhcPTVcuTMncXwQQVESL9DidscaPW+QEtAICRu9PSxFTpJiePV8AI9AsTvXZBY/Pa+wJ9ApNApIILm8S5Y4QXXQwhYFH6csemDP4G3G5v579i5d04mknknQhDYS4HCrCVr/mC3D305KnbCEpvVIia5Onw6WaWw+KAl0Np+FUXbdiMcyoqfUoeRHoFrJ1uRtnBG1/9Mf/3LtElp+VwF2wcd7woJib1vUPwMH4GWQCQJJtBa/V9cPmFD8uQUpMdNGDhY8bNYrobh8acHu270/l0ImJWRt64Wn6WACN9z5gq2lXwPW8pfweT0icP/fH23vO9QLYq3/QKyLBmFQI3CUcT9NdESEEPItKsSN3r7MBaSJoxHWZERM6ZmMLy2gDP8/pd/og418dTL37hFSUpMUC5f+UiWZcnY9s5+ixCwUiCXx2iiJdDNx6f4pgkH8Q3lbxK7h8+enoHha1cRNdMp8axiHxo6+/5bVdk8DSROYIW1X7QEIom3wHD3gEf4vu1bVYEJZeWQ0zJQvmcfyiv2QZak6raG/QWfK4Ez9mTc5v8xPMJfuojoxXmIX/9DOMe+FCWbcHu4BJJ0YEwCx0824bFNW9HesB+CqYu+jepfPYcHF+aoPXS8sQl/+vU2bgmOU2C+qRc9/YrrPPbGBtzavd0nvCxLxui4pJrBm911PFwak4CYA80cj+JCAiGUzYkmxrSY4N2c3GLi6UEIFL/wRxxqkhmHnTEpDQcrfq6ea+hcE8bNy3GFzyq4H22HW1Kd4WMSkg1jmsSRpKj0Rzhy4gNUv/y8Gjrv8SJK3OWScA+fMn/ysVPPvTmeh6nh1TcxBUJ+jEaKYr7N36x7h+Edj0pB6+WrLokn87+BrTt/p4ZPzZ6MM7/8R2//h33vOcNzdwgBMwVMbGvySQmo4a0NqOZccU7YmGXLEfPQUlUid/XT6B8YdIU/99vjsPcOdEhDsfOd4QVCwKB8yp8SWuG1njbTl83DpMWz1PCKAswuWPDI0e8WebyAJBbxNdrF7cls+hBpAb3h3XtehL/3+4u7D35rQwpP4YFTwMJ91rHpQyQFQgmf9sAMNL9Ur4afv/FBjIuPVj+n4YVTwMD96tj0IVICoYYXv/q1VJ1Sl8UveQyaRwErvOB6B5SwKhqP00gI6A0vhsycJ7/KIzxhyHqGN0ADbnNAAYOicRfCFdAb/p50Gbfuc/wy5w1D5lOghk0fuG0USlgVr7sQjoDe8C8WxKGKPy2KjzlvAQb02/sCbh+FApngX1QUtyeSuwDi0hxFByV7L+LIf3r5kvpp4PBr07Hqvn71Y85bgOG6WS2ggA1+4D6eUKKQApVsqngI6KSkqh9HzsoM/3zg8Oz5VQ9E8wjf30YFDGdkeAsCwH18oYRZGXk7C4HuYxcwe6rjQsFovzaEvoFxqNkTOPzMjGikJso8wsF77XYkLx6dAwxWxvBmBIH7aUMJi8J3w0DnTVz7dyvX6KPzVBt+kL8cmzesRq9ps2Z48bRJmOIapS7E4zM2lXNt5CcU6ID7+ocSZkqY2NRN6ysnsHbJEpR8ZwV6t5Yg+iuLELf2KVd48VwXQf3BQGUMb4ZOuH9gKFEIYJfiNrEDcXZHHV4q3YRv5i7ikgM94RlETNgihrcgBHhccCiRCf7VhBK5rAPyr9I/Y/WKPEyfksH/9NjQ2dODhsYzwcLXsypkeBtCRGLRDUUMAMyKHxEx4dtrzyP97nQMygripiQiKi4aSbPvQmKW7+OXF69ntYvBa1iPCYklZEZECsGm4ja0Ops7EJsaj4SprlU+8IJiqIjAFga3Ikx4vvAYkTGALxyWFArlsnbBC9Sz6mI5zWKNRGh3JJY7mjte4GOz+r4tkRbxQQAAAABJRU5ErkJggg=="
+}

agent/templates/sql_assistant.json

@@ -206,7 +206,7 @@
                             "enablePrologue": true,
                             "inputs": {},
                             "mode": "conversational",
-                            "prologue": "Hi! I'm your SQL assistant, what can I do for you?"
+                            "prologue": "Hi! I'm your SQL assistant. What can I do for you?"
                         }
                     },
                     "upstream": []
@@ -319,7 +319,7 @@
                                 "enablePrologue": true,
                                 "inputs": {},
                                 "mode": "conversational",
-                                "prologue": "Hi! I'm your SQL assistant, what can I do for you?"
+                                "prologue": "Hi! I'm your SQL assistant. What can I do for you?"
                             },
                             "label": "Begin",
                             "name": "begin"

agent/tools/base.py

@@ -24,6 +24,7 @@ from api.utils import hash_str2int
 from rag.llm.chat_model import ToolCallSession
 from rag.prompts.prompts import kb_prompt
 from rag.utils.mcp_tool_call_conn import MCPToolCallSession
+from timeit import default_timer as timer
 
 
 class ToolParameter(TypedDict):
@@ -49,12 +50,13 @@ class LLMToolPluginCallSession(ToolCallSession):
 
     def tool_call(self, name: str, arguments: dict[str, Any]) -> Any:
         assert name in self.tools_map, f"LLM tool {name} does not exist"
+        st = timer()
         if isinstance(self.tools_map[name], MCPToolCallSession):
             resp = self.tools_map[name].tool_call(name, arguments, 60)
         else:
             resp = self.tools_map[name].invoke(**arguments)
 
-        self.callback(name, arguments, resp)
+        self.callback(name, arguments, resp, elapsed_time=timer()-st)
         return resp
 
     def get_tool_obj(self, name):

agent/tools/code_exec.py

@@ -67,11 +67,19 @@ class CodeExecParam(ToolParamBase):
             "description": """
 This tool has a sandbox that can execute code written in 'Python'/'Javascript'. It recieves a piece of code and return a Json string.
 Here's a code example for Python(`main` function MUST be included):
-def main(arg1: str, arg2: str) -> dict:
+def main() -> dict:
+    \"\"\"
+    Generate Fibonacci numbers within 100.
+    \"\"\"
+    def fibonacci_recursive(n):
+        if n <= 1:
+            return n
+        else:
+            return fibonacci_recursive(n-1) + fibonacci_recursive(n-2)
     return {
-        "result": arg1 + arg2,
+        "result": fibonacci_recursive(100),
     }
-    
+
 Here's a code example for Javascript(`main` function MUST be included and exported):
 const axios = require('axios');
 async function main(args) {
@@ -148,7 +156,7 @@ class CodeExec(ToolBase, ABC):
             self.set_output("_ERROR", "construct code request error: " + str(e))
 
         try:
-            resp = requests.post(url=f"http://{settings.SANDBOX_HOST}:9385/run", json=code_req, timeout=10)
+            resp = requests.post(url=f"http://{settings.SANDBOX_HOST}:9385/run", json=code_req, timeout=os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60))
             logging.info(f"http://{settings.SANDBOX_HOST}:9385/run", code_req, resp.status_code)
             if resp.status_code != 200:
                 resp.raise_for_status()

agent/tools/exesql.py

@@ -79,6 +79,17 @@ class ExeSQL(ToolBase, ABC):
 
     @timeout(os.environ.get("COMPONENT_EXEC_TIMEOUT", 60))
     def _invoke(self, **kwargs):
+
+        def convert_decimals(obj):
+            from decimal import Decimal
+            if isinstance(obj, Decimal):
+                return float(obj)  # 或 str(obj)
+            elif isinstance(obj, dict):
+                return {k: convert_decimals(v) for k, v in obj.items()}
+            elif isinstance(obj, list):
+                return [convert_decimals(item) for item in obj]
+            return obj
+
         sql = kwargs.get("sql")
         if not sql:
             raise Exception("SQL for `ExeSQL` MUST not be empty.")
@@ -122,7 +133,11 @@ class ExeSQL(ToolBase, ABC):
                 single_res = pd.DataFrame([i for i in cursor.fetchmany(self._param.max_records)])
                 single_res.columns = [i[0] for i in cursor.description]
 
-            sql_res.append(single_res.to_dict(orient='records'))
+            for col in single_res.columns:
+                if pd.api.types.is_datetime64_any_dtype(single_res[col]):
+                    single_res[col] = single_res[col].dt.strftime('%Y-%m-%d')
+
+            sql_res.append(convert_decimals(single_res.to_dict(orient='records')))
             formalized_content.append(single_res.to_markdown(index=False, floatfmt=".6f"))
 
         self.set_output("json", sql_res)
@@ -130,4 +145,4 @@ class ExeSQL(ToolBase, ABC):
         return self.output("formalized_content")
 
     def thoughts(self) -> str:
-        return "Query sent—waiting for the data."
+        return "Query sent—waiting for the data."

agent/tools/retrieval.py

@@ -86,10 +86,16 @@ class Retrieval(ToolBase, ABC):
                 kb_ids.append(id)
                 continue
             kb_nm = self._canvas.get_variable_value(id)
-            e, kb = KnowledgebaseService.get_by_name(kb_nm, self._canvas._tenant_id)
-            if not e:
-                raise Exception(f"Dataset({kb_nm}) does not exist.")
-            kb_ids.append(kb.id)
+            # if kb_nm is a list
+            kb_nm_list = kb_nm if isinstance(kb_nm, list) else [kb_nm]
+            for nm_or_id in kb_nm_list:
+                e, kb = KnowledgebaseService.get_by_name(nm_or_id,
+                                                         self._canvas._tenant_id)
+                if not e:
+                    e, kb = KnowledgebaseService.get_by_id(nm_or_id)
+                    if not e:
+                        raise Exception(f"Dataset({nm_or_id}) does not exist.")
+                kb_ids.append(kb.id)
 
         filtered_kb_ids: list[str] = list(set([kb_id for kb_id in kb_ids if kb_id]))
 
@@ -108,7 +114,9 @@ class Retrieval(ToolBase, ABC):
         if self._param.rerank_id:
             rerank_mdl = LLMBundle(kbs[0].tenant_id, LLMType.RERANK, self._param.rerank_id)
 
-        query = kwargs["query"]
+        vars = self.get_input_elements_from_text(kwargs["query"])
+        vars = {k:o["value"] for k,o in vars.items()}
+        query = self.string_format(kwargs["query"], vars)
         if self._param.cross_languages:
             query = cross_languages(kbs[0].tenant_id, None, query, self._param.cross_languages)
 

api/apps/__init__.py

@@ -29,6 +29,7 @@ from api.db.db_models import close_connection
 from api.db.services import UserService
 from api.utils import CustomJSONEncoder, commands
 
+from flask_mail import Mail
 from flask_session import Session
 from flask_login import LoginManager
 from api import settings
@@ -40,6 +41,7 @@ __all__ = ["app"]
 Request.json = property(lambda self: self.get_json(force=True, silent=True))
 
 app = Flask(__name__)
+smtp_mail_server = Mail()
 
 # Add this at the beginning of your file to configure Swagger UI
 swagger_config = {
@@ -146,16 +148,16 @@ def load_user(web_request):
     if authorization:
         try:
             access_token = str(jwt.loads(authorization))
-            
+
             if not access_token or not access_token.strip():
                 logging.warning("Authentication attempt with empty access token")
                 return None
-            
+
             # Access tokens should be UUIDs (32 hex characters)
             if len(access_token.strip()) < 32:
                 logging.warning(f"Authentication attempt with invalid token format: {len(access_token)} chars")
                 return None
-            
+
             user = UserService.query(
                 access_token=access_token, status=StatusEnum.VALID.value
             )

api/apps/canvas_app.py

@@ -74,11 +74,11 @@ def rm():
 @login_required
 def save():
     req = request.json
-    req["user_id"] = current_user.id
     if not isinstance(req["dsl"], str):
         req["dsl"] = json.dumps(req["dsl"], ensure_ascii=False)
     req["dsl"] = json.loads(req["dsl"])
     if "id" not in req:
+        req["user_id"] = current_user.id
         if UserCanvasService.query(user_id=current_user.id, title=req["title"].strip()):
             return get_data_error_result(message=f"{req['title'].strip()} already exists.")
         req["id"] = get_uuid()
@@ -115,6 +115,12 @@ def getsse(canvas_id):
     if not objs:
         return get_data_error_result(message='Authentication error: API key is invalid!"')
     tenant_id = objs[0].tenant_id
+    if not UserCanvasService.query(user_id=tenant_id, id=canvas_id):
+        return get_json_result(
+            data=False,
+            message='Only owner of canvas authorized for this operation.',
+            code=RetCode.OPERATING_ERROR
+        )
     e, c = UserCanvasService.get_by_id(canvas_id)
     if not e or c.user_id != tenant_id:
         return get_data_error_result(message="canvas not found.")

api/apps/chunk_app.py

@@ -23,15 +23,18 @@ from flask_login import current_user, login_required
 
 from api import settings
 from api.db import LLMType, ParserType
+from api.db.services.dialog_service import meta_filter
 from api.db.services.document_service import DocumentService
 from api.db.services.knowledgebase_service import KnowledgebaseService
 from api.db.services.llm_service import LLMBundle
+from api.db.services.search_service import SearchService
 from api.db.services.user_service import UserTenantService
 from api.utils.api_utils import get_data_error_result, get_json_result, server_error_response, validate_request
 from rag.app.qa import beAdoc, rmPrefix
 from rag.app.tag import label_question
 from rag.nlp import rag_tokenizer, search
 from rag.prompts import cross_languages, keyword_extraction
+from rag.prompts.prompts import gen_meta_filter
 from rag.settings import PAGERANK_FLD
 from rag.utils import rmSpace
 
@@ -288,13 +291,26 @@ def retrieval_test():
     if isinstance(kb_ids, str):
         kb_ids = [kb_ids]
     doc_ids = req.get("doc_ids", [])
-    similarity_threshold = float(req.get("similarity_threshold", 0.0))
-    vector_similarity_weight = float(req.get("vector_similarity_weight", 0.3))
     use_kg = req.get("use_kg", False)
     top = int(req.get("top_k", 1024))
     langs = req.get("cross_languages", [])
     tenant_ids = []
 
+    if req.get("search_id", ""):
+        search_config = SearchService.get_detail(req.get("search_id", "")).get("search_config", {})
+        meta_data_filter = search_config.get("meta_data_filter", {})
+        metas = DocumentService.get_meta_by_kbs(kb_ids)
+        if meta_data_filter.get("method") == "auto":
+            chat_mdl = LLMBundle(current_user.id, LLMType.CHAT, llm_name=search_config.get("chat_id", ""))
+            filters = gen_meta_filter(chat_mdl, metas, question)
+            doc_ids.extend(meta_filter(metas, filters))
+            if not doc_ids:
+                doc_ids = None
+        elif meta_data_filter.get("method") == "manual":
+            doc_ids.extend(meta_filter(metas, meta_data_filter["manual"]))
+            if not doc_ids:
+                doc_ids = None
+
     try:
         tenants = UserTenantService.query(user_id=current_user.id)
         for kb_id in kb_ids:
@@ -327,7 +343,9 @@ def retrieval_test():
 
         labels = label_question(question, [kb])
         ranks = settings.retrievaler.retrieval(question, embd_mdl, tenant_ids, kb_ids, page, size,
-                               similarity_threshold, vector_similarity_weight, top,
+                               float(req.get("similarity_threshold", 0.0)),
+                               float(req.get("vector_similarity_weight", 0.3)),
+                               top,
                                doc_ids, rerank_mdl=rerank_mdl, highlight=req.get("highlight"),
                                rank_feature=labels
                                )

api/apps/conversation_app.py

@@ -17,22 +17,19 @@ import json
 import re
 import traceback
 from copy import deepcopy
-
-import trio
 from flask import Response, request
 from flask_login import current_user, login_required
-
 from api import settings
 from api.db import LLMType
 from api.db.db_models import APIToken
 from api.db.services.conversation_service import ConversationService, structure_answer
-from api.db.services.dialog_service import DialogService, ask, chat
-from api.db.services.knowledgebase_service import KnowledgebaseService
-from api.db.services.llm_service import LLMBundle, TenantService
-from api.db.services.user_service import UserTenantService
+from api.db.services.dialog_service import DialogService, ask, chat, gen_mindmap
+from api.db.services.llm_service import LLMBundle
+from api.db.services.search_service import SearchService
+from api.db.services.tenant_llm_service import TenantLLMService
+from api.db.services.user_service import TenantService, UserTenantService
 from api.utils.api_utils import get_data_error_result, get_json_result, server_error_response, validate_request
-from graphrag.general.mind_map_extractor import MindMapExtractor
-from rag.app.tag import label_question
+from rag.prompts.prompt_template import load_prompt
 from rag.prompts.prompts import chunks_format
 
 
@@ -66,8 +63,14 @@ def set_conversation():
         e, dia = DialogService.get_by_id(req["dialog_id"])
         if not e:
             return get_data_error_result(message="Dialog not found")
-        conv = {"id": conv_id, "dialog_id": req["dialog_id"], "name": name, "message": [{"role": "assistant", "content": dia.prompt_config["prologue"]}],"user_id": current_user.id,
-                "reference":[],}
+        conv = {
+            "id": conv_id,
+            "dialog_id": req["dialog_id"],
+            "name": name,
+            "message": [{"role": "assistant", "content": dia.prompt_config["prologue"]}],
+            "user_id": current_user.id,
+            "reference": [],
+        }
         ConversationService.save(**conv)
         return get_json_result(data=conv)
     except Exception as e:
@@ -174,6 +177,21 @@ def completion():
             continue
         msg.append(m)
     message_id = msg[-1].get("id")
+    chat_model_id = req.get("llm_id", "")
+    req.pop("llm_id", None)
+
+    chat_model_config = {}
+    for model_config in [
+        "temperature",
+        "top_p",
+        "frequency_penalty",
+        "presence_penalty",
+        "max_tokens",
+    ]:
+        config = req.get(model_config)
+        if config:
+            chat_model_config[model_config] = config
+
     try:
         e, conv = ConversationService.get_by_id(req["conversation_id"])
         if not e:
@@ -190,13 +208,23 @@ def completion():
         conv.reference = [r for r in conv.reference if r]
         conv.reference.append({"chunks": [], "doc_aggs": []})
 
+        if chat_model_id:
+            if not TenantLLMService.get_api_key(tenant_id=dia.tenant_id, model_name=chat_model_id):
+                req.pop("chat_model_id", None)
+                req.pop("chat_model_config", None)
+                return get_data_error_result(message=f"Cannot use specified model {chat_model_id}.")
+            dia.llm_id = chat_model_id
+            dia.llm_setting = chat_model_config
+
+        is_embedded = bool(chat_model_id)
         def stream():
             nonlocal dia, msg, req, conv
             try:
                 for ans in chat(dia, msg, True, **req):
                     ans = structure_answer(conv, ans, message_id, conv.id)
                     yield "data:" + json.dumps({"code": 0, "message": "", "data": ans}, ensure_ascii=False) + "\n\n"
-                ConversationService.update_by_id(conv.id, conv.to_dict())
+                if not is_embedded:
+                    ConversationService.update_by_id(conv.id, conv.to_dict())
             except Exception as e:
                 traceback.print_exc()
                 yield "data:" + json.dumps({"code": 500, "message": str(e), "data": {"answer": "**ERROR**: " + str(e), "reference": []}}, ensure_ascii=False) + "\n\n"
@@ -214,7 +242,8 @@ def completion():
             answer = None
             for ans in chat(dia, msg, **req):
                 answer = structure_answer(conv, ans, message_id, conv.id)
-                ConversationService.update_by_id(conv.id, conv.to_dict())
+                if not is_embedded:
+                    ConversationService.update_by_id(conv.id, conv.to_dict())
                 break
             return get_json_result(data=answer)
     except Exception as e:
@@ -310,10 +339,18 @@ def ask_about():
     req = request.json
     uid = current_user.id
 
+    search_id = req.get("search_id", "")
+    search_app = None
+    search_config = {}
+    if search_id:
+        search_app = SearchService.get_detail(search_id)
+    if search_app:
+        search_config = search_app.get("search_config", {})
+
     def stream():
         nonlocal req, uid
         try:
-            for ans in ask(req["question"], req["kb_ids"], uid):
+            for ans in ask(req["question"], req["kb_ids"], uid, search_config=search_config):
                 yield "data:" + json.dumps({"code": 0, "message": "", "data": ans}, ensure_ascii=False) + "\n\n"
         except Exception as e:
             yield "data:" + json.dumps({"code": 500, "message": str(e), "data": {"answer": "**ERROR**: " + str(e), "reference": []}}, ensure_ascii=False) + "\n\n"
@@ -332,18 +369,14 @@ def ask_about():
 @validate_request("question", "kb_ids")
 def mindmap():
     req = request.json
-    kb_ids = req["kb_ids"]
-    e, kb = KnowledgebaseService.get_by_id(kb_ids[0])
-    if not e:
-        return get_data_error_result(message="Knowledgebase not found!")
+    search_id = req.get("search_id", "")
+    search_app = SearchService.get_detail(search_id) if search_id else {}
+    search_config = search_app.get("search_config", {}) if search_app else {}
+    kb_ids = search_config.get("kb_ids", [])
+    kb_ids.extend(req["kb_ids"])
+    kb_ids = list(set(kb_ids))
 
-    embd_mdl = LLMBundle(kb.tenant_id, LLMType.EMBEDDING, llm_name=kb.embd_id)
-    chat_mdl = LLMBundle(current_user.id, LLMType.CHAT)
-    question = req["question"]
-    ranks = settings.retrievaler.retrieval(question, embd_mdl, kb.tenant_id, kb_ids, 1, 12, 0.3, 0.3, aggs=False, rank_feature=label_question(question, [kb]))
-    mindmap = MindMapExtractor(chat_mdl)
-    mind_map = trio.run(mindmap, [c["content_with_weight"] for c in ranks["chunks"]])
-    mind_map = mind_map.output
+    mind_map = gen_mindmap(req["question"], kb_ids, search_app.get("tenant_id", current_user.id), search_config)
     if "error" in mind_map:
         return server_error_response(Exception(mind_map["error"]))
     return get_json_result(data=mind_map)
@@ -354,41 +387,20 @@ def mindmap():
 @validate_request("question")
 def related_questions():
     req = request.json
+
+    search_id = req.get("search_id", "")
+    search_config = {}
+    if search_id:
+        if search_app := SearchService.get_detail(search_id):
+            search_config = search_app.get("search_config", {})
+
     question = req["question"]
-    chat_mdl = LLMBundle(current_user.id, LLMType.CHAT)
-    prompt = """
-Role: You are an AI language model assistant tasked with generating 5-10 related questions based on a user’s original query. These questions should help expand the search query scope and improve search relevance.
 
-Instructions:
-	Input: You are provided with a user’s question.
-	Output: Generate 5-10 alternative questions that are related to the original user question. These alternatives should help retrieve a broader range of relevant documents from a vector database.
-	Context: Focus on rephrasing the original question in different ways, making sure the alternative questions are diverse but still connected to the topic of the original query. Do not create overly obscure, irrelevant, or unrelated questions.
-	Fallback: If you cannot generate any relevant alternatives, do not return any questions.
-	Guidance:
-	1. Each alternative should be unique but still relevant to the original query.
-	2. Keep the phrasing clear, concise, and easy to understand.
-	3. Avoid overly technical jargon or specialized terms unless directly relevant.
-	4. Ensure that each question contributes towards improving search results by broadening the search angle, not narrowing it.
+    chat_id = search_config.get("chat_id", "")
+    chat_mdl = LLMBundle(current_user.id, LLMType.CHAT, chat_id)
 
-Example:
-Original Question: What are the benefits of electric vehicles?
-
-Alternative Questions:
-	1. How do electric vehicles impact the environment?
-	2. What are the advantages of owning an electric car?
-	3. What is the cost-effectiveness of electric vehicles?
-	4. How do electric vehicles compare to traditional cars in terms of fuel efficiency?
-	5. What are the environmental benefits of switching to electric cars?
-	6. How do electric vehicles help reduce carbon emissions?
-	7. Why are electric vehicles becoming more popular?
-	8. What are the long-term savings of using electric vehicles?
-	9. How do electric vehicles contribute to sustainability?
-	10. What are the key benefits of electric vehicles for consumers?
-
-Reason:
-	Rephrasing the original query into multiple alternative questions helps the user explore different aspects of their search topic, improving the quality of search results.
-	These questions guide the search engine to provide a more comprehensive set of relevant documents.
-"""
+    gen_conf = search_config.get("llm_setting", {"temperature": 0.9})
+    prompt = load_prompt("related_question")
     ans = chat_mdl.chat(
         prompt,
         [
@@ -400,6 +412,6 @@ Related search terms:
     """,
             }
         ],
-        {"temperature": 0.9},
+        gen_conf,
     )
     return get_json_result(data=[re.sub(r"^[0-9]\. ", "", a) for a in ans.split("\n") if re.match(r"^[0-9]\. ", a)])

api/apps/dialog_app.py

@@ -16,9 +16,10 @@
 
 from flask import request
 from flask_login import login_required, current_user
+from api.db.services import duplicate_name
 from api.db.services.dialog_service import DialogService
 from api.db import StatusEnum
-from api.db.services.llm_service import TenantLLMService
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.db.services.knowledgebase_service import KnowledgebaseService
 from api.db.services.user_service import TenantService, UserTenantService
 from api import settings
@@ -41,6 +42,15 @@ def set_dialog():
         return get_data_error_result(message="Dialog name can't be empty.")
     if len(name.encode("utf-8")) > 255:
         return get_data_error_result(message=f"Dialog name length is {len(name)} which is larger than 255")
+
+    if is_create and DialogService.query(tenant_id=current_user.id, name=name.strip()):
+        name = name.strip()
+        name = duplicate_name(
+            DialogService.query,
+            name=name,
+            tenant_id=current_user.id,
+            status=StatusEnum.VALID.value)
+
     description = req.get("description", "A helpful dialog")
     icon = req.get("icon", "")
     top_n = req.get("top_n", 6)

api/apps/llm_app.py

@@ -17,7 +17,8 @@ import logging
 import json
 from flask import request
 from flask_login import login_required, current_user
-from api.db.services.llm_service import LLMFactoriesService, TenantLLMService, LLMService
+from api.db.services.tenant_llm_service import LLMFactoriesService, TenantLLMService
+from api.db.services.llm_service import LLMService
 from api import settings
 from api.utils.api_utils import server_error_response, get_data_error_result, validate_request
 from api.db import StatusEnum, LLMType

api/apps/sdk/chat.py

@@ -21,7 +21,7 @@ from api import settings
 from api.db import StatusEnum
 from api.db.services.dialog_service import DialogService
 from api.db.services.knowledgebase_service import KnowledgebaseService
-from api.db.services.llm_service import TenantLLMService
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.db.services.user_service import TenantService
 from api.utils import get_uuid
 from api.utils.api_utils import check_duplicate_ids, get_error_data_result, get_result, token_required
@@ -99,7 +99,7 @@ def create(tenant_id):
       Here is the knowledge base:
       {knowledge}
       The above is the knowledge base.""",
-        "prologue": "Hi! I'm your assistant, what can I do for you?",
+        "prologue": "Hi! I'm your assistant. What can I do for you?",
         "parameters": [{"key": "knowledge", "optional": False}],
         "empty_response": "Sorry! No relevant content was found in the knowledge base!",
         "quote": True,
@@ -150,10 +150,10 @@ def update(tenant_id, chat_id):
     if not DialogService.query(tenant_id=tenant_id, id=chat_id, status=StatusEnum.VALID.value):
         return get_error_data_result(message="You do not own the chat")
     req = request.json
-    ids = req.get("dataset_ids")
+    ids = req.get("dataset_ids", [])
     if "show_quotation" in req:
         req["do_refer"] = req.pop("show_quotation")
-    if ids is not None:
+    if ids:
         for kb_id in ids:
             kbs = KnowledgebaseService.accessible(kb_id=kb_id, user_id=tenant_id)
             if not kbs:

api/apps/sdk/dify_retrieval.py

@@ -24,6 +24,7 @@ from api.db.services.llm_service import LLMBundle
 from api import settings
 from api.utils.api_utils import validate_request, build_error_result, apikey_required
 from rag.app.tag import label_question
+from api.db.services.dialog_service import meta_filter
 
 
 @manager.route('/dify/retrieval', methods=['POST'])  # noqa: F821
@@ -37,18 +38,23 @@ def retrieval(tenant_id):
     retrieval_setting = req.get("retrieval_setting", {})
     similarity_threshold = float(retrieval_setting.get("score_threshold", 0.0))
     top = int(retrieval_setting.get("top_k", 1024))
-
+    metadata_condition = req.get("metadata_condition",{})
+    metas = DocumentService.get_meta_by_kbs([kb_id])
+ 
+    doc_ids = []
     try:
 
         e, kb = KnowledgebaseService.get_by_id(kb_id)
         if not e:
             return build_error_result(message="Knowledgebase not found!", code=settings.RetCode.NOT_FOUND)
 
-        if kb.tenant_id != tenant_id:
-            return build_error_result(message="Knowledgebase not found!", code=settings.RetCode.NOT_FOUND)
-
         embd_mdl = LLMBundle(kb.tenant_id, LLMType.EMBEDDING.value, llm_name=kb.embd_id)
-
+        print(metadata_condition)
+        print("after",convert_conditions(metadata_condition))
+        doc_ids.extend(meta_filter(metas, convert_conditions(metadata_condition)))
+        print("doc_ids",doc_ids)
+        if not doc_ids and metadata_condition is not None:
+            doc_ids = ['-999']
         ranks = settings.retrievaler.retrieval(
             question,
             embd_mdl,
@@ -59,6 +65,7 @@ def retrieval(tenant_id):
             similarity_threshold=similarity_threshold,
             vector_similarity_weight=0.3,
             top=top,
+            doc_ids=doc_ids,
             rank_feature=label_question(question, [kb])
         )
 
@@ -67,6 +74,7 @@ def retrieval(tenant_id):
                                                    [tenant_id],
                                                    [kb_id],
                                                    embd_mdl,
+                                                   doc_ids,
                                                    LLMBundle(kb.tenant_id, LLMType.CHAT))
             if ck["content_with_weight"]:
                 ranks["chunks"].insert(0, ck)
@@ -93,3 +101,20 @@ def retrieval(tenant_id):
             )
         logging.exception(e)
         return build_error_result(message=str(e), code=settings.RetCode.SERVER_ERROR)
+
+def convert_conditions(metadata_condition):
+    if metadata_condition is None:
+        metadata_condition = {}
+    op_mapping = {
+        "is": "=",
+        "not is": "≠"
+    }
+    return [
+    {
+        "op": op_mapping.get(cond["comparison_operator"], cond["comparison_operator"]),
+        "key": cond["name"],
+        "value": cond["value"]
+    }
+    for cond in metadata_condition.get("conditions", [])
+]
+

api/apps/sdk/doc.py

@@ -32,7 +32,8 @@ from api.db.services.document_service import DocumentService
 from api.db.services.file2document_service import File2DocumentService
 from api.db.services.file_service import FileService
 from api.db.services.knowledgebase_service import KnowledgebaseService
-from api.db.services.llm_service import LLMBundle, TenantLLMService
+from api.db.services.llm_service import LLMBundle
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.db.services.task_service import TaskService, queue_tasks
 from api.utils.api_utils import check_duplicate_ids, construct_json_result, get_error_data_result, get_parser_config, get_result, server_error_response, token_required
 from rag.app.qa import beAdoc, rmPrefix

api/apps/sdk/session.py

@@ -21,6 +21,7 @@ import tiktoken
 from flask import Response, jsonify, request
 
 from agent.canvas import Canvas
+from api import settings
 from api.db import LLMType, StatusEnum
 from api.db.db_models import APIToken
 from api.db.services.api_service import API4ConversationService
@@ -28,13 +29,18 @@ from api.db.services.canvas_service import UserCanvasService, completionOpenAI
 from api.db.services.canvas_service import completion as agent_completion
 from api.db.services.conversation_service import ConversationService, iframe_completion
 from api.db.services.conversation_service import completion as rag_completion
-from api.db.services.dialog_service import DialogService, ask, chat
-from api.db.services.file_service import FileService
+from api.db.services.dialog_service import DialogService, ask, chat, gen_mindmap, meta_filter
+from api.db.services.document_service import DocumentService
 from api.db.services.knowledgebase_service import KnowledgebaseService
 from api.db.services.llm_service import LLMBundle
+from api.db.services.search_service import SearchService
+from api.db.services.user_service import UserTenantService
 from api.utils import get_uuid
-from api.utils.api_utils import check_duplicate_ids, get_data_openai, get_error_data_result, get_result, token_required, validate_request
+from api.utils.api_utils import check_duplicate_ids, get_data_openai, get_error_data_result, get_json_result, get_result, server_error_response, token_required, validate_request
+from rag.app.tag import label_question
 from rag.prompts import chunks_format
+from rag.prompts.prompt_template import load_prompt
+from rag.prompts.prompts import cross_languages, gen_meta_filter, keyword_extraction
 
 
 @manager.route("/chats/<chat_id>/sessions", methods=["POST"])  # noqa: F821
@@ -69,11 +75,7 @@ def create(tenant_id, chat_id):
 @manager.route("/agents/<agent_id>/sessions", methods=["POST"])  # noqa: F821
 @token_required
 def create_agent_session(tenant_id, agent_id):
-    req = request.json
-    if not request.is_json:
-        req = request.form
-    files = request.files
-    user_id = request.args.get("user_id", "")
+    user_id = request.args.get("user_id", tenant_id)
     e, cvs = UserCanvasService.get_by_id(agent_id)
     if not e:
         return get_error_data_result("Agent not found.")
@@ -82,45 +84,12 @@ def create_agent_session(tenant_id, agent_id):
     if not isinstance(cvs.dsl, str):
         cvs.dsl = json.dumps(cvs.dsl, ensure_ascii=False)
 
-    canvas = Canvas(cvs.dsl, tenant_id)
+    session_id = get_uuid()
+    canvas = Canvas(cvs.dsl, tenant_id, agent_id)
     canvas.reset()
-    query = canvas.get_preset_param()
-    if query:
-        for ele in query:
-            if not ele["optional"]:
-                if ele["type"] == "file":
-                    if files is None or not files.get(ele["key"]):
-                        return get_error_data_result(f"`{ele['key']}` with type `{ele['type']}` is required")
-                    upload_file = files.get(ele["key"])
-                    file_content = FileService.parse_docs([upload_file], user_id)
-                    file_name = upload_file.filename
-                    ele["value"] = file_name + "\n" + file_content
-                else:
-                    if req is None or not req.get(ele["key"]):
-                        return get_error_data_result(f"`{ele['key']}` with type `{ele['type']}` is required")
-                    ele["value"] = req[ele["key"]]
-            else:
-                if ele["type"] == "file":
-                    if files is not None and files.get(ele["key"]):
-                        upload_file = files.get(ele["key"])
-                        file_content = FileService.parse_docs([upload_file], user_id)
-                        file_name = upload_file.filename
-                        ele["value"] = file_name + "\n" + file_content
-                    else:
-                        if "value" in ele:
-                            ele.pop("value")
-                else:
-                    if req is not None and req.get(ele["key"]):
-                        ele["value"] = req[ele["key"]]
-                    else:
-                        if "value" in ele:
-                            ele.pop("value")
-
-    for ans in canvas.run(stream=False):
-        pass
 
     cvs.dsl = json.loads(str(canvas))
-    conv = {"id": get_uuid(), "dialog_id": cvs.id, "user_id": user_id, "message": [{"role": "assistant", "content": canvas.get_prologue()}], "source": "agent", "dsl": cvs.dsl}
+    conv = {"id": session_id, "dialog_id": cvs.id, "user_id": user_id, "message": [{"role": "assistant", "content": canvas.get_prologue()}], "source": "agent", "dsl": cvs.dsl}
     API4ConversationService.save(**conv)
     conv["agent_id"] = conv.pop("dialog_id")
     return get_result(data=conv)
@@ -487,13 +456,14 @@ def agent_completions(tenant_id, agent_id):
                     except Exception:
                         continue
 
-                if ans.get("event") != "message":
+                if ans.get("event") != "message" or not ans.get("data", {}).get("reference", None):
                     continue
 
                 yield answer
 
             yield "data:[DONE]\n\n"
 
+    if req.get("stream", True):
         resp = Response(generate(), mimetype="text/event-stream")
         resp.headers.add_header("Cache-control", "no-cache")
         resp.headers.add_header("Connection", "keep-alive")
@@ -501,9 +471,17 @@ def agent_completions(tenant_id, agent_id):
         resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
         return resp
 
+    full_content = ""
     for answer in agent_completion(tenant_id=tenant_id, agent_id=agent_id, **req):
         try:
-            ans = json.loads(answer[5:])  # remove "data:"
+            ans = json.loads(answer[5:])
+
+            if ans["event"] == "message":
+                full_content += ans["data"]["content"]
+
+            if ans.get("data", {}).get("reference", None):
+                ans["data"]["content"] = full_content
+                return get_result(data=ans)
         except Exception as e:
             return get_result(data=f"**ERROR**: {str(e)}")
     return get_result(data=ans)
@@ -601,12 +579,12 @@ def list_agent_session(tenant_id, agent_id):
                 if message_num != 0 and messages[message_num]["role"] != "user":
                     chunk_list = []
                     # Add boundary and type checks to prevent KeyError
-                    if (chunk_num < len(conv["reference"]) and
-                            conv["reference"][chunk_num] is not None and
-                            isinstance(conv["reference"][chunk_num], dict) and
-                            "chunks" in conv["reference"][chunk_num]):
+                    if chunk_num < len(conv["reference"]) and conv["reference"][chunk_num] is not None and isinstance(conv["reference"][chunk_num], dict) and "chunks" in conv["reference"][chunk_num]:
                         chunks = conv["reference"][chunk_num]["chunks"]
                         for chunk in chunks:
+                            # Ensure chunk is a dictionary before calling get method
+                            if not isinstance(chunk, dict):
+                                continue
                             new_chunk = {
                                 "id": chunk.get("chunk_id", chunk.get("id")),
                                 "content": chunk.get("content_with_weight", chunk.get("content")),
@@ -840,6 +818,29 @@ def chatbot_completions(dialog_id):
         return get_result(data=answer)
 
 
+@manager.route("/chatbots/<dialog_id>/info", methods=["GET"])  # noqa: F821
+def chatbots_inputs(dialog_id):
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    e, dialog = DialogService.get_by_id(dialog_id)
+    if not e:
+        return get_error_data_result(f"Can't find dialog by ID: {dialog_id}")
+
+    return get_result(
+        data={
+            "title": dialog.name,
+            "avatar": dialog.icon,
+            "prologue": dialog.prompt_config.get("prologue", ""),
+        }
+    )
+
+
 @manager.route("/agentbots/<agent_id>/completions", methods=["POST"])  # noqa: F821
 def agent_bot_completions(agent_id):
     req = request.json
@@ -879,11 +880,231 @@ def begin_inputs(agent_id):
         return get_error_data_result(f"Can't find agent by ID: {agent_id}")
 
     canvas = Canvas(json.dumps(cvs.dsl), objs[0].tenant_id)
-    return get_result(
-        data={
-            "title": cvs.title,
-            "avatar": cvs.avatar,
-            "inputs": canvas.get_component_input_form("begin"),
-            "prologue": canvas.get_prologue()
-        }
+    return get_result(data={"title": cvs.title, "avatar": cvs.avatar, "inputs": canvas.get_component_input_form("begin"), "prologue": canvas.get_prologue(), "mode": canvas.get_mode()})
+
+
+@manager.route("/searchbots/ask", methods=["POST"])  # noqa: F821
+@validate_request("question", "kb_ids")
+def ask_about_embedded():
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    req = request.json
+    uid = objs[0].tenant_id
+
+    search_id = req.get("search_id", "")
+    search_config = {}
+    if search_id:
+        if search_app := SearchService.get_detail(search_id):
+            search_config = search_app.get("search_config", {})
+
+    def stream():
+        nonlocal req, uid
+        try:
+            for ans in ask(req["question"], req["kb_ids"], uid, search_config=search_config):
+                yield "data:" + json.dumps({"code": 0, "message": "", "data": ans}, ensure_ascii=False) + "\n\n"
+        except Exception as e:
+            yield "data:" + json.dumps({"code": 500, "message": str(e), "data": {"answer": "**ERROR**: " + str(e), "reference": []}}, ensure_ascii=False) + "\n\n"
+        yield "data:" + json.dumps({"code": 0, "message": "", "data": True}, ensure_ascii=False) + "\n\n"
+
+    resp = Response(stream(), mimetype="text/event-stream")
+    resp.headers.add_header("Cache-control", "no-cache")
+    resp.headers.add_header("Connection", "keep-alive")
+    resp.headers.add_header("X-Accel-Buffering", "no")
+    resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
+    return resp
+
+
+@manager.route("/searchbots/retrieval_test", methods=["POST"])  # noqa: F821
+@validate_request("kb_id", "question")
+def retrieval_test_embedded():
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    req = request.json
+    page = int(req.get("page", 1))
+    size = int(req.get("size", 30))
+    question = req["question"]
+    kb_ids = req["kb_id"]
+    if isinstance(kb_ids, str):
+        kb_ids = [kb_ids]
+    doc_ids = req.get("doc_ids", [])
+    similarity_threshold = float(req.get("similarity_threshold", 0.0))
+    vector_similarity_weight = float(req.get("vector_similarity_weight", 0.3))
+    use_kg = req.get("use_kg", False)
+    top = int(req.get("top_k", 1024))
+    langs = req.get("cross_languages", [])
+    tenant_ids = []
+
+    tenant_id = objs[0].tenant_id
+    if not tenant_id:
+        return get_error_data_result(message="permission denined.")
+
+    if req.get("search_id", ""):
+        search_config = SearchService.get_detail(req.get("search_id", "")).get("search_config", {})
+        meta_data_filter = search_config.get("meta_data_filter", {})
+        metas = DocumentService.get_meta_by_kbs(kb_ids)
+        if meta_data_filter.get("method") == "auto":
+            chat_mdl = LLMBundle(tenant_id, LLMType.CHAT, llm_name=search_config.get("chat_id", ""))
+            filters = gen_meta_filter(chat_mdl, metas, question)
+            doc_ids.extend(meta_filter(metas, filters))
+            if not doc_ids:
+                doc_ids = None
+        elif meta_data_filter.get("method") == "manual":
+            doc_ids.extend(meta_filter(metas, meta_data_filter["manual"]))
+            if not doc_ids:
+                doc_ids = None
+
+    try:
+        tenants = UserTenantService.query(user_id=tenant_id)
+        for kb_id in kb_ids:
+            for tenant in tenants:
+                if KnowledgebaseService.query(tenant_id=tenant.tenant_id, id=kb_id):
+                    tenant_ids.append(tenant.tenant_id)
+                    break
+            else:
+                return get_json_result(data=False, message="Only owner of knowledgebase authorized for this operation.", code=settings.RetCode.OPERATING_ERROR)
+
+        e, kb = KnowledgebaseService.get_by_id(kb_ids[0])
+        if not e:
+            return get_error_data_result(message="Knowledgebase not found!")
+
+        if langs:
+            question = cross_languages(kb.tenant_id, None, question, langs)
+
+        embd_mdl = LLMBundle(kb.tenant_id, LLMType.EMBEDDING.value, llm_name=kb.embd_id)
+
+        rerank_mdl = None
+        if req.get("rerank_id"):
+            rerank_mdl = LLMBundle(kb.tenant_id, LLMType.RERANK.value, llm_name=req["rerank_id"])
+
+        if req.get("keyword", False):
+            chat_mdl = LLMBundle(kb.tenant_id, LLMType.CHAT)
+            question += keyword_extraction(chat_mdl, question)
+
+        labels = label_question(question, [kb])
+        ranks = settings.retrievaler.retrieval(
+            question, embd_mdl, tenant_ids, kb_ids, page, size, similarity_threshold, vector_similarity_weight, top, doc_ids, rerank_mdl=rerank_mdl, highlight=req.get("highlight"), rank_feature=labels
+        )
+        if use_kg:
+            ck = settings.kg_retrievaler.retrieval(question, tenant_ids, kb_ids, embd_mdl, LLMBundle(kb.tenant_id, LLMType.CHAT))
+            if ck["content_with_weight"]:
+                ranks["chunks"].insert(0, ck)
+
+        for c in ranks["chunks"]:
+            c.pop("vector", None)
+        ranks["labels"] = labels
+
+        return get_json_result(data=ranks)
+    except Exception as e:
+        if str(e).find("not_found") > 0:
+            return get_json_result(data=False, message="No chunk found! Check the chunk status please!", code=settings.RetCode.DATA_ERROR)
+        return server_error_response(e)
+
+
+@manager.route("/searchbots/related_questions", methods=["POST"])  # noqa: F821
+@validate_request("question")
+def related_questions_embedded():
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    req = request.json
+    tenant_id = objs[0].tenant_id
+    if not tenant_id:
+        return get_error_data_result(message="permission denined.")
+
+    search_id = req.get("search_id", "")
+    search_config = {}
+    if search_id:
+        if search_app := SearchService.get_detail(search_id):
+            search_config = search_app.get("search_config", {})
+
+    question = req["question"]
+
+    chat_id = search_config.get("chat_id", "")
+    chat_mdl = LLMBundle(tenant_id, LLMType.CHAT, chat_id)
+
+    gen_conf = search_config.get("llm_setting", {"temperature": 0.9})
+    prompt = load_prompt("related_question")
+    ans = chat_mdl.chat(
+        prompt,
+        [
+            {
+                "role": "user",
+                "content": f"""
+Keywords: {question}
+Related search terms:
+    """,
+            }
+        ],
+        gen_conf,
     )
+    return get_json_result(data=[re.sub(r"^[0-9]\. ", "", a) for a in ans.split("\n") if re.match(r"^[0-9]\. ", a)])
+
+
+@manager.route("/searchbots/detail", methods=["GET"])  # noqa: F821
+def detail_share_embedded():
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    search_id = request.args["search_id"]
+    tenant_id = objs[0].tenant_id
+    if not tenant_id:
+        return get_error_data_result(message="permission denined.")
+    try:
+        tenants = UserTenantService.query(user_id=tenant_id)
+        for tenant in tenants:
+            if SearchService.query(tenant_id=tenant.tenant_id, id=search_id):
+                break
+        else:
+            return get_json_result(data=False, message="Has no permission for this operation.", code=settings.RetCode.OPERATING_ERROR)
+
+        search = SearchService.get_detail(search_id)
+        if not search:
+            return get_error_data_result(message="Can't find this Search App!")
+        return get_json_result(data=search)
+    except Exception as e:
+        return server_error_response(e)
+
+
+@manager.route("/searchbots/mindmap", methods=["POST"])  # noqa: F821
+@validate_request("question", "kb_ids")
+def mindmap():
+    token = request.headers.get("Authorization").split()
+    if len(token) != 2:
+        return get_error_data_result(message='Authorization is not valid!"')
+    token = token[1]
+    objs = APIToken.query(beta=token)
+    if not objs:
+        return get_error_data_result(message='Authentication error: API key is invalid!"')
+
+    tenant_id = objs[0].tenant_id
+    req = request.json
+
+    search_id = req.get("search_id", "")
+    search_app = SearchService.get_detail(search_id) if search_id else {}
+
+    mind_map = gen_mindmap(req["question"], req["kb_ids"], tenant_id, search_app.get("search_config", {}))
+    if "error" in mind_map:
+        return server_error_response(Exception(mind_map["error"]))
+    return get_json_result(data=mind_map)

api/apps/search_app.py

@@ -22,7 +22,6 @@ from api.constants import DATASET_NAME_LIMIT
 from api.db import StatusEnum
 from api.db.db_models import DB
 from api.db.services import duplicate_name
-from api.db.services.knowledgebase_service import KnowledgebaseService
 from api.db.services.search_service import SearchService
 from api.db.services.user_service import TenantService, UserTenantService
 from api.utils import get_uuid
@@ -47,7 +46,7 @@ def create():
         return get_data_error_result(message="Authorizationd identity.")
 
     search_name = search_name.strip()
-    search_name = duplicate_name(KnowledgebaseService.query, name=search_name, tenant_id=current_user.id, status=StatusEnum.VALID.value)
+    search_name = duplicate_name(SearchService.query, name=search_name, tenant_id=current_user.id, status=StatusEnum.VALID.value)
 
     req["id"] = get_uuid()
     req["name"] = search_name
@@ -156,8 +155,9 @@ def list_search_app():
     owner_ids = req.get("owner_ids", [])
     try:
         if not owner_ids:
-            tenants = TenantService.get_joined_tenants_by_user_id(current_user.id)
-            tenants = [m["tenant_id"] for m in tenants]
+            # tenants = TenantService.get_joined_tenants_by_user_id(current_user.id)
+            # tenants = [m["tenant_id"] for m in tenants]
+            tenants = []
             search_apps, total = SearchService.get_by_tenant_ids(tenants, current_user.id, page_number, items_per_page, orderby, desc, keywords)
         else:
             tenants = owner_ids

api/apps/tenant_app.py

@@ -18,12 +18,14 @@ from flask import request
 from flask_login import login_required, current_user
 
 from api import settings
+from api.apps import smtp_mail_server
 from api.db import UserTenantRole, StatusEnum
 from api.db.db_models import UserTenant
 from api.db.services.user_service import UserTenantService, UserService
 
 from api.utils import get_uuid, delta_seconds
 from api.utils.api_utils import get_json_result, validate_request, server_error_response, get_data_error_result
+from api.utils.web_utils import send_invite_email
 
 
 @manager.route("/<tenant_id>/user/list", methods=["GET"])  # noqa: F821
@@ -78,6 +80,20 @@ def create(tenant_id):
         role=UserTenantRole.INVITE,
         status=StatusEnum.VALID.value)
 
+    if smtp_mail_server and settings.SMTP_CONF:
+        from threading import Thread
+
+        user_name = ""
+        _, user = UserService.get_by_id(current_user.id)
+        if user:
+            user_name = user.nickname
+
+        Thread(
+            target=send_invite_email,
+            args=(invite_user_email, settings.MAIL_FRONTEND_URL, tenant_id, user_name or current_user.email),
+            daemon=True
+        ).start()
+
     usr = invite_users[0].to_dict()
     usr = {k: v for k, v in usr.items() if k in ["id", "avatar", "email", "nickname"]}
 

api/apps/user_app.py

@@ -28,7 +28,8 @@ from api.apps.auth import get_auth_client
 from api.db import FileType, UserTenantRole
 from api.db.db_models import TenantLLM
 from api.db.services.file_service import FileService
-from api.db.services.llm_service import LLMService, TenantLLMService
+from api.db.services.llm_service import get_init_tenant_llm
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.db.services.user_service import TenantService, UserService, UserTenantService
 from api.utils import (
     current_timestamp,
@@ -619,57 +620,8 @@ def user_register(user_id, user):
         "size": 0,
         "location": "",
     }
-    tenant_llm = []
 
-    seen = set()
-    factory_configs = []
-    for factory_config in [
-        settings.CHAT_CFG,
-        settings.EMBEDDING_CFG,
-        settings.ASR_CFG,
-        settings.IMAGE2TEXT_CFG,
-        settings.RERANK_CFG,
-    ]:
-        factory_name = factory_config["factory"]
-        if factory_name not in seen:
-            seen.add(factory_name)
-            factory_configs.append(factory_config)
-
-    for factory_config in factory_configs:
-        for llm in LLMService.query(fid=factory_config["factory"]):
-            tenant_llm.append(
-                {
-                    "tenant_id": user_id,
-                    "llm_factory": factory_config["factory"],
-                    "llm_name": llm.llm_name,
-                    "model_type": llm.model_type,
-                    "api_key": factory_config["api_key"],
-                    "api_base": factory_config["base_url"],
-                    "max_tokens": llm.max_tokens if llm.max_tokens else 8192,
-                }
-            )
-
-    if settings.LIGHTEN != 1:
-        for buildin_embedding_model in settings.BUILTIN_EMBEDDING_MODELS:
-            mdlnm, fid = TenantLLMService.split_model_name_and_factory(buildin_embedding_model)
-            tenant_llm.append(
-                {
-                    "tenant_id": user_id,
-                    "llm_factory": fid,
-                    "llm_name": mdlnm,
-                    "model_type": "embedding",
-                    "api_key": "",
-                    "api_base": "",
-                    "max_tokens": 1024 if buildin_embedding_model == "BAAI/bge-large-zh-v1.5@BAAI" else 512,
-                }
-            )
-
-    unique = {}
-    for item in tenant_llm:
-        key = (item["tenant_id"], item["llm_factory"], item["llm_name"])
-        if key not in unique:
-            unique[key] = item
-    tenant_llm = list(unique.values())
+    tenant_llm = get_init_tenant_llm(user_id)
 
     if not UserService.save(**user):
         return

api/db/db_models.py

@@ -742,7 +742,7 @@ class Dialog(DataBaseModel):
     prompt_type = CharField(max_length=16, null=False, default="simple", help_text="simple|advanced", index=True)
     prompt_config = JSONField(
         null=False,
-        default={"system": "", "prologue": "Hi! I'm your assistant, what can I do for you?", "parameters": [], "empty_response": "Sorry! No relevant content was found in the knowledge base!"},
+        default={"system": "", "prologue": "Hi! I'm your assistant. What can I do for you?", "parameters": [], "empty_response": "Sorry! No relevant content was found in the knowledge base!"},
     )
     meta_data_filter = JSONField(null=True, default={})
 
@@ -872,7 +872,7 @@ class Search(DataBaseModel):
         default={
             "kb_ids": [],
             "doc_ids": [],
-            "similarity_threshold": 0.0,
+            "similarity_threshold": 0.2,
             "vector_similarity_weight": 0.3,
             "use_kg": False,
             # rerank settings
@@ -881,11 +881,12 @@ class Search(DataBaseModel):
             # chat settings
             "summary": False,
             "chat_id": "",
+            # Leave it here for reference, don't need to set default values
             "llm_setting": {
-                "temperature": 0.1,
-                "top_p": 0.3,
-                "frequency_penalty": 0.7,
-                "presence_penalty": 0.4,
+                # "temperature": 0.1,
+                # "top_p": 0.3,
+                # "frequency_penalty": 0.7,
+                # "presence_penalty": 0.4,
             },
             "chat_settingcross_languages": [],
             "highlight": False,
@@ -1020,4 +1021,4 @@ def migrate_db():
         migrate(migrator.add_column("dialog", "meta_data_filter", JSONField(null=True, default={})))
     except Exception:
         pass
-    logging.disable(logging.NOTSET)
+    logging.disable(logging.NOTSET)

api/db/init_data.py

@@ -27,7 +27,8 @@ from api.db.services import UserService
 from api.db.services.canvas_service import CanvasTemplateService
 from api.db.services.document_service import DocumentService
 from api.db.services.knowledgebase_service import KnowledgebaseService
-from api.db.services.llm_service import LLMFactoriesService, LLMService, TenantLLMService, LLMBundle
+from api.db.services.tenant_llm_service import LLMFactoriesService, TenantLLMService
+from api.db.services.llm_service import LLMService, LLMBundle, get_init_tenant_llm
 from api.db.services.user_service import TenantService, UserTenantService
 from api import settings
 from api.utils.file_utils import get_project_base_directory
@@ -64,43 +65,7 @@ def init_superuser():
         "role": UserTenantRole.OWNER
     }
 
-    user_id = user_info
-    tenant_llm = []
-
-    seen = set()
-    factory_configs = []
-    for factory_config in [
-        settings.CHAT_CFG["factory"],
-        settings.EMBEDDING_CFG["factory"],
-        settings.ASR_CFG["factory"],
-        settings.IMAGE2TEXT_CFG["factory"],
-        settings.RERANK_CFG["factory"],
-    ]:
-        factory_name = factory_config["factory"]
-        if factory_name not in seen:
-            seen.add(factory_name)
-            factory_configs.append(factory_config)
-
-    for factory_config in factory_configs:
-        for llm in LLMService.query(fid=factory_config["factory"]):
-            tenant_llm.append(
-                {
-                    "tenant_id": user_id,
-                    "llm_factory": factory_config["factory"],
-                    "llm_name": llm.llm_name,
-                    "model_type": llm.model_type,
-                    "api_key": factory_config["api_key"],
-                    "api_base": factory_config["base_url"],
-                    "max_tokens": llm.max_tokens if llm.max_tokens else 8192,
-                }
-            )
-
-    unique = {}
-    for item in tenant_llm:
-        key = (item["tenant_id"], item["llm_factory"], item["llm_name"])
-        if key not in unique:
-            unique[key] = item
-    tenant_llm = list(unique.values())
+    tenant_llm = get_init_tenant_llm(user_info["id"])
 
     if not UserService.save(**user_info):
         logging.error("can't init admin.")

api/db/services/canvas_service.py

@@ -134,6 +134,7 @@ class UserCanvasService(CommonService):
             return False
         return True
 
+
 def completion(tenant_id, agent_id, session_id=None, **kwargs):
     query = kwargs.get("query", "") or kwargs.get("question", "")
     files = kwargs.get("files", [])
@@ -163,7 +164,8 @@ def completion(tenant_id, agent_id, session_id=None, **kwargs):
             "user_id": user_id,
             "message": [],
             "source": "agent",
-            "dsl": cvs.dsl
+            "dsl": cvs.dsl,
+            "reference": []
         }
         API4ConversationService.save(**conv)
         conv = API4Conversation(**conv)
@@ -211,10 +213,8 @@ def completionOpenAI(tenant_id, agent_id, question, session_id=None, stream=True
                     except Exception as e:
                         logging.exception(f"Agent OpenAI-Compatible completionOpenAI parse answer failed: {e}")
                         continue
-
-                if ans.get("event") != "message":
+                if ans.get("event") != "message" or not ans.get("data", {}).get("reference", None):
                     continue
-
                 content_piece = ans["data"]["content"]
                 completion_tokens += len(tiktokenenc.encode(content_piece))
 
@@ -260,7 +260,7 @@ def completionOpenAI(tenant_id, agent_id, question, session_id=None, stream=True
             ):
                 if isinstance(ans, str):
                     ans = json.loads(ans[5:])
-                if ans.get("event") != "message":
+                if ans.get("event") != "message" or not ans.get("data", {}).get("reference", None):
                     continue
                 all_content += ans["data"]["content"]
 

api/db/services/dialog_service.py

@@ -22,6 +22,7 @@ from datetime import datetime
 from functools import partial
 from timeit import default_timer as timer
 
+import trio
 from langfuse import Langfuse
 from peewee import fn
 
@@ -33,13 +34,15 @@ from api.db.services.common_service import CommonService
 from api.db.services.document_service import DocumentService
 from api.db.services.knowledgebase_service import KnowledgebaseService
 from api.db.services.langfuse_service import TenantLangfuseService
-from api.db.services.llm_service import LLMBundle, TenantLLMService
+from api.db.services.llm_service import LLMBundle
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.utils import current_timestamp, datetime_format
+from graphrag.general.mind_map_extractor import MindMapExtractor
 from rag.app.resume import forbidden_select_fields4resume
 from rag.app.tag import label_question
 from rag.nlp.search import index_name
 from rag.prompts import chunks_format, citation_prompt, cross_languages, full_question, kb_prompt, keyword_extraction, message_fit_in
-from rag.prompts.prompts import gen_meta_filter
+from rag.prompts.prompts import gen_meta_filter, PROMPT_JINJA_ENV, ASK_SUMMARY
 from rag.utils import num_tokens_from_string, rmSpace
 from rag.utils.tavily_conn import Tavily
 
@@ -98,7 +101,6 @@ class DialogService(CommonService):
 
         return list(chats.dicts())
 
-
     @classmethod
     @DB.connection_context()
     def get_by_tenant_ids(cls, joined_tenant_ids, user_id, page_number, items_per_page, orderby, desc, keywords, parser_id=None):
@@ -254,10 +256,11 @@ def repair_bad_citation_formats(answer: str, kbinfos: dict, idx: set):
 
 
 def meta_filter(metas: dict, filters: list[dict]):
-    doc_ids = []
+    doc_ids = set([])
+
     def filter_out(v2docs, operator, value):
-        nonlocal doc_ids
-        for input,docids in v2docs.items():
+        ids = []
+        for input, docids in v2docs.items():
             try:
                 input = float(input)
                 value = float(value)
@@ -281,16 +284,24 @@ def meta_filter(metas: dict, filters: list[dict]):
                 ]:
                 try:
                     if all(conds):
-                        doc_ids.extend(docids)
+                        ids.extend(docids)
+                        break
                 except Exception:
                     pass
+        return ids
 
     for k, v2docs in metas.items():
         for f in filters:
             if k != f["key"]:
                 continue
-            filter_out(v2docs, f["op"], f["value"])
-    return doc_ids
+            ids = filter_out(v2docs, f["op"], f["value"])
+            if not doc_ids:
+                doc_ids = set(ids)
+            else:
+                doc_ids = doc_ids & set(ids)
+            if not doc_ids:
+                return []
+    return list(doc_ids)
 
 
 def chat(dialog, messages, stream=True, **kwargs):
@@ -365,8 +376,12 @@ def chat(dialog, messages, stream=True, **kwargs):
         if dialog.meta_data_filter.get("method") == "auto":
             filters = gen_meta_filter(chat_mdl, metas, questions[-1])
             attachments.extend(meta_filter(metas, filters))
+            if not attachments:
+                attachments = None
         elif dialog.meta_data_filter.get("method") == "manual":
             attachments.extend(meta_filter(metas, dialog.meta_data_filter["manual"]))
+            if not attachments:
+                attachments = None
 
     if prompt_config.get("keyword", False):
         questions[-1] += keyword_extraction(chat_mdl, questions[-1])
@@ -375,17 +390,26 @@ def chat(dialog, messages, stream=True, **kwargs):
 
     thought = ""
     kbinfos = {"total": 0, "chunks": [], "doc_aggs": []}
+    knowledges = []
 
-    if "knowledge" not in [p["key"] for p in prompt_config["parameters"]]:
-        knowledges = []
-    else:
+    if attachments is not None and "knowledge" in [p["key"] for p in prompt_config["parameters"]]:
         tenant_ids = list(set([kb.tenant_id for kb in kbs]))
         knowledges = []
         if prompt_config.get("reasoning", False):
             reasoner = DeepResearcher(
                 chat_mdl,
                 prompt_config,
-                partial(retriever.retrieval, embd_mdl=embd_mdl, tenant_ids=tenant_ids, kb_ids=dialog.kb_ids, page=1, page_size=dialog.top_n, similarity_threshold=0.2, vector_similarity_weight=0.3),
+                partial(
+                    retriever.retrieval,
+                    embd_mdl=embd_mdl,
+                    tenant_ids=tenant_ids,
+                    kb_ids=dialog.kb_ids,
+                    page=1,
+                    page_size=dialog.top_n,
+                    similarity_threshold=0.2,
+                    vector_similarity_weight=0.3,
+                    doc_ids=attachments,
+                ),
             )
 
             for think in reasoner.thinking(kbinfos, " ".join(questions)):
@@ -673,7 +697,14 @@ def tts(tts_mdl, text):
     return binascii.hexlify(bin).decode("utf-8")
 
 
-def ask(question, kb_ids, tenant_id, chat_llm_name=None):
+def ask(question, kb_ids, tenant_id, chat_llm_name=None, search_config={}):
+    doc_ids = search_config.get("doc_ids", [])
+    rerank_mdl = None
+    kb_ids = search_config.get("kb_ids", kb_ids)
+    chat_llm_name = search_config.get("chat_id", chat_llm_name)
+    rerank_id = search_config.get("rerank_id", "")
+    meta_data_filter = search_config.get("meta_data_filter")
+
     kbs = KnowledgebaseService.get_by_ids(kb_ids)
     embedding_list = list(set([kb.embd_id for kb in kbs]))
 
@@ -682,30 +713,46 @@ def ask(question, kb_ids, tenant_id, chat_llm_name=None):
 
     embd_mdl = LLMBundle(tenant_id, LLMType.EMBEDDING, embedding_list[0])
     chat_mdl = LLMBundle(tenant_id, LLMType.CHAT, chat_llm_name)
+    if rerank_id:
+        rerank_mdl = LLMBundle(tenant_id, LLMType.RERANK, rerank_id)
     max_tokens = chat_mdl.max_length
     tenant_ids = list(set([kb.tenant_id for kb in kbs]))
-    kbinfos = retriever.retrieval(question, embd_mdl, tenant_ids, kb_ids, 1, 12, 0.1, 0.3, aggs=False, rank_feature=label_question(question, kbs))
+
+    if meta_data_filter:
+        metas = DocumentService.get_meta_by_kbs(kb_ids)
+        if meta_data_filter.get("method") == "auto":
+            filters = gen_meta_filter(chat_mdl, metas, question)
+            doc_ids.extend(meta_filter(metas, filters))
+            if not doc_ids:
+                doc_ids = None
+        elif meta_data_filter.get("method") == "manual":
+            doc_ids.extend(meta_filter(metas, meta_data_filter["manual"]))
+            if not doc_ids:
+                doc_ids = None
+
+    kbinfos = retriever.retrieval(
+        question = question,
+        embd_mdl=embd_mdl,
+        tenant_ids=tenant_ids,
+        kb_ids=kb_ids,
+        page=1,
+        page_size=12,
+        similarity_threshold=search_config.get("similarity_threshold", 0.1),
+        vector_similarity_weight=search_config.get("vector_similarity_weight", 0.3),
+        top=search_config.get("top_k", 1024),
+        doc_ids=doc_ids,
+        aggs=False,
+        rerank_mdl=rerank_mdl,
+        rank_feature=label_question(question, kbs)
+    )
+
     knowledges = kb_prompt(kbinfos, max_tokens)
-    prompt = """
-    Role: You're a smart assistant. Your name is Miss R.
-    Task: Summarize the information from knowledge bases and answer user's question.
-    Requirements and restriction:
-      - DO NOT make things up, especially for numbers.
-      - If the information from knowledge is irrelevant with user's question, JUST SAY: Sorry, no relevant information provided.
-      - Answer with markdown format text.
-      - Answer in language of user's question.
-      - DO NOT make things up, especially for numbers.
+    sys_prompt = PROMPT_JINJA_ENV.from_string(ASK_SUMMARY).render(knowledge="\n".join(knowledges))
 
-    ### Information from knowledge bases
-    %s
-
-    The above is information from knowledge bases.
-
-    """ % "\n".join(knowledges)
     msg = [{"role": "user", "content": question}]
 
     def decorate_answer(answer):
-        nonlocal knowledges, kbinfos, prompt
+        nonlocal knowledges, kbinfos, sys_prompt
         answer, idx = retriever.insert_citations(answer, [ck["content_ltks"] for ck in kbinfos["chunks"]], [ck["vector"] for ck in kbinfos["chunks"]], embd_mdl, tkweight=0.7, vtweight=0.3)
         idx = set([kbinfos["chunks"][int(i)]["doc_id"] for i in idx])
         recall_docs = [d for d in kbinfos["doc_aggs"] if d["doc_id"] in idx]
@@ -723,7 +770,55 @@ def ask(question, kb_ids, tenant_id, chat_llm_name=None):
         return {"answer": answer, "reference": refs}
 
     answer = ""
-    for ans in chat_mdl.chat_streamly(prompt, msg, {"temperature": 0.1}):
+    for ans in chat_mdl.chat_streamly(sys_prompt, msg, {"temperature": 0.1}):
         answer = ans
         yield {"answer": answer, "reference": {}}
     yield decorate_answer(answer)
+
+
+def gen_mindmap(question, kb_ids, tenant_id, search_config={}):
+    meta_data_filter = search_config.get("meta_data_filter", {})
+    doc_ids = search_config.get("doc_ids", [])
+    rerank_id = search_config.get("rerank_id", "")
+    rerank_mdl = None
+    kbs = KnowledgebaseService.get_by_ids(kb_ids)
+    if not kbs:
+        return {"error": "No KB selected"}
+    embedding_list = list(set([kb.embd_id for kb in kbs]))
+    tenant_ids = list(set([kb.tenant_id for kb in kbs]))
+
+    embd_mdl = LLMBundle(tenant_id, LLMType.EMBEDDING, llm_name=embedding_list[0])
+    chat_mdl = LLMBundle(tenant_id, LLMType.CHAT, llm_name=search_config.get("chat_id", ""))
+    if rerank_id:
+        rerank_mdl = LLMBundle(tenant_id, LLMType.RERANK, rerank_id)
+
+    if meta_data_filter:
+        metas = DocumentService.get_meta_by_kbs(kb_ids)
+        if meta_data_filter.get("method") == "auto":
+            filters = gen_meta_filter(chat_mdl, metas, question)
+            doc_ids.extend(meta_filter(metas, filters))
+            if not doc_ids:
+                doc_ids = None
+        elif meta_data_filter.get("method") == "manual":
+            doc_ids.extend(meta_filter(metas, meta_data_filter["manual"]))
+            if not doc_ids:
+                doc_ids = None
+
+    ranks = settings.retrievaler.retrieval(
+        question=question,
+        embd_mdl=embd_mdl,
+        tenant_ids=tenant_ids,
+        kb_ids=kb_ids,
+        page=1,
+        page_size=12,
+        similarity_threshold=search_config.get("similarity_threshold", 0.2),
+        vector_similarity_weight=search_config.get("vector_similarity_weight", 0.3),
+        top=search_config.get("top_k", 1024),
+        doc_ids=doc_ids,
+        aggs=False,
+        rerank_mdl=rerank_mdl,
+        rank_feature=label_question(question, kbs),
+    )
+    mindmap = MindMapExtractor(chat_mdl)
+    mind_map = trio.run(mindmap, [c["content_with_weight"] for c in ranks["chunks"]])
+    return mind_map.output

api/db/services/file_service.py

@@ -227,10 +227,13 @@ class FileService(CommonService):
         #     tenant_id: Tenant ID
         # Returns:
         #     Knowledge base folder dictionary
-        for root in cls.model.select().where((cls.model.tenant_id == tenant_id), (cls.model.parent_id == cls.model.id)):
-            for folder in cls.model.select().where((cls.model.tenant_id == tenant_id), (cls.model.parent_id == root.id), (cls.model.name == KNOWLEDGEBASE_FOLDER_NAME)):
-                return folder.to_dict()
-        assert False, "Can't find the KB folder. Database init error."
+        root_folder = cls.get_root_folder(tenant_id)
+        root_id = root_folder["id"]
+        kb_folder = cls.model.select().where((cls.model.tenant_id == tenant_id), (cls.model.parent_id == root_id), (cls.model.name == KNOWLEDGEBASE_FOLDER_NAME)).first()
+        if not kb_folder:
+            kb_folder = cls.new_a_file_from_kb(tenant_id, KNOWLEDGEBASE_FOLDER_NAME, root_id)
+            return kb_folder
+        return kb_folder.to_dict()
 
     @classmethod
     @DB.connection_context()
@@ -499,10 +502,9 @@ class FileService(CommonService):
     @staticmethod
     def get_blob(user_id, location):
         bname = f"{user_id}-downloads"
-        return  STORAGE_IMPL.get(bname, location)
+        return STORAGE_IMPL.get(bname, location)
 
     @staticmethod
     def put_blob(user_id, location, blob):
         bname = f"{user_id}-downloads"
-        return  STORAGE_IMPL.put(bname, location, blob)
-
+        return STORAGE_IMPL.put(bname, location, blob)

api/db/services/llm_service.py

@@ -18,246 +18,73 @@ import logging
 import re
 from functools import partial
 from typing import Generator
-
-from langfuse import Langfuse
-
-from api import settings
-from api.db import LLMType
-from api.db.db_models import DB, LLM, LLMFactories, TenantLLM
+from api.db.db_models import LLM
 from api.db.services.common_service import CommonService
-from api.db.services.langfuse_service import TenantLangfuseService
-from api.db.services.user_service import TenantService
-from rag.llm import ChatModel, CvModel, EmbeddingModel, RerankModel, Seq2txtModel, TTSModel
-
-
-class LLMFactoriesService(CommonService):
-    model = LLMFactories
+from api.db.services.tenant_llm_service import LLM4Tenant, TenantLLMService
 
 
 class LLMService(CommonService):
     model = LLM
 
 
-class TenantLLMService(CommonService):
-    model = TenantLLM
+def get_init_tenant_llm(user_id):
+    from api import settings
+    tenant_llm = []
 
-    @classmethod
-    @DB.connection_context()
-    def get_api_key(cls, tenant_id, model_name):
-        mdlnm, fid = TenantLLMService.split_model_name_and_factory(model_name)
-        if not fid:
-            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm)
-        else:
-            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm, llm_factory=fid)
+    seen = set()
+    factory_configs = []
+    for factory_config in [
+        settings.CHAT_CFG,
+        settings.EMBEDDING_CFG,
+        settings.ASR_CFG,
+        settings.IMAGE2TEXT_CFG,
+        settings.RERANK_CFG,
+    ]:
+        factory_name = factory_config["factory"]
+        if factory_name not in seen:
+            seen.add(factory_name)
+            factory_configs.append(factory_config)
 
-        if (not objs) and fid:
-            if fid == "LocalAI":
-                mdlnm += "___LocalAI"
-            elif fid == "HuggingFace":
-                mdlnm += "___HuggingFace"
-            elif fid == "OpenAI-API-Compatible":
-                mdlnm += "___OpenAI-API"
-            elif fid == "VLLM":
-                mdlnm += "___VLLM"
-            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm, llm_factory=fid)
-        if not objs:
-            return
-        return objs[0]
-
-    @classmethod
-    @DB.connection_context()
-    def get_my_llms(cls, tenant_id):
-        fields = [cls.model.llm_factory, LLMFactories.logo, LLMFactories.tags, cls.model.model_type, cls.model.llm_name, cls.model.used_tokens]
-        objs = cls.model.select(*fields).join(LLMFactories, on=(cls.model.llm_factory == LLMFactories.name)).where(cls.model.tenant_id == tenant_id, ~cls.model.api_key.is_null()).dicts()
-
-        return list(objs)
-
-    @staticmethod
-    def split_model_name_and_factory(model_name):
-        arr = model_name.split("@")
-        if len(arr) < 2:
-            return model_name, None
-        if len(arr) > 2:
-            return "@".join(arr[0:-1]), arr[-1]
-
-        # model name must be xxx@yyy
-        try:
-            model_factories = settings.FACTORY_LLM_INFOS
-            model_providers = set([f["name"] for f in model_factories])
-            if arr[-1] not in model_providers:
-                return model_name, None
-            return arr[0], arr[-1]
-        except Exception as e:
-            logging.exception(f"TenantLLMService.split_model_name_and_factory got exception: {e}")
-        return model_name, None
-
-    @classmethod
-    @DB.connection_context()
-    def get_model_config(cls, tenant_id, llm_type, llm_name=None):
-        e, tenant = TenantService.get_by_id(tenant_id)
-        if not e:
-            raise LookupError("Tenant not found")
-
-        if llm_type == LLMType.EMBEDDING.value:
-            mdlnm = tenant.embd_id if not llm_name else llm_name
-        elif llm_type == LLMType.SPEECH2TEXT.value:
-            mdlnm = tenant.asr_id
-        elif llm_type == LLMType.IMAGE2TEXT.value:
-            mdlnm = tenant.img2txt_id if not llm_name else llm_name
-        elif llm_type == LLMType.CHAT.value:
-            mdlnm = tenant.llm_id if not llm_name else llm_name
-        elif llm_type == LLMType.RERANK:
-            mdlnm = tenant.rerank_id if not llm_name else llm_name
-        elif llm_type == LLMType.TTS:
-            mdlnm = tenant.tts_id if not llm_name else llm_name
-        else:
-            assert False, "LLM type error"
-
-        model_config = cls.get_api_key(tenant_id, mdlnm)
-        mdlnm, fid = TenantLLMService.split_model_name_and_factory(mdlnm)
-        if not model_config:  # for some cases seems fid mismatch
-            model_config = cls.get_api_key(tenant_id, mdlnm)
-        if model_config:
-            model_config = model_config.to_dict()
-            llm = LLMService.query(llm_name=mdlnm) if not fid else LLMService.query(llm_name=mdlnm, fid=fid)
-            if not llm and fid:  # for some cases seems fid mismatch
-                llm = LLMService.query(llm_name=mdlnm)
-            if llm:
-                model_config["is_tools"] = llm[0].is_tools
-        if not model_config:
-            if llm_type in [LLMType.EMBEDDING, LLMType.RERANK]:
-                llm = LLMService.query(llm_name=mdlnm) if not fid else LLMService.query(llm_name=mdlnm, fid=fid)
-                if llm and llm[0].fid in ["Youdao", "FastEmbed", "BAAI"]:
-                    model_config = {"llm_factory": llm[0].fid, "api_key": "", "llm_name": mdlnm, "api_base": ""}
-            if not model_config:
-                if mdlnm == "flag-embedding":
-                    model_config = {"llm_factory": "Tongyi-Qianwen", "api_key": "", "llm_name": llm_name, "api_base": ""}
-                else:
-                    if not mdlnm:
-                        raise LookupError(f"Type of {llm_type} model is not set.")
-                    raise LookupError("Model({}) not authorized".format(mdlnm))
-        return model_config
-
-    @classmethod
-    @DB.connection_context()
-    def model_instance(cls, tenant_id, llm_type, llm_name=None, lang="Chinese", **kwargs):
-        model_config = TenantLLMService.get_model_config(tenant_id, llm_type, llm_name)
-        kwargs.update({"provider": model_config["llm_factory"]})
-        if llm_type == LLMType.EMBEDDING.value:
-            if model_config["llm_factory"] not in EmbeddingModel:
-                return
-            return EmbeddingModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"])
-
-        if llm_type == LLMType.RERANK:
-            if model_config["llm_factory"] not in RerankModel:
-                return
-            return RerankModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"])
-
-        if llm_type == LLMType.IMAGE2TEXT.value:
-            if model_config["llm_factory"] not in CvModel:
-                return
-            return CvModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], lang, base_url=model_config["api_base"], **kwargs)
-
-        if llm_type == LLMType.CHAT.value:
-            if model_config["llm_factory"] not in ChatModel:
-                return
-            return ChatModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"], **kwargs)
-
-        if llm_type == LLMType.SPEECH2TEXT:
-            if model_config["llm_factory"] not in Seq2txtModel:
-                return
-            return Seq2txtModel[model_config["llm_factory"]](key=model_config["api_key"], model_name=model_config["llm_name"], lang=lang, base_url=model_config["api_base"])
-        if llm_type == LLMType.TTS:
-            if model_config["llm_factory"] not in TTSModel:
-                return
-            return TTSModel[model_config["llm_factory"]](
-                model_config["api_key"],
-                model_config["llm_name"],
-                base_url=model_config["api_base"],
+    for factory_config in factory_configs:
+        for llm in LLMService.query(fid=factory_config["factory"]):
+            tenant_llm.append(
+                {
+                    "tenant_id": user_id,
+                    "llm_factory": factory_config["factory"],
+                    "llm_name": llm.llm_name,
+                    "model_type": llm.model_type,
+                    "api_key": factory_config["api_key"],
+                    "api_base": factory_config["base_url"],
+                    "max_tokens": llm.max_tokens if llm.max_tokens else 8192,
+                }
             )
 
-    @classmethod
-    @DB.connection_context()
-    def increase_usage(cls, tenant_id, llm_type, used_tokens, llm_name=None):
-        e, tenant = TenantService.get_by_id(tenant_id)
-        if not e:
-            logging.error(f"Tenant not found: {tenant_id}")
-            return 0
-
-        llm_map = {
-            LLMType.EMBEDDING.value: tenant.embd_id if not llm_name else llm_name,
-            LLMType.SPEECH2TEXT.value: tenant.asr_id,
-            LLMType.IMAGE2TEXT.value: tenant.img2txt_id,
-            LLMType.CHAT.value: tenant.llm_id if not llm_name else llm_name,
-            LLMType.RERANK.value: tenant.rerank_id if not llm_name else llm_name,
-            LLMType.TTS.value: tenant.tts_id if not llm_name else llm_name,
-        }
-
-        mdlnm = llm_map.get(llm_type)
-        if mdlnm is None:
-            logging.error(f"LLM type error: {llm_type}")
-            return 0
-
-        llm_name, llm_factory = TenantLLMService.split_model_name_and_factory(mdlnm)
-
-        try:
-            num = (
-                cls.model.update(used_tokens=cls.model.used_tokens + used_tokens)
-                .where(cls.model.tenant_id == tenant_id, cls.model.llm_name == llm_name, cls.model.llm_factory == llm_factory if llm_factory else True)
-                .execute()
+    if settings.LIGHTEN != 1:
+        for buildin_embedding_model in settings.BUILTIN_EMBEDDING_MODELS:
+            mdlnm, fid = TenantLLMService.split_model_name_and_factory(buildin_embedding_model)
+            tenant_llm.append(
+                {
+                    "tenant_id": user_id,
+                    "llm_factory": fid,
+                    "llm_name": mdlnm,
+                    "model_type": "embedding",
+                    "api_key": "",
+                    "api_base": "",
+                    "max_tokens": 1024 if buildin_embedding_model == "BAAI/bge-large-zh-v1.5@BAAI" else 512,
+                }
             )
-        except Exception:
-            logging.exception("TenantLLMService.increase_usage got exception,Failed to update used_tokens for tenant_id=%s, llm_name=%s", tenant_id, llm_name)
-            return 0
 
-        return num
-
-    @classmethod
-    @DB.connection_context()
-    def get_openai_models(cls):
-        objs = cls.model.select().where((cls.model.llm_factory == "OpenAI"), ~(cls.model.llm_name == "text-embedding-3-small"), ~(cls.model.llm_name == "text-embedding-3-large")).dicts()
-        return list(objs)
-
-    @staticmethod
-    def llm_id2llm_type(llm_id: str) -> str | None:
-        llm_id, *_ = TenantLLMService.split_model_name_and_factory(llm_id)
-        llm_factories = settings.FACTORY_LLM_INFOS
-        for llm_factory in llm_factories:
-            for llm in llm_factory["llm"]:
-                if llm_id == llm["llm_name"]:
-                    return llm["model_type"].split(",")[-1]
-
-        for llm in LLMService.query(llm_name=llm_id):
-            return llm.model_type
-
-        llm = TenantLLMService.get_or_none(llm_name=llm_id)
-        if llm:
-            return llm.model_type
-        for llm in TenantLLMService.query(llm_name=llm_id):
-            return llm.model_type
+    unique = {}
+    for item in tenant_llm:
+        key = (item["tenant_id"], item["llm_factory"], item["llm_name"])
+        if key not in unique:
+            unique[key] = item
+    return list(unique.values())
 
 
-class LLMBundle:
+class LLMBundle(LLM4Tenant):
     def __init__(self, tenant_id, llm_type, llm_name=None, lang="Chinese", **kwargs):
-        self.tenant_id = tenant_id
-        self.llm_type = llm_type
-        self.llm_name = llm_name
-        self.mdl = TenantLLMService.model_instance(tenant_id, llm_type, llm_name, lang=lang, **kwargs)
-        assert self.mdl, "Can't find model for {}/{}/{}".format(tenant_id, llm_type, llm_name)
-        model_config = TenantLLMService.get_model_config(tenant_id, llm_type, llm_name)
-        self.max_length = model_config.get("max_tokens", 8192)
-
-        self.is_tools = model_config.get("is_tools", False)
-        self.verbose_tool_use = kwargs.get("verbose_tool_use")
-
-        langfuse_keys = TenantLangfuseService.filter_by_tenant(tenant_id=tenant_id)
-        self.langfuse = None
-        if langfuse_keys:
-            langfuse = Langfuse(public_key=langfuse_keys.public_key, secret_key=langfuse_keys.secret_key, host=langfuse_keys.host)
-            if langfuse.auth_check():
-                self.langfuse = langfuse
-                trace_id = self.langfuse.create_trace_id()
-                self.trace_context = {"trace_id": trace_id}
+        super().__init__(tenant_id, llm_type, llm_name, lang, **kwargs)
 
     def bind_tools(self, toolcall_session, tools):
         if not self.is_tools:
@@ -325,7 +152,7 @@ class LLMBundle:
 
     def describe_with_prompt(self, image, prompt):
         if self.langfuse:
-            generation = self.language.start_generation(trace_context=self.trace_context, name="describe_with_prompt", metadata={"model": self.llm_name, "prompt": prompt})
+            generation = self.langfuse.start_generation(trace_context=self.trace_context, name="describe_with_prompt", metadata={"model": self.llm_name, "prompt": prompt})
 
         txt, used_tokens = self.mdl.describe_with_prompt(image, prompt)
         if not TenantLLMService.increase_usage(self.tenant_id, self.llm_type, used_tokens):

api/db/services/search_service.py

@@ -71,6 +71,8 @@ class SearchService(CommonService):
             .first()
             .to_dict()
         )
+        if not search:
+            return {}
         return search
 
     @classmethod

api/db/services/tenant_llm_service.py

@@ -0,0 +1,252 @@
+#
+#  Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+import logging
+from langfuse import Langfuse
+from api import settings
+from api.db import LLMType
+from api.db.db_models import DB, LLMFactories, TenantLLM
+from api.db.services.common_service import CommonService
+from api.db.services.langfuse_service import TenantLangfuseService
+from api.db.services.user_service import TenantService
+from rag.llm import ChatModel, CvModel, EmbeddingModel, RerankModel, Seq2txtModel, TTSModel
+
+
+class LLMFactoriesService(CommonService):
+    model = LLMFactories
+
+
+class TenantLLMService(CommonService):
+    model = TenantLLM
+
+    @classmethod
+    @DB.connection_context()
+    def get_api_key(cls, tenant_id, model_name):
+        mdlnm, fid = TenantLLMService.split_model_name_and_factory(model_name)
+        if not fid:
+            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm)
+        else:
+            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm, llm_factory=fid)
+
+        if (not objs) and fid:
+            if fid == "LocalAI":
+                mdlnm += "___LocalAI"
+            elif fid == "HuggingFace":
+                mdlnm += "___HuggingFace"
+            elif fid == "OpenAI-API-Compatible":
+                mdlnm += "___OpenAI-API"
+            elif fid == "VLLM":
+                mdlnm += "___VLLM"
+            objs = cls.query(tenant_id=tenant_id, llm_name=mdlnm, llm_factory=fid)
+        if not objs:
+            return
+        return objs[0]
+
+    @classmethod
+    @DB.connection_context()
+    def get_my_llms(cls, tenant_id):
+        fields = [cls.model.llm_factory, LLMFactories.logo, LLMFactories.tags, cls.model.model_type, cls.model.llm_name, cls.model.used_tokens]
+        objs = cls.model.select(*fields).join(LLMFactories, on=(cls.model.llm_factory == LLMFactories.name)).where(cls.model.tenant_id == tenant_id, ~cls.model.api_key.is_null()).dicts()
+
+        return list(objs)
+
+    @staticmethod
+    def split_model_name_and_factory(model_name):
+        arr = model_name.split("@")
+        if len(arr) < 2:
+            return model_name, None
+        if len(arr) > 2:
+            return "@".join(arr[0:-1]), arr[-1]
+
+        # model name must be xxx@yyy
+        try:
+            model_factories = settings.FACTORY_LLM_INFOS
+            model_providers = set([f["name"] for f in model_factories])
+            if arr[-1] not in model_providers:
+                return model_name, None
+            return arr[0], arr[-1]
+        except Exception as e:
+            logging.exception(f"TenantLLMService.split_model_name_and_factory got exception: {e}")
+        return model_name, None
+
+    @classmethod
+    @DB.connection_context()
+    def get_model_config(cls, tenant_id, llm_type, llm_name=None):
+        from api.db.services.llm_service import LLMService
+        e, tenant = TenantService.get_by_id(tenant_id)
+        if not e:
+            raise LookupError("Tenant not found")
+
+        if llm_type == LLMType.EMBEDDING.value:
+            mdlnm = tenant.embd_id if not llm_name else llm_name
+        elif llm_type == LLMType.SPEECH2TEXT.value:
+            mdlnm = tenant.asr_id
+        elif llm_type == LLMType.IMAGE2TEXT.value:
+            mdlnm = tenant.img2txt_id if not llm_name else llm_name
+        elif llm_type == LLMType.CHAT.value:
+            mdlnm = tenant.llm_id if not llm_name else llm_name
+        elif llm_type == LLMType.RERANK:
+            mdlnm = tenant.rerank_id if not llm_name else llm_name
+        elif llm_type == LLMType.TTS:
+            mdlnm = tenant.tts_id if not llm_name else llm_name
+        else:
+            assert False, "LLM type error"
+
+        model_config = cls.get_api_key(tenant_id, mdlnm)
+        mdlnm, fid = TenantLLMService.split_model_name_and_factory(mdlnm)
+        if not model_config:  # for some cases seems fid mismatch
+            model_config = cls.get_api_key(tenant_id, mdlnm)
+        if model_config:
+            model_config = model_config.to_dict()
+            llm = LLMService.query(llm_name=mdlnm) if not fid else LLMService.query(llm_name=mdlnm, fid=fid)
+            if not llm and fid:  # for some cases seems fid mismatch
+                llm = LLMService.query(llm_name=mdlnm)
+            if llm:
+                model_config["is_tools"] = llm[0].is_tools
+        if not model_config:
+            if llm_type in [LLMType.EMBEDDING, LLMType.RERANK]:
+                llm = LLMService.query(llm_name=mdlnm) if not fid else LLMService.query(llm_name=mdlnm, fid=fid)
+                if llm and llm[0].fid in ["Youdao", "FastEmbed", "BAAI"]:
+                    model_config = {"llm_factory": llm[0].fid, "api_key": "", "llm_name": mdlnm, "api_base": ""}
+            if not model_config:
+                if mdlnm == "flag-embedding":
+                    model_config = {"llm_factory": "Tongyi-Qianwen", "api_key": "", "llm_name": llm_name, "api_base": ""}
+                else:
+                    if not mdlnm:
+                        raise LookupError(f"Type of {llm_type} model is not set.")
+                    raise LookupError("Model({}) not authorized".format(mdlnm))
+        return model_config
+
+    @classmethod
+    @DB.connection_context()
+    def model_instance(cls, tenant_id, llm_type, llm_name=None, lang="Chinese", **kwargs):
+        model_config = TenantLLMService.get_model_config(tenant_id, llm_type, llm_name)
+        kwargs.update({"provider": model_config["llm_factory"]})
+        if llm_type == LLMType.EMBEDDING.value:
+            if model_config["llm_factory"] not in EmbeddingModel:
+                return
+            return EmbeddingModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"])
+
+        if llm_type == LLMType.RERANK:
+            if model_config["llm_factory"] not in RerankModel:
+                return
+            return RerankModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"])
+
+        if llm_type == LLMType.IMAGE2TEXT.value:
+            if model_config["llm_factory"] not in CvModel:
+                return
+            return CvModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], lang, base_url=model_config["api_base"], **kwargs)
+
+        if llm_type == LLMType.CHAT.value:
+            if model_config["llm_factory"] not in ChatModel:
+                return
+            return ChatModel[model_config["llm_factory"]](model_config["api_key"], model_config["llm_name"], base_url=model_config["api_base"], **kwargs)
+
+        if llm_type == LLMType.SPEECH2TEXT:
+            if model_config["llm_factory"] not in Seq2txtModel:
+                return
+            return Seq2txtModel[model_config["llm_factory"]](key=model_config["api_key"], model_name=model_config["llm_name"], lang=lang, base_url=model_config["api_base"])
+        if llm_type == LLMType.TTS:
+            if model_config["llm_factory"] not in TTSModel:
+                return
+            return TTSModel[model_config["llm_factory"]](
+                model_config["api_key"],
+                model_config["llm_name"],
+                base_url=model_config["api_base"],
+            )
+
+    @classmethod
+    @DB.connection_context()
+    def increase_usage(cls, tenant_id, llm_type, used_tokens, llm_name=None):
+        e, tenant = TenantService.get_by_id(tenant_id)
+        if not e:
+            logging.error(f"Tenant not found: {tenant_id}")
+            return 0
+
+        llm_map = {
+            LLMType.EMBEDDING.value: tenant.embd_id if not llm_name else llm_name,
+            LLMType.SPEECH2TEXT.value: tenant.asr_id,
+            LLMType.IMAGE2TEXT.value: tenant.img2txt_id,
+            LLMType.CHAT.value: tenant.llm_id if not llm_name else llm_name,
+            LLMType.RERANK.value: tenant.rerank_id if not llm_name else llm_name,
+            LLMType.TTS.value: tenant.tts_id if not llm_name else llm_name,
+        }
+
+        mdlnm = llm_map.get(llm_type)
+        if mdlnm is None:
+            logging.error(f"LLM type error: {llm_type}")
+            return 0
+
+        llm_name, llm_factory = TenantLLMService.split_model_name_and_factory(mdlnm)
+
+        try:
+            num = (
+                cls.model.update(used_tokens=cls.model.used_tokens + used_tokens)
+                .where(cls.model.tenant_id == tenant_id, cls.model.llm_name == llm_name, cls.model.llm_factory == llm_factory if llm_factory else True)
+                .execute()
+            )
+        except Exception:
+            logging.exception("TenantLLMService.increase_usage got exception,Failed to update used_tokens for tenant_id=%s, llm_name=%s", tenant_id, llm_name)
+            return 0
+
+        return num
+
+    @classmethod
+    @DB.connection_context()
+    def get_openai_models(cls):
+        objs = cls.model.select().where((cls.model.llm_factory == "OpenAI"), ~(cls.model.llm_name == "text-embedding-3-small"), ~(cls.model.llm_name == "text-embedding-3-large")).dicts()
+        return list(objs)
+
+    @staticmethod
+    def llm_id2llm_type(llm_id: str) -> str | None:
+        from api.db.services.llm_service import LLMService
+        llm_id, *_ = TenantLLMService.split_model_name_and_factory(llm_id)
+        llm_factories = settings.FACTORY_LLM_INFOS
+        for llm_factory in llm_factories:
+            for llm in llm_factory["llm"]:
+                if llm_id == llm["llm_name"]:
+                    return llm["model_type"].split(",")[-1]
+
+        for llm in LLMService.query(llm_name=llm_id):
+            return llm.model_type
+
+        llm = TenantLLMService.get_or_none(llm_name=llm_id)
+        if llm:
+            return llm.model_type
+        for llm in TenantLLMService.query(llm_name=llm_id):
+            return llm.model_type
+
+
+class LLM4Tenant:
+    def __init__(self, tenant_id, llm_type, llm_name=None, lang="Chinese", **kwargs):
+        self.tenant_id = tenant_id
+        self.llm_type = llm_type
+        self.llm_name = llm_name
+        self.mdl = TenantLLMService.model_instance(tenant_id, llm_type, llm_name, lang=lang, **kwargs)
+        assert self.mdl, "Can't find model for {}/{}/{}".format(tenant_id, llm_type, llm_name)
+        model_config = TenantLLMService.get_model_config(tenant_id, llm_type, llm_name)
+        self.max_length = model_config.get("max_tokens", 8192)
+
+        self.is_tools = model_config.get("is_tools", False)
+        self.verbose_tool_use = kwargs.get("verbose_tool_use")
+
+        langfuse_keys = TenantLangfuseService.filter_by_tenant(tenant_id=tenant_id)
+        self.langfuse = None
+        if langfuse_keys:
+            langfuse = Langfuse(public_key=langfuse_keys.public_key, secret_key=langfuse_keys.secret_key, host=langfuse_keys.host)
+            if langfuse.auth_check():
+                self.langfuse = langfuse
+                trace_id = self.langfuse.create_trace_id()
+                self.trace_context = {"trace_id": trace_id}

api/ragflow_server.py

@@ -33,7 +33,7 @@ import uuid
 
 from werkzeug.serving import run_simple
 from api import settings
-from api.apps import app
+from api.apps import app, smtp_mail_server
 from api.db.runtime_config import RuntimeConfig
 from api.db.services.document_service import DocumentService
 from api import utils
@@ -59,11 +59,14 @@ def update_progress():
             if redis_lock.acquire():
                 DocumentService.update_progress()
                 redis_lock.release()
-            stop_event.wait(6)
         except Exception:
             logging.exception("update_progress exception")
         finally:
-            redis_lock.release()
+            try:
+                redis_lock.release()
+            except Exception:
+                logging.exception("update_progress exception")
+            stop_event.wait(6)
 
 def signal_handler(sig, frame):
     logging.info("Received interrupt signal, shutting down...")
@@ -74,11 +77,11 @@ def signal_handler(sig, frame):
 
 if __name__ == '__main__':
     logging.info(r"""
-        ____   ___    ______ ______ __               
+        ____   ___    ______ ______ __
        / __ \ /   |  / ____// ____// /____  _      __
       / /_/ // /| | / / __ / /_   / // __ \| | /| / /
-     / _, _// ___ |/ /_/ // __/  / // /_/ /| |/ |/ / 
-    /_/ |_|/_/  |_|\____//_/    /_/ \____/ |__/|__/                             
+     / _, _// ___ |/ /_/ // __/  / // /_/ /| |/ |/ /
+    /_/ |_|/_/  |_|\____//_/    /_/ \____/ |__/|__/
 
     """)
     logging.info(
@@ -137,6 +140,18 @@ if __name__ == '__main__':
     else:
         threading.Timer(1.0, delayed_start_update_progress).start()
 
+    # init smtp server
+    if settings.SMTP_CONF:
+        app.config["MAIL_SERVER"] = settings.MAIL_SERVER
+        app.config["MAIL_PORT"] = settings.MAIL_PORT
+        app.config["MAIL_USE_SSL"] = settings.MAIL_USE_SSL
+        app.config["MAIL_USE_TLS"] = settings.MAIL_USE_TLS
+        app.config["MAIL_USERNAME"] = settings.MAIL_USERNAME
+        app.config["MAIL_PASSWORD"] = settings.MAIL_PASSWORD
+        app.config["MAIL_DEFAULT_SENDER"] = settings.MAIL_DEFAULT_SENDER
+        smtp_mail_server.init_app(app)
+
+
     # start http server
     try:
         logging.info("RAGFlow HTTP server start...")

api/settings.py

@@ -79,6 +79,16 @@ STRONG_TEST_COUNT = int(os.environ.get("STRONG_TEST_COUNT", "8"))
 
 BUILTIN_EMBEDDING_MODELS = ["BAAI/bge-large-zh-v1.5@BAAI", "maidalun1020/bce-embedding-base_v1@Youdao"]
 
+SMTP_CONF = None
+MAIL_SERVER = ""
+MAIL_PORT = 000
+MAIL_USE_SSL= True
+MAIL_USE_TLS = False
+MAIL_USERNAME = ""
+MAIL_PASSWORD = ""
+MAIL_DEFAULT_SENDER = ()
+MAIL_FRONTEND_URL = ""
+
 
 def get_or_create_secret_key():
     secret_key = os.environ.get("RAGFLOW_SECRET_KEY")
@@ -186,6 +196,21 @@ def init_settings():
         global SANDBOX_HOST
         SANDBOX_HOST = os.environ.get("SANDBOX_HOST", "sandbox-executor-manager")
 
+    global SMTP_CONF, MAIL_SERVER, MAIL_PORT, MAIL_USE_SSL, MAIL_USE_TLS
+    global MAIL_USERNAME, MAIL_PASSWORD, MAIL_DEFAULT_SENDER, MAIL_FRONTEND_URL
+    SMTP_CONF = get_base_config("smtp", {})
+
+    MAIL_SERVER = SMTP_CONF.get("mail_server", "")
+    MAIL_PORT = SMTP_CONF.get("mail_port", 000)
+    MAIL_USE_SSL = SMTP_CONF.get("mail_use_ssl", True)
+    MAIL_USE_TLS = SMTP_CONF.get("mail_use_tls", False)
+    MAIL_USERNAME = SMTP_CONF.get("mail_username", "")
+    MAIL_PASSWORD = SMTP_CONF.get("mail_password", "")
+    mail_default_sender = SMTP_CONF.get("mail_default_sender", [])
+    if mail_default_sender and len(mail_default_sender) >= 2:
+        MAIL_DEFAULT_SENDER = (mail_default_sender[0], mail_default_sender[1])
+    MAIL_FRONTEND_URL = SMTP_CONF.get("mail_frontend_url", "")
+
 
 class CustomEnum(Enum):
     @classmethod

api/utils/api_utils.py

@@ -17,6 +17,7 @@ import asyncio
 import functools
 import json
 import logging
+import os
 import queue
 import random
 import threading
@@ -48,7 +49,8 @@ from werkzeug.http import HTTP_STATUS_CODES
 from api import settings
 from api.constants import REQUEST_MAX_WAIT_SEC, REQUEST_WAIT_SEC
 from api.db.db_models import APIToken
-from api.db.services.llm_service import LLMService, TenantLLMService
+from api.db.services.llm_service import LLMService
+from api.db.services.tenant_llm_service import TenantLLMService
 from api.utils import CustomJSONEncoder, get_uuid, json_dumps
 from rag.utils.mcp_tool_call_conn import MCPToolCallSession, close_multiple_mcp_toolcall_sessions
 
@@ -352,7 +354,7 @@ def get_parser_config(chunk_method, parser_config):
     if not chunk_method:
         chunk_method = "naive"
 
-    # Define default configurations for each chunk method
+    # Define default configurations for each chunking method
     key_mapping = {
         "naive": {"chunk_token_num": 512, "delimiter": r"\n", "html4excel": False, "layout_recognize": "DeepDOC", "raptor": {"use_raptor": False}, "graphrag": {"use_graphrag": False}},
         "qa": {"raptor": {"use_raptor": False}, "graphrag": {"use_graphrag": False}},
@@ -666,7 +668,10 @@ def timeout(seconds: float | int = None, attempts: int = 2, *, exception: Option
 
             for a in range(attempts):
                 try:
-                    result = result_queue.get(timeout=seconds)
+                    if os.environ.get("ENABLE_TIMEOUT_ASSERTION"):
+                        result = result_queue.get(timeout=seconds)
+                    else:
+                        result = result_queue.get()
                     if isinstance(result, Exception):
                         raise result
                     return result
@@ -681,7 +686,10 @@ def timeout(seconds: float | int = None, attempts: int = 2, *, exception: Option
 
             for a in range(attempts):
                 try:
-                    with trio.fail_after(seconds):
+                    if os.environ.get("ENABLE_TIMEOUT_ASSERTION"):
+                        with trio.fail_after(seconds):
+                            return await func(*args, **kwargs)
+                    else:
                         return await func(*args, **kwargs)
                 except trio.TooSlowError:
                     if a < attempts - 1:

api/utils/web_utils.py

@@ -21,6 +21,9 @@ import re
 import socket
 from urllib.parse import urlparse
 
+from api.apps import smtp_mail_server
+from flask_mail import Message
+from flask import render_template_string
 from selenium import webdriver
 from selenium.common.exceptions import TimeoutException
 from selenium.webdriver.chrome.options import Options
@@ -31,6 +34,7 @@ from selenium.webdriver.support.ui import WebDriverWait
 from webdriver_manager.chrome import ChromeDriverManager
 
 
+
 CONTENT_TYPE_MAP = {
     # Office
     "docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
@@ -172,3 +176,26 @@ def get_float(req: dict, key: str, default: float | int = 10.0) -> float:
         return parsed if parsed > 0 else default
     except (TypeError, ValueError):
         return default
+
+
+INVITE_EMAIL_TMPL = """
+<p>Hi {{email}},</p>
+<p>{{inviter}} has invited you to join their team (ID: {{tenant_id}}).</p>
+<p>Click the link below to complete your registration:<br>
+<a href="{{invite_url}}">{{invite_url}}</a></p>
+<p>If you did not request this, please ignore this email.</p>
+"""
+
+def send_invite_email(to_email, invite_url, tenant_id, inviter):
+    from api.apps import  app
+    with app.app_context():
+        msg = Message(subject="RAGFlow Invitation",
+                      recipients=[to_email])
+        msg.html = render_template_string(
+            INVITE_EMAIL_TMPL,
+            email=to_email,
+            invite_url=invite_url,
+            tenant_id=tenant_id,
+            inviter=inviter,
+        )
+        smtp_mail_server.send(msg)

conf/llm_factories.json

@@ -505,6 +505,24 @@
                     "tags": "RE-RANK,4k",
                     "max_tokens": 4000,
                     "model_type": "rerank"
+                },
+                {
+                    "llm_name": "qwen-audio-asr",
+                    "tags": "SPEECH2TEXT,8k",
+                    "max_tokens": 8000,
+                    "model_type": "speech2text"
+                },
+                {
+                    "llm_name": "qwen-audio-asr-latest",
+                    "tags": "SPEECH2TEXT,8k",
+                    "max_tokens": 8000,
+                    "model_type": "speech2text"
+                },
+                {
+                    "llm_name": "qwen-audio-asr-1204",
+                    "tags": "SPEECH2TEXT,8k",
+                    "max_tokens": 8000,
+                    "model_type": "speech2text"
                 }
             ]
         },
@@ -514,23 +532,65 @@
             "tags": "LLM,TEXT EMBEDDING,SPEECH2TEXT,MODERATION",
             "status": "1",
             "llm": [
+                {
+                    "llm_name": "glm-4.5",
+                    "tags": "LLM,CHAT,128K",
+                    "max_tokens": 128000,
+                    "model_type": "chat",
+                    "is_tools": true
+                },
+                {
+                    "llm_name": "glm-4.5-x",
+                    "tags": "LLM,CHAT,128k",
+                    "max_tokens": 128000,
+                    "model_type": "chat",
+                    "is_tools": true
+                },
+                {
+                    "llm_name": "glm-4.5-air",
+                    "tags": "LLM,CHAT,128K",
+                    "max_tokens": 128000,
+                    "model_type": "chat",
+                    "is_tools": true
+                },
+                {
+                    "llm_name": "glm-4.5-airx",
+                    "tags": "LLM,CHAT,128k",
+                    "max_tokens": 128000,
+                    "model_type": "chat",
+                    "is_tools": true
+                },
+                {
+                    "llm_name": "glm-4.5-flash",
+                    "tags": "LLM,CHAT,128k",
+                    "max_tokens": 128000,
+                    "model_type": "chat",
+                    "is_tools": true
+                },
+                {
+                    "llm_name": "glm-4.5v",
+                    "tags": "LLM,IMAGE2TEXT,64,",
+                    "max_tokens": 64000,
+                    "model_type": "image2text",
+                    "is_tools": false
+                },
                 {
                     "llm_name": "glm-4-plus",
-                    "tags": "LLM,CHAT,",
+                    "tags": "LLM,CHAT,128K",
                     "max_tokens": 128000,
                     "model_type": "chat",
                     "is_tools": true
                 },
                 {
                     "llm_name": "glm-4-0520",
-                    "tags": "LLM,CHAT,",
+                    "tags": "LLM,CHAT,128K",
                     "max_tokens": 128000,
                     "model_type": "chat",
                     "is_tools": true
                 },
                 {
                     "llm_name": "glm-4",
-                    "tags": "LLM,CHAT,",
+                    "tags":"LLM,CHAT,128K",
                     "max_tokens": 128000,
                     "model_type": "chat",
                     "is_tools": true
@@ -1146,60 +1206,35 @@
                     "llm_name": "gemini-2.5-flash",
                     "tags": "LLM,CHAT,1024K,IMAGE2TEXT",
                     "max_tokens": 1048576,
-                    "model_type": "image2text",
+                    "model_type": "chat",
                     "is_tools": true
                 },
                 {
                     "llm_name": "gemini-2.5-pro",
                     "tags": "LLM,CHAT,IMAGE2TEXT,1024K",
                     "max_tokens": 1048576,
-                    "model_type": "image2text",
+                    "model_type": "chat",
                     "is_tools": true
                 },
                 {
-                    "llm_name": "gemini-2.5-flash-preview-05-20",
+                    "llm_name": "gemini-2.5-flash-lite",
                     "tags": "LLM,CHAT,1024K,IMAGE2TEXT",
                     "max_tokens": 1048576,
-                    "model_type": "image2text",
+                    "model_type": "chat",
                     "is_tools": true
                 },
                 {
-                    "llm_name": "gemini-2.0-flash-001",
-                    "tags": "LLM,CHAT,1024K",
-                    "max_tokens": 1048576,
-                    "model_type": "image2text",
-                    "is_tools": true
-                },
-                {
-                    "llm_name": "gemini-2.0-flash-thinking-exp-01-21",
+                    "llm_name": "gemini-2.0-flash",
                     "tags": "LLM,CHAT,1024K",
                     "max_tokens": 1048576,
                     "model_type": "chat",
                     "is_tools": true
                 },
                 {
-                    "llm_name": "gemini-1.5-flash",
-                    "tags": "LLM,IMAGE2TEXT,1024K",
+                    "llm_name": "gemini-2.0-flash-lite",
+                    "tags": "LLM,CHAT,1024K",
                     "max_tokens": 1048576,
-                    "model_type": "image2text"
-                },
-                {
-                    "llm_name": "gemini-2.5-pro-preview-05-06",
-                    "tags": "LLM,IMAGE2TEXT,1024K",
-                    "max_tokens": 1048576,
-                    "model_type": "image2text"
-                },
-                {
-                    "llm_name": "gemini-1.5-pro",
-                    "tags": "LLM,IMAGE2TEXT,2048K",
-                    "max_tokens": 2097152,
-                    "model_type": "image2text"
-                },
-                {
-                    "llm_name": "gemini-1.5-flash-8b",
-                    "tags": "LLM,IMAGE2TEXT,1024K",
-                    "max_tokens": 1048576,
-                    "model_type": "image2text",
+                    "model_type": "chat",
                     "is_tools": true
                 },
                 {

conf/service_conf.yaml

@@ -113,3 +113,14 @@ redis:
 #   switch: false
 #   component: false
 #   dataset: false
+# smtp:
+#   mail_server: ""
+#   mail_port: 465
+#   mail_use_ssl: true
+#   mail_use_tls: false
+#   mail_username: ""
+#   mail_password: ""
+#   mail_default_sender:
+#     - "RAGFlow" # display name
+#     - "" # sender email address
+#   mail_frontend_url: "https://your-frontend.example.com"

deepdoc/parser/__init__.py

@@ -14,13 +14,15 @@
 #  limitations under the License.
 #
 
-from .pdf_parser import RAGFlowPdfParser as PdfParser, PlainParser
 from .docx_parser import RAGFlowDocxParser as DocxParser
 from .excel_parser import RAGFlowExcelParser as ExcelParser
-from .ppt_parser import RAGFlowPptParser as PptParser
 from .html_parser import RAGFlowHtmlParser as HtmlParser
 from .json_parser import RAGFlowJsonParser as JsonParser
+from .markdown_parser import MarkdownElementExtractor
 from .markdown_parser import RAGFlowMarkdownParser as MarkdownParser
+from .pdf_parser import PlainParser
+from .pdf_parser import RAGFlowPdfParser as PdfParser
+from .ppt_parser import RAGFlowPptParser as PptParser
 from .txt_parser import RAGFlowTxtParser as TxtParser
 
 __all__ = [
@@ -33,4 +35,6 @@ __all__ = [
     "JsonParser",
     "MarkdownParser",
     "TxtParser",
-]
+    "MarkdownElementExtractor",
+]
+

deepdoc/parser/excel_parser.py

@@ -90,9 +90,17 @@ class RAGFlowExcelParser:
         return wb
 
     def html(self, fnm, chunk_rows=256):
+        from html import escape
+
         file_like_object = BytesIO(fnm) if not isinstance(fnm, str) else fnm
         wb = RAGFlowExcelParser._load_excel_to_workbook(file_like_object)
         tb_chunks = []
+
+        def _fmt(v):
+            if v is None:
+                return ""
+            return str(v).strip()
+
         for sheetname in wb.sheetnames:
             ws = wb[sheetname]
             rows = list(ws.rows)
@@ -101,7 +109,7 @@ class RAGFlowExcelParser:
 
             tb_rows_0 = "<tr>"
             for t in list(rows[0]):
-                tb_rows_0 += f"<th>{t.value}</th>"
+                tb_rows_0 += f"<th>{escape(_fmt(t.value))}</th>"
             tb_rows_0 += "</tr>"
 
             for chunk_i in range((len(rows) - 1) // chunk_rows + 1):
@@ -109,7 +117,7 @@ class RAGFlowExcelParser:
                 tb += f"<table><caption>{sheetname}</caption>"
                 tb += tb_rows_0
                 for r in list(
-                    rows[1 + chunk_i * chunk_rows: 1 + (chunk_i + 1) * chunk_rows]
+                    rows[1 + chunk_i * chunk_rows: min(1 + (chunk_i + 1) * chunk_rows, len(rows))]
                 ):
                     tb += "<tr>"
                     for i, c in enumerate(r):

deepdoc/parser/html_parser.py

@@ -15,35 +15,200 @@
 #  limitations under the License.
 #
 
-from rag.nlp import find_codec
-import readability
-import html_text
+from rag.nlp import find_codec, rag_tokenizer
+import uuid
 import chardet
-
+from bs4 import BeautifulSoup, NavigableString, Tag, Comment
+import html
 
 def get_encoding(file):
     with open(file,'rb') as f:
         tmp = chardet.detect(f.read())
         return tmp['encoding']
 
+BLOCK_TAGS = [
+    "h1", "h2", "h3", "h4", "h5", "h6",
+    "p", "div", "article", "section", "aside",
+    "ul", "ol", "li",
+    "table", "pre", "code", "blockquote",
+    "figure", "figcaption"
+]
+TITLE_TAGS = {"h1": "#", "h2": "##", "h3": "###", "h4": "#####", "h5": "#####", "h6": "######"}
+
 
 class RAGFlowHtmlParser:
-    def __call__(self, fnm, binary=None):
+    def __call__(self, fnm, binary=None, chunk_token_num=None):
         if binary:
             encoding = find_codec(binary)
             txt = binary.decode(encoding, errors="ignore")
         else:
             with open(fnm, "r",encoding=get_encoding(fnm)) as f:
                 txt = f.read()
-        return self.parser_txt(txt)
+        return self.parser_txt(txt, chunk_token_num)
 
     @classmethod
-    def parser_txt(cls, txt):
+    def parser_txt(cls, txt, chunk_token_num):
         if not isinstance(txt, str):
             raise TypeError("txt type should be string!")
-        html_doc = readability.Document(txt)
-        title = html_doc.title()
-        content = html_text.extract_text(html_doc.summary(html_partial=True))
-        txt = f"{title}\n{content}"
-        sections = txt.split("\n")
+
+        temp_sections = []
+        soup = BeautifulSoup(txt, "html5lib")
+        # delete <style> tag
+        for style_tag in soup.find_all(["style", "script"]):
+            style_tag.decompose()
+        # delete <script> tag in <div>
+        for div_tag in soup.find_all("div"):
+            for script_tag in div_tag.find_all("script"):
+                script_tag.decompose()
+        # delete inline style
+        for tag in soup.find_all(True):
+            if 'style' in tag.attrs:
+                del tag.attrs['style']
+        # delete HTML comment
+        for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
+            comment.extract()
+
+        cls.read_text_recursively(soup.body, temp_sections, chunk_token_num=chunk_token_num)
+        block_txt_list, table_list = cls.merge_block_text(temp_sections)
+        sections = cls.chunk_block(block_txt_list, chunk_token_num=chunk_token_num)
+        for table in table_list:
+            sections.append(table.get("content", ""))
         return sections
+
+    @classmethod
+    def split_table(cls, html_table, chunk_token_num=512):
+        soup = BeautifulSoup(html_table, "html.parser")
+        rows = soup.find_all("tr")
+        tables = []
+        current_table = []
+        current_count = 0
+        table_str_list = []
+        for row in rows:
+            tks_str = rag_tokenizer.tokenize(str(row))
+            token_count = len(tks_str.split(" ")) if tks_str else 0
+            if current_count + token_count > chunk_token_num:
+                tables.append(current_table)
+                current_table = []
+                current_count = 0
+            current_table.append(row)
+            current_count += token_count
+        if current_table:
+            tables.append(current_table)
+
+        for table_rows in tables:
+            new_table = soup.new_tag("table")
+            for row in table_rows:
+                new_table.append(row)
+            table_str_list.append(str(new_table))
+
+        return table_str_list
+
+    @classmethod
+    def read_text_recursively(cls, element, parser_result, chunk_token_num=512, parent_name=None, block_id=None):
+        if isinstance(element, NavigableString):
+            content = element.strip()
+
+            def is_valid_html(content):
+                try:
+                    soup = BeautifulSoup(content, "html.parser")
+                    return bool(soup.find())
+                except Exception:
+                    return False
+
+            return_info = []
+            if content:
+                if is_valid_html(content):
+                    soup = BeautifulSoup(content, "html.parser")
+                    child_info = cls.read_text_recursively(soup, parser_result, chunk_token_num, element.name, block_id)
+                    parser_result.extend(child_info)
+                else:
+                    info = {"content": element.strip(), "tag_name": "inner_text", "metadata": {"block_id": block_id}}
+                    if parent_name:
+                        info["tag_name"] = parent_name
+                    return_info.append(info)
+            return return_info
+        elif isinstance(element, Tag):
+
+            if str.lower(element.name) == "table":
+                table_info_list = []
+                table_id = str(uuid.uuid1())
+                table_list = [html.unescape(str(element))]
+                for t in table_list:
+                    table_info_list.append({"content": t, "tag_name": "table",
+                                            "metadata": {"table_id": table_id, "index": table_list.index(t)}})
+                return table_info_list
+            else:
+                block_id = None
+                if str.lower(element.name) in BLOCK_TAGS:
+                    block_id = str(uuid.uuid1())
+                for child in element.children:
+                    child_info = cls.read_text_recursively(child, parser_result, chunk_token_num, element.name,
+                                                           block_id)
+                    parser_result.extend(child_info)
+        return []
+
+    @classmethod
+    def merge_block_text(cls, parser_result):
+        block_content = []
+        current_content = ""
+        table_info_list = []
+        lask_block_id = None
+        for item in parser_result:
+            content = item.get("content")
+            tag_name = item.get("tag_name")
+            title_flag = tag_name in TITLE_TAGS
+            block_id = item.get("metadata", {}).get("block_id")
+            if block_id:
+                if title_flag:
+                    content = f"{TITLE_TAGS[tag_name]} {content}"
+                if lask_block_id != block_id:
+                    if lask_block_id is not None:
+                        block_content.append(current_content)
+                    current_content = content
+                    lask_block_id = block_id
+                else:
+                    current_content += (" " if current_content else "") + content
+            else:
+                if tag_name == "table":
+                    table_info_list.append(item)
+                else:
+                    current_content += (" " if current_content else "" + content)
+        if current_content:
+            block_content.append(current_content)
+        return block_content, table_info_list
+
+    @classmethod
+    def chunk_block(cls, block_txt_list, chunk_token_num=512):
+        chunks = []
+        current_block = ""
+        current_token_count = 0
+
+        for block in block_txt_list:
+            tks_str = rag_tokenizer.tokenize(block)
+            block_token_count = len(tks_str.split(" ")) if tks_str else 0
+            if block_token_count > chunk_token_num:
+                if current_block:
+                    chunks.append(current_block)
+                start = 0
+                tokens = tks_str.split(" ")
+                while start < len(tokens):
+                    end = start + chunk_token_num
+                    split_tokens = tokens[start:end]
+                    chunks.append(" ".join(split_tokens))
+                    start = end
+                current_block = ""
+                current_token_count = 0
+            else:
+                if current_token_count + block_token_count <= chunk_token_num:
+                    current_block += ("\n" if current_block else "") + block
+                    current_token_count += block_token_count
+                else:
+                    chunks.append(current_block)
+                    current_block = block
+                    current_token_count = block_token_count
+
+        if current_block:
+            chunks.append(current_block)
+
+        return chunks
+

deepdoc/parser/markdown_parser.py

@@ -17,8 +17,10 @@
 
 import re
 
+import mistune
 from markdown import markdown
 
+
 class RAGFlowMarkdownParser:
     def __init__(self, chunk_token_num=128):
         self.chunk_token_num = int(chunk_token_num)
@@ -35,40 +37,44 @@ class RAGFlowMarkdownParser:
                 table_list.append(raw_table)
                 if separate_tables:
                     # Skip this match (i.e., remove it)
-                    new_text += working_text[last_end:match.start()] + "\n\n"
+                    new_text += working_text[last_end : match.start()] + "\n\n"
                 else:
                     # Replace with rendered HTML
-                    html_table = markdown(raw_table, extensions=['markdown.extensions.tables']) if render else raw_table
-                    new_text += working_text[last_end:match.start()] + html_table + "\n\n"
+                    html_table = markdown(raw_table, extensions=["markdown.extensions.tables"]) if render else raw_table
+                    new_text += working_text[last_end : match.start()] + html_table + "\n\n"
                 last_end = match.end()
             new_text += working_text[last_end:]
             return new_text
 
-        if "|" in markdown_text: # for optimize performance
+        if "|" in markdown_text:  # for optimize performance
             # Standard Markdown table
             border_table_pattern = re.compile(
-                r'''
+                r"""
                 (?:\n|^)
                 (?:\|.*?\|.*?\|.*?\n)
                 (?:\|(?:\s*[:-]+[-| :]*\s*)\|.*?\n)
                 (?:\|.*?\|.*?\|.*?\n)+
-            ''', re.VERBOSE)
+            """,
+                re.VERBOSE,
+            )
             working_text = replace_tables_with_rendered_html(border_table_pattern, tables)
 
             # Borderless Markdown table
             no_border_table_pattern = re.compile(
-                r'''
+                r"""
                 (?:\n|^)
                 (?:\S.*?\|.*?\n)
                 (?:(?:\s*[:-]+[-| :]*\s*).*?\n)
                 (?:\S.*?\|.*?\n)+
-                ''', re.VERBOSE)
+                """,
+                re.VERBOSE,
+            )
             working_text = replace_tables_with_rendered_html(no_border_table_pattern, tables)
 
-        if "<table>" in working_text.lower(): # for optimize performance
-            #HTML table extraction - handle possible html/body wrapper tags
+        if "<table>" in working_text.lower():  # for optimize performance
+            # HTML table extraction - handle possible html/body wrapper tags
             html_table_pattern = re.compile(
-            r'''
+                r"""
             (?:\n|^)
             \s*
             (?:
@@ -83,9 +89,10 @@ class RAGFlowMarkdownParser:
             )
             \s*
             (?=\n|$)
-            ''',
-            re.VERBOSE | re.DOTALL | re.IGNORECASE
+            """,
+                re.VERBOSE | re.DOTALL | re.IGNORECASE,
             )
+
             def replace_html_tables():
                 nonlocal working_text
                 new_text = ""
@@ -94,9 +101,9 @@ class RAGFlowMarkdownParser:
                     raw_table = match.group()
                     tables.append(raw_table)
                     if separate_tables:
-                        new_text += working_text[last_end:match.start()] + "\n\n"
+                        new_text += working_text[last_end : match.start()] + "\n\n"
                     else:
-                        new_text += working_text[last_end:match.start()] + raw_table + "\n\n"
+                        new_text += working_text[last_end : match.start()] + raw_table + "\n\n"
                     last_end = match.end()
                 new_text += working_text[last_end:]
                 working_text = new_text
@@ -104,3 +111,163 @@ class RAGFlowMarkdownParser:
             replace_html_tables()
 
         return working_text, tables
+
+
+class MarkdownElementExtractor:
+    def __init__(self, markdown_content):
+        self.markdown_content = markdown_content
+        self.lines = markdown_content.split("\n")
+        self.ast_parser = mistune.create_markdown(renderer="ast")
+        self.ast_nodes = self.ast_parser(markdown_content)
+
+    def extract_elements(self):
+        """Extract individual elements (headers, code blocks, lists, etc.)"""
+        sections = []
+
+        i = 0
+        while i < len(self.lines):
+            line = self.lines[i]
+
+            if re.match(r"^#{1,6}\s+.*$", line):
+                # header
+                element = self._extract_header(i)
+                sections.append(element["content"])
+                i = element["end_line"] + 1
+            elif line.strip().startswith("```"):
+                # code block
+                element = self._extract_code_block(i)
+                sections.append(element["content"])
+                i = element["end_line"] + 1
+            elif re.match(r"^\s*[-*+]\s+.*$", line) or re.match(r"^\s*\d+\.\s+.*$", line):
+                # list block
+                element = self._extract_list_block(i)
+                sections.append(element["content"])
+                i = element["end_line"] + 1
+            elif line.strip().startswith(">"):
+                # blockquote
+                element = self._extract_blockquote(i)
+                sections.append(element["content"])
+                i = element["end_line"] + 1
+            elif line.strip():
+                # text block (paragraphs and inline elements until next block element)
+                element = self._extract_text_block(i)
+                sections.append(element["content"])
+                i = element["end_line"] + 1
+            else:
+                i += 1
+
+        sections = [section for section in sections if section.strip()]
+        return sections
+
+    def _extract_header(self, start_pos):
+        return {
+            "type": "header",
+            "content": self.lines[start_pos],
+            "start_line": start_pos,
+            "end_line": start_pos,
+        }
+
+    def _extract_code_block(self, start_pos):
+        end_pos = start_pos
+        content_lines = [self.lines[start_pos]]
+
+        # Find the end of the code block
+        for i in range(start_pos + 1, len(self.lines)):
+            content_lines.append(self.lines[i])
+            end_pos = i
+            if self.lines[i].strip().startswith("```"):
+                break
+
+        return {
+            "type": "code_block",
+            "content": "\n".join(content_lines),
+            "start_line": start_pos,
+            "end_line": end_pos,
+        }
+
+    def _extract_list_block(self, start_pos):
+        end_pos = start_pos
+        content_lines = []
+
+        i = start_pos
+        while i < len(self.lines):
+            line = self.lines[i]
+            # check if this line is a list item or continuation of a list
+            if (
+                re.match(r"^\s*[-*+]\s+.*$", line)
+                or re.match(r"^\s*\d+\.\s+.*$", line)
+                or (i > start_pos and not line.strip())
+                or (i > start_pos and re.match(r"^\s{2,}[-*+]\s+.*$", line))
+                or (i > start_pos and re.match(r"^\s{2,}\d+\.\s+.*$", line))
+                or (i > start_pos and re.match(r"^\s+\w+.*$", line))
+            ):
+                content_lines.append(line)
+                end_pos = i
+                i += 1
+            else:
+                break
+
+        return {
+            "type": "list_block",
+            "content": "\n".join(content_lines),
+            "start_line": start_pos,
+            "end_line": end_pos,
+        }
+
+    def _extract_blockquote(self, start_pos):
+        end_pos = start_pos
+        content_lines = []
+
+        i = start_pos
+        while i < len(self.lines):
+            line = self.lines[i]
+            if line.strip().startswith(">") or (i > start_pos and not line.strip()):
+                content_lines.append(line)
+                end_pos = i
+                i += 1
+            else:
+                break
+
+        return {
+            "type": "blockquote",
+            "content": "\n".join(content_lines),
+            "start_line": start_pos,
+            "end_line": end_pos,
+        }
+
+    def _extract_text_block(self, start_pos):
+        """Extract a text block (paragraphs, inline elements) until next block element"""
+        end_pos = start_pos
+        content_lines = [self.lines[start_pos]]
+
+        i = start_pos + 1
+        while i < len(self.lines):
+            line = self.lines[i]
+            # stop if we encounter a block element
+            if re.match(r"^#{1,6}\s+.*$", line) or line.strip().startswith("```") or re.match(r"^\s*[-*+]\s+.*$", line) or re.match(r"^\s*\d+\.\s+.*$", line) or line.strip().startswith(">"):
+                break
+            elif not line.strip():
+                # check if the next line is a block element
+                if i + 1 < len(self.lines) and (
+                    re.match(r"^#{1,6}\s+.*$", self.lines[i + 1])
+                    or self.lines[i + 1].strip().startswith("```")
+                    or re.match(r"^\s*[-*+]\s+.*$", self.lines[i + 1])
+                    or re.match(r"^\s*\d+\.\s+.*$", self.lines[i + 1])
+                    or self.lines[i + 1].strip().startswith(">")
+                ):
+                    break
+                else:
+                    content_lines.append(line)
+                    end_pos = i
+                    i += 1
+            else:
+                content_lines.append(line)
+                end_pos = i
+                i += 1
+
+        return {
+            "type": "text_block",
+            "content": "\n".join(content_lines),
+            "start_line": start_pos,
+            "end_line": end_pos,
+        }

docker/.env

@@ -93,13 +93,13 @@ REDIS_PASSWORD=infini_rag_flow
 SVR_HTTP_PORT=9380
 
 # The RAGFlow Docker image to download.
-# Defaults to the v0.20.1-slim edition, which is the RAGFlow Docker image without embedding models.
-RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1-slim
+# Defaults to the v0.20.4-slim edition, which is the RAGFlow Docker image without embedding models.
+RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4-slim
 #
 # To download the RAGFlow Docker image with embedding models, uncomment the following line instead:
-# RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1
+# RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4
 #
-# The Docker image of the v0.20.1 edition includes built-in embedding models:
+# The Docker image of the v0.20.4 edition includes built-in embedding models:
 #   - BAAI/bge-large-zh-v1.5
 #   - maidalun1020/bce-embedding-base_v1
 #

docker/README.md

@@ -79,8 +79,8 @@ The [.env](./.env) file contains important environment variables for Docker.
 - `RAGFLOW-IMAGE`  
   The Docker image edition. Available editions:  
   
-  - `infiniflow/ragflow:v0.20.1-slim` (default): The RAGFlow Docker image without embedding models.  
-  - `infiniflow/ragflow:v0.20.1`: The RAGFlow Docker image with embedding models including:
+  - `infiniflow/ragflow:v0.20.4-slim` (default): The RAGFlow Docker image without embedding models.  
+  - `infiniflow/ragflow:v0.20.4`: The RAGFlow Docker image with embedding models including:
     - Built-in embedding models:
       - `BAAI/bge-large-zh-v1.5` 
       - `maidalun1020/bce-embedding-base_v1`

docker/nginx/proxy.conf

@@ -6,3 +6,7 @@ proxy_set_header Connection "";
 proxy_buffering off;
 proxy_read_timeout 3600s;
 proxy_send_timeout 3600s;
+proxy_buffer_size 1024k;
+proxy_buffers 16 1024k;
+proxy_busy_buffers_size 2048k;
+proxy_temp_file_write_size 2048k;

docs/configurations.md

@@ -99,8 +99,8 @@ RAGFlow utilizes MinIO as its object storage solution, leveraging its scalabilit
 - `RAGFLOW-IMAGE`  
   The Docker image edition. Available editions:  
   
-  - `infiniflow/ragflow:v0.20.1-slim` (default): The RAGFlow Docker image without embedding models.  
-  - `infiniflow/ragflow:v0.20.1`: The RAGFlow Docker image with embedding models including:
+  - `infiniflow/ragflow:v0.20.4-slim` (default): The RAGFlow Docker image without embedding models.  
+  - `infiniflow/ragflow:v0.20.4`: The RAGFlow Docker image with embedding models including:
     - Built-in embedding models:
       - `BAAI/bge-large-zh-v1.5` 
       - `maidalun1020/bce-embedding-base_v1`

docs/develop/acquire_ragflow_api_key.md

@@ -11,7 +11,7 @@ An API key is required for the RAGFlow server to authenticate your HTTP/Python o
 2. Click **API** to switch to the **API** page.
 3. Obtain a RAGFlow API key:
 
-![ragflow_api_key](https://github.com/user-attachments/assets/f461ed61-04c6-4faf-b3d8-6b5fa56be4e7)
+![ragflow_api_key](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/ragflow_api_key.jpg)
 
 :::tip NOTE
 See the [RAGFlow HTTP API reference](../references/http_api_reference.md) or the [RAGFlow Python API reference](../references/python_api_reference.md) for a complete reference of RAGFlow's HTTP or Python APIs.

docs/develop/build_docker_image.mdx

@@ -77,7 +77,7 @@ After building the infiniflow/ragflow:nightly-slim image, you are ready to launc
 
 1. Edit Docker Compose Configuration
 
-Open the `docker/.env` file. Find the `RAGFLOW_IMAGE` setting and change the image reference from `infiniflow/ragflow:v0.20.1-slim` to `infiniflow/ragflow:nightly-slim` to use the pre-built image.
+Open the `docker/.env` file. Find the `RAGFLOW_IMAGE` setting and change the image reference from `infiniflow/ragflow:v0.20.4-slim` to `infiniflow/ragflow:nightly-slim` to use the pre-built image.
 
 
 2. Launch the Service

docs/faq.mdx

@@ -30,17 +30,17 @@ The "garbage in garbage out" status quo remains unchanged despite the fact that
 
 Each RAGFlow release is available in two editions:
 
-- **Slim edition**: excludes built-in embedding models and is identified by a **-slim** suffix added to the version name. Example: `infiniflow/ragflow:v0.20.1-slim`
-- **Full edition**: includes built-in embedding models and has no suffix added to the version name. Example: `infiniflow/ragflow:v0.20.1`
+- **Slim edition**: excludes built-in embedding models and is identified by a **-slim** suffix added to the version name. Example: `infiniflow/ragflow:v0.20.4-slim`
+- **Full edition**: includes built-in embedding models and has no suffix added to the version name. Example: `infiniflow/ragflow:v0.20.4`
 
 ---
 
 ### Which embedding models can be deployed locally?
 
-RAGFlow offers two Docker image editions, `v0.20.1-slim` and `v0.20.1`:  
+RAGFlow offers two Docker image editions, `v0.20.4-slim` and `v0.20.4`:  
   
-- `infiniflow/ragflow:v0.20.1-slim` (default): The RAGFlow Docker image without embedding models.  
-- `infiniflow/ragflow:v0.20.1`: The RAGFlow Docker image with embedding models including:
+- `infiniflow/ragflow:v0.20.4-slim` (default): The RAGFlow Docker image without embedding models.  
+- `infiniflow/ragflow:v0.20.4`: The RAGFlow Docker image with embedding models including:
   - Built-in embedding models:
     - `BAAI/bge-large-zh-v1.5`
     - `maidalun1020/bce-embedding-base_v1`

docs/guides/agent/agent_component_reference/agent.mdx

@@ -9,7 +9,7 @@ The component equipped with reasoning, tool usage, and multi-agent collaboration
 
 ---
 
-An **Agent** component fine-tunes the LLM and sets its prompt. From v0.20.1 onwards, an **Agent** component is able to work independently and with the following capabilities:
+An **Agent** component fine-tunes the LLM and sets its prompt. From v0.20.4 onwards, an **Agent** component is able to work independently and with the following capabilities:
 
 - Autonomous reasoning with reflection and adjustment based on environmental feedback.
 - Use of tools or subagents to complete tasks.

docs/guides/agent/agent_component_reference/retrieval.mdx

@@ -9,7 +9,7 @@ A component that retrieves information from specified datasets.
 
 ## Scenarios
 
-A **Retrieval** component is essential in most RAG scenarios, where information is extracted from designated knowledge bases before being sent to the LLM for content generation. As of v0.20.1, a **Retrieval** component can operate either as a workflow component or as a tool of an **Agent**, enabling the Agent to control its invocation and search queries.
+A **Retrieval** component is essential in most RAG scenarios, where information is extracted from designated knowledge bases before being sent to the LLM for content generation. As of v0.20.4, a **Retrieval** component can operate either as a workflow component or as a tool of an **Agent**, enabling the Agent to control its invocation and search queries.
 
 ## Configurations
 

docs/guides/chat/start_chat.md

@@ -48,7 +48,7 @@ You start an AI conversation by creating an assistant.
      - If no target language is selected, the system will search only in the language of your query, which may cause relevant information in other languages to be missed.
    - **Variable** refers to the variables (keys) to be used in the system prompt. `{knowledge}` is a reserved variable. Click **Add** to add more variables for the system prompt.
       - If you are uncertain about the logic behind **Variable**, leave it *as-is*.
-      - As of v0.20.1, if you add custom variables here, the only way you can pass in their values is to call:
+      - As of v0.20.4, if you add custom variables here, the only way you can pass in their values is to call:
          - HTTP method [Converse with chat assistant](../../references/http_api_reference.md#converse-with-chat-assistant), or
          - Python method [Converse with chat assistant](../../references/python_api_reference.md#converse-with-chat-assistant).
 

docs/guides/dataset/configure_knowledge_base.md

@@ -128,7 +128,7 @@ See [Run retrieval test](./run_retrieval_test.md) for details.
 
 ## Search for knowledge base
 
-As of RAGFlow v0.20.1, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
+As of RAGFlow v0.20.4, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
 
 ![search knowledge base](https://github.com/infiniflow/ragflow/assets/93570324/836ae94c-2438-42be-879e-c7ad2a59693e)
 

docs/guides/manage_files.md

@@ -87,4 +87,4 @@ RAGFlow's file management allows you to download an uploaded file:
 
 ![download_file](https://github.com/infiniflow/ragflow/assets/93570324/cf3b297f-7d9b-4522-bf5f-4f45743e4ed5)
 
-> As of RAGFlow v0.20.1, bulk download is not supported, nor can you download an entire folder. 
+> As of RAGFlow v0.20.4, bulk download is not supported, nor can you download an entire folder. 

docs/guides/tracing.mdx

@@ -18,7 +18,7 @@ RAGFlow ships with a built-in [Langfuse](https://langfuse.com) integration so th
 Langfuse stores traces, spans and prompt payloads in a purpose-built observability backend and offers filtering and visualisations on top.  
 
 :::info NOTE
-• RAGFlow **≥ 0.20.1** (contains the Langfuse connector)  
+• RAGFlow **≥ 0.20.4** (contains the Langfuse connector)  
 • A Langfuse workspace (cloud or self-hosted) with a _Project Public Key_ and _Secret Key_
 :::
 

docs/guides/upgrade_ragflow.mdx

@@ -66,10 +66,10 @@ To upgrade RAGFlow, you must upgrade **both** your code **and** your Docker imag
    git clone https://github.com/infiniflow/ragflow.git
    ```
 
-2. Switch to the latest, officially published release, e.g., `v0.20.1`:
+2. Switch to the latest, officially published release, e.g., `v0.20.4`:
 
    ```bash
-   git checkout -f v0.20.1
+   git checkout -f v0.20.4
    ```
 
 3. Update **ragflow/docker/.env**:
@@ -83,14 +83,14 @@ To upgrade RAGFlow, you must upgrade **both** your code **and** your Docker imag
   <TabItem value="slim">
 
 ```bash
-RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1-slim
+RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4-slim
 ```
 
   </TabItem>
   <TabItem value="full">
 
 ```bash
-RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1
+RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4
 ```
 
   </TabItem>
@@ -114,10 +114,10 @@ No, you do not need to. Upgrading RAGFlow in itself will *not* remove your uploa
 1. From an environment with Internet access, pull the required Docker image.
 2. Save the Docker image to a **.tar** file.
    ```bash
-   docker save -o ragflow.v0.20.1.tar infiniflow/ragflow:v0.20.1
+   docker save -o ragflow.v0.20.4.tar infiniflow/ragflow:v0.20.4
    ```
 3. Copy the **.tar** file to the target server.
 4. Load the **.tar** file into Docker:
    ```bash
-   docker load -i ragflow.v0.20.1.tar
+   docker load -i ragflow.v0.20.4.tar
    ```

docs/quickstart.mdx

@@ -44,7 +44,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
 
    `vm.max_map_count`. This value sets the maximum number of memory map areas a process may have. Its default value is 65530. While most applications require fewer than a thousand maps, reducing this value can result in abnormal behaviors, and the system will throw out-of-memory errors when a process reaches the limitation.
 
-   RAGFlow v0.20.1 uses Elasticsearch or [Infinity](https://github.com/infiniflow/infinity) for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning of the Elasticsearch component.
+   RAGFlow v0.20.4 uses Elasticsearch or [Infinity](https://github.com/infiniflow/infinity) for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning of the Elasticsearch component.
 
 <Tabs
   defaultValue="linux"
@@ -184,13 +184,13 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
    ```bash
    $ git clone https://github.com/infiniflow/ragflow.git
    $ cd ragflow/docker
-   $ git checkout -f v0.20.1
+   $ git checkout -f v0.20.4
    ```
 
 3. Use the pre-built Docker images and start up the server:
 
    :::tip NOTE
-   The command below downloads the `v0.20.1-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.20.1-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.1` for the full edition `v0.20.1`.
+   The command below downloads the `v0.20.4-slim` edition of the RAGFlow Docker image. Refer to the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.20.4-slim`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. For example: set `RAGFLOW_IMAGE=infiniflow/ragflow:v0.20.4` for the full edition `v0.20.4`.
    :::
 
    ```bash
@@ -207,8 +207,8 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
 
 | RAGFlow image tag   | Image size (GB) | Has embedding models and Python packages? | Stable?                  |
 | ------------------- | --------------- | ----------------------------------------- | ------------------------ |
-| `v0.20.1`           | &approx;9       | :heavy_check_mark:                        | Stable release           |
-| `v0.20.1-slim`      | &approx;2       | ❌                                        | Stable release           |
+| `v0.20.4`           | &approx;9       | :heavy_check_mark:                        | Stable release           |
+| `v0.20.4-slim`      | &approx;2       | ❌                                        | Stable release           |
 | `nightly`           | &approx;9       | :heavy_check_mark:                        | *Unstable* nightly build |
 | `nightly-slim`      | &approx;2       | ❌                                        | *Unstable* nightly build |
 
@@ -217,7 +217,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
 ```
 
 :::danger IMPORTANT
-The embedding models included in `v0.20.1` and `nightly` are:
+The embedding models included in `v0.20.4` and `nightly` are:
 
 - BAAI/bge-large-zh-v1.5
 - maidalun1020/bce-embedding-base_v1

docs/references/glossary.mdx

@@ -19,7 +19,7 @@ import TOCInline from '@theme/TOCInline';
 
 ### Cross-language search
 
-Cross-language search (also known as cross-lingual retrieval) is a feature introduced in version 0.20.1. It enables users to submit queries in one language (for example, English) and retrieve relevant documents written in other languages such as Chinese or Spanish. This feature is enabled by the system’s default chat model, which translates queries to ensure accurate matching of semantic meaning across languages.
+Cross-language search (also known as cross-lingual retrieval) is a feature introduced in version 0.20.4. It enables users to submit queries in one language (for example, English) and retrieve relevant documents written in other languages such as Chinese or Spanish. This feature is enabled by the system’s default chat model, which translates queries to ensure accurate matching of semantic meaning across languages.
 
 By enabling cross-language search, users can effortlessly access a broader range of information regardless of language barriers, significantly enhancing the system’s usability and inclusiveness.
 

docs/references/python_api_reference.md

@@ -5,7 +5,7 @@ slug: /python_api_reference
 
 # Python API
 
-A complete reference for RAGFlow's Python APIs. Before proceeding, please ensure you [have your RAGFlow API key ready for authentication](../guides/models/llm_api_key_setup.md).
+A complete reference for RAGFlow's Python APIs. Before proceeding, please ensure you [have your RAGFlow API key ready for authentication](https://ragflow.io/docs/dev/acquire_ragflow_api_key).
 
 :::tip NOTE
 Run the following command to download the Python SDK:

docs/release_notes.md

@@ -9,8 +9,8 @@ Key features, improvements and bug fixes in the latest releases.
 
 :::info
 Each RAGFlow release is available in two editions:
-- **Slim edition**: excludes built-in embedding models and is identified by a **-slim** suffix added to the version name. Example: `infiniflow/ragflow:v0.20.1-slim`
-- **Full edition**: includes built-in embedding models and has no suffix added to the version name. Example: `infiniflow/ragflow:v0.20.1`
+- **Slim edition**: excludes built-in embedding models and is identified by a **-slim** suffix added to the version name. Example: `infiniflow/ragflow:v0.20.4-slim`
+- **Full edition**: includes built-in embedding models and has no suffix added to the version name. Example: `infiniflow/ragflow:v0.20.4`
 :::
 
 :::danger IMPORTANT
@@ -22,6 +22,70 @@ The embedding models included in a full edition are:
 These two embedding models are optimized specifically for English and Chinese, so performance may be compromised if you use them to embed documents in other languages.
 :::
 
+## v0.20.4
+
+Released on August 27, 2025.
+
+### Improvements
+
+- Agent component: Completes Chinese localization for the Agent component.
+- Introduces the `ENABLE_TIMEOUT_ASSERTION` environment variable to enable or disable timeout assertions for file parsing tasks.
+- Dataset:
+  - Improves Markdown file parsing, with AST support to avoid unintended chunking.
+  - Enhances HTML parsing, supporting bs4-based HTML tag traversal.
+
+### Added models
+
+ZHIPU GLM-4.5
+
+### New Agent templates
+
+Ecommerce Customer Service Workflow: A template designed to handle enquiries about product features and multi-product comparisons using the internal knowledge base, as well as to manage installation appointment bookings.
+
+### Fixed issues
+
+- Dataset:  
+  - Unable to share resources with the team.
+  - Inappropriate restrictions on the number and size of uploaded files.
+- Chat:
+  - Unable to preview referenced files in responses.
+  - Unable to send out messages after file uploads.
+- An OAuth2 authentication failure.
+- A logical error in multi-conditioned metadata searches within a dataset.
+- Citations infinitely increased in multi-turn conversations.
+
+## v0.20.3
+
+Released on August 20, 2025.
+
+### Improvements
+
+- Revamps the user interface for the **Datasets**, **Chat**, and **Search** pages.  
+- Search and Chat: Introduces document-level metadata filtering, allowing automatic or manual filtering during chats or searches.
+- Search: Supports creating search apps tailored to various business scenarios
+- Chat: Supports comparing answer performance of up to three chat model settings on a single **Chat** page.
+- Agent:  
+  - Implements a toggle in the **Agent** component to enable or disable citation.  
+  - Introduces a drag-and-drop method for creating components.  
+- Documentation: Corrects inaccuracies in the API reference.
+
+### New Agent templates
+
+- Report Agent: A template for generating summary reports in internal question-answering scenarios, supporting the display of tables and formulae.  [#9427](https://github.com/infiniflow/ragflow/pull/9427)
+
+### Fixed issues
+
+- The timeout mechanism introduced in v0.20.0 caused tasks like GraphRAG to halt.
+- Predefined opening greeting in the **Agent** component was missing during conversations.  
+- An automatic line break issue in the prompt editor.  
+- A memory leak issue caused by PyPDF. [#9469](https://github.com/infiniflow/ragflow/pull/9469)
+
+### API changes
+
+#### Deprecated
+
+[Create session with agent](./references/http_api_reference.md#create-session-with-agent)
+
 ## v0.20.1
 
 Released on August 8, 2025.
@@ -182,7 +246,7 @@ From this release onwards, if you still see RAGFlow's responses being cut short
 
 - Unable to add models via Ollama/Xinference, an issue introduced in v0.17.1.
 
-### Related APIs
+### API changes
 
 #### HTTP APIs
 
@@ -243,7 +307,7 @@ The following is a screenshot of a conversation that integrates Deep Research:
 
 ![Image](https://github.com/user-attachments/assets/165b88ff-1f5d-4fb8-90e2-c836b25e32e9)
 
-### Related APIs
+### API changes
 
 #### HTTP APIs
 
@@ -318,7 +382,7 @@ This release fixes the following issues:
 - Using the **Table** parsing method results in information loss.
 - Miscellaneous API issues.
 
-### Related APIs
+### API changes
 
 #### HTTP APIs
 
@@ -354,7 +418,7 @@ Released on December 18, 2024.
 - Upgrades the Document Layout Analysis model in DeepDoc.
 - Significantly enhances the retrieval performance when using [Infinity](https://github.com/infiniflow/infinity) as document engine.
 
-### Related APIs
+### API changes
 
 #### HTTP APIs
 
@@ -411,7 +475,7 @@ This approach eliminates the need to manually update **service_config.yaml** aft
 Ensure that you [upgrade **both** your code **and** Docker image to this release](https://ragflow.io/docs/dev/upgrade_ragflow#upgrade-ragflow-to-the-most-recent-officially-published-release) before trying this new approach.
 :::
 
-### Related APIs
+### API changes
 
 #### HTTP APIs
 
@@ -570,7 +634,7 @@ While we also test RAGFlow on ARM64 platforms, we do not maintain RAGFlow Docker
 If you are on an ARM platform, follow [this guide](./develop/build_docker_image.mdx) to build a RAGFlow Docker image.
 :::
 
-### Related APIs
+### API changes
 
 #### HTTP API
 
@@ -591,7 +655,7 @@ Released on May 21, 2024.
 - Supports monitoring of system components, including Elasticsearch, MySQL, Redis, and MinIO.
 - Supports disabling **Layout Recognition** in the GENERAL chunking method to reduce file chunking time.
 
-### Related APIs
+### API changes
 
 #### HTTP API
 

graphrag/entity_resolution.py

@@ -15,6 +15,7 @@
 #
 import logging
 import itertools
+import os
 import re
 from dataclasses import dataclass
 from typing import Any, Callable
@@ -106,7 +107,8 @@ class EntityResolution(Extractor):
             nonlocal remain_candidates_to_resolve, callback
             async with semaphore:
                 try:
-                    with trio.move_on_after(180) as cancel_scope:
+                    enable_timeout_assertion = os.environ.get("ENABLE_TIMEOUT_ASSERTION")
+                    with trio.move_on_after(280 if enable_timeout_assertion else 1000000000) as cancel_scope:
                         await self._resolve_candidate(candidate_batch, result_set, result_lock)
                         remain_candidates_to_resolve = remain_candidates_to_resolve - len(candidate_batch[1])
                         callback(msg=f"Resolved {len(candidate_batch[1])} pairs, {remain_candidates_to_resolve} are remained to resolve. ")
@@ -169,7 +171,8 @@ class EntityResolution(Extractor):
         logging.info(f"Created resolution prompt {len(text)} bytes for {len(candidate_resolution_i[1])} entity pairs of type {candidate_resolution_i[0]}")
         async with chat_limiter:
             try:
-                with trio.move_on_after(120) as cancel_scope:
+                enable_timeout_assertion = os.environ.get("ENABLE_TIMEOUT_ASSERTION")
+                with trio.move_on_after(280 if enable_timeout_assertion else 1000000000) as cancel_scope:
                     response = await trio.to_thread.run_sync(self._chat, text, [{"role": "user", "content": "Output:"}], {})
                 if cancel_scope.cancelled_caught:
                     logging.warning("_resolve_candidate._chat timeout, skipping...")

graphrag/general/community_reports_extractor.py

@@ -7,6 +7,7 @@ Reference:
 
 import logging
 import json
+import os
 import re
 from typing import Callable
 from dataclasses import dataclass
@@ -51,6 +52,7 @@ class CommunityReportsExtractor(Extractor):
         self._max_report_length = max_report_length or 1500
 
     async def __call__(self, graph: nx.Graph, callback: Callable | None = None):
+        enable_timeout_assertion = os.environ.get("ENABLE_TIMEOUT_ASSERTION")
         for node_degree in graph.degree:
             graph.nodes[str(node_degree[0])]["rank"] = int(node_degree[1])
 
@@ -92,7 +94,7 @@ class CommunityReportsExtractor(Extractor):
             text = perform_variable_replacements(self._extraction_prompt, variables=prompt_variables)
             async with chat_limiter:
                 try:
-                    with trio.move_on_after(80) as cancel_scope:
+                    with trio.move_on_after(180 if enable_timeout_assertion else 1000000000) as cancel_scope:
                         response = await trio.to_thread.run_sync( self._chat, text, [{"role": "user", "content": "Output:"}], {})
                     if cancel_scope.cancelled_caught:
                         logging.warning("extract_community_report._chat timeout, skipping...")

graphrag/general/extractor.py

@@ -47,7 +47,7 @@ class Extractor:
         self._language = language
         self._entity_types = entity_types or DEFAULT_ENTITY_TYPES
 
-    @timeout(60*5)
+    @timeout(60*20)
     def _chat(self, system, history, gen_conf={}):
         hist = deepcopy(history)
         conf = deepcopy(gen_conf)

graphrag/general/index.py

@@ -15,6 +15,8 @@
 #
 import json
 import logging
+import os
+
 import networkx as nx
 import trio
 
@@ -49,6 +51,7 @@ async def run_graphrag(
     embedding_model,
     callback,
 ):
+    enable_timeout_assertion=os.environ.get("ENABLE_TIMEOUT_ASSERTION")
     start = trio.current_time()
     tenant_id, kb_id, doc_id = row["tenant_id"], str(row["kb_id"]), row["doc_id"]
     chunks = []
@@ -57,20 +60,22 @@ async def run_graphrag(
     ):
         chunks.append(d["content_with_weight"])
 
-    subgraph = await generate_subgraph(
-        LightKGExt
-        if "method" not in row["kb_parser_config"].get("graphrag", {}) or row["kb_parser_config"]["graphrag"]["method"] != "general"
-        else GeneralKGExt,
-        tenant_id,
-        kb_id,
-        doc_id,
-        chunks,
-        language,
-        row["kb_parser_config"]["graphrag"].get("entity_types", []),
-        chat_model,
-        embedding_model,
-        callback,
-    )
+    with trio.fail_after(max(120, len(chunks)*60*10) if enable_timeout_assertion else 10000000000):
+        subgraph = await generate_subgraph(
+            LightKGExt
+            if "method" not in row["kb_parser_config"].get("graphrag", {}) or row["kb_parser_config"]["graphrag"]["method"] != "general"
+            else GeneralKGExt,
+            tenant_id,
+            kb_id,
+            doc_id,
+            chunks,
+            language,
+            row["kb_parser_config"]["graphrag"].get("entity_types", []),
+            chat_model,
+            embedding_model,
+            callback,
+        )
+
     if not subgraph:
         return
 
@@ -125,7 +130,6 @@ async def run_graphrag(
     return
 
 
-@timeout(60*60, 1)
 async def generate_subgraph(
     extractor: Extractor,
     tenant_id: str,

graphrag/light/graph_prompt.py

@@ -130,7 +130,36 @@ Output:
 
 PROMPTS[
     "entiti_continue_extraction"
-] = """MANY entities were missed in the last extraction.  Add them below using the same format:
+] = """
+MANY entities and relationships were missed in the last extraction. Please find only the missing entities and relationships from previous text.
+
+---Remember Steps---
+
+1. Identify all entities. For each identified entity, extract the following information:
+- entity_name: Name of the entity, use same language as input text. If English, capitalized the name
+- entity_type: One of the following types: [{entity_types}]
+- entity_description: Provide a comprehensive description of the entity's attributes and activities *based solely on the information present in the input text*. **Do not infer or hallucinate information not explicitly stated.** If the text provides insufficient information to create a comprehensive description, state "Description not available in text."
+Format each entity as ("entity"{tuple_delimiter}<entity_name>{tuple_delimiter}<entity_type>{tuple_delimiter}<entity_description>)
+
+2. From the entities identified in step 1, identify all pairs of (source_entity, target_entity) that are *clearly related* to each other.
+For each pair of related entities, extract the following information:
+- source_entity: name of the source entity, as identified in step 1
+- target_entity: name of the target entity, as identified in step 1
+- relationship_description: explanation as to why you think the source entity and the target entity are related to each other
+- relationship_strength: a numeric score indicating strength of the relationship between the source entity and target entity
+- relationship_keywords: one or more high-level key words that summarize the overarching nature of the relationship, focusing on concepts or themes rather than specific details
+Format each relationship as ("relationship"{tuple_delimiter}<source_entity>{tuple_delimiter}<target_entity>{tuple_delimiter}<relationship_description>{tuple_delimiter}<relationship_keywords>{tuple_delimiter}<relationship_strength>)
+
+3. Identify high-level key words that summarize the main concepts, themes, or topics of the entire text. These should capture the overarching ideas present in the document.
+Format the content-level key words as ("content_keywords"{tuple_delimiter}<high_level_keywords>)
+
+4. Return output in {language} as a single list of all the entities and relationships identified in steps 1 and 2. Use **{record_delimiter}** as the list delimiter.
+
+5. When finished, output {completion_delimiter}
+
+---Output---
+
+Add new entities and relations below using the same format, and do not include entities and relations that have been previously extracted. :
 """
 
 PROMPTS[
@@ -252,4 +281,4 @@ When handling information with timestamps:
 - List up to 5 most important reference sources at the end under "References", clearly indicating whether each source is from Knowledge Graph (KG) or Vector Data (VD)
   Format: [KG/VD] Source content
 
-Add sections and commentary to the response as appropriate for the length and format. If the provided information is insufficient to answer the question, clearly state that you don't know or cannot provide an answer in the same language as the user's question."""
+Add sections and commentary to the response as appropriate for the length and format. If the provided information is insufficient to answer the question, clearly state that you don't know or cannot provide an answer in the same language as the user's question."""

graphrag/utils.py

@@ -307,6 +307,7 @@ def chunk_id(chunk):
 
 async def graph_node_to_chunk(kb_id, embd_mdl, ent_name, meta, chunks):
     global chat_limiter
+    enable_timeout_assertion=os.environ.get("ENABLE_TIMEOUT_ASSERTION")
     chunk = {
         "id": get_uuid(),
         "important_kwd": [ent_name],
@@ -324,7 +325,7 @@ async def graph_node_to_chunk(kb_id, embd_mdl, ent_name, meta, chunks):
     ebd = get_embed_cache(embd_mdl.llm_name, ent_name)
     if ebd is None:
         async with chat_limiter:
-            with trio.fail_after(3):
+            with trio.fail_after(3 if enable_timeout_assertion else 30000000):
                 ebd, _ = await trio.to_thread.run_sync(lambda: embd_mdl.encode([ent_name]))
         ebd = ebd[0]
         set_embed_cache(embd_mdl.llm_name, ent_name, ebd)
@@ -362,6 +363,7 @@ def get_relation(tenant_id, kb_id, from_ent_name, to_ent_name, size=1):
 
 
 async def graph_edge_to_chunk(kb_id, embd_mdl, from_ent_name, to_ent_name, meta, chunks):
+    enable_timeout_assertion=os.environ.get("ENABLE_TIMEOUT_ASSERTION")
     chunk = {
         "id": get_uuid(),
         "from_entity_kwd": from_ent_name,
@@ -380,7 +382,7 @@ async def graph_edge_to_chunk(kb_id, embd_mdl, from_ent_name, to_ent_name, meta,
     ebd = get_embed_cache(embd_mdl.llm_name, txt)
     if ebd is None:
         async with chat_limiter:
-            with trio.fail_after(3):
+            with trio.fail_after(3 if enable_timeout_assertion else 300000000):
                 ebd, _ = await trio.to_thread.run_sync(lambda: embd_mdl.encode([txt+f": {meta['description']}"]))
         ebd = ebd[0]
         set_embed_cache(embd_mdl.llm_name, txt, ebd)
@@ -514,9 +516,10 @@ async def set_graph(tenant_id: str, kb_id: str, embd_mdl, graph: nx.Graph, chang
         callback(msg=f"set_graph converted graph change to {len(chunks)} chunks in {now - start:.2f}s.")
     start = now
 
+    enable_timeout_assertion=os.environ.get("ENABLE_TIMEOUT_ASSERTION")
     es_bulk_size = 4
     for b in range(0, len(chunks), es_bulk_size):
-        with trio.fail_after(3):
+        with trio.fail_after(3 if enable_timeout_assertion else 30000000):
             doc_store_result = await trio.to_thread.run_sync(lambda: settings.docStoreConn.insert(chunks[b:b + es_bulk_size], search.index_name(tenant_id), kb_id))
         if b % 100 == es_bulk_size and callback:
             callback(msg=f"Insert chunks: {b}/{len(chunks)}")

helm/templates/elasticsearch.yaml

@@ -44,9 +44,21 @@ spec:
         checksum/config-es: {{ include (print $.Template.BasePath "/elasticsearch-config.yaml") . | sha256sum }}
         checksum/config-env: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.elasticsearch.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.elasticsearch.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       initContainers:
       - name: fix-data-volume-permissions
-        image: alpine
+        image: {{ .Values.elasticsearch.initContainers.alpine.repository }}:{{ .Values.elasticsearch.initContainers.alpine.tag }}
+        {{- with .Values.elasticsearch.initContainers.alpine.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         command:
         - sh
         - -c
@@ -55,14 +67,20 @@ spec:
           - mountPath: /usr/share/elasticsearch/data
             name: es-data
       - name: sysctl
-        image: busybox
+        image: {{ .Values.elasticsearch.initContainers.busybox.repository }}:{{ .Values.elasticsearch.initContainers.busybox.tag }}
+        {{- with .Values.elasticsearch.initContainers.busybox.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         securityContext:
           privileged: true
           runAsUser: 0
         command: ["sysctl", "-w", "vm.max_map_count=262144"]
       containers:
       - name: elasticsearch
-        image: elasticsearch:{{ .Values.env.STACK_VERSION }}
+        image: {{ .Values.elasticsearch.image.repository }}:{{ .Values.elasticsearch.image.tag }}
+        {{- with .Values.elasticsearch.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         envFrom:
           - secretRef:
               name: {{ include "ragflow.fullname" . }}-env-config

helm/templates/infinity.yaml

@@ -43,9 +43,21 @@ spec:
       annotations:
         checksum/config: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.infinity.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.infinity.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       containers:
       - name: infinity
         image: {{ .Values.infinity.image.repository }}:{{ .Values.infinity.image.tag }}
+        {{- with .Values.infinity.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         envFrom:
           - secretRef:
               name: {{ include "ragflow.fullname" . }}-env-config

helm/templates/minio.yaml

@@ -43,9 +43,21 @@ spec:
         {{- include "ragflow.labels" . | nindent 8 }}
         app.kubernetes.io/component: minio
     spec:
+      {{- if or .Values.imagePullSecrets .Values.minio.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.minio.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       containers:
       - name: minio
         image: {{ .Values.minio.image.repository }}:{{ .Values.minio.image.tag }}
+        {{- with .Values.minio.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         envFrom:
           - secretRef:
               name: {{ include "ragflow.fullname" . }}-env-config

helm/templates/mysql.yaml

@@ -44,9 +44,21 @@ spec:
         checksum/config-mysql: {{ include (print $.Template.BasePath "/mysql-config.yaml") . | sha256sum }}
         checksum/config-env: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.mysql.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.mysql.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       containers:
       - name: mysql
         image: {{ .Values.mysql.image.repository }}:{{ .Values.mysql.image.tag }}
+        {{- with .Values.mysql.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         envFrom:
           - secretRef:
               name: {{ include "ragflow.fullname" . }}-env-config

helm/templates/opensearch.yaml

@@ -44,9 +44,21 @@ spec:
         checksum/config-opensearch: {{ include (print $.Template.BasePath "/opensearch-config.yaml") . | sha256sum }}
         checksum/config-env: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.opensearch.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.opensearch.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       initContainers:
       - name: fix-data-volume-permissions
-        image: alpine
+        image: {{ .Values.opensearch.initContainers.alpine.repository }}:{{ .Values.opensearch.initContainers.alpine.tag }}
+        {{- with .Values.opensearch.initContainers.alpine.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         command:
         - sh
         - -c
@@ -55,7 +67,10 @@ spec:
           - mountPath: /usr/share/opensearch/data
             name: opensearch-data
       - name: sysctl
-        image: busybox
+        image: {{ .Values.opensearch.initContainers.busybox.repository }}:{{ .Values.opensearch.initContainers.busybox.tag }}
+        {{- with .Values.opensearch.initContainers.busybox.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         securityContext:
           privileged: true
           runAsUser: 0
@@ -63,6 +78,9 @@ spec:
       containers:
       - name: opensearch
         image: {{ .Values.opensearch.image.repository }}:{{ .Values.opensearch.image.tag }}
+        {{- with .Values.opensearch.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         envFrom:
           - secretRef:
               name: {{ include "ragflow.fullname" . }}-env-config

helm/templates/ragflow.yaml

@@ -25,9 +25,21 @@ spec:
         checksum/config-env: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
         checksum/config-ragflow: {{ include (print $.Template.BasePath "/ragflow_config.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.ragflow.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.ragflow.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       containers:
       - name: ragflow
-        image: {{ .Values.env.RAGFLOW_IMAGE }}
+        image: {{ .Values.ragflow.image.repository }}:{{ .Values.ragflow.image.tag }}
+        {{- with .Values.ragflow.image.pullPolicy }}
+        imagePullPolicy: {{ . }}
+        {{- end }}
         ports:
           - containerPort: 80
             name: http

helm/templates/redis.yaml

@@ -40,10 +40,22 @@ spec:
       annotations:
         checksum/config-env: {{ include (print $.Template.BasePath "/env.yaml") . | sha256sum }}
     spec:
+      {{- if or .Values.imagePullSecrets .Values.redis.image.pullSecrets }}
+      imagePullSecrets:
+        {{- with .Values.imagePullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- with .Values.redis.image.pullSecrets }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      {{- end }}
       terminationGracePeriodSeconds: 60
       containers:
         - name: redis
           image: {{ .Values.redis.image.repository }}:{{ .Values.redis.image.tag }}
+          {{- with .Values.redis.image.pullPolicy }}
+          imagePullPolicy: {{ . }}
+          {{- end }}
           command:
             - "sh"
             - "-c"

helm/values.yaml

@@ -1,4 +1,8 @@
 # Based on docker compose .env file
+
+# Global image pull secrets configuration
+imagePullSecrets: []
+
 env:
   # The type of doc engine to use.
   # Available options:
@@ -32,31 +36,6 @@ env:
   # The password for Redis
   REDIS_PASSWORD: infini_rag_flow_helm
 
-  # The RAGFlow Docker image to download.
-  # Defaults to the v0.20.1-slim edition, which is the RAGFlow Docker image without embedding models.
-  RAGFLOW_IMAGE: infiniflow/ragflow:v0.20.1-slim
-  #
-  # To download the RAGFlow Docker image with embedding models, uncomment the following line instead:
-  # RAGFLOW_IMAGE: infiniflow/ragflow:v0.20.1
-  #
-  # The Docker image of the v0.20.1 edition includes:
-  # - Built-in embedding models:
-  #   - BAAI/bge-large-zh-v1.5
-  #   - BAAI/bge-reranker-v2-m3
-  #   - maidalun1020/bce-embedding-base_v1
-  #   - maidalun1020/bce-reranker-base_v1
-  # - Embedding models that will be downloaded once you select them in the RAGFlow UI:
-  #   - BAAI/bge-base-en-v1.5
-  #   - BAAI/bge-large-en-v1.5
-  #   - BAAI/bge-small-en-v1.5
-  #   - BAAI/bge-small-zh-v1.5
-  #   - jinaai/jina-embeddings-v2-base-en
-  #   - jinaai/jina-embeddings-v2-small-en
-  #   - nomic-ai/nomic-embed-text-v1.5
-  #   - sentence-transformers/all-MiniLM-L6-v2
-  #
-  #
-
   # The local time zone.
   TIMEZONE: "Asia/Shanghai"
 
@@ -75,7 +54,11 @@ env:
   EMBEDDING_BATCH_SIZE: 16
 
 ragflow:
-
+  image:
+    repository: infiniflow/ragflow
+    tag: v0.20.4-slim
+    pullPolicy: IfNotPresent
+    pullSecrets: []
   # Optional service configuration overrides
   # to be written to local.service_conf.yaml
   # inside the RAGFlow container
@@ -114,6 +97,8 @@ infinity:
   image:
     repository: infiniflow/infinity
     tag: v0.6.0-dev5
+    pullPolicy: IfNotPresent
+    pullSecrets: []
   storage:
     className:
     capacity: 5Gi
@@ -124,6 +109,20 @@ infinity:
     type: ClusterIP
 
 elasticsearch:
+  image:
+    repository: elasticsearch
+    tag: "8.11.3"
+    pullPolicy: IfNotPresent
+    pullSecrets: []
+  initContainers:
+    alpine:
+      repository: alpine
+      tag: latest
+      pullPolicy: IfNotPresent
+    busybox:
+      repository: busybox
+      tag: latest
+      pullPolicy: IfNotPresent
   storage:
     className:
     capacity: 20Gi
@@ -140,6 +139,17 @@ opensearch:
   image:
     repository: opensearchproject/opensearch
     tag: 2.19.1
+    pullPolicy: IfNotPresent
+    pullSecrets: []
+  initContainers:
+    alpine:
+      repository: alpine
+      tag: latest
+      pullPolicy: IfNotPresent
+    busybox:
+      repository: busybox
+      tag: latest
+      pullPolicy: IfNotPresent
   storage:
     className:
     capacity: 20Gi
@@ -156,6 +166,8 @@ minio:
   image:
     repository: quay.io/minio/minio
     tag: RELEASE.2023-12-20T01-00-02Z
+    pullPolicy: IfNotPresent
+    pullSecrets: []
   storage:
     className:
     capacity: 5Gi
@@ -169,6 +181,8 @@ mysql:
   image:
     repository: mysql
     tag: 8.0.39
+    pullPolicy: IfNotPresent
+    pullSecrets: []
   storage:
     className:
     capacity: 5Gi
@@ -182,6 +196,8 @@ redis:
   image:
     repository: valkey/valkey
     tag: 8
+    pullPolicy: IfNotPresent
+    pullSecrets: []
   storage:
     className:
     capacity: 5Gi

mcp/server/server.py

@@ -16,6 +16,9 @@
 
 import json
 import logging
+import random
+import time
+from collections import OrderedDict
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 from functools import wraps
@@ -53,6 +56,13 @@ JSON_RESPONSE = True
 
 
 class RAGFlowConnector:
+    _MAX_DATASET_CACHE = 32
+    _MAX_DOCUMENT_CACHE = 128
+    _CACHE_TTL = 300
+
+    _dataset_metadata_cache: OrderedDict[str, tuple[dict, float | int]] = OrderedDict()  # "dataset_id" -> (metadata, expiry_ts)
+    _document_metadata_cache: OrderedDict[str, tuple[list[tuple[str, dict]], float | int]] = OrderedDict()  # "dataset_id" -> ([(document_id, doc_metadata)], expiry_ts)
+
     def __init__(self, base_url: str, version="v1"):
         self.base_url = base_url
         self.version = version
@@ -72,6 +82,43 @@ class RAGFlowConnector:
         res = requests.get(url=self.api_url + path, params=params, headers=self.authorization_header, json=json)
         return res
 
+    def _is_cache_valid(self, ts):
+        return time.time() < ts
+
+    def _get_expiry_timestamp(self):
+        offset = random.randint(-30, 30)
+        return time.time() + self._CACHE_TTL + offset
+
+    def _get_cached_dataset_metadata(self, dataset_id):
+        entry = self._dataset_metadata_cache.get(dataset_id)
+        if entry:
+            data, ts = entry
+            if self._is_cache_valid(ts):
+                self._dataset_metadata_cache.move_to_end(dataset_id)
+                return data
+        return None
+
+    def _set_cached_dataset_metadata(self, dataset_id, metadata):
+        self._dataset_metadata_cache[dataset_id] = (metadata, self._get_expiry_timestamp())
+        self._dataset_metadata_cache.move_to_end(dataset_id)
+        if len(self._dataset_metadata_cache) > self._MAX_DATASET_CACHE:
+            self._dataset_metadata_cache.popitem(last=False)
+
+    def _get_cached_document_metadata_by_dataset(self, dataset_id):
+        entry = self._document_metadata_cache.get(dataset_id)
+        if entry:
+            data_list, ts = entry
+            if self._is_cache_valid(ts):
+                self._document_metadata_cache.move_to_end(dataset_id)
+                return {doc_id: doc_meta for doc_id, doc_meta in data_list}
+        return None
+
+    def _set_cached_document_metadata_by_dataset(self, dataset_id, doc_id_meta_list):
+        self._document_metadata_cache[dataset_id] = (doc_id_meta_list, self._get_expiry_timestamp())
+        self._document_metadata_cache.move_to_end(dataset_id)
+        if len(self._document_metadata_cache) > self._MAX_DOCUMENT_CACHE:
+            self._document_metadata_cache.popitem(last=False)
+
     def list_datasets(self, page: int = 1, page_size: int = 1000, orderby: str = "create_time", desc: bool = True, id: str | None = None, name: str | None = None):
         res = self._get("/datasets", {"page": page, "page_size": page_size, "orderby": orderby, "desc": desc, "id": id, "name": name})
         if not res:
@@ -87,10 +134,38 @@ class RAGFlowConnector:
         return ""
 
     def retrieval(
-        self, dataset_ids, document_ids=None, question="", page=1, page_size=30, similarity_threshold=0.2, vector_similarity_weight=0.3, top_k=1024, rerank_id: str | None = None, keyword: bool = False
+        self,
+        dataset_ids,
+        document_ids=None,
+        question="",
+        page=1,
+        page_size=30,
+        similarity_threshold=0.2,
+        vector_similarity_weight=0.3,
+        top_k=1024,
+        rerank_id: str | None = None,
+        keyword: bool = False,
+        force_refresh: bool = False,
     ):
         if document_ids is None:
             document_ids = []
+        
+        # If no dataset_ids provided or empty list, get all available dataset IDs
+        if not dataset_ids:
+            dataset_list_str = self.list_datasets()
+            dataset_ids = []
+            
+            # Parse the dataset list to extract IDs
+            if dataset_list_str:
+                for line in dataset_list_str.strip().split('\n'):
+                    if line.strip():
+                        try:
+                            dataset_info = json.loads(line.strip())
+                            dataset_ids.append(dataset_info["id"])
+                        except (json.JSONDecodeError, KeyError):
+                            # Skip malformed lines
+                            continue
+        
         data_json = {
             "page": page,
             "page_size": page_size,
@@ -110,12 +185,127 @@ class RAGFlowConnector:
 
         res = res.json()
         if res.get("code") == 0:
+            data = res["data"]
             chunks = []
-            for chunk_data in res["data"].get("chunks"):
-                chunks.append(json.dumps(chunk_data, ensure_ascii=False))
-            return [types.TextContent(type="text", text="\n".join(chunks))]
+
+            # Cache document metadata and dataset information
+            document_cache, dataset_cache = self._get_document_metadata_cache(dataset_ids, force_refresh=force_refresh)
+
+            # Process chunks with enhanced field mapping including per-chunk metadata
+            for chunk_data in data.get("chunks", []):
+                enhanced_chunk = self._map_chunk_fields(chunk_data, dataset_cache, document_cache)
+                chunks.append(enhanced_chunk)
+
+            # Build structured response (no longer need response-level document_metadata)
+            response = {
+                "chunks": chunks,
+                "pagination": {
+                    "page": data.get("page", page),
+                    "page_size": data.get("page_size", page_size),
+                    "total_chunks": data.get("total", len(chunks)),
+                    "total_pages": (data.get("total", len(chunks)) + page_size - 1) // page_size,
+                },
+                "query_info": {
+                    "question": question,
+                    "similarity_threshold": similarity_threshold,
+                    "vector_weight": vector_similarity_weight,
+                    "keyword_search": keyword,
+                    "dataset_count": len(dataset_ids),
+                },
+            }
+
+            return [types.TextContent(type="text", text=json.dumps(response, ensure_ascii=False))]
+
         raise Exception([types.TextContent(type="text", text=res.get("message"))])
 
+    def _get_document_metadata_cache(self, dataset_ids, force_refresh=False):
+        """Cache document metadata for all documents in the specified datasets"""
+        document_cache = {}
+        dataset_cache = {}
+
+        try:
+            for dataset_id in dataset_ids:
+                dataset_meta = None if force_refresh else self._get_cached_dataset_metadata(dataset_id)
+                if not dataset_meta:
+                    # First get dataset info for name
+                    dataset_res = self._get("/datasets", {"id": dataset_id, "page_size": 1})
+                    if dataset_res and dataset_res.status_code == 200:
+                        dataset_data = dataset_res.json()
+                        if dataset_data.get("code") == 0 and dataset_data.get("data"):
+                            dataset_info = dataset_data["data"][0]
+                            dataset_meta = {"name": dataset_info.get("name", "Unknown"), "description": dataset_info.get("description", "")}
+                            self._set_cached_dataset_metadata(dataset_id, dataset_meta)
+                if dataset_meta:
+                    dataset_cache[dataset_id] = dataset_meta
+
+                docs = None if force_refresh else self._get_cached_document_metadata_by_dataset(dataset_id)
+                if docs is None:
+                    docs_res = self._get(f"/datasets/{dataset_id}/documents")
+                    docs_data = docs_res.json()
+                    if docs_data.get("code") == 0 and docs_data.get("data", {}).get("docs"):
+                        doc_id_meta_list = []
+                        docs = {}
+                        for doc in docs_data["data"]["docs"]:
+                            doc_id = doc.get("id")
+                            if not doc_id:
+                                continue
+                            doc_meta = {
+                                "document_id": doc_id,
+                                "name": doc.get("name", ""),
+                                "location": doc.get("location", ""),
+                                "type": doc.get("type", ""),
+                                "size": doc.get("size"),
+                                "chunk_count": doc.get("chunk_count"),
+                                # "chunk_method": doc.get("chunk_method", ""),
+                                "create_date": doc.get("create_date", ""),
+                                "update_date": doc.get("update_date", ""),
+                                # "process_begin_at": doc.get("process_begin_at", ""),
+                                # "process_duration": doc.get("process_duration"),
+                                # "progress": doc.get("progress"),
+                                # "progress_msg": doc.get("progress_msg", ""),
+                                # "status": doc.get("status", ""),
+                                # "run": doc.get("run", ""),
+                                "token_count": doc.get("token_count"),
+                                # "source_type": doc.get("source_type", ""),
+                                "thumbnail": doc.get("thumbnail", ""),
+                                "dataset_id": doc.get("dataset_id", dataset_id),
+                                "meta_fields": doc.get("meta_fields", {}),
+                                # "parser_config": doc.get("parser_config", {})
+                            }
+                            doc_id_meta_list.append((doc_id, doc_meta))
+                            docs[doc_id] = doc_meta
+                        self._set_cached_document_metadata_by_dataset(dataset_id, doc_id_meta_list)
+                if docs:
+                    document_cache.update(docs)
+
+        except Exception:
+            # Gracefully handle metadata cache failures
+            pass
+
+        return document_cache, dataset_cache
+
+    def _map_chunk_fields(self, chunk_data, dataset_cache, document_cache):
+        """Preserve all original API fields and add per-chunk document metadata"""
+        # Start with ALL raw data from API (preserve everything like original version)
+        mapped = dict(chunk_data)
+
+        # Add dataset name enhancement
+        dataset_id = chunk_data.get("dataset_id") or chunk_data.get("kb_id")
+        if dataset_id and dataset_id in dataset_cache:
+            mapped["dataset_name"] = dataset_cache[dataset_id]["name"]
+        else:
+            mapped["dataset_name"] = "Unknown"
+
+        # Add document name convenience field
+        mapped["document_name"] = chunk_data.get("document_keyword", "")
+
+        # Add per-chunk document metadata
+        document_id = chunk_data.get("document_id")
+        if document_id and document_id in document_cache:
+            mapped["document_metadata"] = document_cache[document_id]
+
+        return mapped
+
 
 class RAGFlowCtx:
     def __init__(self, connector: RAGFlowConnector):
@@ -195,7 +385,58 @@ async def list_tools(*, connector) -> list[types.Tool]:
                         "items": {"type": "string"},
                         "description": "Optional array of document IDs to search within."
                     },
-                    "question": {"type": "string", "description": "The question or query to search for."},
+                    "question": {
+                        "type": "string",
+                        "description": "The question or query to search for."
+                    },
+                    "page": {
+                        "type": "integer",
+                        "description": "Page number for pagination",
+                        "default": 1,
+                        "minimum": 1,
+                    },
+                    "page_size": {
+                        "type": "integer",
+                        "description": "Number of results to return per page (default: 10, max recommended: 50 to avoid token limits)",
+                        "default": 10,
+                        "minimum": 1,
+                        "maximum": 100,
+                    },
+                    "similarity_threshold": {
+                        "type": "number",
+                        "description": "Minimum similarity threshold for results",
+                        "default": 0.2,
+                        "minimum": 0.0,
+                        "maximum": 1.0,
+                    },
+                    "vector_similarity_weight": {
+                        "type": "number",
+                        "description": "Weight for vector similarity vs term similarity",
+                        "default": 0.3,
+                        "minimum": 0.0,
+                        "maximum": 1.0,
+                    },
+                    "keyword": {
+                        "type": "boolean",
+                        "description": "Enable keyword-based search",
+                        "default": False,
+                    },
+                    "top_k": {
+                        "type": "integer",
+                        "description": "Maximum results to consider before ranking",
+                        "default": 1024,
+                        "minimum": 1,
+                        "maximum": 1024,
+                    },
+                    "rerank_id": {
+                        "type": "string",
+                        "description": "Optional reranking model identifier",
+                    },
+                    "force_refresh": {
+                        "type": "boolean",
+                        "description": "Set to true only if fresh dataset and document metadata is explicitly required. Otherwise, cached metadata is used (default: false).",
+                        "default": False,
+                    },
                 },
                 "required": ["question"],
             },
@@ -209,6 +450,16 @@ async def call_tool(name: str, arguments: dict, *, connector) -> list[types.Text
     if name == "ragflow_retrieval":
         document_ids = arguments.get("document_ids", [])
         dataset_ids = arguments.get("dataset_ids", [])
+        question = arguments.get("question", "")
+        page = arguments.get("page", 1)
+        page_size = arguments.get("page_size", 10)
+        similarity_threshold = arguments.get("similarity_threshold", 0.2)
+        vector_similarity_weight = arguments.get("vector_similarity_weight", 0.3)
+        keyword = arguments.get("keyword", False)
+        top_k = arguments.get("top_k", 1024)
+        rerank_id = arguments.get("rerank_id")
+        force_refresh = arguments.get("force_refresh", False)
+
         
         # If no dataset_ids provided or empty list, get all available dataset IDs
         if not dataset_ids:
@@ -229,7 +480,15 @@ async def call_tool(name: str, arguments: dict, *, connector) -> list[types.Text
         return connector.retrieval(
             dataset_ids=dataset_ids,
             document_ids=document_ids,
-            question=arguments["question"],
+            question=question,
+            page=page,
+            page_size=page_size,
+            similarity_threshold=similarity_threshold,
+            vector_similarity_weight=vector_similarity_weight,
+            keyword=keyword,
+            top_k=top_k,
+            rerank_id=rerank_id,
+            force_refresh=force_refresh,
         )
     raise ValueError(f"Tool not found: {name}")
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ragflow"
-version = "0.20.1"
+version = "0.20.4"
 description = "[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data."
 authors = [{ name = "Zhichang Yu", email = "yuzhichang@gmail.com" }]
 license-files = ["LICENSE"]
@@ -43,9 +43,9 @@ dependencies = [
     "groq==0.9.0",
     "hanziconv==0.3.2",
     "html-text==0.6.2",
-    "httpx==0.27.2",
+    "httpx[socks]==0.27.2",
     "huggingface-hub>=0.25.0,<0.26.0",
-    "infinity-sdk==0.6.0-dev4",
+    "infinity-sdk==0.6.0.dev5",
     "infinity-emb>=0.0.66,<0.0.67",
     "itsdangerous==2.1.2",
     "json-repair==0.35.0",
@@ -73,7 +73,7 @@ dependencies = [
     "pyclipper==1.3.0.post5",
     "pycryptodomex==3.20.0",
     "pymysql>=1.1.1,<2.0.0",
-    "pypdf>=5.0.0,<6.0.0",
+    "pypdf==6.0.0",
     "python-dotenv==1.0.1",
     "python-dateutil==2.8.2",
     "python-pptx>=1.0.2,<2.0.0",
@@ -130,6 +130,7 @@ dependencies = [
     "click>=8.1.8",
     "python-calamine>=0.4.0",
     "litellm>=1.74.15.post1",
+    "flask-mail>=0.10.0",
 ]
 
 [project.optional-dependencies]

rag/app/audio.py

@@ -14,31 +14,48 @@
 #  limitations under the License.
 #
 
+import os
 import re
+import tempfile
 
 from api.db import LLMType
-from rag.nlp import rag_tokenizer
 from api.db.services.llm_service import LLMBundle
-from rag.nlp import tokenize
+from rag.nlp import rag_tokenizer, tokenize
 
 
 def chunk(filename, binary, tenant_id, lang, callback=None, **kwargs):
-    doc = {
-        "docnm_kwd": filename,
-        "title_tks": rag_tokenizer.tokenize(re.sub(r"\.[a-zA-Z]+$", "", filename))
-    }
+    doc = {"docnm_kwd": filename, "title_tks": rag_tokenizer.tokenize(re.sub(r"\.[a-zA-Z]+$", "", filename))}
     doc["title_sm_tks"] = rag_tokenizer.fine_grained_tokenize(doc["title_tks"])
 
     # is it English
     eng = lang.lower() == "english"  # is_english(sections)
     try:
+        _, ext = os.path.splitext(filename)
+        if not ext:
+            raise RuntimeError("No extension detected.")
+
+        if ext not in [".da", ".wave", ".wav", ".mp3", ".wav", ".aac", ".flac", ".ogg", ".aiff", ".au", ".midi", ".wma", ".realaudio", ".vqf", ".oggvorbis", ".aac", ".ape"]:
+            raise RuntimeError(f"Extension {ext} is not supported yet.")
+
+        tmp_path = ""
+        with tempfile.NamedTemporaryFile(suffix=ext, delete=False) as tmpf:
+            tmpf.write(binary)
+            tmpf.flush()
+            tmp_path = os.path.abspath(tmpf.name)
+
         callback(0.1, "USE Sequence2Txt LLM to transcription the audio")
         seq2txt_mdl = LLMBundle(tenant_id, LLMType.SPEECH2TEXT, lang=lang)
-        ans = seq2txt_mdl.transcription(binary)
+        ans = seq2txt_mdl.transcription(tmp_path)
         callback(0.8, "Sequence2Txt LLM respond: %s ..." % ans[:32])
+
         tokenize(doc, ans, eng)
         return [doc]
     except Exception as e:
         callback(prog=-1, msg=str(e))
-
+    finally:
+        if tmp_path and os.path.exists(tmp_path):
+            try:
+                os.unlink(tmp_path)
+            except Exception:
+                pass
     return []

rag/app/naive.py

@@ -22,13 +22,15 @@ from timeit import default_timer as timer
 
 from docx import Document
 from docx.image.exceptions import InvalidImageStreamError, UnexpectedEndOfFileError, UnrecognizedImageError
+from docx.opc.pkgreader import _SerializedRelationships, _SerializedRelationship
+from docx.opc.oxml import parse_xml
 from markdown import markdown
 from PIL import Image
 from tika import parser
 
 from api.db import LLMType
 from api.db.services.llm_service import LLMBundle
-from deepdoc.parser import DocxParser, ExcelParser, HtmlParser, JsonParser, MarkdownParser, PdfParser, TxtParser
+from deepdoc.parser import DocxParser, ExcelParser, HtmlParser, JsonParser, MarkdownElementExtractor, MarkdownParser, PdfParser, TxtParser
 from deepdoc.parser.figure_parser import VisionFigureParser, vision_figure_parser_figure_data_wrapper
 from deepdoc.parser.pdf_parser import PlainParser, VisionParser
 from rag.nlp import concat_img, find_codec, naive_merge, naive_merge_with_images, naive_merge_docx, rag_tokenizer, tokenize_chunks, tokenize_chunks_with_images, tokenize_table
@@ -47,8 +49,8 @@ class Docx(DocxParser):
         if not embed:
             return None
         embed = embed[0]
-        related_part = document.part.related_parts[embed]
         try:
+            related_part = document.part.related_parts[embed]
             image_blob = related_part.image.blob
         except UnrecognizedImageError:
             logging.info("Unrecognized image format. Skipping image.")
@@ -62,6 +64,9 @@ class Docx(DocxParser):
         except UnicodeDecodeError:
             logging.info("The recognized image stream appears to be corrupted. Skipping image.")
             return None
+        except Exception:
+            logging.info("The recognized image stream appears to be corrupted. Skipping image.")
+            return None
         try:
             image = Image.open(BytesIO(image_blob)).convert('RGB')
             return image
@@ -284,7 +289,7 @@ class Pdf(PdfParser):
             return [(b["text"], self._line_tag(b, zoomin)) for b in self.boxes], tbls, figures
         else:
             tbls = self._extract_table_figure(True, zoomin, True, True)
-            # self._naive_vertical_merge()
+            self._naive_vertical_merge()
             self._concat_downward()
             # self._filter_forpages()
             logging.info("layouts cost: {}s".format(timer() - first_start))
@@ -345,21 +350,32 @@ class Markdown(MarkdownParser):
         else:
             with open(filename, "r") as f:
                 txt = f.read()
+
         remainder, tables = self.extract_tables_and_remainder(f'{txt}\n', separate_tables=separate_tables)
-        sections = []
+
+        extractor = MarkdownElementExtractor(txt)
+        element_sections = extractor.extract_elements()
+        sections = [(element, "") for element in element_sections]
+
         tbls = []
-        for sec in remainder.split("\n"):
-            if sec.strip().find("#") == 0:
-                sections.append((sec, ""))
-            elif sections and sections[-1][0].strip().find("#") == 0:
-                sec_, _ = sections.pop(-1)
-                sections.append((sec_ + "\n" + sec, ""))
-            else:
-                sections.append((sec, ""))
         for table in tables:
             tbls.append(((None, markdown(table, extensions=['markdown.extensions.tables'])), ""))
         return sections, tbls
 
+def load_from_xml_v2(baseURI, rels_item_xml):
+    """
+    Return |_SerializedRelationships| instance loaded with the
+    relationships contained in *rels_item_xml*. Returns an empty
+    collection if *rels_item_xml* is |None|.
+    """
+    srels = _SerializedRelationships()
+    if rels_item_xml is not None:
+        rels_elm = parse_xml(rels_item_xml)
+        for rel_elm in rels_elm.Relationship_lst:
+            if rel_elm.target_ref in ('../NULL', 'NULL'):
+                continue
+            srels._srels.append(_SerializedRelationship(baseURI, rel_elm))
+    return srels
 
 def chunk(filename, binary=None, from_page=0, to_page=100000,
           lang="Chinese", callback=None, **kwargs):
@@ -391,6 +407,8 @@ def chunk(filename, binary=None, from_page=0, to_page=100000,
         except Exception:
             vision_model = None
 
+        # fix "There is no item named 'word/NULL' in the archive", referring to https://github.com/python-openxml/python-docx/issues/1105#issuecomment-1298075246
+        _SerializedRelationships.load_from_xml = load_from_xml_v2
         sections, tables = Docx()(filename, binary)
 
         if vision_model:
@@ -469,6 +487,7 @@ def chunk(filename, binary=None, from_page=0, to_page=100000,
             sections = [(_, "") for _ in excel_parser.html(binary, 12) if _]
         else:
             sections = [(_, "") for _ in excel_parser(binary) if _]
+        parser_config["chunk_token_num"] = 12800
 
     elif re.search(r"\.(txt|py|js|java|c|cpp|h|php|go|ts|sh|cs|kt|sql)$", filename, re.IGNORECASE):
         callback(0.1, "Start to parse.")
@@ -498,7 +517,8 @@ def chunk(filename, binary=None, from_page=0, to_page=100000,
 
     elif re.search(r"\.(htm|html)$", filename, re.IGNORECASE):
         callback(0.1, "Start to parse.")
-        sections = HtmlParser()(filename, binary)
+        chunk_token_num = int(parser_config.get("chunk_token_num", 128))
+        sections = HtmlParser()(filename, binary, chunk_token_num)
         sections = [(_, "") for _ in sections if _]
         callback(0.8, "Finish parsing.")
 

rag/llm/__init__.py

@@ -36,6 +36,7 @@ class SupportedLiteLLMProvider(StrEnum):
     Nvidia = "NVIDIA"
     TogetherAI = "TogetherAI"
     Anthropic = "Anthropic"
+    Ollama = "Ollama"
 
 
 FACTORY_DEFAULT_BASE_URL = {
@@ -59,6 +60,7 @@ LITELLM_PROVIDER_PREFIX = {
     SupportedLiteLLMProvider.Nvidia: "nvidia_nim/",
     SupportedLiteLLMProvider.TogetherAI: "together_ai/",
     SupportedLiteLLMProvider.Anthropic: "",  # don't need a prefix
+    SupportedLiteLLMProvider.Ollama: "ollama_chat/",
 }
 
 ChatModel = globals().get("ChatModel", {})

rag/llm/chat_model.py

@@ -29,7 +29,6 @@ import json_repair
 import litellm
 import openai
 import requests
-from ollama import Client
 from openai import OpenAI
 from openai.lib.azure import AzureOpenAI
 from strenum import StrEnum
@@ -112,6 +111,32 @@ class Base(ABC):
     def _clean_conf(self, gen_conf):
         if "max_tokens" in gen_conf:
             del gen_conf["max_tokens"]
+
+        allowed_conf = {
+            "temperature",
+            "max_completion_tokens",
+            "top_p",
+            "stream",
+            "stream_options",
+            "stop",
+            "n",
+            "presence_penalty",
+            "frequency_penalty",
+            "functions",
+            "function_call",
+            "logit_bias",
+            "user",
+            "response_format",
+            "seed",
+            "tools",
+            "tool_choice",
+            "logprobs",
+            "top_logprobs",
+            "extra_headers",
+        }
+
+        gen_conf = {k: v for k, v in gen_conf.items() if k in allowed_conf}
+
         return gen_conf
 
     def _chat(self, history, gen_conf, **kwargs):
@@ -657,73 +682,6 @@ class ZhipuChat(Base):
         return super().chat_streamly_with_tools(system, history, gen_conf)
 
 
-class OllamaChat(Base):
-    _FACTORY_NAME = "Ollama"
-
-    def __init__(self, key, model_name, base_url=None, **kwargs):
-        super().__init__(key, model_name, base_url=base_url, **kwargs)
-
-        self.client = Client(host=base_url) if not key or key == "x" else Client(host=base_url, headers={"Authorization": f"Bearer {key}"})
-        self.model_name = model_name
-        self.keep_alive = kwargs.get("ollama_keep_alive", int(os.environ.get("OLLAMA_KEEP_ALIVE", -1)))
-
-    def _clean_conf(self, gen_conf):
-        options = {}
-        if "max_tokens" in gen_conf:
-            options["num_predict"] = gen_conf["max_tokens"]
-        for k in ["temperature", "top_p", "presence_penalty", "frequency_penalty"]:
-            if k not in gen_conf:
-                continue
-            options[k] = gen_conf[k]
-        return options
-
-    def _chat(self, history, gen_conf={}, **kwargs):
-        # Calculate context size
-        ctx_size = self._calculate_dynamic_ctx(history)
-
-        gen_conf["num_ctx"] = ctx_size
-        response = self.client.chat(model=self.model_name, messages=history, options=gen_conf, keep_alive=self.keep_alive)
-        ans = response["message"]["content"].strip()
-        token_count = response.get("eval_count", 0) + response.get("prompt_eval_count", 0)
-        return ans, token_count
-
-    def chat_streamly(self, system, history, gen_conf={}, **kwargs):
-        if system:
-            history.insert(0, {"role": "system", "content": system})
-        if "max_tokens" in gen_conf:
-            del gen_conf["max_tokens"]
-        try:
-            # Calculate context size
-            ctx_size = self._calculate_dynamic_ctx(history)
-            options = {"num_ctx": ctx_size}
-            if "temperature" in gen_conf:
-                options["temperature"] = gen_conf["temperature"]
-            if "max_tokens" in gen_conf:
-                options["num_predict"] = gen_conf["max_tokens"]
-            if "top_p" in gen_conf:
-                options["top_p"] = gen_conf["top_p"]
-            if "presence_penalty" in gen_conf:
-                options["presence_penalty"] = gen_conf["presence_penalty"]
-            if "frequency_penalty" in gen_conf:
-                options["frequency_penalty"] = gen_conf["frequency_penalty"]
-
-            ans = ""
-            try:
-                response = self.client.chat(model=self.model_name, messages=history, stream=True, options=options, keep_alive=self.keep_alive)
-                for resp in response:
-                    if resp["done"]:
-                        token_count = resp.get("prompt_eval_count", 0) + resp.get("eval_count", 0)
-                        yield token_count
-                    ans = resp["message"]["content"]
-                    yield ans
-            except Exception as e:
-                yield ans + "\n**ERROR**: " + str(e)
-            yield 0
-        except Exception as e:
-            yield "**ERROR**: " + str(e)
-            yield 0
-
-
 class LocalAIChat(Base):
     _FACTORY_NAME = "LocalAI"
 
@@ -1396,7 +1354,7 @@ class Ai302Chat(Base):
 
 
 class LiteLLMBase(ABC):
-    _FACTORY_NAME = ["Tongyi-Qianwen", "Bedrock", "Moonshot", "xAI", "DeepInfra", "Groq", "Cohere", "Gemini", "DeepSeek", "NVIDIA", "TogetherAI", "Anthropic"]
+    _FACTORY_NAME = ["Tongyi-Qianwen", "Bedrock", "Moonshot", "xAI", "DeepInfra", "Groq", "Cohere", "Gemini", "DeepSeek", "NVIDIA", "TogetherAI", "Anthropic", "Ollama"]
 
     def __init__(self, key, model_name, base_url=None, **kwargs):
         self.timeout = int(os.environ.get("LM_TIMEOUT_SECONDS", 600))

rag/llm/cv_model.py

@@ -68,7 +68,7 @@ class Base(ABC):
             pmpt.append({
                 "type": "image_url",
                 "image_url": {
-                    "url": f"data:image/jpeg;base64,{img}" if img[:4] != "data" else img
+                    "url": img if isinstance(img, str) and img.startswith("data:") else f"data:image/png;base64,{img}"
                 }
             })
         return pmpt
@@ -109,16 +109,33 @@ class Base(ABC):
 
     @staticmethod
     def image2base64(image):
+        # Return a data URL with the correct MIME to avoid provider mismatches
         if isinstance(image, bytes):
-            return base64.b64encode(image).decode("utf-8")
+            # Best-effort magic number sniffing
+            mime = "image/png"
+            if len(image) >= 2 and image[0] == 0xFF and image[1] == 0xD8:
+                mime = "image/jpeg"
+            b64 = base64.b64encode(image).decode("utf-8")
+            return f"data:{mime};base64,{b64}"
         if isinstance(image, BytesIO):
-            return base64.b64encode(image.getvalue()).decode("utf-8")
+            data = image.getvalue()
+            mime = "image/png"
+            if len(data) >= 2 and data[0] == 0xFF and data[1] == 0xD8:
+                mime = "image/jpeg"
+            b64 = base64.b64encode(data).decode("utf-8")
+            return f"data:{mime};base64,{b64}"
         buffered = BytesIO()
+        fmt = "JPEG"
         try:
             image.save(buffered, format="JPEG")
         except Exception:
+            buffered = BytesIO()  # reset buffer before saving PNG
             image.save(buffered, format="PNG")
-        return base64.b64encode(buffered.getvalue()).decode("utf-8")
+            fmt = "PNG"
+        data = buffered.getvalue()
+        b64 = base64.b64encode(data).decode("utf-8")
+        mime = f"image/{fmt.lower()}"
+        return f"data:{mime};base64,{b64}"
 
     def prompt(self, b64):
         return [
@@ -372,6 +389,16 @@ class OllamaCV(Base):
         self.keep_alive = kwargs.get("ollama_keep_alive", int(os.environ.get("OLLAMA_KEEP_ALIVE", -1)))
         Base.__init__(self, **kwargs)
 
+
+    def _clean_img(self, img):
+        if not isinstance(img, str):
+            return img
+
+        #remove the header like "data/*;base64,"
+        if img.startswith("data:") and ";base64," in img:
+            img = img.split(";base64,")[1]
+        return img
+
     def _clean_conf(self, gen_conf):
         options = {}
         if "temperature" in gen_conf:
@@ -390,9 +417,12 @@ class OllamaCV(Base):
             hist.insert(0, {"role": "system", "content": system})
         if not images:
             return hist
+        temp_images = []
+        for img in images:
+            temp_images.append(self._clean_img(img))
         for his in hist:
             if his["role"] == "user":
-                his["images"] = images
+                his["images"] = temp_images
                 break
         return hist
 
@@ -509,24 +539,24 @@ class GeminiCV(Base):
         return res.text, res.usage_metadata.total_token_count
 
     def chat(self, system, history, gen_conf, images=[]):
-        from transformers import GenerationConfig
+        generation_config = dict(temperature=gen_conf.get("temperature", 0.3), top_p=gen_conf.get("top_p", 0.7))
         try:
             response = self.model.generate_content(
                 self._form_history(system, history, images),
-                generation_config=GenerationConfig(temperature=gen_conf.get("temperature", 0.3), top_p=gen_conf.get("top_p", 0.7)))
+                generation_config=generation_config)
             ans = response.text
             return ans, response.usage_metadata.total_token_count
         except Exception as e:
             return "**ERROR**: " + str(e), 0
 
     def chat_streamly(self, system, history, gen_conf, images=[]):
-        from transformers import GenerationConfig
         ans = ""
         response = None
         try:
+            generation_config = dict(temperature=gen_conf.get("temperature", 0.3), top_p=gen_conf.get("top_p", 0.7))
             response = self.model.generate_content(
                 self._form_history(system, history, images),
-                generation_config=GenerationConfig(temperature=gen_conf.get("temperature", 0.3), top_p=gen_conf.get("top_p", 0.7)),
+                generation_config=generation_config,
                 stream=True,
             )
 
@@ -542,7 +572,7 @@ class GeminiCV(Base):
             yield response.usage_metadata.total_token_count
         else:
             yield 0
-            
+
 
 class NvidiaCV(Base):
     _FACTORY_NAME = "NVIDIA"
@@ -661,8 +691,8 @@ class AnthropicCV(Base):
                         "type": "image",
                         "source": {
                             "type": "base64",
-                            "media_type": "image/jpeg" if img[:4] != "data" else img.split(":")[1].split(";")[0],
-                            "data": img if img[:4] != "data" else img.split(",")[1]
+                            "media_type": (img.split(":")[1].split(";")[0] if isinstance(img, str) and img[:4] == "data" else "image/png"),
+                            "data": (img.split(",")[1] if isinstance(img, str) and img[:4] == "data" else img)
                         },
                     }
             )

rag/llm/rerank_model.py

@@ -44,14 +44,17 @@ class Base(ABC):
         raise NotImplementedError("Please implement encode method!")
 
     def total_token_count(self, resp):
-        try:
-            return resp.usage.total_tokens
-        except Exception:
-            pass
-        try:
-            return resp["usage"]["total_tokens"]
-        except Exception:
-            pass
+        if hasattr(resp, "usage") and hasattr(resp.usage, "total_tokens"):
+            try:
+                return resp.usage.total_tokens
+            except Exception:
+                pass
+
+        if 'usage' in resp and 'total_tokens' in resp['usage']:
+            try:
+                return resp["usage"]["total_tokens"]
+            except Exception:
+                pass
         return 0
 
 
@@ -100,7 +103,7 @@ class DefaultRerank(Base):
         old_dynamic_batch_size = self._dynamic_batch_size
         if max_batch_size is not None:
             self._dynamic_batch_size = max_batch_size
-        res = np.array([], dtype=float)
+        res = np.array(len(pairs), dtype=float)
         i = 0
         while i < len(pairs):
             cur_i = i
@@ -111,7 +114,7 @@ class DefaultRerank(Base):
                 try:
                     # call subclass implemented batch processing calculation
                     batch_scores = self._compute_batch_scores(pairs[i : i + current_batch])
-                    res = np.append(res, batch_scores)
+                    res[i : i + current_batch] = batch_scores
                     i += current_batch
                     self._dynamic_batch_size = min(self._dynamic_batch_size * 2, 8)
                     break
@@ -125,8 +128,8 @@ class DefaultRerank(Base):
                         raise
             if retry_count >= max_retries:
                 raise RuntimeError("max retry times, still cannot process batch, please check your GPU memory")
-            self.torch_empty_cache()
-
+            
+        self.torch_empty_cache()
         self._dynamic_batch_size = old_dynamic_batch_size
         return np.array(res)
 
@@ -482,9 +485,10 @@ class VoyageRerank(Base):
         self.model_name = model_name
 
     def similarity(self, query: str, texts: list):
-        rank = np.zeros(len(texts), dtype=float)
         if not texts:
-            return rank, 0
+            return np.array([]), 0
+        rank = np.zeros(len(texts), dtype=float)
+
         res = self.client.rerank(query=query, documents=texts, model=self.model_name, top_k=len(texts))
         try:
             for r in res.results:

rag/llm/sequence2txt_model.py

@@ -35,8 +35,9 @@ class Base(ABC):
         """
         pass
 
-    def transcription(self, audio, **kwargs):
-        transcription = self.client.audio.transcriptions.create(model=self.model_name, file=audio, response_format="text")
+    def transcription(self, audio_path, **kwargs):
+        audio_file = open(audio_path, "rb")
+        transcription = self.client.audio.transcriptions.create(model=self.model_name, file=audio_file)
         return transcription.text.strip(), num_tokens_from_string(transcription.text.strip())
 
     def audio2base64(self, audio):
@@ -50,7 +51,7 @@ class Base(ABC):
 class GPTSeq2txt(Base):
     _FACTORY_NAME = "OpenAI"
 
-    def __init__(self, key, model_name="whisper-1", base_url="https://api.openai.com/v1"):
+    def __init__(self, key, model_name="whisper-1", base_url="https://api.openai.com/v1", **kwargs):
         if not base_url:
             base_url = "https://api.openai.com/v1"
         self.client = OpenAI(api_key=key, base_url=base_url)
@@ -60,27 +61,38 @@ class GPTSeq2txt(Base):
 class QWenSeq2txt(Base):
     _FACTORY_NAME = "Tongyi-Qianwen"
 
-    def __init__(self, key, model_name="paraformer-realtime-8k-v1", **kwargs):
+    def __init__(self, key, model_name="qwen-audio-asr", **kwargs):
         import dashscope
 
         dashscope.api_key = key
         self.model_name = model_name
 
-    def transcription(self, audio, format):
-        from http import HTTPStatus
+    def transcription(self, audio_path):
+        if "paraformer" in self.model_name or "sensevoice" in self.model_name:
+            return f"**ERROR**: model {self.model_name} is not suppported yet.", 0
 
-        from dashscope.audio.asr import Recognition
+        from dashscope import MultiModalConversation
 
-        recognition = Recognition(model=self.model_name, format=format, sample_rate=16000, callback=None)
-        result = recognition.call(audio)
+        audio_path = f"file://{audio_path}"
+        messages = [
+            {
+                "role": "user",
+                "content": [{"audio": audio_path}],
+            }
+        ]
 
-        ans = ""
-        if result.status_code == HTTPStatus.OK:
-            for sentence in result.get_sentence():
-                ans += sentence.text.decode("utf-8") + "\n"
-            return ans, num_tokens_from_string(ans)
-
-        return "**ERROR**: " + result.message, 0
+        response = None
+        full_content = ""
+        try:
+            response = MultiModalConversation.call(model="qwen-audio-asr", messages=messages, result_format="message", stream=True)
+            for response in response:
+                try:
+                    full_content += response["output"]["choices"][0]["message"].content[0]["text"]
+                except Exception:
+                    pass
+            return full_content, num_tokens_from_string(full_content)
+        except Exception as e:
+            return "**ERROR**: " + str(e), 0
 
 
 class AzureSeq2txt(Base):
@@ -212,6 +224,7 @@ class GiteeSeq2txt(Base):
         self.client = OpenAI(api_key=key, base_url=base_url)
         self.model_name = model_name
 
+
 class DeepInfraSeq2txt(Base):
     _FACTORY_NAME = "DeepInfra"
 

rag/nlp/__init__.py

@@ -611,10 +611,6 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
                 if re.match(f"^{dels}$", sub_sec):
                     continue
                 add_chunk(sub_sec, image)
-        
-        for img in images:
-            if isinstance(img, Image.Image):
-                img.close()
 
     return cks, result_images
 

rag/prompts/ask_summary.md

@@ -0,0 +1,14 @@
+Role: You're a smart assistant. Your name is Miss R.
+Task: Summarize the information from knowledge bases and answer user's question.
+Requirements and restriction:
+  - DO NOT make things up, especially for numbers.
+  - If the information from knowledge is irrelevant with user's question, JUST SAY: Sorry, no relevant information provided.
+  - Answer with markdown format text.
+  - Answer in language of user's question.
+  - DO NOT make things up, especially for numbers.
+
+### Information from knowledge bases
+
+{{ knowledge }}
+
+The above is information from knowledge bases.

rag/prompts/citation_prompt.md

@@ -105,4 +105,5 @@ REMEMBER:
 - Cite FACTS, not opinions or transitions
 - Each citation supports the ENTIRE sentence
 - When in doubt, ask: "Would a fact-checker need to verify this?"
-- Place citations at sentence end, before punctuation
+- Place citations at sentence end, before punctuation
+- Format likes this is FORBIDDEN: [ID:0, ID:5, ID:...]. It MUST be seperated like, [ID:0][ID:5]...

rag/prompts/prompts.py

@@ -114,6 +114,8 @@ def kb_prompt(kbinfos, max_tokens, hash_id=False):
     docs = {d.id: d.meta_fields for d in docs}
 
     def draw_node(k, line):
+        if line is not None and not isinstance(line, str):
+            line = str(line)
         if not line:
             return ""
         return f"\n├── {k}: " + re.sub(r"\n+", " ", line, flags=re.DOTALL)
@@ -150,6 +152,7 @@ REFLECT = load_prompt("reflect")
 SUMMARY4MEMORY = load_prompt("summary4memory")
 RANK_MEMORY = load_prompt("rank_memory")
 META_FILTER = load_prompt("meta_filter")
+ASK_SUMMARY = load_prompt("ask_summary")
 
 PROMPT_JINJA_ENV = jinja2.Environment(autoescape=False, trim_blocks=True, lstrip_blocks=True)
 
@@ -197,7 +200,7 @@ def question_proposal(chat_mdl, content, topn=3):
 def full_question(tenant_id=None, llm_id=None, messages=[], language=None, chat_mdl=None):
     from api.db import LLMType
     from api.db.services.llm_service import LLMBundle
-    from api.db.services.llm_service import TenantLLMService
+    from api.db.services.tenant_llm_service import TenantLLMService
 
     if not chat_mdl:
         if TenantLLMService.llm_id2llm_type(llm_id) == "image2text":
@@ -231,7 +234,7 @@ def full_question(tenant_id=None, llm_id=None, messages=[], language=None, chat_
 def cross_languages(tenant_id, llm_id, query, languages=[]):
     from api.db import LLMType
     from api.db.services.llm_service import LLMBundle
-    from api.db.services.llm_service import TenantLLMService
+    from api.db.services.tenant_llm_service import TenantLLMService
 
     if llm_id and TenantLLMService.llm_id2llm_type(llm_id) == "image2text":
         chat_mdl = LLMBundle(tenant_id, LLMType.IMAGE2TEXT, llm_id)

rag/prompts/related_question.md

@@ -0,0 +1,55 @@
+# Role
+You are an AI language model assistant tasked with generating **5-10 related questions** based on a user’s original query.
+These questions should help **expand the search query scope** and **improve search relevance**.
+
+---
+
+## Instructions
+
+**Input:**
+You are provided with a **user’s question**.
+
+**Output:**
+Generate **5-10 alternative questions** that are **related** to the original user question.
+These alternatives should help retrieve a **broader range of relevant documents** from a vector database.
+
+**Context:**
+Focus on **rephrasing** the original question in different ways, ensuring the alternative questions are **diverse but still connected** to the topic of the original query.
+Do **not** create overly obscure, irrelevant, or unrelated questions.
+
+**Fallback:**
+If you cannot generate any relevant alternatives, do **not** return any questions.
+
+---
+
+## Guidance
+
+1. Each alternative should be **unique** but still **relevant** to the original query.
+2. Keep the phrasing **clear, concise, and easy to understand**.
+3. Avoid overly technical jargon or specialized terms **unless directly relevant**.
+4. Ensure that each question **broadens** the search angle, **not narrows** it.
+
+---
+
+## Example
+
+**Original Question:**
+> What are the benefits of electric vehicles?
+
+**Alternative Questions:**
+1. How do electric vehicles impact the environment?
+2. What are the advantages of owning an electric car?
+3. What is the cost-effectiveness of electric vehicles?
+4. How do electric vehicles compare to traditional cars in terms of fuel efficiency?
+5. What are the environmental benefits of switching to electric cars?
+6. How do electric vehicles help reduce carbon emissions?
+7. Why are electric vehicles becoming more popular?
+8. What are the long-term savings of using electric vehicles?
+9. How do electric vehicles contribute to sustainability?
+10. What are the key benefits of electric vehicles for consumers?
+
+---
+
+## Reason
+Rephrasing the original query into multiple alternative questions helps the user explore **different aspects** of their search topic, improving the **quality of search results**.
+These questions guide the search engine to provide a **more comprehensive set** of relevant documents.

rag/raptor.py

@@ -42,9 +42,12 @@ class RecursiveAbstractiveProcessing4TreeOrganizedRetrieval:
         self._prompt = prompt
         self._max_token = max_token
 
-    @timeout(60)
+    @timeout(60*20)
     async def _chat(self, system, history, gen_conf):
-        response = get_llm_cache(self._llm_model.llm_name, system, history, gen_conf)
+        response = await trio.to_thread.run_sync(
+            lambda: get_llm_cache(self._llm_model.llm_name, system, history, gen_conf)
+        )
+
         if response:
             return response
         response = await trio.to_thread.run_sync(
@@ -53,19 +56,23 @@ class RecursiveAbstractiveProcessing4TreeOrganizedRetrieval:
         response = re.sub(r"^.*</think>", "", response, flags=re.DOTALL)
         if response.find("**ERROR**") >= 0:
             raise Exception(response)
-        set_llm_cache(self._llm_model.llm_name, system, response, history, gen_conf)
+        await trio.to_thread.run_sync(
+            lambda: set_llm_cache(self._llm_model.llm_name, system, response, history, gen_conf)
+        )
         return response
 
-    @timeout(2)
+    @timeout(20)
     async def _embedding_encode(self, txt):
-        response = get_embed_cache(self._embd_model.llm_name, txt)
+        response = await trio.to_thread.run_sync(
+            lambda: get_embed_cache(self._embd_model.llm_name, txt)
+        )
         if response is not None:
             return response
         embds, _ = await trio.to_thread.run_sync(lambda: self._embd_model.encode([txt]))
         if len(embds) < 1 or len(embds[0]) < 1:
             raise Exception("Embedding error: ")
         embds = embds[0]
-        set_embed_cache(self._embd_model.llm_name, txt, embds)
+        await trio.to_thread.run_sync(lambda: set_embed_cache(self._embd_model.llm_name, txt, embds))
         return embds
 
     def _get_optimal_clusters(self, embeddings: np.ndarray, random_state: int):
@@ -86,7 +93,7 @@ class RecursiveAbstractiveProcessing4TreeOrganizedRetrieval:
         layers = [(0, len(chunks))]
         start, end = 0, len(chunks)
 
-        @timeout(60)
+        @timeout(60*20)
         async def summarize(ck_idx: list[int]):
             nonlocal chunks
             texts = [chunks[i][0] for i in ck_idx]

rag/svr/task_executor.py

@@ -21,7 +21,7 @@ import sys
 import threading
 import time
 
-from api.utils.api_utils import timeout, is_strong_enough
+from api.utils.api_utils import timeout
 from api.utils.log_utils import init_root_logger, get_project_base_directory
 from graphrag.general.index import run_graphrag
 from graphrag.utils import get_llm_cache, set_llm_cache, get_tags_from_cache, set_tags_to_cache
@@ -302,7 +302,7 @@ async def build_chunks(task, progress_callback):
                     # If the image is in RGBA mode, convert it to RGB mode before saving it in JPEG format.
                     if d["image"].mode in ("RGBA", "P"):
                         converted_image = d["image"].convert("RGB")
-                        d["image"].close()  # Close original image
+                        #d["image"].close()  # Close original image
                         d["image"] = converted_image
                     try:
                         d["image"].save(output_buffer, format='JPEG')
@@ -478,8 +478,6 @@ async def embedding(docs, mdl, parser_config=None, callback=None):
 
 @timeout(3600)
 async def run_raptor(row, chat_mdl, embd_mdl, vector_size, callback=None):
-    # Pressure test for GraphRAG task
-    await is_strong_enough(chat_mdl, embd_mdl)
     chunks = []
     vctr_nm = "q_%d_vec"%vector_size
     for d in settings.retrievaler.chunk_list(row["doc_id"], row["tenant_id"], [str(row["kb_id"])],
@@ -520,7 +518,7 @@ async def run_raptor(row, chat_mdl, embd_mdl, vector_size, callback=None):
     return res, tk_count
 
 
-@timeout(60*60, 1)
+@timeout(60*60*2, 1)
 async def do_handle_task(task):
     task_id = task["id"]
     task_from_page = task["from_page"]
@@ -553,7 +551,6 @@ async def do_handle_task(task):
     try:
         # bind embedding model
         embedding_model = LLMBundle(task_tenant_id, LLMType.EMBEDDING, llm_name=task_embedding_id, lang=task_language)
-        await is_strong_enough(None, embedding_model)
         vts, _ = embedding_model.encode(["ok"])
         vector_size = len(vts[0])
     except Exception as e:
@@ -568,7 +565,6 @@ async def do_handle_task(task):
     if task.get("task_type", "") == "raptor":
         # bind LLM for raptor
         chat_model = LLMBundle(task_tenant_id, LLMType.CHAT, llm_name=task_llm_id, lang=task_language)
-        await is_strong_enough(chat_model, None)
         # run RAPTOR
         async with kg_limiter:
             chunks, token_count = await run_raptor(task, chat_model, embedding_model, vector_size, progress_callback)
@@ -580,7 +576,6 @@ async def do_handle_task(task):
         graphrag_conf = task["kb_parser_config"].get("graphrag", {})
         start_ts = timer()
         chat_model = LLMBundle(task_tenant_id, LLMType.CHAT, llm_name=task_llm_id, lang=task_language)
-        await is_strong_enough(chat_model, None)
         with_resolution = graphrag_conf.get("resolution", False)
         with_community = graphrag_conf.get("community", False)
         async with kg_limiter: