Compare commits

...

25 Commits

Author SHA1 Message Date
675d18d359 Add docs category file (#12359)
### What problem does this PR solve?

As title.

### Type of change

- [x] Documentation Update

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-31 13:56:11 +08:00
750335978c Fix: Batch parsing problem (#12358)
### What problem does this PR solve?

Fix: Batch parsing problem

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-31 13:52:50 +08:00
ae7c623a35 fix(rag/prompts): Restructure metadata extraction rules for precision (#12360)
### What problem does this PR solve?

- Simplified and consolidated extraction rules
- Emphasized strict evidence-based extraction only
- Strengthened enum handling and hallucination prevention
- Clarified output requirements for empty results

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-31 13:52:33 +08:00
f24bdc0f83 Remove doc of health check (#12363)
### What problem does this PR solve?

System web page is disabled since v0.22.0, and the health check API is
also described in API reference. This document is obsolete.

### Type of change

- [x] Documentation Update

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-31 13:49:38 +08:00
07ef35b7e6 Docs: Update version references to v0.23.1 in READMEs and docs (#12349)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.23.0 to v0.23.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-12-31 12:49:42 +08:00
7c9823a1ff Update release notes (#12356)
### What problem does this PR solve?

As title

### Type of change

- [x] Documentation Update

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-31 12:49:09 +08:00
a0c3bcf798 [Bug] Don't display not used component status in admin status dashboard (#12355)
### What problem does this PR solve?

Currently, all components in configs are displayed even they are not
used. This PR is to fix it.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-31 12:40:52 +08:00
1a4a7d1705 Fix: apply kb configured llm issue. (#12354)
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-31 12:40:28 +08:00
f141947085 [Feature] Admin: sort user list by email (#12350)
### What problem does this PR solve?

Sort the user list by user email address.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-31 11:55:18 +08:00
a07e947644 Feat: Fixed the issue where the newly created agent begin node displayed "undefined". #10427 (#12348)
### What problem does this PR solve?
Feat: Fixed the issue where the newly created agent begin node displayed
"undefined". #10427

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2025-12-31 11:19:33 +08:00
ae4692a845 Fix: Bug fixed (#12345)
### What problem does this PR solve?

Fix: Bug fixed
- Memory type multilingual display
- Name modification is prohibited in Data source
- Jump directly to Metadata settings

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-31 10:43:57 +08:00
7dac269429 fix: correct session reference initialization to prevent dialogue misalignment (#12343)
## Summary

Fixes #12311

Changes the `reference` field initialization from `[{}]` to `[]` in
session creation.

### Problem

When creating a session via the SDK API, the `reference` field was
incorrectly initialized as `[{}]`. This caused:
- First dialogue round: Empty reference
- Second dialogue round: Reference pointing to first round's data
- Overall misalignment between dialogue rounds and their references

### Solution

Changed the initialization to `[]` (empty list), which:
- Matches the `Conversation` model's expected default
- Ensures references grow correctly one-to-one with assistant responses
- Aligns with the service layer's expectations

### Testing

After applying this fix:
1. Create a session via `POST /api/v1/chats/{conversation_id}/sessions`
2. Send multiple questions via `POST
/api/v1/chats/{conversation_id}/completions`
3. View the conversation on web - references should now align correctly
with each dialogue round
2025-12-31 10:25:49 +08:00
ec5575dce2 fix(admin-ui): pagination auto reset to first page when after refetching data (#12339)
### What problem does this PR solve?

Admin user enabling/disabling a user will causing user list reset to
first page

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-31 09:39:13 +08:00
6fee60e110 Docs: What is RAG & What is Agent context engine (#12341)
### Type of change

- [x] Documentation Update
2025-12-30 21:29:21 +08:00
52f91c2388 Refine: image/table context. (#12336)
### What problem does this PR solve?

#12303

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-30 20:24:27 +08:00
348265afc1 Feat: Display mode at the begin node #10427 (#12326)
### What problem does this PR solve?

Feat: Display mode at the begin node #10427
### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2025-12-30 19:53:24 +08:00
a7e466142d Fix: Dataset parse logic (#12330)
### What problem does this PR solve?

Fix: Dataset  logic of parser

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-30 19:53:00 +08:00
2fccf3924d Feat: Adapt the theme of the documentation page. #10427 (#12337)
### What problem does this PR solve?

Feat: Adapt the theme of the documentation page. #10427
### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2025-12-30 19:35:44 +08:00
4705d07e11 fix: malformed dynamic translation key chunk.docType.${chunkType} (#12329)
### What problem does this PR solve?

Back-end may returns empty array on `"doc_type_kwd"` property which
causes translation key malformed.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-12-30 19:33:20 +08:00
68be3b9a3d Update release workflow (#12335)
### What problem does this PR solve?

As title

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-30 19:06:27 +08:00
e2d17d808b Potential fix for code scanning alert no. 62: Workflow does not contain permissions (#12334)
Potential fix for
[https://github.com/infiniflow/ragflow/security/code-scanning/62](https://github.com/infiniflow/ragflow/security/code-scanning/62)

In general, the fix is to explicitly declare a `permissions:` block so
the GITHUB_TOKEN used by this workflow only has the scopes required:
read access to repository contents and write access to
contents/releases. Since this workflow creates or moves tags and
creates/overwrites releases via `softprops/action-gh-release`, it needs
`contents: write`. There is no evidence that it needs other elevated
scopes (issues, pull-requests, actions, etc.), so these should remain at
their default of `none` by omission.

The best minimal fix without changing existing functionality is to add a
workflow-level `permissions:` block near the top of
`.github/workflows/release.yml`, after `name:` and before `on:` (or
anywhere at the root level, but this is conventional). This will apply
to all jobs (there is only `jobs.release`) and ensure that the
GITHUB_TOKEN has only `contents: write`. No additional imports or
methods are needed because this is a YAML configuration change only.

Concretely:
- Edit `.github/workflows/release.yml`.
- Insert:

```yaml
permissions:
  contents: write
```

between line 2 (empty line after `name: release`) and line 3 (`on:`). No
other lines need to be changed.


_Suggested fixes powered by Copilot Autofix. Review carefully before
merging._

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-30 18:59:51 +08:00
95edbd43ba Update model providers (#12333)
### What problem does this PR solve?

As title

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-30 18:51:35 +08:00
b96d553cd8 Update release workflow (#12327)
### What problem does this PR solve?

As title.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-12-30 17:25:27 +08:00
bffdb5fb11 Feat: add IMAP data source integration with configuration and sync capabilities (#12316)
### What problem does this PR solve?
issue:
#12217 [#12313](https://github.com/infiniflow/ragflow/issues/12313)
change:
add IMAP data source integration with configuration and sync
capabilities

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-12-30 17:09:13 +08:00
109e782493 Feat: On the agent page and chat page, you can only select knowledge bases that use the same embedding model. #12320 (#12321)
### What problem does this PR solve?

Feat: On the agent page and chat page, you can only select knowledge
bases that use the same embedding model. #12320

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2025-12-30 17:08:30 +08:00
81 changed files with 1904 additions and 640 deletions

View File

@ -10,6 +10,12 @@ on:
tags: tags:
- "v*.*.*" # normal release - "v*.*.*" # normal release
permissions:
contents: write
actions: read
checks: read
statuses: read
# https://docs.github.com/en/actions/using-jobs/using-concurrency # https://docs.github.com/en/actions/using-jobs/using-concurrency
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
@ -76,6 +82,14 @@ jobs:
# The body field does not support environment variable substitution directly. # The body field does not support environment variable substitution directly.
body_path: release_body.md body_path: release_body.md
- name: Build and push image
run: |
sudo docker login --username infiniflow --password-stdin <<< ${{ secrets.DOCKERHUB_TOKEN }}
sudo docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -t infiniflow/ragflow:${RELEASE_TAG} -f Dockerfile .
sudo docker tag infiniflow/ragflow:${RELEASE_TAG} infiniflow/ragflow:latest
sudo docker push infiniflow/ragflow:${RELEASE_TAG}
sudo docker push infiniflow/ragflow:latest
- name: Build and push ragflow-sdk - name: Build and push ragflow-sdk
if: startsWith(github.ref, 'refs/tags/v') if: startsWith(github.ref, 'refs/tags/v')
run: | run: |
@ -85,11 +99,3 @@ jobs:
if: startsWith(github.ref, 'refs/tags/v') if: startsWith(github.ref, 'refs/tags/v')
run: | run: |
cd admin/client && uv build && uv publish --token ${{ secrets.PYPI_API_TOKEN }} cd admin/client && uv build && uv publish --token ${{ secrets.PYPI_API_TOKEN }}
- name: Build and push image
run: |
sudo docker login --username infiniflow --password-stdin <<< ${{ secrets.DOCKERHUB_TOKEN }}
sudo docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -t infiniflow/ragflow:${RELEASE_TAG} -f Dockerfile .
sudo docker tag infiniflow/ragflow:${RELEASE_TAG} infiniflow/ragflow:latest
sudo docker push infiniflow/ragflow:${RELEASE_TAG}
sudo docker push infiniflow/ragflow:latest

View File

@ -22,7 +22,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Document</a> | <a href="https://ragflow.io/docs/dev/">Document</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -188,12 +188,12 @@ releases! 🌟
> All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64. > All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64.
> If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system. > If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system.
> The command below downloads the `v0.23.0` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.23.0`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server. > The command below downloads the `v0.23.1` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.23.1`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases) # Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version. # This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
@ -396,7 +396,7 @@ docker build --platform linux/amd64 \
## 📜 Roadmap ## 📜 Roadmap
See the [RAGFlow Roadmap 2025](https://github.com/infiniflow/ragflow/issues/4214) See the [RAGFlow Roadmap 2026](https://github.com/infiniflow/ragflow/issues/12241)
## 🏄 Community ## 🏄 Community

View File

@ -22,7 +22,7 @@
<img alt="Lencana Daring" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Lencana Daring" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Rilis%20Terbaru" alt="Rilis Terbaru"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Rilis%20Terbaru" alt="Rilis Terbaru">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Dokumentasi</a> | <a href="https://ragflow.io/docs/dev/">Dokumentasi</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Peta Jalan</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Peta Jalan</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -188,12 +188,12 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io).
> Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64. > Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64.
> Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image). > Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image).
> Perintah di bawah ini mengunduh edisi v0.23.0 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.23.0, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server. > Perintah di bawah ini mengunduh edisi v0.23.1 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.23.1, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# Opsional: gunakan tag stabil (lihat releases: https://github.com/infiniflow/ragflow/releases) # Opsional: gunakan tag stabil (lihat releases: https://github.com/infiniflow/ragflow/releases)
# This steps ensures the **entrypoint.sh** file in the code matches the Docker image version. # This steps ensures the **entrypoint.sh** file in the code matches the Docker image version.
@ -368,7 +368,7 @@ docker build --platform linux/amd64 \
## 📜 Roadmap ## 📜 Roadmap
Lihat [Roadmap RAGFlow 2025](https://github.com/infiniflow/ragflow/issues/4214) Lihat [Roadmap RAGFlow 2026](https://github.com/infiniflow/ragflow/issues/12241)
## 🏄 Komunitas ## 🏄 Komunitas

View File

@ -22,7 +22,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Document</a> | <a href="https://ragflow.io/docs/dev/">Document</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -168,12 +168,12 @@
> 現在、公式に提供されているすべての Docker イメージは x86 アーキテクチャ向けにビルドされており、ARM64 用の Docker イメージは提供されていません。 > 現在、公式に提供されているすべての Docker イメージは x86 アーキテクチャ向けにビルドされており、ARM64 用の Docker イメージは提供されていません。
> ARM64 アーキテクチャのオペレーティングシステムを使用している場合は、[このドキュメント](https://ragflow.io/docs/dev/build_docker_image)を参照して Docker イメージを自分でビルドしてください。 > ARM64 アーキテクチャのオペレーティングシステムを使用している場合は、[このドキュメント](https://ragflow.io/docs/dev/build_docker_image)を参照して Docker イメージを自分でビルドしてください。
> 以下のコマンドは、RAGFlow Docker イメージの v0.23.0 エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.23.0 とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。 > 以下のコマンドは、RAGFlow Docker イメージの v0.23.1 エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.23.1 とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# 任意: 安定版タグを利用 (一覧: https://github.com/infiniflow/ragflow/releases) # 任意: 安定版タグを利用 (一覧: https://github.com/infiniflow/ragflow/releases)
# この手順は、コード内の entrypoint.sh ファイルが Docker イメージのバージョンと一致していることを確認します。 # この手順は、コード内の entrypoint.sh ファイルが Docker イメージのバージョンと一致していることを確認します。
@ -368,7 +368,7 @@ docker build --platform linux/amd64 \
## 📜 ロードマップ ## 📜 ロードマップ
[RAGFlow ロードマップ 2025](https://github.com/infiniflow/ragflow/issues/4214) を参照 [RAGFlow ロードマップ 2026](https://github.com/infiniflow/ragflow/issues/12241) を参照
## 🏄 コミュニティ ## 🏄 コミュニティ

View File

@ -22,7 +22,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Document</a> | <a href="https://ragflow.io/docs/dev/">Document</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -170,12 +170,12 @@
> 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다. > 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다.
> ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image). > ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image).
> 아래 명령어는 RAGFlow Docker 이미지의 v0.23.0 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.23.0과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오. > 아래 명령어는 RAGFlow Docker 이미지의 v0.23.1 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.23.1과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases) # Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# 이 단계는 코드의 entrypoint.sh 파일이 Docker 이미지 버전과 일치하도록 보장합니다. # 이 단계는 코드의 entrypoint.sh 파일이 Docker 이미지 버전과 일치하도록 보장합니다.
@ -372,7 +372,7 @@ docker build --platform linux/amd64 \
## 📜 로드맵 ## 📜 로드맵
[RAGFlow 로드맵 2025](https://github.com/infiniflow/ragflow/issues/4214)을 확인하세요. [RAGFlow 로드맵 2026](https://github.com/infiniflow/ragflow/issues/12241)을 확인하세요.
## 🏄 커뮤니티 ## 🏄 커뮤니티

View File

@ -22,7 +22,7 @@
<img alt="Badge Estático" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Badge Estático" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Última%20Relese" alt="Última Versão"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Última%20Relese" alt="Última Versão">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Documentação</a> | <a href="https://ragflow.io/docs/dev/">Documentação</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -188,12 +188,12 @@ Experimente nossa demo em [https://demo.ragflow.io](https://demo.ragflow.io).
> Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64. > Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64.
> Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema. > Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema.
> O comando abaixo baixa a edição`v0.23.0` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.23.0`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor. > O comando abaixo baixa a edição`v0.23.1` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.23.1`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# Opcional: use uma tag estável (veja releases: https://github.com/infiniflow/ragflow/releases) # Opcional: use uma tag estável (veja releases: https://github.com/infiniflow/ragflow/releases)
# Esta etapa garante que o arquivo entrypoint.sh no código corresponda à versão da imagem do Docker. # Esta etapa garante que o arquivo entrypoint.sh no código corresponda à versão da imagem do Docker.
@ -385,7 +385,7 @@ docker build --platform linux/amd64 \
## 📜 Roadmap ## 📜 Roadmap
Veja o [RAGFlow Roadmap 2025](https://github.com/infiniflow/ragflow/issues/4214) Veja o [RAGFlow Roadmap 2026](https://github.com/infiniflow/ragflow/issues/12241)
## 🏄 Comunidade ## 🏄 Comunidade

View File

@ -22,7 +22,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Document</a> | <a href="https://ragflow.io/docs/dev/">Document</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -187,12 +187,12 @@
> 所有 Docker 映像檔都是為 x86 平台建置的。目前,我們不提供 ARM64 平台的 Docker 映像檔。 > 所有 Docker 映像檔都是為 x86 平台建置的。目前,我們不提供 ARM64 平台的 Docker 映像檔。
> 如果您使用的是 ARM64 平台,請使用 [這份指南](https://ragflow.io/docs/dev/build_docker_image) 來建置適合您系統的 Docker 映像檔。 > 如果您使用的是 ARM64 平台,請使用 [這份指南](https://ragflow.io/docs/dev/build_docker_image) 來建置適合您系統的 Docker 映像檔。
> 執行以下指令會自動下載 RAGFlow Docker 映像 `v0.23.0`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.23.0` 的 Docker 映像,請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。 > 執行以下指令會自動下載 RAGFlow Docker 映像 `v0.23.1`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.23.1` 的 Docker 映像,請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# 可選使用穩定版標籤查看發佈https://github.com/infiniflow/ragflow/releases # 可選使用穩定版標籤查看發佈https://github.com/infiniflow/ragflow/releases
# 此步驟確保程式碼中的 entrypoint.sh 檔案與 Docker 映像版本一致。 # 此步驟確保程式碼中的 entrypoint.sh 檔案與 Docker 映像版本一致。
@ -399,7 +399,7 @@ docker build --platform linux/amd64 \
## 📜 路線圖 ## 📜 路線圖
詳見 [RAGFlow Roadmap 2025](https://github.com/infiniflow/ragflow/issues/4214) 。 詳見 [RAGFlow Roadmap 2026](https://github.com/infiniflow/ragflow/issues/12241) 。
## 🏄 開源社群 ## 🏄 開源社群

View File

@ -22,7 +22,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99"> <img alt="Static Badge" src="https://img.shields.io/badge/Online-Demo-4e6b99">
</a> </a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank"> <a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.0"> <img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.23.1">
</a> </a>
<a href="https://github.com/infiniflow/ragflow/releases/latest"> <a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release"> <img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@ -37,7 +37,7 @@
<h4 align="center"> <h4 align="center">
<a href="https://ragflow.io/docs/dev/">Document</a> | <a href="https://ragflow.io/docs/dev/">Document</a> |
<a href="https://github.com/infiniflow/ragflow/issues/4214">Roadmap</a> | <a href="https://github.com/infiniflow/ragflow/issues/12241">Roadmap</a> |
<a href="https://twitter.com/infiniflowai">Twitter</a> | <a href="https://twitter.com/infiniflowai">Twitter</a> |
<a href="https://discord.gg/NjYzJD3GM3">Discord</a> | <a href="https://discord.gg/NjYzJD3GM3">Discord</a> |
<a href="https://demo.ragflow.io">Demo</a> <a href="https://demo.ragflow.io">Demo</a>
@ -188,12 +188,12 @@
> 请注意,目前官方提供的所有 Docker 镜像均基于 x86 架构构建,并不提供基于 ARM64 的 Docker 镜像。 > 请注意,目前官方提供的所有 Docker 镜像均基于 x86 架构构建,并不提供基于 ARM64 的 Docker 镜像。
> 如果你的操作系统是 ARM64 架构,请参考[这篇文档](https://ragflow.io/docs/dev/build_docker_image)自行构建 Docker 镜像。 > 如果你的操作系统是 ARM64 架构,请参考[这篇文档](https://ragflow.io/docs/dev/build_docker_image)自行构建 Docker 镜像。
> 运行以下命令会自动下载 RAGFlow Docker 镜像 `v0.23.0`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.23.0` 的 Docker 镜像,请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。 > 运行以下命令会自动下载 RAGFlow Docker 镜像 `v0.23.1`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.23.1` 的 Docker 镜像,请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。
```bash ```bash
$ cd ragflow/docker $ cd ragflow/docker
# git checkout v0.23.0 # git checkout v0.23.1
# 可选使用稳定版本标签查看发布https://github.com/infiniflow/ragflow/releases # 可选使用稳定版本标签查看发布https://github.com/infiniflow/ragflow/releases
# 这一步确保代码中的 entrypoint.sh 文件与 Docker 镜像的版本保持一致。 # 这一步确保代码中的 entrypoint.sh 文件与 Docker 镜像的版本保持一致。
@ -402,7 +402,7 @@ docker build --platform linux/amd64 \
## 📜 路线图 ## 📜 路线图
详见 [RAGFlow Roadmap 2025](https://github.com/infiniflow/ragflow/issues/4214) 。 详见 [RAGFlow Roadmap 2026](https://github.com/infiniflow/ragflow/issues/12241) 。
## 🏄 开源社区 ## 🏄 开源社区

View File

@ -48,7 +48,7 @@ It consists of a server-side Service and a command-line client (CLI), both imple
1. Ensure the Admin Service is running. 1. Ensure the Admin Service is running.
2. Install ragflow-cli. 2. Install ragflow-cli.
```bash ```bash
pip install ragflow-cli==0.23.0 pip install ragflow-cli==0.23.1
``` ```
3. Launch the CLI client: 3. Launch the CLI client:
```bash ```bash

View File

@ -370,7 +370,7 @@ class AdminCLI(Cmd):
res_json = response.json() res_json = response.json()
error_code = res_json.get("code", -1) error_code = res_json.get("code", -1)
if error_code == 0: if error_code == 0:
self.session.headers.update({"Content-Type": "application/json", "Authorization": response.headers["Authorization"], "User-Agent": "RAGFlow-CLI/0.23.0"}) self.session.headers.update({"Content-Type": "application/json", "Authorization": response.headers["Authorization"], "User-Agent": "RAGFlow-CLI/0.23.1"})
print("Authentication successful.") print("Authentication successful.")
return True return True
else: else:

View File

@ -1,6 +1,6 @@
[project] [project]
name = "ragflow-cli" name = "ragflow-cli"
version = "0.23.0" version = "0.23.1"
description = "Admin Service's client of [RAGFlow](https://github.com/infiniflow/ragflow). The Admin Service provides user management and system monitoring. " description = "Admin Service's client of [RAGFlow](https://github.com/infiniflow/ragflow). The Admin Service provides user management and system monitoring. "
authors = [{ name = "Lynn", email = "lynn_inf@hotmail.com" }] authors = [{ name = "Lynn", email = "lynn_inf@hotmail.com" }]
license = { text = "Apache License, Version 2.0" } license = { text = "Apache License, Version 2.0" }

2
admin/client/uv.lock generated
View File

@ -196,7 +196,7 @@ wheels = [
[[package]] [[package]]
name = "ragflow-cli" name = "ragflow-cli"
version = "0.23.0" version = "0.23.1"
source = { virtual = "." } source = { virtual = "." }
dependencies = [ dependencies = [
{ name = "beartype" }, { name = "beartype" },

View File

@ -13,6 +13,8 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# #
import os
import logging import logging
import re import re
from werkzeug.security import check_password_hash from werkzeug.security import check_password_hash
@ -179,10 +181,14 @@ class ServiceMgr:
@staticmethod @staticmethod
def get_all_services(): def get_all_services():
doc_engine = os.getenv('DOC_ENGINE', 'elasticsearch')
result = [] result = []
configs = SERVICE_CONFIGS.configs configs = SERVICE_CONFIGS.configs
for service_id, config in enumerate(configs): for service_id, config in enumerate(configs):
config_dict = config.to_dict() config_dict = config.to_dict()
if config_dict['service_type'] == 'retrieval':
if config_dict['extra']['retrieval_type'] != doc_engine:
continue
try: try:
service_detail = ServiceMgr.get_service_details(service_id) service_detail = ServiceMgr.get_service_details(service_id)
if "status" in service_detail: if "status" in service_detail:

View File

@ -60,7 +60,7 @@ async def create(tenant_id, chat_id):
"name": req.get("name", "New session"), "name": req.get("name", "New session"),
"message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}], "message": [{"role": "assistant", "content": dia[0].prompt_config.get("prologue")}],
"user_id": req.get("user_id", ""), "user_id": req.get("user_id", ""),
"reference": [{}], "reference": [],
} }
if not conv.get("name"): if not conv.get("name"):
return get_error_data_result(message="`name` can not be empty.") return get_error_data_result(message="`name` can not be empty.")

View File

@ -164,7 +164,7 @@ class UserService(CommonService):
@classmethod @classmethod
@DB.connection_context() @DB.connection_context()
def get_all_users(cls): def get_all_users(cls):
users = cls.model.select() users = cls.model.select().order_by(cls.model.email)
return list(users) return list(users)

View File

@ -132,6 +132,7 @@ class FileSource(StrEnum):
ASANA = "asana" ASANA = "asana"
GITHUB = "github" GITHUB = "github"
GITLAB = "gitlab" GITLAB = "gitlab"
IMAP = "imap"
class PipelineTaskType(StrEnum): class PipelineTaskType(StrEnum):
PARSE = "Parse" PARSE = "Parse"

View File

@ -38,6 +38,7 @@ from .webdav_connector import WebDAVConnector
from .moodle_connector import MoodleConnector from .moodle_connector import MoodleConnector
from .airtable_connector import AirtableConnector from .airtable_connector import AirtableConnector
from .asana_connector import AsanaConnector from .asana_connector import AsanaConnector
from .imap_connector import ImapConnector
from .config import BlobType, DocumentSource from .config import BlobType, DocumentSource
from .models import Document, TextSection, ImageSection, BasicExpertInfo from .models import Document, TextSection, ImageSection, BasicExpertInfo
from .exceptions import ( from .exceptions import (
@ -75,4 +76,5 @@ __all__ = [
"UnexpectedValidationError", "UnexpectedValidationError",
"AirtableConnector", "AirtableConnector",
"AsanaConnector", "AsanaConnector",
"ImapConnector"
] ]

View File

@ -1,6 +1,6 @@
from datetime import datetime, timezone from datetime import datetime, timezone
import logging import logging
from typing import Any from typing import Any, Generator
import requests import requests
@ -8,8 +8,8 @@ from pyairtable import Api as AirtableApi
from common.data_source.config import AIRTABLE_CONNECTOR_SIZE_THRESHOLD, INDEX_BATCH_SIZE, DocumentSource from common.data_source.config import AIRTABLE_CONNECTOR_SIZE_THRESHOLD, INDEX_BATCH_SIZE, DocumentSource
from common.data_source.exceptions import ConnectorMissingCredentialError from common.data_source.exceptions import ConnectorMissingCredentialError
from common.data_source.interfaces import LoadConnector from common.data_source.interfaces import LoadConnector, PollConnector
from common.data_source.models import Document, GenerateDocumentsOutput from common.data_source.models import Document, GenerateDocumentsOutput, SecondsSinceUnixEpoch
from common.data_source.utils import extract_size_bytes, get_file_ext from common.data_source.utils import extract_size_bytes, get_file_ext
class AirtableClientNotSetUpError(PermissionError): class AirtableClientNotSetUpError(PermissionError):
@ -19,7 +19,7 @@ class AirtableClientNotSetUpError(PermissionError):
) )
class AirtableConnector(LoadConnector): class AirtableConnector(LoadConnector, PollConnector):
""" """
Lightweight Airtable connector. Lightweight Airtable connector.
@ -132,6 +132,26 @@ class AirtableConnector(LoadConnector):
if batch: if batch:
yield batch yield batch
def poll_source(self, start: SecondsSinceUnixEpoch, end: SecondsSinceUnixEpoch) -> Generator[list[Document], None, None]:
"""Poll source to get documents"""
start_dt = datetime.fromtimestamp(start, tz=timezone.utc)
end_dt = datetime.fromtimestamp(end, tz=timezone.utc)
for batch in self.load_from_state():
filtered: list[Document] = []
for doc in batch:
if not doc.doc_updated_at:
continue
doc_dt = doc.doc_updated_at.astimezone(timezone.utc)
if start_dt <= doc_dt < end_dt:
filtered.append(doc)
if filtered:
yield filtered
if __name__ == "__main__": if __name__ == "__main__":
import os import os

View File

@ -57,6 +57,7 @@ class DocumentSource(str, Enum):
ASANA = "asana" ASANA = "asana"
GITHUB = "github" GITHUB = "github"
GITLAB = "gitlab" GITLAB = "gitlab"
IMAP = "imap"
class FileOrigin(str, Enum): class FileOrigin(str, Enum):
@ -266,6 +267,10 @@ ASANA_CONNECTOR_SIZE_THRESHOLD = int(
os.environ.get("ASANA_CONNECTOR_SIZE_THRESHOLD", 10 * 1024 * 1024) os.environ.get("ASANA_CONNECTOR_SIZE_THRESHOLD", 10 * 1024 * 1024)
) )
IMAP_CONNECTOR_SIZE_THRESHOLD = int(
os.environ.get("IMAP_CONNECTOR_SIZE_THRESHOLD", 10 * 1024 * 1024)
)
_USER_NOT_FOUND = "Unknown Confluence User" _USER_NOT_FOUND = "Unknown Confluence User"
_COMMENT_EXPANSION_FIELDS = ["body.storage.value"] _COMMENT_EXPANSION_FIELDS = ["body.storage.value"]

View File

@ -0,0 +1,724 @@
import copy
import email
from email.header import decode_header
import imaplib
import logging
import os
import re
from datetime import datetime, timedelta
from datetime import timezone
from email.message import Message
from email.utils import collapse_rfc2231_value, parseaddr
from enum import Enum
from typing import Any
from typing import cast
import bs4
from pydantic import BaseModel
from common.data_source.config import IMAP_CONNECTOR_SIZE_THRESHOLD, DocumentSource
from common.data_source.interfaces import CheckpointOutput, CheckpointedConnectorWithPermSync, CredentialsConnector, CredentialsProviderInterface
from common.data_source.models import BasicExpertInfo, ConnectorCheckpoint, Document, ExternalAccess, SecondsSinceUnixEpoch
_DEFAULT_IMAP_PORT_NUMBER = int(os.environ.get("IMAP_PORT", 993))
_IMAP_OKAY_STATUS = "OK"
_PAGE_SIZE = 100
_USERNAME_KEY = "imap_username"
_PASSWORD_KEY = "imap_password"
class Header(str, Enum):
SUBJECT_HEADER = "subject"
FROM_HEADER = "from"
TO_HEADER = "to"
CC_HEADER = "cc"
DELIVERED_TO_HEADER = (
"Delivered-To" # Used in mailing lists instead of the "to" header.
)
DATE_HEADER = "date"
MESSAGE_ID_HEADER = "Message-ID"
class EmailHeaders(BaseModel):
"""
Model for email headers extracted from IMAP messages.
"""
id: str
subject: str
sender: str
recipients: str | None
cc: str | None
date: datetime
@classmethod
def from_email_msg(cls, email_msg: Message) -> "EmailHeaders":
def _decode(header: str, default: str | None = None) -> str | None:
value = email_msg.get(header, default)
if not value:
return None
decoded_fragments = decode_header(value)
decoded_strings: list[str] = []
for decoded_value, encoding in decoded_fragments:
if isinstance(decoded_value, bytes):
try:
decoded_strings.append(
decoded_value.decode(encoding or "utf-8", errors="replace")
)
except LookupError:
decoded_strings.append(
decoded_value.decode("utf-8", errors="replace")
)
elif isinstance(decoded_value, str):
decoded_strings.append(decoded_value)
else:
decoded_strings.append(str(decoded_value))
return "".join(decoded_strings)
def _parse_date(date_str: str | None) -> datetime | None:
if not date_str:
return None
try:
return email.utils.parsedate_to_datetime(date_str)
except (TypeError, ValueError):
return None
message_id = _decode(header=Header.MESSAGE_ID_HEADER)
if not message_id:
message_id = f"<generated-{uuid.uuid4()}@imap.local>"
# It's possible for the subject line to not exist or be an empty string.
subject = _decode(header=Header.SUBJECT_HEADER) or "Unknown Subject"
from_ = _decode(header=Header.FROM_HEADER)
to = _decode(header=Header.TO_HEADER)
if not to:
to = _decode(header=Header.DELIVERED_TO_HEADER)
cc = _decode(header=Header.CC_HEADER)
date_str = _decode(header=Header.DATE_HEADER)
date = _parse_date(date_str=date_str)
if not date:
date = datetime.now(tz=timezone.utc)
# If any of the above are `None`, model validation will fail.
# Therefore, no guards (i.e.: `if <header> is None: raise RuntimeError(..)`) were written.
return cls.model_validate(
{
"id": message_id,
"subject": subject,
"sender": from_,
"recipients": to,
"cc": cc,
"date": date,
}
)
class CurrentMailbox(BaseModel):
mailbox: str
todo_email_ids: list[str]
# An email has a list of mailboxes.
# Each mailbox has a list of email-ids inside of it.
#
# Usage:
# To use this checkpointer, first fetch all the mailboxes.
# Then, pop a mailbox and fetch all of its email-ids.
# Then, pop each email-id and fetch its content (and parse it, etc..).
# When you have popped all email-ids for this mailbox, pop the next mailbox and repeat the above process until you're done.
#
# For initial checkpointing, set both fields to `None`.
class ImapCheckpoint(ConnectorCheckpoint):
todo_mailboxes: list[str] | None = None
current_mailbox: CurrentMailbox | None = None
class LoginState(str, Enum):
LoggedIn = "logged_in"
LoggedOut = "logged_out"
class ImapConnector(
CredentialsConnector,
CheckpointedConnectorWithPermSync,
):
def __init__(
self,
host: str,
port: int = _DEFAULT_IMAP_PORT_NUMBER,
mailboxes: list[str] | None = None,
) -> None:
self._host = host
self._port = port
self._mailboxes = mailboxes
self._credentials: dict[str, Any] | None = None
@property
def credentials(self) -> dict[str, Any]:
if not self._credentials:
raise RuntimeError(
"Credentials have not been initialized; call `set_credentials_provider` first"
)
return self._credentials
def _get_mail_client(self) -> imaplib.IMAP4_SSL:
"""
Returns a new `imaplib.IMAP4_SSL` instance.
The `imaplib.IMAP4_SSL` object is supposed to be an "ephemeral" object; it's not something that you can login,
logout, then log back into again. I.e., the following will fail:
```py
mail_client.login(..)
mail_client.logout();
mail_client.login(..)
```
Therefore, you need a fresh, new instance in order to operate with IMAP. This function gives one to you.
# Notes
This function will throw an error if the credentials have not yet been set.
"""
def get_or_raise(name: str) -> str:
value = self.credentials.get(name)
if not value:
raise RuntimeError(f"Credential item {name=} was not found")
if not isinstance(value, str):
raise RuntimeError(
f"Credential item {name=} must be of type str, instead received {type(name)=}"
)
return value
username = get_or_raise(_USERNAME_KEY)
password = get_or_raise(_PASSWORD_KEY)
mail_client = imaplib.IMAP4_SSL(host=self._host, port=self._port)
status, _data = mail_client.login(user=username, password=password)
if status != _IMAP_OKAY_STATUS:
raise RuntimeError(f"Failed to log into imap server; {status=}")
return mail_client
def _load_from_checkpoint(
self,
start: SecondsSinceUnixEpoch,
end: SecondsSinceUnixEpoch,
checkpoint: ImapCheckpoint,
include_perm_sync: bool,
) -> CheckpointOutput[ImapCheckpoint]:
checkpoint = cast(ImapCheckpoint, copy.deepcopy(checkpoint))
checkpoint.has_more = True
mail_client = self._get_mail_client()
if checkpoint.todo_mailboxes is None:
# This is the dummy checkpoint.
# Fill it with mailboxes first.
if self._mailboxes:
checkpoint.todo_mailboxes = _sanitize_mailbox_names(self._mailboxes)
else:
fetched_mailboxes = _fetch_all_mailboxes_for_email_account(
mail_client=mail_client
)
if not fetched_mailboxes:
raise RuntimeError(
"Failed to find any mailboxes for this email account"
)
checkpoint.todo_mailboxes = _sanitize_mailbox_names(fetched_mailboxes)
return checkpoint
if (
not checkpoint.current_mailbox
or not checkpoint.current_mailbox.todo_email_ids
):
if not checkpoint.todo_mailboxes:
checkpoint.has_more = False
return checkpoint
mailbox = checkpoint.todo_mailboxes.pop()
email_ids = _fetch_email_ids_in_mailbox(
mail_client=mail_client,
mailbox=mailbox,
start=start,
end=end,
)
checkpoint.current_mailbox = CurrentMailbox(
mailbox=mailbox,
todo_email_ids=email_ids,
)
_select_mailbox(
mail_client=mail_client, mailbox=checkpoint.current_mailbox.mailbox
)
current_todos = cast(
list, copy.deepcopy(checkpoint.current_mailbox.todo_email_ids[:_PAGE_SIZE])
)
checkpoint.current_mailbox.todo_email_ids = (
checkpoint.current_mailbox.todo_email_ids[_PAGE_SIZE:]
)
for email_id in current_todos:
email_msg = _fetch_email(mail_client=mail_client, email_id=email_id)
if not email_msg:
logging.warning(f"Failed to fetch message {email_id=}; skipping")
continue
email_headers = EmailHeaders.from_email_msg(email_msg=email_msg)
msg_dt = email_headers.date
if msg_dt.tzinfo is None:
msg_dt = msg_dt.replace(tzinfo=timezone.utc)
else:
msg_dt = msg_dt.astimezone(timezone.utc)
start_dt = datetime.fromtimestamp(start, tz=timezone.utc)
end_dt = datetime.fromtimestamp(end, tz=timezone.utc)
if not (start_dt < msg_dt <= end_dt):
continue
email_doc = _convert_email_headers_and_body_into_document(
email_msg=email_msg,
email_headers=email_headers,
include_perm_sync=include_perm_sync,
)
yield email_doc
attachments = extract_attachments(email_msg)
for att in attachments:
yield attachment_to_document(email_doc, att, email_headers)
return checkpoint
# impls for BaseConnector
def load_credentials(self, credentials: dict[str, Any]) -> dict[str, Any] | None:
self._credentials = credentials
return None
def validate_connector_settings(self) -> None:
self._get_mail_client()
# impls for CredentialsConnector
def set_credentials_provider(
self, credentials_provider: CredentialsProviderInterface
) -> None:
self._credentials = credentials_provider.get_credentials()
# impls for CheckpointedConnector
def load_from_checkpoint(
self,
start: SecondsSinceUnixEpoch,
end: SecondsSinceUnixEpoch,
checkpoint: ImapCheckpoint,
) -> CheckpointOutput[ImapCheckpoint]:
return self._load_from_checkpoint(
start=start, end=end, checkpoint=checkpoint, include_perm_sync=False
)
def build_dummy_checkpoint(self) -> ImapCheckpoint:
return ImapCheckpoint(has_more=True)
def validate_checkpoint_json(self, checkpoint_json: str) -> ImapCheckpoint:
return ImapCheckpoint.model_validate_json(json_data=checkpoint_json)
# impls for CheckpointedConnectorWithPermSync
def load_from_checkpoint_with_perm_sync(
self,
start: SecondsSinceUnixEpoch,
end: SecondsSinceUnixEpoch,
checkpoint: ImapCheckpoint,
) -> CheckpointOutput[ImapCheckpoint]:
return self._load_from_checkpoint(
start=start, end=end, checkpoint=checkpoint, include_perm_sync=True
)
def _fetch_all_mailboxes_for_email_account(mail_client: imaplib.IMAP4_SSL) -> list[str]:
status, mailboxes_data = mail_client.list('""', "*")
if status != _IMAP_OKAY_STATUS:
raise RuntimeError(f"Failed to fetch mailboxes; {status=}")
mailboxes = []
for mailboxes_raw in mailboxes_data:
if isinstance(mailboxes_raw, bytes):
mailboxes_str = mailboxes_raw.decode()
elif isinstance(mailboxes_raw, str):
mailboxes_str = mailboxes_raw
else:
logging.warning(
f"Expected the mailbox data to be of type str, instead got {type(mailboxes_raw)=} {mailboxes_raw}; skipping"
)
continue
# The mailbox LIST response output can be found here:
# https://www.rfc-editor.org/rfc/rfc3501.html#section-7.2.2
#
# The general format is:
# `(<name-attributes>) <hierarchy-delimiter> <mailbox-name>`
#
# The below regex matches on that pattern; from there, we select the 3rd match (index 2), which is the mailbox-name.
match = re.match(r'\([^)]*\)\s+"([^"]+)"\s+"?(.+?)"?$', mailboxes_str)
if not match:
logging.warning(
f"Invalid mailbox-data formatting structure: {mailboxes_str=}; skipping"
)
continue
mailbox = match.group(2)
mailboxes.append(mailbox)
if not mailboxes:
logging.warning(
"No mailboxes parsed from LIST response; falling back to INBOX"
)
return ["INBOX"]
return mailboxes
def _select_mailbox(mail_client: imaplib.IMAP4_SSL, mailbox: str) -> bool:
try:
status, _ = mail_client.select(mailbox=mailbox, readonly=True)
if status != _IMAP_OKAY_STATUS:
return False
return True
except Exception:
return False
def _fetch_email_ids_in_mailbox(
mail_client: imaplib.IMAP4_SSL,
mailbox: str,
start: SecondsSinceUnixEpoch,
end: SecondsSinceUnixEpoch,
) -> list[str]:
if not _select_mailbox(mail_client, mailbox):
logging.warning(f"Skip mailbox: {mailbox}")
return []
start_dt = datetime.fromtimestamp(start, tz=timezone.utc)
end_dt = datetime.fromtimestamp(end, tz=timezone.utc) + timedelta(days=1)
start_str = start_dt.strftime("%d-%b-%Y")
end_str = end_dt.strftime("%d-%b-%Y")
search_criteria = f'(SINCE "{start_str}" BEFORE "{end_str}")'
status, email_ids_byte_array = mail_client.search(None, search_criteria)
if status != _IMAP_OKAY_STATUS or not email_ids_byte_array:
raise RuntimeError(f"Failed to fetch email ids; {status=}")
email_ids: bytes = email_ids_byte_array[0]
return [email_id.decode() for email_id in email_ids.split()]
def _fetch_email(mail_client: imaplib.IMAP4_SSL, email_id: str) -> Message | None:
status, msg_data = mail_client.fetch(message_set=email_id, message_parts="(RFC822)")
if status != _IMAP_OKAY_STATUS or not msg_data:
return None
data = msg_data[0]
if not isinstance(data, tuple):
raise RuntimeError(
f"Message data should be a tuple; instead got a {type(data)=} {data=}"
)
_, raw_email = data
return email.message_from_bytes(raw_email)
def _convert_email_headers_and_body_into_document(
email_msg: Message,
email_headers: EmailHeaders,
include_perm_sync: bool,
) -> Document:
sender_name, sender_addr = _parse_singular_addr(raw_header=email_headers.sender)
to_addrs = (
_parse_addrs(email_headers.recipients)
if email_headers.recipients
else []
)
cc_addrs = (
_parse_addrs(email_headers.cc)
if email_headers.cc
else []
)
all_participants = to_addrs + cc_addrs
expert_info_map = {
recipient_addr: BasicExpertInfo(
display_name=recipient_name, email=recipient_addr
)
for recipient_name, recipient_addr in all_participants
}
if sender_addr not in expert_info_map:
expert_info_map[sender_addr] = BasicExpertInfo(
display_name=sender_name, email=sender_addr
)
email_body = _parse_email_body(email_msg=email_msg, email_headers=email_headers)
primary_owners = list(expert_info_map.values())
external_access = (
ExternalAccess(
external_user_emails=set(expert_info_map.keys()),
external_user_group_ids=set(),
is_public=False,
)
if include_perm_sync
else None
)
return Document(
id=email_headers.id,
title=email_headers.subject,
blob=email_body,
size_bytes=len(email_body),
semantic_identifier=email_headers.subject,
metadata={},
extension='.txt',
doc_updated_at=email_headers.date,
source=DocumentSource.IMAP,
primary_owners=primary_owners,
external_access=external_access,
)
def extract_attachments(email_msg: Message, max_bytes: int = IMAP_CONNECTOR_SIZE_THRESHOLD):
attachments = []
if not email_msg.is_multipart():
return attachments
for part in email_msg.walk():
if part.get_content_maintype() == "multipart":
continue
disposition = (part.get("Content-Disposition") or "").lower()
filename = part.get_filename()
if not (
disposition.startswith("attachment")
or (disposition.startswith("inline") and filename)
):
continue
payload = part.get_payload(decode=True)
if not payload:
continue
if len(payload) > max_bytes:
continue
attachments.append({
"filename": filename or "attachment.bin",
"content_type": part.get_content_type(),
"content_bytes": payload,
"size_bytes": len(payload),
})
return attachments
def decode_mime_filename(raw: str | None) -> str | None:
if not raw:
return None
try:
raw = collapse_rfc2231_value(raw)
except Exception:
pass
parts = decode_header(raw)
decoded = []
for value, encoding in parts:
if isinstance(value, bytes):
decoded.append(value.decode(encoding or "utf-8", errors="replace"))
else:
decoded.append(value)
return "".join(decoded)
def attachment_to_document(
parent_doc: Document,
att: dict,
email_headers: EmailHeaders,
):
raw_filename = att["filename"]
filename = decode_mime_filename(raw_filename) or "attachment.bin"
ext = "." + filename.split(".")[-1] if "." in filename else ""
return Document(
id=f"{parent_doc.id}#att:{filename}",
source=DocumentSource.IMAP,
semantic_identifier=filename,
extension=ext,
blob=att["content_bytes"],
size_bytes=att["size_bytes"],
doc_updated_at=email_headers.date,
primary_owners=parent_doc.primary_owners,
metadata={
"parent_email_id": parent_doc.id,
"parent_subject": email_headers.subject,
"attachment_filename": filename,
"attachment_content_type": att["content_type"],
},
)
def _parse_email_body(
email_msg: Message,
email_headers: EmailHeaders,
) -> str:
body = None
for part in email_msg.walk():
if part.is_multipart():
# Multipart parts are *containers* for other parts, not the actual content itself.
# Therefore, we skip until we find the individual parts instead.
continue
charset = part.get_content_charset() or "utf-8"
try:
raw_payload = part.get_payload(decode=True)
if not isinstance(raw_payload, bytes):
logging.warning(
"Payload section from email was expected to be an array of bytes, instead got "
f"{type(raw_payload)=}, {raw_payload=}"
)
continue
body = raw_payload.decode(charset)
break
except (UnicodeDecodeError, LookupError) as e:
logging.warning(f"Could not decode part with charset {charset}. Error: {e}")
continue
if not body:
logging.warning(
f"Email with {email_headers.id=} has an empty body; returning an empty string"
)
return ""
soup = bs4.BeautifulSoup(markup=body, features="html.parser")
return " ".join(str_section for str_section in soup.stripped_strings)
def _sanitize_mailbox_names(mailboxes: list[str]) -> list[str]:
"""
Mailboxes with special characters in them must be enclosed by double-quotes, as per the IMAP protocol.
Just to be safe, we wrap *all* mailboxes with double-quotes.
"""
return [f'"{mailbox}"' for mailbox in mailboxes if mailbox]
def _parse_addrs(raw_header: str) -> list[tuple[str, str]]:
addrs = raw_header.split(",")
name_addr_pairs = [parseaddr(addr=addr) for addr in addrs if addr]
return [(name, addr) for name, addr in name_addr_pairs if addr]
def _parse_singular_addr(raw_header: str) -> tuple[str, str]:
addrs = _parse_addrs(raw_header=raw_header)
if not addrs:
return ("Unknown", "unknown@example.com")
elif len(addrs) >= 2:
raise RuntimeError(
f"Expected a singular address, but instead got multiple; {raw_header=} {addrs=}"
)
return addrs[0]
if __name__ == "__main__":
import time
import uuid
from types import TracebackType
from common.data_source.utils import load_all_docs_from_checkpoint_connector
class OnyxStaticCredentialsProvider(
CredentialsProviderInterface["OnyxStaticCredentialsProvider"]
):
"""Implementation (a very simple one!) to handle static credentials."""
def __init__(
self,
tenant_id: str | None,
connector_name: str,
credential_json: dict[str, Any],
):
self._tenant_id = tenant_id
self._connector_name = connector_name
self._credential_json = credential_json
self._provider_key = str(uuid.uuid4())
def __enter__(self) -> "OnyxStaticCredentialsProvider":
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_value: BaseException | None,
traceback: TracebackType | None,
) -> None:
pass
def get_tenant_id(self) -> str | None:
return self._tenant_id
def get_provider_key(self) -> str:
return self._provider_key
def get_credentials(self) -> dict[str, Any]:
return self._credential_json
def set_credentials(self, credential_json: dict[str, Any]) -> None:
self._credential_json = credential_json
def is_dynamic(self) -> bool:
return False
# from tests.daily.connectors.utils import load_all_docs_from_checkpoint_connector
# from onyx.connectors.credentials_provider import OnyxStaticCredentialsProvider
host = os.environ.get("IMAP_HOST")
mailboxes_str = os.environ.get("IMAP_MAILBOXES","INBOX")
username = os.environ.get("IMAP_USERNAME")
password = os.environ.get("IMAP_PASSWORD")
mailboxes = (
[mailbox.strip() for mailbox in mailboxes_str.split(",")]
if mailboxes_str
else []
)
if not host:
raise RuntimeError("`IMAP_HOST` must be set")
imap_connector = ImapConnector(
host=host,
mailboxes=mailboxes,
)
imap_connector.set_credentials_provider(
OnyxStaticCredentialsProvider(
tenant_id=None,
connector_name=DocumentSource.IMAP,
credential_json={
_USERNAME_KEY: username,
_PASSWORD_KEY: password,
},
)
)
END = time.time()
START = END - 1 * 24 * 60 * 60
for doc in load_all_docs_from_checkpoint_connector(
connector=imap_connector,
start=START,
end=END,
):
print(doc.id,doc.extension)

View File

@ -476,11 +476,13 @@ class RAGFlowPdfParser:
self.boxes = bxs self.boxes = bxs
def _naive_vertical_merge(self, zoomin=3): def _naive_vertical_merge(self, zoomin=3):
bxs = self._assign_column(self.boxes, zoomin) #bxs = self._assign_column(self.boxes, zoomin)
bxs = self.boxes
grouped = defaultdict(list) grouped = defaultdict(list)
for b in bxs: for b in bxs:
grouped[(b["page_number"], b.get("col_id", 0))].append(b) # grouped[(b["page_number"], b.get("col_id", 0))].append(b)
grouped[(b["page_number"], "x")].append(b)
merged_boxes = [] merged_boxes = []
for (pg, col), bxs in grouped.items(): for (pg, col), bxs in grouped.items():
@ -551,7 +553,7 @@ class RAGFlowPdfParser:
merged_boxes.extend(bxs) merged_boxes.extend(bxs)
self.boxes = sorted(merged_boxes, key=lambda x: (x["page_number"], x.get("col_id", 0), x["top"])) #self.boxes = sorted(merged_boxes, key=lambda x: (x["page_number"], x.get("col_id", 0), x["top"]))
def _final_reading_order_merge(self, zoomin=3): def _final_reading_order_merge(self, zoomin=3):
if not self.boxes: if not self.boxes:
@ -1206,7 +1208,7 @@ class RAGFlowPdfParser:
start = timer() start = timer()
self._text_merge() self._text_merge()
self._concat_downward() self._concat_downward()
#self._naive_vertical_merge(zoomin) self._naive_vertical_merge(zoomin)
if callback: if callback:
callback(0.92, "Text merged ({:.2f}s)".format(timer() - start)) callback(0.92, "Text merged ({:.2f}s)".format(timer() - start))

View File

@ -137,11 +137,11 @@ ADMIN_SVR_HTTP_PORT=9381
SVR_MCP_PORT=9382 SVR_MCP_PORT=9382
# The RAGFlow Docker image to download. v0.22+ doesn't include embedding models. # The RAGFlow Docker image to download. v0.22+ doesn't include embedding models.
RAGFLOW_IMAGE=infiniflow/ragflow:v0.23.0 RAGFLOW_IMAGE=infiniflow/ragflow:v0.23.1
# If you cannot download the RAGFlow Docker image: # If you cannot download the RAGFlow Docker image:
# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:v0.23.0 # RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:v0.23.1
# RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:v0.23.0 # RAGFLOW_IMAGE=registry.cn-hangzhou.aliyuncs.com/infiniflow/ragflow:v0.23.1
# #
# - For the `nightly` edition, uncomment either of the following: # - For the `nightly` edition, uncomment either of the following:
# RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:nightly # RAGFLOW_IMAGE=swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow:nightly

View File

@ -77,7 +77,7 @@ The [.env](./.env) file contains important environment variables for Docker.
- `SVR_HTTP_PORT` - `SVR_HTTP_PORT`
The port used to expose RAGFlow's HTTP API service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `9380`. The port used to expose RAGFlow's HTTP API service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `9380`.
- `RAGFLOW-IMAGE` - `RAGFLOW-IMAGE`
The Docker image edition. Defaults to `infiniflow/ragflow:v0.23.0`. The RAGFlow Docker image does not include embedding models. The Docker image edition. Defaults to `infiniflow/ragflow:v0.23.1`. The RAGFlow Docker image does not include embedding models.
> [!TIP] > [!TIP]

View File

@ -0,0 +1,8 @@
{
"label": "Basics",
"position": 2,
"link": {
"type": "generated-index",
"description": "Basic concepts."
}
}

View File

@ -0,0 +1,61 @@
---
sidebar_position: 2
slug: /what_is_agent_context_engine
---
# What is Agent context engine?
From 2025, a silent revolution began beneath the dazzling surface of AI Agents. While the world marveled at agents that could write code, analyze data, and automate workflows, a fundamental bottleneck emerged: why do even the most advanced agents still stumble on simple questions, forget previous conversations, or misuse available tools?
The answer lies not in the intelligence of the Large Language Model (LLM) itself, but in the quality of the Context it receives. An LLM, no matter how powerful, is only as good as the information we feed it. Todays cutting-edge agents are often crippled by a cumbersome, manual, and error-prone process of context assembly—a process known as Context Engineering.
This is where the Agent Context Engine comes in. It is not merely an incremental improvement but a foundational shift, representing the evolution of RAG from a singular technique into the core data and intelligence substrate for the entire Agent ecosystem.
## Beyond the hype: The reality of today's "intelligent" Agents
Today, the “intelligence” behind most AI Agents hides a mountain of human labor. Developers must:
- Hand-craft elaborate prompt templates
- Hard-code document-retrieval logic for every task
- Juggle tool descriptions, conversation history, and knowledge snippets inside a tiny context window
- Repeat the whole process for each new scenario
This pattern is called Context Engineering. It is deeply tied to expert know-how, almost impossible to scale, and prohibitively expensive to maintain. When an enterprise needs to keep dozens of distinct agents alive, the artisanal workshop model collapses under its own weight.
The mission of an Agent Context Engine is to turn Context Engineering from an “art” into an industrial-grade science.
Deconstructing the Agent Context Engine
So, what exactly is an Agent Context Engine? It is a unified, intelligent, and automated platform responsible for the end-to-end process of assembling the optimal context for an LLM or Agent at the moment of inference. It moves from artisanal crafting to industrialized production.
At its core, an Agent Context Engine is built on a triumvirate of next-generation retrieval capabilities, seamlessly integrated into a single service layer:
1. The Knowledge Core (Advanced RAG): This is the evolution of traditional RAG. It moves beyond simple chunk-and-embed to intelligently process static, private enterprise knowledge. Techniques like TreeRAG (building LLM-generated document outlines for "locate-then-expand" retrieval) and GraphRAG (extracting entity networks to find semantically distant connections) work to close the "semantic gap." The engines Ingestion Pipeline acts as the ETL for unstructured data, parsing multi-format documents and using LLMs to enrich content with summaries, metadata, and structure before indexing.
2. The Memory Layer: An Agents intelligence is defined by its ability to learn from interaction. The Memory Layer is a specialized retrieval system for dynamic, episodic data: conversation history, user preferences, and the agents own internal state (e.g., "waiting for human input"). It manages the lifecycle of this data—storing raw dialogue, triggering summarization into semantic memory, and retrieving relevant past interactions to provide continuity and personalization. Technologically, it is a close sibling to RAG, but focused on a temporal stream of data.
3. The Tool Orchestrator: As MCP (Model Context Protocol) enables the connection of hundreds of internal services as tools, a new problem arises: tool selection. The Context Engine solves this with Tool Retrieval. Instead of dumping all tool descriptions into the prompt, it maintains an index of tools and—critically—an index of Playbooks or Guidelines (best practices on when and how to use tools). For a given task, it retrieves only the most relevant tools and instructions, transforming the LLMs job from "searching a haystack" to "following a recipe."
## Why we need a dedicated engine? The case for a unified substrate
The necessity of an Agent Context Engine becomes clear when we examine the alternative: siloed, manually wired components.
- The Data Silo Problem: Knowledge, memory, and tools reside in separate systems, requiring complex integration for each new agent.
- The Assembly Line Bottleneck: Developers spend more time on context plumbing than on agent logic, slowing innovation to a crawl.
- The "Context Ownership" Dilemma: In manually engineered systems, context logic is buried in code, owned by developers, and opaque to business users. An Engine makes context a configurable, observable, and customer-owned asset.
The shift from Context Engineering to a Context Platform/Engine marks the maturation of enterprise AI, as summarized in the table below:
| Dimension | Context engineering (present) | Context engineering/Platform (future) |
| ------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Context creation | Manual, artisanal work by developers and prompt engineers. | Automated, driven by intelligent ingestion pipelines and configurable rules. |
| Context delivery | Hard-coded prompts and static retrieval logic embedded in agent workflows. | Dynamic, real-time retrieval and assembly based on the agent's live state and intent. |
| Context maintenance | A development and operational burden, logic locked in code. | A manageable platform function, with visibility and control returned to the business. |
## RAGFlow: A resolute march toward the context engine of Agents
This is the future RAGFlow is forging.
We left behind the label of “yet another RAG system” long ago. From DeepDoc—our deeply-optimized, multimodal document parser—to the bleeding-edge architectures that bridge semantic chasms in complex RAG scenarios, all the way to a full-blown, enterprise-grade ingestion pipeline, every evolutionary step RAGFlow takes is a deliberate stride toward the ultimate form: an Agentic Context Engine.
We believe tomorrows enterprise AI advantage will hinge not on who owns the largest model, but on who can feed that model the highest-quality, most real-time, and most relevant context. An Agentic Context Engine is the critical infrastructure that turns this vision into reality.
In the paradigm shift from “hand-crafted prompts” to “intelligent context,” RAGFlow is determined to be the most steadfast propeller and enabler. We invite every developer, enterprise, and researcher who cares about the future of AI agents to follow RAGFlows journey—so together we can witness and build the cornerstone of the next-generation AI stack.

107
docs/basics/rag.md Normal file
View File

@ -0,0 +1,107 @@
---
sidebar_position: 1
slug: /what_is_rag
---
# What is Retreival-Augmented-Generation (RAG)
Since large language models (LLMs) became the focus of technology, their ability to handle general knowledge has been astonishing. However, when questions shift to internal corporate documents, proprietary knowledge bases, or real-time data, the limitations of LLMs become glaringly apparent: they cannot access private information outside their training data. Retrieval-Augmented Generation (RAG) was born precisely to address this core need. Before an LLM generates an answer, it first retrieves the most relevant context from an external knowledge base and inputs it as "reference material" to the LLM, thereby guiding it to produce accurate answers. In short, RAG elevates LLMs from "relying on memory" to "having evidence to rely on," significantly improving their accuracy and trustworthiness in specialized fields and real-time information queries.
## Why RAG is important?
Although LLMs excel in language understanding and generation, they have inherent limitations:
- Static Knowledge: The model's knowledge is based on a data snapshot from its training time and cannot be automatically updated, making it difficult to perceive the latest information.
- Blind Spot to External Data: They cannot directly access corporate private documents, real-time information streams, or domain-specific content.
- Hallucination Risk: When lacking accurate evidence, they may still fabricate plausible-sounding but false answers to maintain conversational fluency.
The introduction of RAG provides LLMs with real-time, credible "factual grounding." Its core mechanism is divided into two stages:
- Retrieval Stage: Based on the user's question, quickly retrieve the most relevant documents or data fragments from an external knowledge base.
- Generation Stage: The LLM organizes and generates the final answer by incorporating the retrieved information as context, combined with its own linguistic capabilities.
This upgrades LLMs from "speaking from memory" to "speaking with documentation," significantly enhancing reliability in professional and enterprise-level applications.
## How RAG works?
Retrieval-Augmented Generation enables LLMs to generate higher-quality responses by leveraging real-time, external, or private data sources through the introduction of an information retrieval mechanism. Its workflow can be divided into following key steps:
### Data processing and vectorization
The knowledge required by RAG comes from unstructured data in various formats, such as documents, database records, or API return content. This data typically needs to be chunked, then transformed into vectors via an embedding model, and stored in a vector database.
Why is Chunking Needed? Indexing entire documents directly faces the following problems:
- Decreased Retrieval Precision: Vectorizing long documents leads to semantic "averaging," losing details.
- Context Length Limitation: LLMs have a finite context window, requiring filtering of the most relevant parts for input.
- Cost and Efficiency: Embedding computation and retrieval costs are higher for long texts.
Therefore, an intelligent chunking strategy is key to balancing information integrity, retrieval granularity, and computational efficiency.
### Retrieve relevant information
The user's query is also converted into a vector to perform semantic relevance searches (e.g., calculating cosine similarity) in the vector database, matching and recalling the most relevant text fragments.
### Context construction and answer generation
The retrieved relevant content is added to the LLM's context as factual grounding, and the LLM finally generates the answer. Therefore, RAG can be seen as Context Engineering 1.0 for automated context construction.
## Deep dive into existing RAG architecture: beyond vector retrieval
An industrial-grade RAG system is far from being as simple as "vector search + LLM"; its complexity and challenges are primarily embedded in the retrieval process.
### Data complexity: multimodal document processing
Core Challenge: Corporate knowledge mostly exists in the form of multimodal documents containing text, charts, tables, and formulas. Simple OCR extraction loses a large amount of semantic information.
Advanced Practice: Leading solutions, such as RAGFlow, tend to use Visual Language Models (VLM) or specialized parsing models like DeepDoc to "translate" multimodal documents into unimodal text rich in structural and semantic information. Converting multimodal information into high-quality unimodal text has become standard practice for advanced RAG.
### The complexity of chunking: the trade-off between precision and context
A simple "chunk-embed-retrieve" pipeline has an inherent contradiction:
- Semantic Matching requires small text chunks to ensure clear semantic focus.
- Context Understanding requires large text chunks to ensure complete and coherent information.
This forces system design into a difficult trade-off between "precise but fragmented" and "complete but vague."
Advanced Practice: Leading solutions, such as RAGFlow, employ semantic enhancement techniques like constructing semantic tables of contents and knowledge graphs. These not only address semantic fragmentation caused by physical chunking but also enable the discovery of relevant content across documents based on entity-relationship networks.
### Why is a vector database insufficient for serving RAG?
Vector databases excel at semantic similarity search, but RAG requires precise and reliable answers, demanding more capabilities from the retrieval system:
- Hybrid Search: Relying solely on vector retrieval may miss exact keyword matches (e.g., product codes, regulation numbers). Hybrid search, combining vector retrieval with keyword retrieval (BM25), ensures both semantic breadth and keyword precision.
- Tensor or Multi-Vector Representation: To support cross-modal data, employing tensor or multi-vector representation has become an important trend.
- Metadata Filtering: Filtering based on attributes like date, department, and type is a rigid requirement in business scenarios.
Therefore, the retrieval layer of RAG is a composite system based on vector search but must integrate capabilities like full-text search, re-ranking, and metadata filtering.
## RAG and memory: Retrieval from the same source but different streams
Within the agent framework, the essence of the memory mechanism is the same as RAG: both retrieve relevant information from storage based on current needs. The key difference lies in the data source:
- RAG: Targets pre-existing static or dynamic private data provided by the user in advance (e.g., documents, databases).
- Memory: Targets dynamic data generated or perceived by the agent in real-time during interaction (e.g., conversation history, environmental state, tool execution results).
They are highly consistent at the technical base (e.g., vector retrieval, keyword matching) and can be seen as the same retrieval capability applied in different scenarios ("existing knowledge" vs. "interaction memory"). A complete agent system often includes both an RAG module for inherent knowledge and a Memory module for interaction history.
## RAG applications
RAG has demonstrated clear value in several typical scenarios:
1. Enterprise Knowledge Q&A and Internal Search
By vectorizing corporate private data and combining it with an LLM, RAG can directly return natural language answers based on authoritative sources, rather than document lists. While meeting intelligent Q&A needs, it inherently aligns with corporate requirements for data security, access control, and compliance.
2. Complex Document Understanding and Professional Q&A
For structurally complex documents like contracts and regulations, the value of RAG lies in its ability to generate accurate, verifiable answers while maintaining context integrity. Its system accuracy largely depends on text chunking and semantic understanding strategies.
3. Dynamic Knowledge Fusion and Decision Support
In business scenarios requiring the synthesis of information from multiple sources, RAG evolves into a knowledge orchestration and reasoning support system for business decisions. Through a multi-path recall mechanism, it fuses knowledge from different systems and formats, maintaining factual consistency and logical controllability during the generation phase.
## The future of RAG
The evolution of RAG is unfolding along several clear paths:
1. RAG as the data foundation for Agents
RAG and agents have an architecture vs. scenario relationship. For agents to achieve autonomous and reliable decision-making and execution, they must rely on accurate and timely knowledge. RAG provides them with a standardized capability to access private domain knowledge and is an inevitable choice for building knowledge-aware agents.
2. Advanced RAG: Using LLMs to optimize retrieval itself
The core feature of next-generation RAG is fully utilizing the reasoning capabilities of LLMs to optimize the retrieval process, such as rewriting queries, summarizing or fusing results, or implementing intelligent routing. Empowering every aspect of retrieval with LLMs is key to breaking through current performance bottlenecks.
3. Towards context engineering 2.0
Current RAG can be viewed as Context Engineering 1.0, whose core is assembling static knowledge context for single Q&A tasks. The forthcoming Context Engineering 2.0 will extend with RAG technology at its core, becoming a system that automatically and dynamically assembles comprehensive context for agents. The context fused by this system will come not only from documents but also include interaction memory, available tools/skills, and real-time environmental information. This marks the transition of agent development from a "handicraft workshop" model to the industrial starting point of automated context engineering.
The essence of RAG is to build a dedicated, efficient, and trustworthy external data interface for large language models; its core is Retrieval, not Generation. Starting from the practical need to solve private data access, its technical depth is reflected in the optimization of retrieval for complex unstructured data. With its deep integration into agent architectures and its development towards automated context engineering, RAG is evolving from a technology that improves Q&A quality into the core infrastructure for building the next generation of trustworthy, controllable, and scalable intelligent applications.

View File

@ -99,7 +99,7 @@ RAGFlow utilizes MinIO as its object storage solution, leveraging its scalabilit
- `SVR_HTTP_PORT` - `SVR_HTTP_PORT`
The port used to expose RAGFlow's HTTP API service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `9380`. The port used to expose RAGFlow's HTTP API service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `9380`.
- `RAGFLOW-IMAGE` - `RAGFLOW-IMAGE`
The Docker image edition. Defaults to `infiniflow/ragflow:v0.23.0` (the RAGFlow Docker image without embedding models). The Docker image edition. Defaults to `infiniflow/ragflow:v0.23.1` (the RAGFlow Docker image without embedding models).
:::tip NOTE :::tip NOTE
If you cannot download the RAGFlow Docker image, try the following mirrors. If you cannot download the RAGFlow Docker image, try the following mirrors.

View File

@ -47,7 +47,7 @@ After building the infiniflow/ragflow:nightly image, you are ready to launch a f
1. Edit Docker Compose Configuration 1. Edit Docker Compose Configuration
Open the `docker/.env` file. Find the `RAGFLOW_IMAGE` setting and change the image reference from `infiniflow/ragflow:v0.23.0` to `infiniflow/ragflow:nightly` to use the pre-built image. Open the `docker/.env` file. Find the `RAGFLOW_IMAGE` setting and change the image reference from `infiniflow/ragflow:v0.23.1` to `infiniflow/ragflow:nightly` to use the pre-built image.
2. Launch the Service 2. Launch the Service

View File

@ -133,7 +133,7 @@ See [Run retrieval test](./run_retrieval_test.md) for details.
## Search for dataset ## Search for dataset
As of RAGFlow v0.23.0, the search feature is still in a rudimentary form, supporting only dataset search by name. As of RAGFlow v0.23.1, the search feature is still in a rudimentary form, supporting only dataset search by name.
![search dataset](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/search_datasets.jpg) ![search dataset](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/search_datasets.jpg)

View File

@ -87,4 +87,4 @@ RAGFlow's file management allows you to download an uploaded file:
![download_file](https://github.com/infiniflow/ragflow/assets/93570324/cf3b297f-7d9b-4522-bf5f-4f45743e4ed5) ![download_file](https://github.com/infiniflow/ragflow/assets/93570324/cf3b297f-7d9b-4522-bf5f-4f45743e4ed5)
> As of RAGFlow v0.23.0, bulk download is not supported, nor can you download an entire folder. > As of RAGFlow v0.23.1, bulk download is not supported, nor can you download an entire folder.

View File

@ -46,7 +46,7 @@ The Admin CLI and Admin Service form a client-server architectural suite for RAG
2. Install ragflow-cli. 2. Install ragflow-cli.
```bash ```bash
pip install ragflow-cli==0.23.0 pip install ragflow-cli==0.23.1
``` ```
3. Launch the CLI client: 3. Launch the CLI client:

View File

@ -0,0 +1,8 @@
{
"label": "Migration",
"position": 5,
"link": {
"type": "generated-index",
"description": "RAGFlow migration guide"
}
}

View File

@ -1,109 +0,0 @@
---
sidebar_position: 8
slug: /run_health_check
---
# Monitoring
Double-check the health status of RAGFlow's dependencies.
---
The operation of RAGFlow depends on four services:
- **Elasticsearch** (default) or [Infinity](https://github.com/infiniflow/infinity) as the document engine
- **MySQL**
- **Redis**
- **MinIO** for object storage
If an exception or error occurs related to any of the above services, such as `Exception: Can't connect to ES cluster`, refer to this document to check their health status.
You can also click you avatar in the top right corner of the page **>** System to view the visualized health status of RAGFlow's core services. The following screenshot shows that all services are 'green' (running healthily). The task executor displays the *cumulative* number of completed and failed document parsing tasks from the past 30 minutes:
![system_status_page](https://github.com/user-attachments/assets/b0c1a11e-93e3-4947-b17a-1bfb4cdab6e4)
Services with a yellow or red light are not running properly. The following is a screenshot of the system page after running `docker stop ragflow-es-10`:
![es_failed](https://github.com/user-attachments/assets/06056540-49f5-48bf-9cc9-a7086bc75790)
You can click on a specific 30-second time interval to view the details of completed and failed tasks:
![done_tasks](https://github.com/user-attachments/assets/49b25ec4-03af-48cf-b2e5-c892f6eaa261)
![done_vs_failed](https://github.com/user-attachments/assets/eaa928d0-a31c-4072-adea-046091e04599)
## API Health Check
In addition to checking the system dependencies from the **avatar > System** page in the UI, you can directly query the backend health check endpoint:
```bash
http://IP_OF_YOUR_MACHINE/v1/system/healthz
```
Here `<port>` refers to the actual port of your backend service (e.g., `7897`, `9222`, etc.).
Key points:
- **No login required** (no `@login_required` decorator)
- Returns results in JSON format
- If all dependencies are healthy → HTTP **200 OK**
- If any dependency fails → HTTP **500 Internal Server Error**
### Example 1: All services healthy (HTTP 200)
```bash
http://127.0.0.1/v1/system/healthz
```
Response:
```http
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 120
```
Explanation:
- Database (MySQL/Postgres), Redis, document engine (Elasticsearch/Infinity), and object storage (MinIO) are all healthy.
- The `status` field returns `"ok"`.
### Example 2: One service unhealthy (HTTP 500)
For example, if Redis is down:
Response:
```http
HTTP/1.1 500 INTERNAL SERVER ERROR
Content-Type: application/json
Content-Length: 300
```
Explanation:
- `redis` is marked as `"nok"`, with detailed error info under `_meta.redis.error`.
- The overall `status` is `"nok"`, so the endpoint returns 500.
---
This endpoint allows you to monitor RAGFlows core dependencies programmatically in scripts or external monitoring systems, without relying on the frontend UI.
"redis": "nok",
"doc_engine": "ok",
"storage": "ok",
"status": "nok",
"_meta": {
"redis": {
"elapsed": "5.2",
"error": "Lost connection!"
}
}
}
```
Explanation:
- `redis` is marked as `"nok"`, with detailed error info under `_meta.redis.error`.
- The overall `status` is `"nok"`, so the endpoint returns 500.
---
This endpoint allows you to monitor RAGFlows core dependencies programmatically in scripts or external monitoring systems, without relying on the frontend UI.

View File

@ -60,16 +60,16 @@ To upgrade RAGFlow, you must upgrade **both** your code **and** your Docker imag
git pull git pull
``` ```
3. Switch to the latest, officially published release, e.g., `v0.23.0`: 3. Switch to the latest, officially published release, e.g., `v0.23.1`:
```bash ```bash
git checkout -f v0.23.0 git checkout -f v0.23.1
``` ```
4. Update **ragflow/docker/.env**: 4. Update **ragflow/docker/.env**:
```bash ```bash
RAGFLOW_IMAGE=infiniflow/ragflow:v0.23.0 RAGFLOW_IMAGE=infiniflow/ragflow:v0.23.1
``` ```
5. Update the RAGFlow image and restart RAGFlow: 5. Update the RAGFlow image and restart RAGFlow:
@ -90,10 +90,10 @@ No, you do not need to. Upgrading RAGFlow in itself will *not* remove your uploa
1. From an environment with Internet access, pull the required Docker image. 1. From an environment with Internet access, pull the required Docker image.
2. Save the Docker image to a **.tar** file. 2. Save the Docker image to a **.tar** file.
```bash ```bash
docker save -o ragflow.v0.23.0.tar infiniflow/ragflow:v0.23.0 docker save -o ragflow.v0.23.1.tar infiniflow/ragflow:v0.23.1
``` ```
3. Copy the **.tar** file to the target server. 3. Copy the **.tar** file to the target server.
4. Load the **.tar** file into Docker: 4. Load the **.tar** file into Docker:
```bash ```bash
docker load -i ragflow.v0.23.0.tar docker load -i ragflow.v0.23.1.tar
``` ```

View File

@ -46,7 +46,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
`vm.max_map_count`. This value sets the maximum number of memory map areas a process may have. Its default value is 65530. While most applications require fewer than a thousand maps, reducing this value can result in abnormal behaviors, and the system will throw out-of-memory errors when a process reaches the limitation. `vm.max_map_count`. This value sets the maximum number of memory map areas a process may have. Its default value is 65530. While most applications require fewer than a thousand maps, reducing this value can result in abnormal behaviors, and the system will throw out-of-memory errors when a process reaches the limitation.
RAGFlow v0.23.0 uses Elasticsearch or [Infinity](https://github.com/infiniflow/infinity) for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning of the Elasticsearch component. RAGFlow v0.23.1 uses Elasticsearch or [Infinity](https://github.com/infiniflow/infinity) for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning of the Elasticsearch component.
<Tabs <Tabs
defaultValue="linux" defaultValue="linux"
@ -186,7 +186,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
```bash ```bash
$ git clone https://github.com/infiniflow/ragflow.git $ git clone https://github.com/infiniflow/ragflow.git
$ cd ragflow/docker $ cd ragflow/docker
$ git checkout -f v0.23.0 $ git checkout -f v0.23.1
``` ```
3. Use the pre-built Docker images and start up the server: 3. Use the pre-built Docker images and start up the server:
@ -202,7 +202,7 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
| RAGFlow image tag | Image size (GB) | Stable? | | RAGFlow image tag | Image size (GB) | Stable? |
| ------------------- | --------------- | ------------------------ | | ------------------- | --------------- | ------------------------ |
| v0.23.0 | &approx;2 | Stable release | | v0.23.1 | &approx;2 | Stable release |
| nightly | &approx;2 | _Unstable_ nightly build | | nightly | &approx;2 | _Unstable_ nightly build |
```mdx-code-block ```mdx-code-block

View File

@ -13,61 +13,58 @@ A complete list of models supported by RAGFlow, which will continue to expand.
<APITable> <APITable>
``` ```
| Provider | Chat | Embedding | Rerank | Img2txt | Speech2txt | TTS | | Provider | LLM | Image2Text | Speech2text | TTS | Embedding | Rerank | OCR |
| --------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | | --------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |
| Anthropic | :heavy_check_mark: | | | | | | | Anthropic | :heavy_check_mark: | | | | | | |
| Azure-OpenAI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | | | Azure-OpenAI | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | |
| BAAI | | :heavy_check_mark: | :heavy_check_mark: | | | | | BaiChuan | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| BaiChuan | :heavy_check_mark: | :heavy_check_mark: | | | | | | BaiduYiyan | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| BaiduYiyan | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Bedrock | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| Bedrock | :heavy_check_mark: | :heavy_check_mark: | | | | | | Cohere | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| Cohere | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | DeepSeek | :heavy_check_mark: | | | | | | |
| DeepSeek | :heavy_check_mark: | | | | | | | Fish Audio | | | | :heavy_check_mark: | | | |
| FastEmbed | | :heavy_check_mark: | | | | | | Gemini | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | |
| Fish Audio | | | | | | :heavy_check_mark: | | Google Cloud | :heavy_check_mark: | | | | | | |
| Gemini | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | | | GPUStack | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
| Google Cloud | :heavy_check_mark: | | | | | | | Groq | :heavy_check_mark: | | | | | | |
| GPUStack | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | | HuggingFace | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| Groq | :heavy_check_mark: | | | | | | | Jina | | | | | :heavy_check_mark: | :heavy_check_mark: | |
| HuggingFace | :heavy_check_mark: | :heavy_check_mark: | | | | | | LocalAI | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | |
| Jina | | :heavy_check_mark: | :heavy_check_mark: | | | | | LongCat | :heavy_check_mark: | | | | | | |
| LeptonAI | :heavy_check_mark: | | | | | | | LM-Studio | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | |
| LocalAI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | | | MiniMax | :heavy_check_mark: | | | | | | |
| LM-Studio | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | | | MinerU | | | | | | | :heavy_check_mark: |
| MiniMax | :heavy_check_mark: | | | | | | | Mistral | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| Mistral | :heavy_check_mark: | :heavy_check_mark: | | | | | | ModelScope | :heavy_check_mark: | | | | | | |
| ModelScope | :heavy_check_mark: | | | | | | | Moonshot | :heavy_check_mark: | :heavy_check_mark: | | | | | |
| Moonshot | :heavy_check_mark: | | | :heavy_check_mark: | | | | NovitaAI | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| Novita AI | :heavy_check_mark: | :heavy_check_mark: | | | | | | NVIDIA | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| NVIDIA | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Ollama | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | |
| Ollama | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | | | OpenAI | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |
| OpenAI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | OpenAI-API-Compatible | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| OpenAI-API-Compatible | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | OpenRouter | :heavy_check_mark: | :heavy_check_mark: | | | | | |
| OpenRouter | :heavy_check_mark: | | | :heavy_check_mark: | | | | Replicate | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| PerfXCloud | :heavy_check_mark: | :heavy_check_mark: | | | | | | PPIO | :heavy_check_mark: | | | | | | |
| Replicate | :heavy_check_mark: | :heavy_check_mark: | | | | | | SILICONFLOW | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| PPIO | :heavy_check_mark: | | | | | | | StepFun | :heavy_check_mark: | | | | | | |
| SILICONFLOW | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Tencent Hunyuan | :heavy_check_mark: | | | | | | |
| StepFun | :heavy_check_mark: | | | | | | | Tencent Cloud | | | :heavy_check_mark: | | | | |
| Tencent Hunyuan | :heavy_check_mark: | | | | | | | TogetherAI | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| Tencent Cloud | | | | | :heavy_check_mark: | | | TokenPony | :heavy_check_mark: | | | | | | |
| TogetherAI | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Tongyi-Qianwen | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
| Tongyi-Qianwen | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Upstage | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| Upstage | :heavy_check_mark: | :heavy_check_mark: | | | | | | VLLM | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| VLLM | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | VolcEngine | :heavy_check_mark: | | | | | | |
| VolcEngine | :heavy_check_mark: | | | | | | | Voyage AI | | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| Voyage AI | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Xinference | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
| Xinference | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | XunFei Spark | :heavy_check_mark: | | | :heavy_check_mark: | | | |
| XunFei Spark | :heavy_check_mark: | | | | | :heavy_check_mark: | | xAI | :heavy_check_mark: | :heavy_check_mark: | | | | | |
| xAI | :heavy_check_mark: | | | :heavy_check_mark: | | | | ZHIPU-AI | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | |
| Youdao | | :heavy_check_mark: | :heavy_check_mark: | | | | | DeepInfra | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |
| ZHIPU-AI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | | | 302.AI | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | |
| 01.AI | :heavy_check_mark: | | | | | | | CometAPI | :heavy_check_mark: | | | | :heavy_check_mark: | | |
| DeepInfra | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | | DeerAPI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | | |
| 302.AI | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | | Jiekou.AI | :heavy_check_mark: | | | | :heavy_check_mark: | :heavy_check_mark: | |
| CometAPI | :heavy_check_mark: | :heavy_check_mark: | | | | |
| DeerAPI | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | :heavy_check_mark: |
| Jiekou.AI | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |
```mdx-code-block ```mdx-code-block
</APITable> </APITable>

View File

@ -7,6 +7,24 @@ slug: /release_notes
Key features, improvements and bug fixes in the latest releases. Key features, improvements and bug fixes in the latest releases.
## v0.23.1
Released on December 31, 2025.
### Fixed issues
- Resolved an issue where the RAGFlow Server would fail to start if an empty memory object existed, and corrected the inability to delete a newly created empty Memory.
- Improved the stability of memory extraction across all memory types after selection.
- Fixed MDX file parsing support.
### Data sources
- GitHub
- Gitlab
- Asana
- IMAP
## v0.23.0 ## v0.23.0
Released on December 27, 2025. Released on December 27, 2025.

View File

@ -77,7 +77,7 @@ env:
ragflow: ragflow:
image: image:
repository: infiniflow/ragflow repository: infiniflow/ragflow
tag: v0.23.0 tag: v0.23.1
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
pullSecrets: [] pullSecrets: []
# Optional service configuration overrides # Optional service configuration overrides

View File

@ -1,6 +1,6 @@
[project] [project]
name = "ragflow" name = "ragflow"
version = "0.23.0" version = "0.23.1"
description = "[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data." description = "[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data."
authors = [{ name = "Zhichang Yu", email = "yuzhichang@gmail.com" }] authors = [{ name = "Zhichang Yu", email = "yuzhichang@gmail.com" }]
license-files = ["LICENSE"] license-files = ["LICENSE"]

View File

@ -40,7 +40,7 @@ from deepdoc.parser.docling_parser import DoclingParser
from deepdoc.parser.tcadp_parser import TCADPParser from deepdoc.parser.tcadp_parser import TCADPParser
from common.parser_config_utils import normalize_layout_recognizer from common.parser_config_utils import normalize_layout_recognizer
from rag.nlp import concat_img, find_codec, naive_merge, naive_merge_with_images, naive_merge_docx, rag_tokenizer, \ from rag.nlp import concat_img, find_codec, naive_merge, naive_merge_with_images, naive_merge_docx, rag_tokenizer, \
tokenize_chunks, tokenize_chunks_with_images, tokenize_table, attach_media_context tokenize_chunks, tokenize_chunks_with_images, tokenize_table, attach_media_context, append_context2table_image4pdf
def by_deepdoc(filename, binary=None, from_page=0, to_page=100000, lang="Chinese", callback=None, pdf_cls=None, def by_deepdoc(filename, binary=None, from_page=0, to_page=100000, lang="Chinese", callback=None, pdf_cls=None,
@ -487,7 +487,7 @@ class Pdf(PdfParser):
tbls = self._extract_table_figure(True, zoomin, True, True) tbls = self._extract_table_figure(True, zoomin, True, True)
self._naive_vertical_merge() self._naive_vertical_merge()
self._concat_downward() self._concat_downward()
self._final_reading_order_merge() # self._final_reading_order_merge()
# self._filter_forpages() # self._filter_forpages()
logging.info("layouts cost: {}s".format(timer() - first_start)) logging.info("layouts cost: {}s".format(timer() - first_start))
return [(b["text"], self._line_tag(b, zoomin)) for b in self.boxes], tbls return [(b["text"], self._line_tag(b, zoomin)) for b in self.boxes], tbls
@ -776,6 +776,9 @@ def chunk(filename, binary=None, from_page=0, to_page=100000, lang="Chinese", ca
if not sections and not tables: if not sections and not tables:
return [] return []
if table_context_size or image_context_size:
tables = append_context2table_image4pdf(sections, tables, image_context_size)
if name in ["tcadp", "docling", "mineru"]: if name in ["tcadp", "docling", "mineru"]:
parser_config["chunk_token_num"] = 0 parser_config["chunk_token_num"] = 0
@ -1006,8 +1009,8 @@ def chunk(filename, binary=None, from_page=0, to_page=100000, lang="Chinese", ca
res.extend(embed_res) res.extend(embed_res)
if url_res: if url_res:
res.extend(url_res) res.extend(url_res)
if table_context_size or image_context_size: #if table_context_size or image_context_size:
attach_media_context(res, table_context_size, image_context_size) # attach_media_context(res, table_context_size, image_context_size)
return res return res

View File

@ -16,7 +16,7 @@
import logging import logging
import random import random
from collections import Counter from collections import Counter, defaultdict
from common.token_utils import num_tokens_from_string from common.token_utils import num_tokens_from_string
import re import re
@ -667,6 +667,94 @@ def attach_media_context(chunks, table_context_size=0, image_context_size=0):
return chunks return chunks
def append_context2table_image4pdf(sections: list, tabls: list, table_context_size=0):
from deepdoc.parser import PdfParser
if table_context_size <=0:
return tabls
page_bucket = defaultdict(list)
for i, (txt, poss) in enumerate(sections):
poss = PdfParser.extract_positions(poss)
for page, left, right, top, bottom in poss:
page = page[0]
page_bucket[page].append(((left, top, right, bottom), txt))
def upper_context(page, i):
txt = ""
if page not in page_bucket:
i = -1
while num_tokens_from_string(txt) < table_context_size:
if i < 0:
page -= 1
if page < 0 or page not in page_bucket:
break
i = len(page_bucket[page]) -1
blks = page_bucket[page]
(_, _, _, _), cnt = blks[i]
txts = re.split(r"([。!?\n]|\. )", cnt, flags=re.DOTALL)[::-1]
for j in range(0, len(txts), 2):
txt = (txts[j+1] if j+1<len(txts) else "") + txts[j] + txt
if num_tokens_from_string(txt) > table_context_size:
break
i -= 1
return txt
def lower_context(page, i):
txt = ""
if page not in page_bucket:
return txt
while num_tokens_from_string(txt) < table_context_size:
if i >= len(page_bucket[page]):
page += 1
if page not in page_bucket:
break
i = 0
blks = page_bucket[page]
(_, _, _, _), cnt = blks[i]
txts = re.split(r"([。!?\n]|\. )", cnt, flags=re.DOTALL)
for j in range(0, len(txts), 2):
txt += txts[j] + (txts[j+1] if j+1<len(txts) else "")
if num_tokens_from_string(txt) > table_context_size:
break
i += 1
return txt
res = []
for (img, tb), poss in tabls:
page, left, top, right, bott = poss[0]
_page, _left, _top, _right, _bott = poss[-1]
if isinstance(tb, list):
tb = "\n".join(tb)
i = 0
blks = page_bucket.get(page, [])
_tb = tb
while i < len(blks):
if i + 1 >= len(blks):
if _page > page:
page += 1
i = 0
blks = page_bucket.get(page, [])
continue
tb = upper_context(page, i) + tb + lower_context(page+1, 0)
break
(_, t, r, b), txt = blks[i]
if b > top:
break
(_, _t, _r, _b), _txt = blks[i+1]
if _t < _bott:
i += 1
continue
tb = upper_context(page, i) + tb + lower_context(page, i)
break
if _tb == tb:
tb = upper_context(page, -1) + tb + lower_context(page+1, 0)
res.append(((img, tb), poss))
return res
def add_positions(d, poss): def add_positions(d, poss):
if not poss: if not poss:
return return

View File

@ -729,6 +729,8 @@ TOC_FROM_TEXT_USER = load_prompt("toc_from_text_user")
# Generate TOC from text chunks with text llms # Generate TOC from text chunks with text llms
async def gen_toc_from_text(txt_info: dict, chat_mdl, callback=None): async def gen_toc_from_text(txt_info: dict, chat_mdl, callback=None):
if callback:
callback(msg="")
try: try:
ans = await gen_json( ans = await gen_json(
PROMPT_JINJA_ENV.from_string(TOC_FROM_TEXT_SYSTEM).render(), PROMPT_JINJA_ENV.from_string(TOC_FROM_TEXT_SYSTEM).render(),
@ -738,8 +740,6 @@ async def gen_toc_from_text(txt_info: dict, chat_mdl, callback=None):
gen_conf={"temperature": 0.0, "top_p": 0.9} gen_conf={"temperature": 0.0, "top_p": 0.9}
) )
txt_info["toc"] = ans if ans and not isinstance(ans, str) else [] txt_info["toc"] = ans if ans and not isinstance(ans, str) else []
if callback:
callback(msg="")
except Exception as e: except Exception as e:
logging.exception(e) logging.exception(e)

View File

@ -1,14 +1,11 @@
## Role: Metadata extraction expert ## Role: Metadata extraction expert.
## Constraints: ## Rules:
- Core Directive: Extract important structured information from the given content. Output ONLY a valid JSON string. No Markdown (e.g., ```json), no explanations, and no notes. - Strict Evidence Only: Extract a value ONLY if it is explicitly mentioned in the Content.
- Schema Parsing: In the `properties` object provided in Schema, the attribute name (e.g., 'author') is the target Key. Extract values based on the `description`; if no `description` is provided, refer to the key's literal meaning. - Enum Filter: For any field with an 'enum' list, the list acts as a strict filter. If no element from the list (or its direct synonym) is found in the Content, you MUST NOT extract that field.
- Extraction Rules: Extract only when there is an explicit semantic correlation. If multiple values or data points match a field's definition, extract and include all of them. Strictly follow the Schema below and only output matched key-value pairs. If the content is irrelevant or no matching information is identified, you **MUST** output {}. - No Meta-Inference: Do not infer values based on the document's nature, format, or category. If the text does not literally state the information, treat it as missing.
- Data Source: Extraction must be based solely on content below. Semantic mapping (synonyms) is allowed, but strictly prohibit hallucinations or fabricated facts. - Zero-Hallucination: Never invent information or pick a "likely" value from the enum to fill a field.
- Empty Result: If no matches are found for any field, or if the content is irrelevant, output ONLY {}.
## Enum Rules (Triggered ONLY if an enum list is present): - Output: ONLY a valid JSON string. No Markdown, no notes.
- Value Lock: All extracted values MUST strictly match the provided enum list.
- Normalization: Map synonyms or variants in the text back to the standard enum value (e.g., "Dec" to "December").
- Fallback: Output {} if no explicit match or synonym is identified.
## Schema for extraction: ## Schema for extraction:
{{ schema }} {{ schema }}

View File

@ -49,6 +49,7 @@ from common.data_source import (
WebDAVConnector, WebDAVConnector,
AirtableConnector, AirtableConnector,
AsanaConnector, AsanaConnector,
ImapConnector
) )
from common.constants import FileSource, TaskStatus from common.constants import FileSource, TaskStatus
from common.data_source.config import INDEX_BATCH_SIZE from common.data_source.config import INDEX_BATCH_SIZE
@ -915,6 +916,70 @@ class Github(SyncBase):
return async_wrapper() return async_wrapper()
class IMAP(SyncBase):
SOURCE_NAME: str = FileSource.IMAP
async def _generate(self, task):
from common.data_source.config import DocumentSource
from common.data_source.interfaces import StaticCredentialsProvider
self.connector = ImapConnector(
host=self.conf.get("imap_host"),
port=self.conf.get("imap_port"),
mailboxes=self.conf.get("imap_mailbox"),
)
credentials_provider = StaticCredentialsProvider(tenant_id=task["tenant_id"], connector_name=DocumentSource.IMAP, credential_json=self.conf["credentials"])
self.connector.set_credentials_provider(credentials_provider)
end_time = datetime.now(timezone.utc).timestamp()
if task["reindex"] == "1" or not task["poll_range_start"]:
start_time = end_time - self.conf.get("poll_range",30) * 24 * 60 * 60
begin_info = "totally"
else:
start_time = task["poll_range_start"].timestamp()
begin_info = f"from {task['poll_range_start']}"
raw_batch_size = self.conf.get("sync_batch_size") or self.conf.get("batch_size") or INDEX_BATCH_SIZE
try:
batch_size = int(raw_batch_size)
except (TypeError, ValueError):
batch_size = INDEX_BATCH_SIZE
if batch_size <= 0:
batch_size = INDEX_BATCH_SIZE
def document_batches():
checkpoint = self.connector.build_dummy_checkpoint()
pending_docs = []
iterations = 0
iteration_limit = 100_000
while checkpoint.has_more:
wrapper = CheckpointOutputWrapper()
doc_generator = wrapper(self.connector.load_from_checkpoint(start_time, end_time, checkpoint))
for document, failure, next_checkpoint in doc_generator:
if failure is not None:
logging.warning("IMAP connector failure: %s", getattr(failure, "failure_message", failure))
continue
if document is not None:
pending_docs.append(document)
if len(pending_docs) >= batch_size:
yield pending_docs
pending_docs = []
if next_checkpoint is not None:
checkpoint = next_checkpoint
iterations += 1
if iterations > iteration_limit:
raise RuntimeError("Too many iterations while loading IMAP documents.")
if pending_docs:
yield pending_docs
logging.info(
"Connect to IMAP: host(%s) port(%s) user(%s) folder(%s) %s",
self.conf["imap_host"],
self.conf["imap_port"],
self.conf["credentials"]["imap_username"],
self.conf["imap_mailbox"],
begin_info
)
return document_batches()
class Gitlab(SyncBase): class Gitlab(SyncBase):
@ -977,6 +1042,7 @@ func_factory = {
FileSource.BOX: BOX, FileSource.BOX: BOX,
FileSource.AIRTABLE: Airtable, FileSource.AIRTABLE: Airtable,
FileSource.ASANA: Asana, FileSource.ASANA: Asana,
FileSource.IMAP: IMAP,
FileSource.GITHUB: Github, FileSource.GITHUB: Github,
FileSource.GITLAB: Gitlab, FileSource.GITLAB: Gitlab,
} }

View File

@ -332,6 +332,9 @@ async def build_chunks(task, progress_callback):
async def doc_keyword_extraction(chat_mdl, d, topn): async def doc_keyword_extraction(chat_mdl, d, topn):
cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "keywords", {"topn": topn}) cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "keywords", {"topn": topn})
if not cached: if not cached:
if has_canceled(task["id"]):
progress_callback(-1, msg="Task has been canceled.")
return
async with chat_limiter: async with chat_limiter:
cached = await keyword_extraction(chat_mdl, d["content_with_weight"], topn) cached = await keyword_extraction(chat_mdl, d["content_with_weight"], topn)
set_llm_cache(chat_mdl.llm_name, d["content_with_weight"], cached, "keywords", {"topn": topn}) set_llm_cache(chat_mdl.llm_name, d["content_with_weight"], cached, "keywords", {"topn": topn})
@ -362,6 +365,9 @@ async def build_chunks(task, progress_callback):
async def doc_question_proposal(chat_mdl, d, topn): async def doc_question_proposal(chat_mdl, d, topn):
cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "question", {"topn": topn}) cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "question", {"topn": topn})
if not cached: if not cached:
if has_canceled(task["id"]):
progress_callback(-1, msg="Task has been canceled.")
return
async with chat_limiter: async with chat_limiter:
cached = await question_proposal(chat_mdl, d["content_with_weight"], topn) cached = await question_proposal(chat_mdl, d["content_with_weight"], topn)
set_llm_cache(chat_mdl.llm_name, d["content_with_weight"], cached, "question", {"topn": topn}) set_llm_cache(chat_mdl.llm_name, d["content_with_weight"], cached, "question", {"topn": topn})
@ -392,6 +398,9 @@ async def build_chunks(task, progress_callback):
cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "metadata", cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], "metadata",
task["parser_config"]["metadata"]) task["parser_config"]["metadata"])
if not cached: if not cached:
if has_canceled(task["id"]):
progress_callback(-1, msg="Task has been canceled.")
return
async with chat_limiter: async with chat_limiter:
cached = await gen_metadata(chat_mdl, cached = await gen_metadata(chat_mdl,
metadata_schema(task["parser_config"]["metadata"]), metadata_schema(task["parser_config"]["metadata"]),
@ -457,6 +466,9 @@ async def build_chunks(task, progress_callback):
async def doc_content_tagging(chat_mdl, d, topn_tags): async def doc_content_tagging(chat_mdl, d, topn_tags):
cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], all_tags, {"topn": topn_tags}) cached = get_llm_cache(chat_mdl.llm_name, d["content_with_weight"], all_tags, {"topn": topn_tags})
if not cached: if not cached:
if has_canceled(task["id"]):
progress_callback(-1, msg="Task has been canceled.")
return
picked_examples = random.choices(examples, k=2) if len(examples) > 2 else examples picked_examples = random.choices(examples, k=2) if len(examples) > 2 else examples
if not picked_examples: if not picked_examples:
picked_examples.append({"content": "This is an example", TAG_FLD: {'example': 1}}) picked_examples.append({"content": "This is an example", TAG_FLD: {'example': 1}})
@ -890,6 +902,7 @@ async def do_handle_task(task):
task_embedding_id = task["embd_id"] task_embedding_id = task["embd_id"]
task_language = task["language"] task_language = task["language"]
task_llm_id = task["parser_config"].get("llm_id") or task["llm_id"] task_llm_id = task["parser_config"].get("llm_id") or task["llm_id"]
task["llm_id"] = task_llm_id
task_dataset_id = task["kb_id"] task_dataset_id = task["kb_id"]
task_doc_id = task["doc_id"] task_doc_id = task["doc_id"]
task_document_name = task["name"] task_document_name = task["name"]

View File

@ -1,6 +1,6 @@
[project] [project]
name = "ragflow-sdk" name = "ragflow-sdk"
version = "0.23.0" version = "0.23.1"
description = "Python client sdk of [RAGFlow](https://github.com/infiniflow/ragflow). RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding." description = "Python client sdk of [RAGFlow](https://github.com/infiniflow/ragflow). RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding."
authors = [{ name = "Zhichang Yu", email = "yuzhichang@gmail.com" }] authors = [{ name = "Zhichang Yu", email = "yuzhichang@gmail.com" }]
license = { text = "Apache License, Version 2.0" } license = { text = "Apache License, Version 2.0" }

2
sdk/python/uv.lock generated
View File

@ -353,7 +353,7 @@ wheels = [
[[package]] [[package]]
name = "ragflow-sdk" name = "ragflow-sdk"
version = "0.23.0" version = "0.23.1"
source = { virtual = "." } source = { virtual = "." }
dependencies = [ dependencies = [
{ name = "beartype" }, { name = "beartype" },

2
uv.lock generated
View File

@ -6163,7 +6163,7 @@ wheels = [
[[package]] [[package]]
name = "ragflow" name = "ragflow"
version = "0.23.0" version = "0.23.1"
source = { virtual = "." } source = { virtual = "." }
dependencies = [ dependencies = [
{ name = "aiosmtplib" }, { name = "aiosmtplib" },

View File

@ -0,0 +1,7 @@
<svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24"
stroke-linecap="round" stroke-linejoin="round"
class="text-text-04" height="32" width="32"
xmlns="http://www.w3.org/2000/svg">
<path d="M4 4h16c1.1 0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1 0-2-.9-2-2V6c0-1.1.9-2 2-2z"></path>
<polyline points="22,6 12,13 2,6"></polyline>
</svg>

After

Width:  |  Height:  |  Size: 360 B

View File

@ -1,3 +1,4 @@
import { useIsDarkTheme } from '@/components/theme-provider';
import { useSetModalState, useTranslate } from '@/hooks/common-hooks'; import { useSetModalState, useTranslate } from '@/hooks/common-hooks';
import { LangfuseCard } from '@/pages/user-setting/setting-model/langfuse'; import { LangfuseCard } from '@/pages/user-setting/setting-model/langfuse';
import apiDoc from '@parent/docs/references/http_api_reference.md'; import apiDoc from '@parent/docs/references/http_api_reference.md';
@ -28,6 +29,8 @@ const ApiContent = ({
const { handlePreview } = usePreviewChat(idKey); const { handlePreview } = usePreviewChat(idKey);
const isDarkTheme = useIsDarkTheme();
return ( return (
<div className="pb-2"> <div className="pb-2">
<Flex vertical gap={'middle'}> <Flex vertical gap={'middle'}>
@ -47,7 +50,10 @@ const ApiContent = ({
<div style={{ position: 'relative' }}> <div style={{ position: 'relative' }}>
<MarkdownToc content={apiDoc} /> <MarkdownToc content={apiDoc} />
</div> </div>
<MarkdownPreview source={apiDoc}></MarkdownPreview> <MarkdownPreview
source={apiDoc}
wrapperElement={{ 'data-color-mode': isDarkTheme ? 'dark' : 'light' }}
></MarkdownPreview>
</Flex> </Flex>
{apiKeyVisible && ( {apiKeyVisible && (
<ChatApiKeyModal <ChatApiKeyModal

View File

@ -1,79 +1,72 @@
import { DocumentParserType } from '@/constants/knowledge'; import { DocumentParserType } from '@/constants/knowledge';
import { useTranslate } from '@/hooks/common-hooks';
import { useFetchKnowledgeList } from '@/hooks/use-knowledge-request'; import { useFetchKnowledgeList } from '@/hooks/use-knowledge-request';
import { IKnowledge } from '@/interfaces/database/knowledge';
import { useBuildQueryVariableOptions } from '@/pages/agent/hooks/use-get-begin-query'; import { useBuildQueryVariableOptions } from '@/pages/agent/hooks/use-get-begin-query';
import { UserOutlined } from '@ant-design/icons';
import { Avatar as AntAvatar, Form, Select, Space } from 'antd';
import { toLower } from 'lodash'; import { toLower } from 'lodash';
import { useMemo } from 'react'; import { useEffect, useMemo, useState } from 'react';
import { useFormContext } from 'react-hook-form'; import { useFormContext } from 'react-hook-form';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { RAGFlowAvatar } from './ragflow-avatar'; import { RAGFlowAvatar } from './ragflow-avatar';
import { FormControl, FormField, FormItem, FormLabel } from './ui/form'; import { FormControl, FormField, FormItem, FormLabel } from './ui/form';
import { MultiSelect } from './ui/multi-select'; import { MultiSelect, MultiSelectOptionType } from './ui/multi-select';
interface KnowledgeBaseItemProps {
label?: string;
tooltipText?: string;
name?: string;
required?: boolean;
onChange?(): void;
}
const KnowledgeBaseItem = ({
label,
tooltipText,
name,
required = true,
onChange,
}: KnowledgeBaseItemProps) => {
const { t } = useTranslate('chat');
const { list: knowledgeList } = useFetchKnowledgeList(true);
const filteredKnowledgeList = knowledgeList.filter(
(x) => x.parser_id !== DocumentParserType.Tag,
);
const knowledgeOptions = filteredKnowledgeList.map((x) => ({
label: (
<Space>
<AntAvatar size={20} icon={<UserOutlined />} src={x.avatar} />
{x.name}
</Space>
),
value: x.id,
}));
return (
<Form.Item
label={label || t('knowledgeBases')}
name={name || 'kb_ids'}
tooltip={tooltipText || t('knowledgeBasesTip')}
rules={[
{
required,
message: t('knowledgeBasesMessage'),
type: 'array',
},
]}
>
<Select
mode="multiple"
options={knowledgeOptions}
placeholder={t('knowledgeBasesMessage')}
onChange={onChange}
></Select>
</Form.Item>
);
};
export default KnowledgeBaseItem;
function buildQueryVariableOptionsByShowVariable(showVariable?: boolean) { function buildQueryVariableOptionsByShowVariable(showVariable?: boolean) {
return showVariable ? useBuildQueryVariableOptions : () => []; return showVariable ? useBuildQueryVariableOptions : () => [];
} }
export function useDisableDifferenceEmbeddingDataset() {
const [datasetOptions, setDatasetOptions] = useState<MultiSelectOptionType[]>(
[],
);
const [datasetSelectEmbedId, setDatasetSelectEmbedId] = useState('');
const { list: datasetListOrigin } = useFetchKnowledgeList(true);
useEffect(() => {
const datasetListMap = datasetListOrigin
.filter((x) => x.parser_id !== DocumentParserType.Tag)
.map((item: IKnowledge) => {
return {
label: item.name,
icon: () => (
<RAGFlowAvatar
className="size-4"
avatar={item.avatar}
name={item.name}
/>
),
suffix: (
<div className="text-xs px-4 p-1 bg-bg-card text-text-secondary rounded-lg border border-bg-card">
{item.embd_id}
</div>
),
value: item.id,
disabled:
item.embd_id !== datasetSelectEmbedId &&
datasetSelectEmbedId !== '',
};
});
setDatasetOptions(datasetListMap);
}, [datasetListOrigin, datasetSelectEmbedId]);
const handleDatasetSelectChange = (
value: string[],
onChange: (value: string[]) => void,
) => {
if (value.length) {
const data = datasetListOrigin?.find((item) => item.id === value[0]);
setDatasetSelectEmbedId(data?.embd_id ?? '');
} else {
setDatasetSelectEmbedId('');
}
onChange?.(value);
};
return {
datasetOptions,
handleDatasetSelectChange,
};
}
export function KnowledgeBaseFormField({ export function KnowledgeBaseFormField({
showVariable = false, showVariable = false,
}: { }: {
@ -82,22 +75,12 @@ export function KnowledgeBaseFormField({
const form = useFormContext(); const form = useFormContext();
const { t } = useTranslation(); const { t } = useTranslation();
const { list: knowledgeList } = useFetchKnowledgeList(true); const { datasetOptions, handleDatasetSelectChange } =
useDisableDifferenceEmbeddingDataset();
const filteredKnowledgeList = knowledgeList.filter(
(x) => x.parser_id !== DocumentParserType.Tag,
);
const nextOptions = buildQueryVariableOptionsByShowVariable(showVariable)(); const nextOptions = buildQueryVariableOptionsByShowVariable(showVariable)();
const knowledgeOptions = filteredKnowledgeList.map((x) => ({ const knowledgeOptions = datasetOptions;
label: x.name,
value: x.id,
icon: () => (
<RAGFlowAvatar className="size-4 mr-2" avatar={x.avatar} name={x.name} />
),
}));
const options = useMemo(() => { const options = useMemo(() => {
if (showVariable) { if (showVariable) {
return [ return [
@ -140,11 +123,14 @@ export function KnowledgeBaseFormField({
<FormControl> <FormControl>
<MultiSelect <MultiSelect
options={options} options={options}
onValueChange={field.onChange} onValueChange={(value) => {
handleDatasetSelectChange(value, field.onChange);
}}
placeholder={t('chat.knowledgeBasesMessage')} placeholder={t('chat.knowledgeBasesMessage')}
variant="inverted" variant="inverted"
maxCount={100} maxCount={100}
defaultValue={field.value} defaultValue={field.value}
showSelectAll={false}
{...field} {...field}
/> />
</FormControl> </FormControl>

View File

@ -109,6 +109,19 @@ export const SelectWithSearch = forwardRef<
} }
}, [options, value]); }, [options, value]);
const showSearch = useMemo(() => {
if (Array.isArray(options) && options.length > 5) {
return true;
}
if (Array.isArray(options)) {
const optionsNum = options.reduce((acc, option) => {
return acc + (option?.options?.length || 0);
}, 0);
return optionsNum > 5;
}
return false;
}, [options]);
const handleSelect = useCallback( const handleSelect = useCallback(
(val: string) => { (val: string) => {
setValue(val); setValue(val);
@ -179,7 +192,7 @@ export const SelectWithSearch = forwardRef<
align="start" align="start"
> >
<Command className="p-5"> <Command className="p-5">
{options && options.length > 5 && ( {showSearch && (
<CommandInput <CommandInput
placeholder={t('common.search') + '...'} placeholder={t('common.search') + '...'}
className=" placeholder:text-text-disabled" className=" placeholder:text-text-disabled"

View File

@ -1,35 +1,19 @@
import { toast } from 'sonner'; import { ExternalToast, toast } from 'sonner';
const duration = { duration: 2500 }; const configuration: ExternalToast = { duration: 2500, position: 'top-center' };
const message = { const message = {
success: (msg: string) => { success: (msg: string) => {
toast.success(msg, { toast.success(msg, configuration);
position: 'top-center',
closeButton: false,
...duration,
});
}, },
error: (msg: string) => { error: (msg: string) => {
toast.error(msg, { toast.error(msg, configuration);
position: 'top-center',
closeButton: false,
...duration,
});
}, },
warning: (msg: string) => { warning: (msg: string) => {
toast.warning(msg, { toast.warning(msg, configuration);
position: 'top-center',
closeButton: false,
...duration,
});
}, },
info: (msg: string) => { info: (msg: string) => {
toast.info(msg, { toast.info(msg, configuration);
position: 'top-center',
closeButton: false,
...duration,
});
}, },
}; };
export default message; export default message;

View File

@ -211,3 +211,14 @@ export const WebhookJWTAlgorithmList = [
'ps512', 'ps512',
'none', 'none',
] as const; ] as const;
export enum AgentDialogueMode {
Conversational = 'conversational',
Task = 'task',
Webhook = 'Webhook',
}
export const initialBeginValues = {
mode: AgentDialogueMode.Conversational,
prologue: `Hi! I'm your assistant. What can I do for you?`,
};

View File

@ -1,7 +1,7 @@
import { FileUploadProps } from '@/components/file-upload'; import { FileUploadProps } from '@/components/file-upload';
import { useHandleFilterSubmit } from '@/components/list-filter-bar/use-handle-filter-submit'; import { useHandleFilterSubmit } from '@/components/list-filter-bar/use-handle-filter-submit';
import message from '@/components/ui/message'; import message from '@/components/ui/message';
import { AgentGlobals } from '@/constants/agent'; import { AgentGlobals, initialBeginValues } from '@/constants/agent';
import { import {
IAgentLogsRequest, IAgentLogsRequest,
IAgentLogsResponse, IAgentLogsResponse,
@ -76,6 +76,7 @@ export const EmptyDsl = {
data: { data: {
label: 'Begin', label: 'Begin',
name: 'begin', name: 'begin',
form: initialBeginValues,
}, },
sourcePosition: 'left', sourcePosition: 'left',
targetPosition: 'right', targetPosition: 'right',

View File

@ -116,7 +116,7 @@ export interface ITenantInfo {
tts_id: string; tts_id: string;
} }
export type ChunkDocType = 'image' | 'table'; export type ChunkDocType = 'image' | 'table' | 'text';
export interface IChunk { export interface IChunk {
available_int: number; // Whether to enable, 0: not enabled, 1: enabled available_int: number; // Whether to enable, 0: not enabled, 1: enabled

View File

@ -147,6 +147,8 @@ Procedural Memory: Learned skills, habits, and automated procedures.`,
action: 'Action', action: 'Action',
}, },
config: { config: {
memorySizeTooltip: `Accounts for each message's content + its embedding vector (≈ Content + Dimensions × 8 Bytes).
Example: A 1 KB message with 1024-dim embedding uses ~9 KB. The 5 MB default limit holds ~500 such messages.`,
avatar: 'Avatar', avatar: 'Avatar',
description: 'Description', description: 'Description',
memorySize: 'Memory size', memorySize: 'Memory size',
@ -181,6 +183,8 @@ Procedural Memory: Learned skills, habits, and automated procedures.`,
}, },
knowledgeDetails: { knowledgeDetails: {
metadata: { metadata: {
toMetadataSetting: 'Generation settings',
toMetadataSettingTip: 'Set auto-metadata in Configuration.',
descriptionTip: descriptionTip:
'Provide descriptions or examples to guide LLM extract values for this field. If left empty, it will rely on the field name.', 'Provide descriptions or examples to guide LLM extract values for this field. If left empty, it will rely on the field name.',
restrictTDefinedValuesTip: restrictTDefinedValuesTip:
@ -939,6 +943,8 @@ Example: Virtual Hosted Style`,
'Connect GitLab to sync repositories, issues, merge requests, and related documentation.', 'Connect GitLab to sync repositories, issues, merge requests, and related documentation.',
asanaDescription: asanaDescription:
'Connect to Asana and synchronize files from a specified workspace.', 'Connect to Asana and synchronize files from a specified workspace.',
imapDescription:
'Connect to your IMAP mailbox to sync emails for knowledge retrieval.',
dropboxAccessTokenTip: dropboxAccessTokenTip:
'Generate a long-lived access token in the Dropbox App Console with files.metadata.read, files.content.read, and sharing.read scopes.', 'Generate a long-lived access token in the Dropbox App Console with files.metadata.read, files.content.read, and sharing.read scopes.',
moodleDescription: moodleDescription:

View File

@ -755,6 +755,8 @@ export default {
'Подключите GitLab для синхронизации репозиториев, задач, merge requests и связанной документации.', 'Подключите GitLab для синхронизации репозиториев, задач, merge requests и связанной документации.',
asanaDescription: asanaDescription:
'Подключите Asana и синхронизируйте файлы из рабочего пространства.', 'Подключите Asana и синхронизируйте файлы из рабочего пространства.',
imapDescription:
'Подключите почтовый ящик IMAP для синхронизации писем из указанных почтовых ящиков (mailboxes) с целью поиска и анализа знаний.',
google_driveDescription: google_driveDescription:
'Подключите ваш Google Drive через OAuth и синхронизируйте определенные папки или диски.', 'Подключите ваш Google Drive через OAuth и синхронизируйте определенные папки или диски.',
gmailDescription: gmailDescription:

View File

@ -124,12 +124,11 @@ export default {
forgetMessageTip: '确定遗忘吗?', forgetMessageTip: '确定遗忘吗?',
messageDescription: '记忆提取使用高级设置中的提示词和温度值进行配置。', messageDescription: '记忆提取使用高级设置中的提示词和温度值进行配置。',
copied: '已复制!', copied: '已复制!',
contentEmbed: '内容嵌入',
content: '内容', content: '内容',
delMessageWarn: `遗忘后,代理将无法检索此消息。`, delMessageWarn: `遗忘后,代理将无法检索此消息。`,
forgetMessage: '遗忘消息', forgetMessage: '遗忘消息',
sessionId: '会话ID', sessionId: '会话ID',
agent: '代理', agent: '智能体',
type: '类型', type: '类型',
validDate: '有效日期', validDate: '有效日期',
forgetAt: '遗忘于', forgetAt: '遗忘于',
@ -138,6 +137,8 @@ export default {
action: '操作', action: '操作',
}, },
config: { config: {
memorySizeTooltip: `记录每条消息的内容 + 其嵌入向量(≈ 内容 + 维度 × 8 字节)。
例如:一条带有 1024 维嵌入的 1 KB 消息大约使用 9 KB。5 MB 的默认限制大约可容纳 500 条此类消息。`,
avatar: '头像', avatar: '头像',
description: '描述', description: '描述',
memorySize: '记忆大小', memorySize: '记忆大小',
@ -172,6 +173,8 @@ export default {
}, },
knowledgeDetails: { knowledgeDetails: {
metadata: { metadata: {
toMetadataSettingTip: '在配置中设置自动元数据',
toMetadataSetting: '生成设置',
descriptionTip: descriptionTip:
'提供描述或示例来指导大语言模型为此字段提取值。如果留空,将依赖字段名称。', '提供描述或示例来指导大语言模型为此字段提取值。如果留空,将依赖字段名称。',
restrictTDefinedValuesTip: restrictTDefinedValuesTip:
@ -867,6 +870,8 @@ General实体和关系提取提示来自 GitHub - microsoft/graphrag基于
gitlabDescription: gitlabDescription:
'连接 GitLab同步仓库、Issue、合并请求MR及相关文档内容。', '连接 GitLab同步仓库、Issue、合并请求MR及相关文档内容。',
asanaDescription: '连接 Asana同步工作区中的文件。', asanaDescription: '连接 Asana同步工作区中的文件。',
imapDescription:
'连接你的 IMAP 邮箱同步指定mailboxes中的邮件用于知识检索与分析',
r2Description: '连接你的 Cloudflare R2 存储桶以导入和同步文件。', r2Description: '连接你的 Cloudflare R2 存储桶以导入和同步文件。',
dropboxAccessTokenTip: dropboxAccessTokenTip:
'请在 Dropbox App Console 生成 Access Token并勾选 files.metadata.read、files.content.read、sharing.read 等必要权限。', '请在 Dropbox App Console 生成 Access Token并勾选 files.metadata.read、files.content.read、sharing.read 等必要权限。',

View File

@ -1,4 +1,4 @@
import { useMemo, useState } from 'react'; import { useLayoutEffect, useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { useNavigate } from 'umi'; import { useNavigate } from 'umi';
@ -12,7 +12,12 @@ import {
useReactTable, useReactTable,
} from '@tanstack/react-table'; } from '@tanstack/react-table';
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; import {
keepPreviousData,
useMutation,
useQuery,
useQueryClient,
} from '@tanstack/react-query';
import { import {
LucideClipboardList, LucideClipboardList,
@ -125,12 +130,14 @@ function AdminUserManagement() {
queryFn: async () => (await listRoles()).data.data.roles, queryFn: async () => (await listRoles()).data.data.roles,
enabled: IS_ENTERPRISE, enabled: IS_ENTERPRISE,
retry: false, retry: false,
placeholderData: keepPreviousData,
}); });
const { data: usersList } = useQuery({ const { data: usersList } = useQuery({
queryKey: ['admin/listUsers'], queryKey: ['admin/listUsers'],
queryFn: async () => (await listUsers()).data.data, queryFn: async () => (await listUsers()).data.data,
retry: false, retry: false,
placeholderData: keepPreviousData,
}); });
// Delete user mutation // Delete user mutation
@ -354,8 +361,16 @@ function AdminUserManagement() {
getFilteredRowModel: getFilteredRowModel(), getFilteredRowModel: getFilteredRowModel(),
getSortedRowModel: getSortedRowModel(), getSortedRowModel: getSortedRowModel(),
getPaginationRowModel: getPaginationRowModel(), getPaginationRowModel: getPaginationRowModel(),
autoResetPageIndex: false,
}); });
useLayoutEffect(() => {
if (table.getState().pagination.pageIndex > table.getPageCount()) {
table.setPageIndex(Math.max(0, table.getPageCount() - 1));
}
}, [usersList, table]);
return ( return (
<> <>
<Card className="!shadow-none relative h-full bg-transparent overflow-hidden"> <Card className="!shadow-none relative h-full bg-transparent overflow-hidden">
@ -538,7 +553,7 @@ function AdminUserManagement() {
<CardFooter className="flex items-center justify-end"> <CardFooter className="flex items-center justify-end">
<RAGFlowPagination <RAGFlowPagination
total={usersList?.length ?? 0} total={table.getFilteredRowModel().rows.length}
current={table.getState().pagination.pageIndex + 1} current={table.getState().pagination.pageIndex + 1}
pageSize={table.getState().pagination.pageSize} pageSize={table.getState().pagination.pageSize}
onChange={(page, pageSize) => { onChange={(page, pageSize) => {

View File

@ -1,16 +1,19 @@
import { IBeginNode } from '@/interfaces/database/flow'; import { BaseNode } from '@/interfaces/database/flow';
import { cn } from '@/lib/utils'; import { cn } from '@/lib/utils';
import { NodeProps, Position } from '@xyflow/react'; import { NodeProps, Position } from '@xyflow/react';
import get from 'lodash/get'; import get from 'lodash/get';
import { memo } from 'react'; import { memo } from 'react';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { import {
AgentDialogueMode,
BeginQueryType, BeginQueryType,
BeginQueryTypeIconMap, BeginQueryTypeIconMap,
NodeHandleId, NodeHandleId,
Operator, Operator,
} from '../../constant'; } from '../../constant';
import { BeginQuery } from '../../interface'; import { BeginFormSchemaType } from '../../form/begin-form/schema';
import { useBuildWebhookUrl } from '../../hooks/use-build-webhook-url';
import { useIsPipeline } from '../../hooks/use-is-pipeline';
import OperatorIcon from '../../operator-icon'; import OperatorIcon from '../../operator-icon';
import { LabelCard } from './card'; import { LabelCard } from './card';
import { CommonHandle } from './handle'; import { CommonHandle } from './handle';
@ -18,10 +21,21 @@ import { RightHandleStyle } from './handle-icon';
import styles from './index.less'; import styles from './index.less';
import { NodeWrapper } from './node-wrapper'; import { NodeWrapper } from './node-wrapper';
// TODO: do not allow other nodes to connect to this node function InnerBeginNode({
function InnerBeginNode({ data, id, selected }: NodeProps<IBeginNode>) { data,
id,
selected,
}: NodeProps<BaseNode<BeginFormSchemaType>>) {
const { t } = useTranslation(); const { t } = useTranslation();
const inputs: Record<string, BeginQuery> = get(data, 'form.inputs', {}); const inputs = get(data, 'form.inputs', {});
const mode = data.form?.mode;
const isWebhookMode = mode === AgentDialogueMode.Webhook;
const url = useBuildWebhookUrl();
const isPipeline = useIsPipeline();
return ( return (
<NodeWrapper selected={selected} id={id}> <NodeWrapper selected={selected} id={id}>
@ -34,29 +48,46 @@ function InnerBeginNode({ data, id, selected }: NodeProps<IBeginNode>) {
id={NodeHandleId.Start} id={NodeHandleId.Start}
></CommonHandle> ></CommonHandle>
<section className="flex items-center gap-2"> <section className="flex items-center gap-2">
<OperatorIcon name={data.label as Operator}></OperatorIcon> <OperatorIcon name={data.label as Operator}></OperatorIcon>
<div className="truncate text-center font-semibold text-sm"> <div className="truncate text-center font-semibold text-sm">
{t(`flow.begin`)} {t(`flow.begin`)}
</div> </div>
</section> </section>
<section className={cn(styles.generateParameters, 'flex gap-2 flex-col')}> {isPipeline || (
{Object.entries(inputs).map(([key, val], idx) => { <div className="text-accent-primary mt-2 p-1 bg-bg-accent w-fit rounded-sm text-xs">
const Icon = BeginQueryTypeIconMap[val.type as BeginQueryType]; {t(`flow.${isWebhookMode ? 'webhook.name' : mode}`)}
return ( </div>
<LabelCard key={idx} className={cn('flex gap-1.5 items-center')}> )}
<Icon className="size-3.5" /> {isWebhookMode ? (
<label htmlFor="" className="text-accent-primary text-sm italic"> <LabelCard className="mt-2 flex gap-1 items-center">
{key} <span className="font-bold">URL</span>
</label> <span className="flex-1 truncate">{url}</span>
<LabelCard className="py-0.5 truncate flex-1"> </LabelCard>
{val.name} ) : (
<section
className={cn(styles.generateParameters, 'flex gap-2 flex-col')}
>
{Object.entries(inputs).map(([key, val], idx) => {
const Icon = BeginQueryTypeIconMap[val.type as BeginQueryType];
return (
<LabelCard key={idx} className={cn('flex gap-1.5 items-center')}>
<Icon className="size-3.5" />
<label
htmlFor=""
className="text-accent-primary text-sm italic"
>
{key}
</label>
<LabelCard className="py-0.5 truncate flex-1">
{val.name}
</LabelCard>
<span className="flex-1">{val.optional ? 'Yes' : 'No'}</span>
</LabelCard> </LabelCard>
<span className="flex-1">{val.optional ? 'Yes' : 'No'}</span> );
</LabelCard> })}
); </section>
})} )}
</section>
</NodeWrapper> </NodeWrapper>
); );
} }

View File

@ -15,19 +15,15 @@ import {
initialLlmBaseValues, initialLlmBaseValues,
} from '@/constants/agent'; } from '@/constants/agent';
export { export {
AgentDialogueMode,
AgentStructuredOutputField, AgentStructuredOutputField,
JsonSchemaDataType, JsonSchemaDataType,
Operator, Operator,
initialBeginValues,
} from '@/constants/agent'; } from '@/constants/agent';
export * from './pipeline'; export * from './pipeline';
export enum AgentDialogueMode {
Conversational = 'conversational',
Task = 'task',
Webhook = 'Webhook',
}
import { ModelVariableType } from '@/constants/knowledge'; import { ModelVariableType } from '@/constants/knowledge';
import { t } from 'i18next'; import { t } from 'i18next';
@ -109,11 +105,6 @@ export const initialRetrievalValues = {
}, },
}; };
export const initialBeginValues = {
mode: AgentDialogueMode.Conversational,
prologue: `Hi! I'm your assistant. What can I do for you?`,
};
export const initialRewriteQuestionValues = { export const initialRewriteQuestionValues = {
...initialLlmBaseValues, ...initialLlmBaseValues,
language: '', language: '',
@ -750,6 +741,8 @@ export const NodeMap = {
[Operator.Loop]: 'loopNode', [Operator.Loop]: 'loopNode',
[Operator.LoopStart]: 'loopStartNode', [Operator.LoopStart]: 'loopStartNode',
[Operator.ExitLoop]: 'exitLoopNode', [Operator.ExitLoop]: 'exitLoopNode',
[Operator.ExcelProcessor]: 'ragNode',
[Operator.PDFGenerator]: 'ragNode',
}; };
export enum BeginQueryType { export enum BeginQueryType {

View File

@ -3,13 +3,16 @@ import { CopyToClipboardWithText } from '@/components/copy-to-clipboard';
import NumberInput from '@/components/originui/number-input'; import NumberInput from '@/components/originui/number-input';
import { SelectWithSearch } from '@/components/originui/select-with-search'; import { SelectWithSearch } from '@/components/originui/select-with-search';
import { RAGFlowFormItem } from '@/components/ragflow-form'; import { RAGFlowFormItem } from '@/components/ragflow-form';
import { Label } from '@/components/ui/label';
import { MultiSelect } from '@/components/ui/multi-select'; import { MultiSelect } from '@/components/ui/multi-select';
import { Separator } from '@/components/ui/separator';
import { Textarea } from '@/components/ui/textarea'; import { Textarea } from '@/components/ui/textarea';
import { useBuildWebhookUrl } from '@/pages/agent/hooks/use-build-webhook-url';
import { buildOptions } from '@/utils/form'; import { buildOptions } from '@/utils/form';
import { upperFirst } from 'lodash';
import { useCallback } from 'react'; import { useCallback } from 'react';
import { useFormContext, useWatch } from 'react-hook-form'; import { useFormContext, useWatch } from 'react-hook-form';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { useParams } from 'umi';
import { import {
RateLimitPerList, RateLimitPerList,
WebhookMaxBodySize, WebhookMaxBodySize,
@ -22,7 +25,10 @@ import { Auth } from './auth';
import { WebhookRequestSchema } from './request-schema'; import { WebhookRequestSchema } from './request-schema';
import { WebhookResponse } from './response'; import { WebhookResponse } from './response';
const RateLimitPerOptions = buildOptions(RateLimitPerList); const RateLimitPerOptions = RateLimitPerList.map((x) => ({
value: x,
label: upperFirst(x),
}));
const RequestLimitMap = { const RequestLimitMap = {
[WebhookRateLimitPer.Second]: 100, [WebhookRateLimitPer.Second]: 100,
@ -33,7 +39,6 @@ const RequestLimitMap = {
export function WebHook() { export function WebHook() {
const { t } = useTranslation(); const { t } = useTranslation();
const { id } = useParams();
const form = useFormContext(); const form = useFormContext();
const rateLimitPer = useWatch({ const rateLimitPer = useWatch({
@ -45,7 +50,7 @@ export function WebHook() {
return RequestLimitMap[rateLimitPer as keyof typeof RequestLimitMap] ?? 100; return RequestLimitMap[rateLimitPer as keyof typeof RequestLimitMap] ?? 100;
}, []); }, []);
const text = `${location.protocol}//${location.host}/api/v1/webhook/${id}`; const text = useBuildWebhookUrl();
return ( return (
<> <>
@ -74,33 +79,36 @@ export function WebHook() {
></SelectWithSearch> ></SelectWithSearch>
</RAGFlowFormItem> </RAGFlowFormItem>
<Auth></Auth> <Auth></Auth>
<RAGFlowFormItem <section>
name="security.rate_limit.limit" <Label>{t('flow.webhook.limit')}</Label>
label={t('flow.webhook.limit')} <div className="flex items-center mt-1 gap-2">
> <RAGFlowFormItem
<NumberInput name="security.rate_limit.limit"
max={getLimitRateLimitPerMax(rateLimitPer)} className="flex-1"
className="w-full" >
></NumberInput> <NumberInput
</RAGFlowFormItem> max={getLimitRateLimitPerMax(rateLimitPer)}
<RAGFlowFormItem className="w-full"
name="security.rate_limit.per" ></NumberInput>
label={t('flow.webhook.per')} </RAGFlowFormItem>
> <Separator className="w-2" />
{(field) => ( <RAGFlowFormItem name="security.rate_limit.per">
<SelectWithSearch {(field) => (
options={RateLimitPerOptions} <SelectWithSearch
value={field.value} options={RateLimitPerOptions}
onChange={(val) => { value={field.value}
field.onChange(val); onChange={(val) => {
form.setValue( field.onChange(val);
'security.rate_limit.limit', form.setValue(
getLimitRateLimitPerMax(val), 'security.rate_limit.limit',
); getLimitRateLimitPerMax(val),
}} );
></SelectWithSearch> }}
)} ></SelectWithSearch>
</RAGFlowFormItem> )}
</RAGFlowFormItem>
</div>
</section>
<RAGFlowFormItem <RAGFlowFormItem
name="security.max_body_size" name="security.max_body_size"
label={t('flow.webhook.maxBodySize')} label={t('flow.webhook.maxBodySize')}

View File

@ -179,6 +179,8 @@ export const useInitializeOperatorParams = () => {
[Operator.Loop]: initialLoopValues, [Operator.Loop]: initialLoopValues,
[Operator.LoopStart]: {}, [Operator.LoopStart]: {},
[Operator.ExitLoop]: {}, [Operator.ExitLoop]: {},
[Operator.PDFGenerator]: {},
[Operator.ExcelProcessor]: {},
}; };
}, [llmId]); }, [llmId]);

View File

@ -0,0 +1,8 @@
import { useParams } from 'umi';
export function useBuildWebhookUrl() {
const { id } = useParams();
const text = `${location.protocol}//${location.host}/api/v1/webhook/${id}`;
return text;
}

View File

@ -8,7 +8,7 @@ import {
TooltipContent, TooltipContent,
TooltipTrigger, TooltipTrigger,
} from '@/components/ui/tooltip'; } from '@/components/ui/tooltip';
import { IChunk } from '@/interfaces/database/knowledge'; import type { ChunkDocType, IChunk } from '@/interfaces/database/knowledge';
import { cn } from '@/lib/utils'; import { cn } from '@/lib/utils';
import { CheckedState } from '@radix-ui/react-checkbox'; import { CheckedState } from '@radix-ui/react-checkbox';
import classNames from 'classnames'; import classNames from 'classnames';
@ -67,6 +67,10 @@ const ChunkCard = ({
setEnabled(available === 1); setEnabled(available === 1);
}, [available]); }, [available]);
const chunkType =
((item.doc_type_kwd &&
String(item.doc_type_kwd)?.toLowerCase()) as ChunkDocType) || 'text';
return ( return (
<Card <Card
className={classNames('relative flex-none', styles.chunkCard, { className={classNames('relative flex-none', styles.chunkCard, {
@ -81,9 +85,7 @@ const ChunkCard = ({
bg-bg-card rounded-bl-2xl rounded-tr-lg bg-bg-card rounded-bl-2xl rounded-tr-lg
border-l-0.5 border-b-0.5 border-border-button" border-l-0.5 border-b-0.5 border-border-button"
> >
{t( {t(`chunk.docType.${chunkType}`)}
`chunk.docType.${item.doc_type_kwd ? String(item.doc_type_kwd).toLowerCase() : 'text'}`,
)}
</span> </span>
<div className="flex items-start justify-between gap-2"> <div className="flex items-start justify-between gap-2">

View File

@ -22,6 +22,7 @@ import { Switch } from '@/components/ui/switch';
import { Textarea } from '@/components/ui/textarea'; import { Textarea } from '@/components/ui/textarea';
import { useFetchChunk } from '@/hooks/use-chunk-request'; import { useFetchChunk } from '@/hooks/use-chunk-request';
import { IModalProps } from '@/interfaces/common'; import { IModalProps } from '@/interfaces/common';
import type { ChunkDocType } from '@/interfaces/database/knowledge';
import React, { useCallback, useEffect, useState } from 'react'; import React, { useCallback, useEffect, useState } from 'react';
import { FieldValues, FormProvider, useForm } from 'react-hook-form'; import { FieldValues, FormProvider, useForm } from 'react-hook-form';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
@ -151,20 +152,25 @@ const ChunkCreatingModal: React.FC<IModalProps<any> & kFProps> = ({
<FormField <FormField
control={form.control} control={form.control}
name="doc_type_kwd" name="doc_type_kwd"
render={({ field }) => ( render={({ field }) => {
<FormItem> const chunkType =
<FormLabel>{t(`chunk.type`)}</FormLabel> ((field.value &&
<FormControl> String(field.value)?.toLowerCase()) as ChunkDocType) ||
<Input 'text';
type="text"
value={t( return (
`chunk.docType.${field.value ? String(field.value).toLowerCase() : 'text'}`, <FormItem>
)} <FormLabel>{t(`chunk.type`)}</FormLabel>
readOnly <FormControl>
/> <Input
</FormControl> type="text"
</FormItem> value={t(`chunk.docType.${chunkType}`)}
)} readOnly
/>
</FormControl>
</FormItem>
);
}}
/> />
)} )}

View File

@ -15,6 +15,7 @@ import {
TableRow, TableRow,
} from '@/components/ui/table'; } from '@/components/ui/table';
import { useSetModalState } from '@/hooks/common-hooks'; import { useSetModalState } from '@/hooks/common-hooks';
import { Routes } from '@/routes';
import { import {
ColumnDef, ColumnDef,
flexRender, flexRender,
@ -27,6 +28,7 @@ import {
import { Plus, Settings, Trash2 } from 'lucide-react'; import { Plus, Settings, Trash2 } from 'lucide-react';
import { useCallback, useEffect, useMemo, useState } from 'react'; import { useCallback, useEffect, useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { useHandleMenuClick } from '../../sidebar/hooks';
import { import {
MetadataDeleteMap, MetadataDeleteMap,
MetadataType, MetadataType,
@ -78,7 +80,7 @@ export const ManageMetadataModal = (props: IManageModalProps) => {
addUpdateValue, addUpdateValue,
addDeleteValue, addDeleteValue,
} = useManageMetaDataModal(originalTableData, metadataType, otherData); } = useManageMetaDataModal(originalTableData, metadataType, otherData);
const { handleMenuClick } = useHandleMenuClick();
const { const {
visible: manageValuesVisible, visible: manageValuesVisible,
showModal: showManageValuesModal, showModal: showManageValuesModal,
@ -335,67 +337,87 @@ export const ManageMetadataModal = (props: IManageModalProps) => {
success?.(res); success?.(res);
}} }}
> >
<div className="flex flex-col gap-2"> <>
<div className="flex items-center justify-between"> <div className="flex flex-col gap-2">
<div>{t('knowledgeDetails.metadata.metadata')}</div> <div className="flex items-center justify-between">
{isCanAdd && ( <div>{t('knowledgeDetails.metadata.metadata')}</div>
<Button {metadataType === MetadataType.Manage && false && (
variant={'ghost'} <Button
className="border border-border-button" variant={'ghost'}
onClick={handAddValueRow} className="border border-border-button"
> type="button"
<Plus /> onClick={handleMenuClick(Routes.DataSetSetting, {
</Button> openMetadata: true,
)} })}
</div> >
<Table rootClassName="max-h-[800px]"> {t('knowledgeDetails.metadata.toMetadataSetting')}
<TableHeader> </Button>
{table.getHeaderGroups().map((headerGroup) => ( )}
<TableRow key={headerGroup.id}> {isCanAdd && (
{headerGroup.headers.map((header) => ( <Button
<TableHead key={header.id}> variant={'ghost'}
{header.isPlaceholder className="border border-border-button"
? null type="button"
: flexRender( onClick={handAddValueRow}
header.column.columnDef.header, >
header.getContext(), <Plus />
)} </Button>
</TableHead> )}
))} </div>
</TableRow> <Table rootClassName="max-h-[800px]">
))} <TableHeader>
</TableHeader> {table.getHeaderGroups().map((headerGroup) => (
<TableBody className="relative"> <TableRow key={headerGroup.id}>
{table.getRowModel().rows?.length ? ( {headerGroup.headers.map((header) => (
table.getRowModel().rows.map((row) => ( <TableHead key={header.id}>
<TableRow {header.isPlaceholder
key={row.id} ? null
data-state={row.getIsSelected() && 'selected'} : flexRender(
className="group" header.column.columnDef.header,
> header.getContext(),
{row.getVisibleCells().map((cell) => ( )}
<TableCell key={cell.id}> </TableHead>
{flexRender(
cell.column.columnDef.cell,
cell.getContext(),
)}
</TableCell>
))} ))}
</TableRow> </TableRow>
)) ))}
) : ( </TableHeader>
<TableRow> <TableBody className="relative">
<TableCell {table.getRowModel().rows?.length ? (
colSpan={columns.length} table.getRowModel().rows.map((row) => (
className="h-24 text-center" <TableRow
> key={row.id}
<Empty type={EmptyType.Data} /> data-state={row.getIsSelected() && 'selected'}
</TableCell> className="group"
</TableRow> >
)} {row.getVisibleCells().map((cell) => (
</TableBody> <TableCell key={cell.id}>
</Table> {flexRender(
</div> cell.column.columnDef.cell,
cell.getContext(),
)}
</TableCell>
))}
</TableRow>
))
) : (
<TableRow>
<TableCell
colSpan={columns.length}
className="h-24 text-center"
>
<Empty type={EmptyType.Data} />
</TableCell>
</TableRow>
)}
</TableBody>
</Table>
</div>
{metadataType === MetadataType.Manage && (
<div className=" absolute bottom-6 left-5 text-text-secondary text-sm">
{t('knowledgeDetails.metadata.toMetadataSettingTip')}
</div>
)}
</>
</Modal> </Modal>
{manageValuesVisible && ( {manageValuesVisible && (
<ManageValuesModal <ManageValuesModal

View File

@ -25,12 +25,13 @@ import { useComposeLlmOptionsByModelTypes } from '@/hooks/use-llm-request';
import { cn } from '@/lib/utils'; import { cn } from '@/lib/utils';
import { t } from 'i18next'; import { t } from 'i18next';
import { Settings } from 'lucide-react'; import { Settings } from 'lucide-react';
import { useMemo, useState } from 'react'; import { useCallback, useEffect, useMemo, useState } from 'react';
import { import {
ControllerRenderProps, ControllerRenderProps,
FieldValues, FieldValues,
useFormContext, useFormContext,
} from 'react-hook-form'; } from 'react-hook-form';
import { useLocation } from 'umi';
import { import {
MetadataType, MetadataType,
useManageMetadata, useManageMetadata,
@ -368,6 +369,7 @@ export function AutoMetadata({
otherData?: Record<string, any>; otherData?: Record<string, any>;
}) { }) {
// get metadata field // get metadata field
const location = useLocation();
const form = useFormContext(); const form = useFormContext();
const { const {
manageMetadataVisible, manageMetadataVisible,
@ -377,6 +379,29 @@ export function AutoMetadata({
config: metadataConfig, config: metadataConfig,
} = useManageMetadata(); } = useManageMetadata();
const handleClickOpenMetadata = useCallback(() => {
const metadata = form.getValues('parser_config.metadata');
const tableMetaData = util.metaDataSettingJSONToMetaDataTableData(metadata);
showManageMetadataModal({
metadata: tableMetaData,
isCanAdd: true,
type: type,
record: otherData,
});
}, [form, otherData, showManageMetadataModal, type]);
useEffect(() => {
const locationState = location.state as
| { openMetadata?: boolean }
| undefined;
if (locationState?.openMetadata) {
setTimeout(() => {
handleClickOpenMetadata();
}, 100);
locationState.openMetadata = false;
}
}, [location, handleClickOpenMetadata]);
const autoMetadataField: FormFieldConfig = { const autoMetadataField: FormFieldConfig = {
name: 'parser_config.enable_metadata', name: 'parser_config.enable_metadata',
label: t('knowledgeConfiguration.autoMetadata'), label: t('knowledgeConfiguration.autoMetadata'),
@ -386,21 +411,7 @@ export function AutoMetadata({
tooltip: t('knowledgeConfiguration.autoMetadataTip'), tooltip: t('knowledgeConfiguration.autoMetadataTip'),
render: (fieldProps: ControllerRenderProps) => ( render: (fieldProps: ControllerRenderProps) => (
<div className="flex items-center justify-between"> <div className="flex items-center justify-between">
<Button <Button type="button" variant="ghost" onClick={handleClickOpenMetadata}>
type="button"
variant="ghost"
onClick={() => {
const metadata = form.getValues('parser_config.metadata');
const tableMetaData =
util.metaDataSettingJSONToMetaDataTableData(metadata);
showManageMetadataModal({
metadata: tableMetaData,
isCanAdd: true,
type: type,
record: otherData,
});
}}
>
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
<Settings /> <Settings />
{t('knowledgeConfiguration.settings')} {t('knowledgeConfiguration.settings')}

View File

@ -79,6 +79,7 @@ export default function Dataset() {
useRowSelection(); useRowSelection();
const { const {
chunkNum,
list, list,
visible: reparseDialogVisible, visible: reparseDialogVisible,
hideModal: hideReparseDialogModal, hideModal: hideReparseDialogModal,
@ -216,9 +217,9 @@ export default function Dataset() {
{reparseDialogVisible && ( {reparseDialogVisible && (
<ReparseDialog <ReparseDialog
// hidden={isZeroChunk || isRunning} // hidden={isZeroChunk || isRunning}
hidden={true} hidden={false}
handleOperationIconClick={handleOperationIconClick} handleOperationIconClick={handleOperationIconClick}
chunk_num={0} chunk_num={chunkNum}
visible={reparseDialogVisible} visible={reparseDialogVisible}
hideModal={hideReparseDialogModal} hideModal={hideReparseDialogModal}
></ReparseDialog> ></ReparseDialog>

View File

@ -183,7 +183,7 @@ export function ParsingStatusCell({
)} )}
{reparseDialogVisible && ( {reparseDialogVisible && (
<ReparseDialog <ReparseDialog
hidden={isZeroChunk || isRunning} hidden={isRunning}
// hidden={false} // hidden={false}
handleOperationIconClick={handleOperationIconClick} handleOperationIconClick={handleOperationIconClick}
chunk_num={chunk_num} chunk_num={chunk_num}

View File

@ -2,12 +2,14 @@ import { ConfirmDeleteDialog } from '@/components/confirm-delete-dialog';
import { import {
DynamicForm, DynamicForm,
DynamicFormRef, DynamicFormRef,
FormFieldConfig,
FormFieldType, FormFieldType,
} from '@/components/dynamic-form'; } from '@/components/dynamic-form';
import { Checkbox } from '@/components/ui/checkbox'; import { Checkbox } from '@/components/ui/checkbox';
import { DialogProps } from '@radix-ui/react-dialog'; import { DialogProps } from '@radix-ui/react-dialog';
import { t } from 'i18next'; import { memo, useCallback, useEffect, useRef, useState } from 'react';
import { memo, useCallback, useEffect, useRef } from 'react'; import { ControllerRenderProps } from 'react-hook-form';
import { useTranslation } from 'react-i18next';
export const ReparseDialog = memo( export const ReparseDialog = memo(
({ ({
@ -26,18 +28,77 @@ export const ReparseDialog = memo(
hideModal: () => void; hideModal: () => void;
hidden?: boolean; hidden?: boolean;
}) => { }) => {
// const [formInstance, setFormInstance] = useState<DynamicFormRef | null>( const [defaultValues, setDefaultValues] = useState<any>(null);
// null, const [fields, setFields] = useState<FormFieldConfig[]>([]);
// ); const { t } = useTranslation();
const handleOperationIconClickRef = useRef(handleOperationIconClick);
const hiddenRef = useRef(hidden);
// const formCallbackRef = useCallback((node: DynamicFormRef | null) => { useEffect(() => {
// if (node) { handleOperationIconClickRef.current = handleOperationIconClick;
// setFormInstance(node); hiddenRef.current = hidden;
// console.log('Form instance assigned:', node); });
// } else {
// console.log('Form instance removed'); useEffect(() => {
// } if (hiddenRef.current) {
// }, []); handleOperationIconClickRef.current();
}
}, []);
useEffect(() => {
setDefaultValues({
delete: chunk_num > 0,
apply_kb: false,
});
const deleteField = {
name: 'delete',
label: '',
type: FormFieldType.Checkbox,
render: (fieldProps: ControllerRenderProps) => (
<div className="flex items-center text-text-secondary p-5 border border-border-button rounded-lg">
<Checkbox
{...fieldProps}
checked={fieldProps.value}
onCheckedChange={(checked: boolean) => {
fieldProps.onChange(checked);
}}
/>
<span className="ml-2">
{chunk_num > 0
? t(`knowledgeDetails.redo`, {
chunkNum: chunk_num,
})
: t('knowledgeDetails.redoAll')}
</span>
</div>
),
};
const applyKBField = {
name: 'apply_kb',
label: '',
type: FormFieldType.Checkbox,
defaultValue: false,
render: (fieldProps: ControllerRenderProps) => (
<div className="flex items-center text-text-secondary p-5 border border-border-button rounded-lg">
<Checkbox
{...fieldProps}
checked={fieldProps.value}
onCheckedChange={(checked: boolean) => {
fieldProps.onChange(checked);
}}
/>
<span className="ml-2">
{t('knowledgeDetails.applyAutoMetadataSettings')}
</span>
</div>
),
};
if (chunk_num > 0) {
setFields([deleteField, applyKBField]);
}
if (chunk_num <= 0) {
setFields([applyKBField]);
}
}, [chunk_num, t]);
const formCallbackRef = useRef<DynamicFormRef>(null); const formCallbackRef = useRef<DynamicFormRef>(null);
@ -68,12 +129,6 @@ export const ReparseDialog = memo(
} }
}, [formCallbackRef, handleOperationIconClick]); }, [formCallbackRef, handleOperationIconClick]);
useEffect(() => {
if (hidden) {
handleOperationIconClick();
}
}, []);
return ( return (
<ConfirmDeleteDialog <ConfirmDeleteDialog
title={t(`knowledgeDetails.parseFile`)} title={t(`knowledgeDetails.parseFile`)}
@ -91,48 +146,8 @@ export const ReparseDialog = memo(
console.log('submit', data); console.log('submit', data);
}} }}
ref={formCallbackRef} ref={formCallbackRef}
fields={[ fields={fields}
{ defaultValues={defaultValues}
name: 'delete',
label: '',
type: FormFieldType.Checkbox,
render: (fieldProps) => (
<div className="flex items-center text-text-secondary p-5 border border-border-button rounded-lg">
<Checkbox
{...fieldProps}
onCheckedChange={(checked: boolean) => {
fieldProps.onChange(checked);
}}
/>
<span className="ml-2">
{chunk_num > 0
? t(`knowledgeDetails.redo`, {
chunkNum: chunk_num,
})
: t('knowledgeDetails.redoAll')}
</span>
</div>
),
},
{
name: 'apply_kb',
label: '',
type: FormFieldType.Checkbox,
render: (fieldProps) => (
<div className="flex items-center text-text-secondary p-5 border border-border-button rounded-lg">
<Checkbox
{...fieldProps}
onCheckedChange={(checked: boolean) => {
fieldProps.onChange(checked);
}}
/>
<span className="ml-2">
{t('knowledgeDetails.applyAutoMetadataSettings')}
</span>
</div>
),
},
]}
> >
{/* <DynamicForm.CancelButton {/* <DynamicForm.CancelButton
handleCancel={() => handleOperationIconClick(false)} handleCancel={() => handleOperationIconClick(false)}

View File

@ -10,7 +10,7 @@ import {
} from '@/hooks/use-document-request'; } from '@/hooks/use-document-request';
import { IDocumentInfo } from '@/interfaces/database/document'; import { IDocumentInfo } from '@/interfaces/database/document';
import { Ban, CircleCheck, CircleX, Play, Trash2 } from 'lucide-react'; import { Ban, CircleCheck, CircleX, Play, Trash2 } from 'lucide-react';
import { useCallback } from 'react'; import { useCallback, useMemo } from 'react';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'react-i18next';
import { toast } from 'sonner'; import { toast } from 'sonner';
import { DocumentType, RunningStatus } from './constant'; import { DocumentType, RunningStatus } from './constant';
@ -32,6 +32,16 @@ export function useBulkOperateDataset({
const { setDocumentStatus } = useSetDocumentStatus(); const { setDocumentStatus } = useSetDocumentStatus();
const { removeDocument } = useRemoveDocument(); const { removeDocument } = useRemoveDocument();
const { visible, showModal, hideModal } = useSetModalState(); const { visible, showModal, hideModal } = useSetModalState();
const chunkNum = useMemo(() => {
if (!documents.length) {
return 0;
}
return documents.reduce((acc, cur) => {
return acc + cur.chunk_num;
}, 0);
}, [documents]);
const runDocument = useCallback( const runDocument = useCallback(
async (run: number, option?: { delete: boolean; apply_kb: boolean }) => { async (run: number, option?: { delete: boolean; apply_kb: boolean }) => {
const nonVirtualKeys = selectedRowKeys.filter( const nonVirtualKeys = selectedRowKeys.filter(
@ -132,5 +142,5 @@ export function useBulkOperateDataset({
}, },
]; ];
return { list, visible, hideModal, showModal, handleRunClick }; return { chunkNum, list, visible, hideModal, showModal, handleRunClick };
} }

View File

@ -38,21 +38,27 @@ interface ProcessLogModalProps {
} }
const InfoItem: React.FC<{ const InfoItem: React.FC<{
overflowTip?: boolean;
label: string; label: string;
value: string | React.ReactNode; value: string | React.ReactNode;
className?: string; className?: string;
}> = ({ label, value, className = '' }) => { }> = ({ label, value, className = '', overflowTip = false }) => {
return ( return (
<div className={`flex flex-col mb-4 ${className}`}> <div className={`flex flex-col mb-4 ${className}`}>
<span className="text-text-secondary text-sm">{label}</span> <span className="text-text-secondary text-sm">{label}</span>
<Tooltip> {overflowTip && (
<TooltipTrigger asChild> <Tooltip>
<span className="text-text-primary mt-1 truncate w-full"> <TooltipTrigger asChild>
{value} <span className="text-text-primary mt-1 truncate w-full">
</span> {value}
</TooltipTrigger> </span>
<TooltipContent>{value}</TooltipContent> </TooltipTrigger>
</Tooltip> <TooltipContent>{value}</TooltipContent>
</Tooltip>
)}
{!overflowTip && (
<span className="text-text-primary mt-1 truncate w-full">{value}</span>
)}
</div> </div>
); );
}; };
@ -139,6 +145,7 @@ const ProcessLogModal: React.FC<ProcessLogModalProps> = ({
return ( return (
<div className="w-1/2" key={key}> <div className="w-1/2" key={key}>
<InfoItem <InfoItem
overflowTip={true}
label={t(key)} label={t(key)}
value={logInfo[key as keyof typeof logInfo]} value={logInfo[key as keyof typeof logInfo]}
/> />

View File

@ -7,8 +7,8 @@ export const useHandleMenuClick = () => {
const { id } = useParams(); const { id } = useParams();
const handleMenuClick = useCallback( const handleMenuClick = useCallback(
(key: Routes) => () => { (key: Routes, data?: any) => () => {
navigate(`${Routes.DatasetBase}${key}/${id}`); navigate(`${Routes.DatasetBase}${key}/${id}`, { state: data });
}, },
[id, navigate], [id, navigate],
); );

View File

@ -18,6 +18,7 @@ import {
} from '@/components/ui/table'; } from '@/components/ui/table';
import { Pagination } from '@/interfaces/common'; import { Pagination } from '@/interfaces/common';
import { replaceText } from '@/pages/dataset/process-log-modal'; import { replaceText } from '@/pages/dataset/process-log-modal';
import { MemoryOptions } from '@/pages/memories/constants';
import { import {
ColumnDef, ColumnDef,
ColumnFiltersState, ColumnFiltersState,
@ -99,7 +100,12 @@ export function MemoryTable({
header: () => <span>{t('memory.messages.type')}</span>, header: () => <span>{t('memory.messages.type')}</span>,
cell: ({ row }) => ( cell: ({ row }) => (
<div className="text-sm font-medium capitalize"> <div className="text-sm font-medium capitalize">
{row.getValue('message_type')} {row.getValue('message_type')
? MemoryOptions(t).find(
(item) =>
item.value === (row.getValue('message_type') as string),
)?.label
: row.getValue('message_type')}
</div> </div>
), ),
}, },
@ -117,13 +123,13 @@ export function MemoryTable({
<div className="text-sm ">{row.getValue('forget_at')}</div> <div className="text-sm ">{row.getValue('forget_at')}</div>
), ),
}, },
{ // {
accessorKey: 'source_id', // accessorKey: 'source_id',
header: () => <span>{t('memory.messages.source')}</span>, // header: () => <span>{t('memory.messages.source')}</span>,
cell: ({ row }) => ( // cell: ({ row }) => (
<div className="text-sm ">{row.getValue('source_id')}</div> // <div className="text-sm ">{row.getValue('source_id')}</div>
), // ),
}, // },
{ {
accessorKey: 'status', accessorKey: 'status',
header: () => <span>{t('memory.messages.enable')}</span>, header: () => <span>{t('memory.messages.enable')}</span>,

View File

@ -92,6 +92,7 @@ export const MemoryModelForm = () => {
label: t('memory.config.memorySize') + ' (Bytes)', label: t('memory.config.memorySize') + ' (Bytes)',
type: FormFieldType.Number, type: FormFieldType.Number,
horizontal: true, horizontal: true,
tooltip: t('memory.config.memorySizeTooltip'),
// placeholder: t('memory.config.memorySizePlaceholder'), // placeholder: t('memory.config.memorySizePlaceholder'),
required: false, required: false,
}} }}

View File

@ -27,6 +27,7 @@ export enum DataSourceKey {
AIRTABLE = 'airtable', AIRTABLE = 'airtable',
GITLAB = 'gitlab', GITLAB = 'gitlab',
ASANA = 'asana', ASANA = 'asana',
IMAP = 'imap',
GITHUB = 'github', GITHUB = 'github',
// SHAREPOINT = 'sharepoint', // SHAREPOINT = 'sharepoint',
// SLACK = 'slack', // SLACK = 'slack',
@ -127,6 +128,11 @@ export const generateDataSourceInfo = (t: TFunction) => {
description: t(`setting.${DataSourceKey.GITHUB}Description`), description: t(`setting.${DataSourceKey.GITHUB}Description`),
icon: <SvgIcon name={'data-source/github'} width={38} />, icon: <SvgIcon name={'data-source/github'} width={38} />,
}, },
[DataSourceKey.IMAP]: {
name: 'IMAP',
description: t(`setting.${DataSourceKey.IMAP}Description`),
icon: <SvgIcon name={'data-source/imap'} width={38} />,
},
}; };
}; };
@ -654,7 +660,7 @@ export const DataSourceFormFields = {
{ {
label: 'Access Token', label: 'Access Token',
name: 'config.credentials.airtable_access_token', name: 'config.credentials.airtable_access_token',
type: FormFieldType.Text, type: FormFieldType.Password,
required: true, required: true,
}, },
{ {
@ -722,7 +728,7 @@ export const DataSourceFormFields = {
{ {
label: 'API Token', label: 'API Token',
name: 'config.credentials.asana_api_token_secret', name: 'config.credentials.asana_api_token_secret',
type: FormFieldType.Text, type: FormFieldType.Password,
required: true, required: true,
}, },
{ {
@ -778,6 +784,44 @@ export const DataSourceFormFields = {
defaultValue: false, defaultValue: false,
}, },
], ],
[DataSourceKey.IMAP]: [
{
label: 'Username',
name: 'config.credentials.imap_username',
type: FormFieldType.Text,
required: true,
},
{
label: 'Password',
name: 'config.credentials.imap_password',
type: FormFieldType.Password,
required: true,
},
{
label: 'Host',
name: 'config.imap_host',
type: FormFieldType.Text,
required: true,
},
{
label: 'Port',
name: 'config.imap_port',
type: FormFieldType.Number,
required: true,
},
{
label: 'Mailboxes',
name: 'config.imap_mailbox',
type: FormFieldType.Tag,
required: false,
},
{
label: 'Poll Range',
name: 'config.poll_range',
type: FormFieldType.Number,
required: false,
},
],
}; };
export const DataSourceFormDefaultValues = { export const DataSourceFormDefaultValues = {
@ -1017,4 +1061,19 @@ export const DataSourceFormDefaultValues = {
}, },
}, },
}, },
[DataSourceKey.IMAP]: {
name: '',
source: DataSourceKey.IMAP,
config: {
name: '',
imap_host: '',
imap_port: 993,
imap_mailbox: [],
poll_range: 30,
credentials: {
imap_username: '',
imap_password: '',
},
},
},
}; };

View File

@ -127,9 +127,21 @@ const SourceDetailPage = () => {
}, []); }, []);
useEffect(() => { useEffect(() => {
const baseFields = DataSourceFormBaseFields.map((field) => {
if (field.name === 'name') {
return {
...field,
disabled: true,
};
} else {
return {
...field,
};
}
});
if (detail) { if (detail) {
const fields = [ const fields = [
...DataSourceFormBaseFields, ...baseFields,
...DataSourceFormFields[ ...DataSourceFormFields[
detail.source as keyof typeof DataSourceFormFields detail.source as keyof typeof DataSourceFormFields
], ],

View File

@ -14,6 +14,7 @@ import { ChevronsDown, ChevronsUp, Trash2 } from 'lucide-react';
import { FC } from 'react'; import { FC } from 'react';
import { isLocalLlmFactory } from '../../utils'; import { isLocalLlmFactory } from '../../utils';
import { useHandleDeleteFactory, useHandleEnableLlm } from '../hooks'; import { useHandleDeleteFactory, useHandleEnableLlm } from '../hooks';
import { mapModelKey } from './un-add-model';
interface IModelCardProps { interface IModelCardProps {
item: LlmItem; item: LlmItem;
@ -145,7 +146,8 @@ export const ModelProviderCard: FC<IModelCardProps> = ({
key={index} key={index}
className="px-2 py-1 text-xs bg-bg-card text-text-secondary rounded-md" className="px-2 py-1 text-xs bg-bg-card text-text-secondary rounded-md"
> >
{tag} {mapModelKey[tag.trim() as keyof typeof mapModelKey] ||
tag.trim()}
</span> </span>
))} ))}
</div> </div>

View File

@ -7,7 +7,21 @@ import { useTranslate } from '@/hooks/common-hooks';
import { useSelectLlmList } from '@/hooks/use-llm-request'; import { useSelectLlmList } from '@/hooks/use-llm-request';
import { ArrowUpRight, Plus } from 'lucide-react'; import { ArrowUpRight, Plus } from 'lucide-react';
import { FC, useMemo, useState } from 'react'; import { FC, useMemo, useState } from 'react';
export const mapModelKey = {
IMAGE2TEXT: 'VLM',
'TEXT EMBEDDING': 'Embedding',
SPEECH2TEXT: 'ASR',
'TEXT RE-RANK': 'Rerank',
};
const orderMap: Record<TagType, number> = {
LLM: 1,
'TEXT EMBEDDING': 2,
'TEXT RE-RANK': 3,
TTS: 4,
SPEECH2TEXT: 5,
IMAGE2TEXT: 6,
MODERATION: 7,
};
type TagType = type TagType =
| 'LLM' | 'LLM'
| 'TEXT EMBEDDING' | 'TEXT EMBEDDING'
@ -18,16 +32,6 @@ type TagType =
| 'MODERATION'; | 'MODERATION';
const sortTags = (tags: string) => { const sortTags = (tags: string) => {
const orderMap: Record<TagType, number> = {
LLM: 1,
'TEXT EMBEDDING': 2,
'TEXT RE-RANK': 3,
TTS: 4,
SPEECH2TEXT: 5,
IMAGE2TEXT: 6,
MODERATION: 7,
};
return tags return tags
.split(',') .split(',')
.map((tag) => tag.trim()) .map((tag) => tag.trim())
@ -64,7 +68,10 @@ export const AvailableModels: FC<{
factoryList.forEach((model) => { factoryList.forEach((model) => {
model.tags.split(',').forEach((tag) => tagsSet.add(tag.trim())); model.tags.split(',').forEach((tag) => tagsSet.add(tag.trim()));
}); });
return Array.from(tagsSet).sort(); return Array.from(tagsSet).sort(
(a, b) =>
(orderMap[a as TagType] || 999) - (orderMap[b as TagType] || 999),
);
}, [factoryList]); }, [factoryList]);
const handleTagClick = (tag: string) => { const handleTagClick = (tag: string) => {
@ -114,7 +121,7 @@ export const AvailableModels: FC<{
: 'text-text-secondary border-none bg-bg-card' : 'text-text-secondary border-none bg-bg-card'
}`} }`}
> >
{tag} {mapModelKey[tag.trim() as keyof typeof mapModelKey] || tag.trim()}
</Button> </Button>
))} ))}
</div> </div>
@ -162,7 +169,9 @@ export const AvailableModels: FC<{
key={index} key={index}
className="px-1 flex items-center h-5 text-xs bg-bg-card text-text-secondary rounded-md" className="px-1 flex items-center h-5 text-xs bg-bg-card text-text-secondary rounded-md"
> >
{tag} {/* {tag} */}
{mapModelKey[tag.trim() as keyof typeof mapModelKey] ||
tag.trim()}
</span> </span>
))} ))}
</div> </div>