Commit Graph

78 Commits

Author SHA1 Message Date
0aa3c4cdae Docs: Update version references to v0.20.2 in READMEs and docs (#9559)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.20.1 to v0.20.2
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-19 17:26:49 +08:00
c8bbf7452d Env: Update dependencies for proxy support (#9519)
### What problem does this PR solve?

- Update httpx dependency to include socks support in pyproject.toml
- Update lockfile with new socksio dependency

### Type of change

- [x] Update dependencies for proxy support
2025-08-18 12:04:16 +08:00
99df0766fe Feat: add SMTP support for user invitation emails (#9479)
### What problem does this PR solve?

Add SMTP support for user invitation emails

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-08-15 18:12:20 +08:00
b6e34e3aa7 Fix: PyPDF's Manipulated FlateDecode streams can exhaust RAM (#9469)
### What problem does this PR solve?

#3951
#8463 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-08-14 13:45:19 +08:00
569ab011c4 Add fallback to use 'calamine' parse engine in excel_parser.py (#9374)
### What problem does this PR solve?

add fallback to `calamine` engine when parse error raised using the
default `openpyxl` / `xlrd` engine.
e.g. the following error can be fixed:
```
Traceback (most recent call last):
  File "/ragflow/deepdoc/parser/excel_parser.py", line 53, in _load_excel_to_workbook
    df = pd.read_excel(file_like_object)
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 495, in read_excel
    io = ExcelFile(
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 1567, in __init__
    self._reader = self._engines[engine](
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_xlrd.py", line 46, in __init__
    super().__init__(
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 573, in __init__
    self.book = self.load_workbook(self.handles.handle, engine_kwargs)
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_xlrd.py", line 63, in load_workbook
    return open_workbook(file_contents=data, **engine_kwargs)
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/__init__.py", line 172, in open_workbook
    bk = open_workbook_xls(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/book.py", line 68, in open_workbook_xls
    bk.biff2_8_load(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/book.py", line 641, in biff2_8_load
    cd.locate_named_stream(UNICODE_LITERAL(qname))
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/compdoc.py", line 398, in locate_named_stream
    result = self._locate_stream(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/compdoc.py", line 429, in _locate_stream
    raise CompDocError("%s corruption: seen[%d] == %d" % (qname, s, self.seen[s]))
xlrd.compdoc.CompDocError: Workbook corruption: seen[2] == 4
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-08-12 12:41:33 +08:00
83771e500c Refa: migrate chat models to LiteLLM (#9394)
### What problem does this PR solve?

All models pass the mock response tests, which means that if a model can
return the correct response, everything should work as expected.
However, not all models have been fully tested in a real environment,
the real API_KEY. I suggest actively monitoring the refactored models
over the coming period to ensure they work correctly and fixing them
step by step, or waiting to merge until most have been tested in
practical environment.

### Type of change

- [x] Refactoring
2025-08-12 10:59:20 +08:00
b9eeb8e64f Docs: Update version references to v0.20.1 in READMEs and docs (#9335)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.20.0 to v0.20.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-08 18:17:25 +08:00
95534f5cf2 Docs: Update version references to v0.20.0 in READMEs and docs (#9164)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.19.1 to v0.20.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-01 20:41:44 +08:00
d9fe279dde Feat: Redesign and refactor agent module (#9113)
### What problem does this PR solve?

#9082 #6365

<u> **WARNING: it's not compatible with the older version of `Agent`
module, which means that `Agent` from older versions can not work
anymore.**</u>

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-07-30 19:41:09 +08:00
ad177951e9 Bump to infinity v0.6.0-dev4 (#9013)
### What problem does this PR solve?

Bump to infinity v0.6.0-dev4.
WARNNING: infinity v0.6.0-dev4 has very different meta data format with
older versions. You have to destroy infinity data volume are restart
infinity container if there's existing data.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-07-23 19:27:57 +08:00
f6ae570417 fix redundancy package import. (#8754)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [x] Performance Improvement
- [ ] Other (please describe):

Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
2025-07-11 19:11:52 +08:00
03656da4dd Refa: upgrade MCP SDK to v1.9.4 (#8421)
### What problem does this PR solve?

Upgrade MCP SDK to v1.9.4 (latest).

### Type of change

- [x] Refactoring
2025-06-23 16:53:59 +08:00
7e87eb2e23 Docs: Update version references to v0.19.1 in READMEs and docs (#8366)
### What problem does this PR solve?

- Update Docker image version badges and references from v0.19.0 to
v0.19.1
- Modify version mentions in all localized README files (id, ja, ko,
pt_br, tzh, zh)
- Update version in docker/README.md and related documentation files
- Includes updates to Helm values and Python SDK dependencies

### Type of change

- [x] Documentation Update
2025-06-19 14:39:27 +08:00
44287fb05f Oss support opendal(including mysql) (#8204)
### What problem does this PR solve?

#8074
Oss support opendal(including mysql)

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2025-06-12 11:37:42 +08:00
52c814b89d Refa: Move HTTP API tests to top-level test directory (#8042)
### What problem does this PR solve?

Move test cases only - CI still runs tests under sdk/python

### Type of change

- [x] Refactoring
2025-06-04 13:16:17 +08:00
e702431fcb Feat: sync test group to top pyproject.toml (#8015)
### What problem does this PR solve?

sync test group from sdk/python/pyproject.toml to top pyproject.toml

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-06-03 15:21:06 +08:00
aaefc3f44c update xgboost and dep scripts for local build on MacOS (#7857)
### What problem does this PR solve?

There are two main changes:

1. Update xgboost to 1.6.0 to build the project on MacOS with Apple
chips, this change refers to the issue:
https://github.com/infiniflow/ragflow/issues/5114.

2. When `use_china_mirrors` is set in `download_deps.py`, the names of
chrome files downloaded by the script will be different from the file
names used in Dockerfile, so I added the file name in `get_urls`
function to solve this problem.

I think it's better to add testing for Docker image
`infiniflow/ragflow_deps` to the test workflow, but since the workflow
is currently running on a self-hosted runner, I'm not sure how to modify
it.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-27 09:28:52 +08:00
590b9dabab Docs: update for v0.19.0 (#7823)
### What problem does this PR solve?

update for v0.19.0

### Type of change

- [x] Documentation Update
2025-05-23 18:25:47 +08:00
a1f06a4fdc Feat: Support tool calling in Generate component (#7572)
### What problem does this PR solve?

Hello, our use case requires LLM agent to invoke some tools, so I made a
simple implementation here.

This PR does two things:

1. A simple plugin mechanism based on `pluginlib`:

This mechanism lives in the `plugin` directory. It will only load
plugins from `plugin/embedded_plugins` for now.

A sample plugin `bad_calculator.py` is placed in
`plugin/embedded_plugins/llm_tools`, it accepts two numbers `a` and `b`,
then give a wrong result `a + b + 100`.

In the future, it can load plugins from external location with little
code change.

Plugins are divided into different types. The only plugin type supported
in this PR is `llm_tools`, which must implement the `LLMToolPlugin`
class in the `plugin/llm_tool_plugin.py`.
More plugin types can be added in the future.

2. A tool selector in the `Generate` component:

Added a tool selector to select one or more tools for LLM:


![image](https://github.com/user-attachments/assets/74a21fdf-9333-4175-991b-43df6524c5dc)

And with the `bad_calculator` tool, it results this with the `qwen-max`
model:


![image](https://github.com/user-attachments/assets/93aff9c4-8550-414a-90a2-1a15a5249d94)


### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
2025-05-16 16:32:19 +08:00
0b4d366514 Fix: Setuptools project.license as a TOML table deprecation (#7652)
TOML-table-based project.license is deprecated as per PEP 639, see:
https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license-and-license-files

### What problem does this PR solve?

The following error when building project (e.g. `uv build`)

```
SetuptoolsDeprecationWarning: `project.license` as a TOML table is deprecated
!!

        ********************************************************************************
        Please use a simple string containing a SPDX expression for `project.license`. You can also use `project.license-files`. (Both options available on setuptools>=77.0.0).

        By 2026-Feb-18, you need to update your project and remove deprecated calls
        or your builds will no longer be supported.

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-15 12:23:29 +08:00
e7a84bdac2 Fix: setuptools package definitions (#7654)
### What problem does this PR solve?
For `uv package`/`uv pip install ".[full]"`, bug introduced in #6370:

* Removes erroneous (non-package) directories (`helm`, `flask_session`)
* Adds `mcp.server` package
* Resolves "warning: package would be ignored" ambiguity by changing
`sdk` to `sdk.python.ragflow_sdk`
* Resolves "error: package directory 'intergrations' does not exist" by
including `intergrations.chatgpt-on-wechat.plugins` explicitly
* Also rearranges packages in alphabetical order, for DX.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-15 12:22:56 +08:00
c8c3b756b0 Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140)
### What problem does this PR solve?

This PR adds the support for latest OpenSearch2.19.1 as the store engine
& search engine option for RAGFlow.

### Main Benefit

1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is
much better than Elasticsearch
2. For search, OpenSearch2.19.1 supports full-text
search、vector_search、hybrid_search those are similar with Elasticsearch
on schema
3. For store, OpenSearch2.19.1 stores text、vector those are quite
simliar with Elasticsearch on schema

### Changes

- Support opensearch_python_connetor. I make a lot of adaptions since
the schema and api/method between ES and Opensearch differs in many
ways(especially the knn_search has a significant gap) :
rag/utils/opensearch_coon.py
- Support static config adaptions by changing:
conf/service_conf.yaml、api/settings.py、rag/settings.py
- Supprt some store&search schema changes between OpenSearch and ES:
conf/os_mapping.json
- Support OpenSearch python sdk : pyproject.toml
- Support docker config for OpenSearch2.19.1 :
docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template

### How to use
- I didn't change the priority that ES as the default doc/search engine.
Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it
will work.


### Others
Our team tested a lot of docs in our environment by using OpenSearch as
the vector database ,it works very well.
All the conifg for OpenSearch is necessary.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Yongteng Lei <yongtengrey@outlook.com>
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
2025-04-24 16:03:31 +08:00
03672df691 Docs: update for v0.18.0 (#7223)
### What problem does this PR solve?

update for v0.18.0

### Type of change

- [x] Documentation Update
2025-04-23 12:02:50 +08:00
68b9dae6c0 Feat: mcp server (#7084)
### What problem does this PR solve?

Add MCP support with a client example.

Issue link: #4344

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-04-21 09:43:20 +08:00
6bf26e2a81 Optimize graphrag again (#6513)
### What problem does this PR solve?

Removed set_entity and set_relation to avoid accessing doc engine during
graph computation.
Introduced GraphChange to avoid writing unchanged chunks.

### Type of change

- [x] Performance Improvement
2025-03-26 15:34:42 +08:00
85eb367ede Feat: add basic Langfuse support for LLM module (#6443)
### What problem does this PR solve?

#6155

Add basic Langfuse support for LLM module.

A trace example:

<img width="755" alt="image"
src="https://github.com/user-attachments/assets/25c1f852-5116-486c-a47f-6097187142ca"
/>


### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-03-24 13:18:47 +08:00
4091af4560 Fix: multiple top-level packages error in Python project (#6370)
### What problem does this PR solve?

This PR resolves the issue of multiple top-level packages being detected
in the Python project, which caused errors when using uv pip install.
The problem occurred because the project had multiple directories files
at the root level, leading to a flat-layout error.
To fix this, the pyproject.toml file was updated to explicitly list the
packages using the [tool.setuptools] section. This ensures that the
correct packages are included during installation, avoiding the
flat-layout error.
Type of change

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-03-21 18:44:49 +08:00
674f94228b Chore: unify Ruff config and enable async checks (ASYNC, TRIO) (#6351)
### What problem does this PR solve?

Unify Ruff config and enable async checks (ASYNC, TRIO)

### Type of change

- [x] CI/CD or tooling improvement
2025-03-20 22:31:18 +08:00
4df4bf68a2 DOCS: for release. (#6023)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-03-13 15:09:29 +08:00
d44739283c Docs: prepare docs for release v0.17.1 (#5900)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-03-11 14:39:41 +08:00
c813c1ff4c Made task_executor async to speedup parsing (#5530)
### What problem does this PR solve?

Made task_executor async to speedup parsing

### Type of change

- [x] Performance Improvement
2025-03-03 18:59:49 +08:00
d6836444c9 DOC: for release. (#5472)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2025-03-02 18:47:06 +08:00
37aacb3960 Refa: drop useless fasttext (#5470)
### What problem does this PR solve?

This patch drop useless fastext which is seems useless in the code base 
and its very kind of hard install
should close #4498


### Type of change

- [x] Refactoring

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-02-28 14:30:56 +08:00
53b9e7b52f Add tavily as web searh tool. (#5349)
### What problem does this PR solve?

#5198

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-02-26 10:21:04 +08:00
fe9e9a644f Preparation for release. (#4739)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-02-06 15:15:13 +08:00
3411d0a2ce Added cuda_is_available (#4725)
### What problem does this PR solve?

Added cuda_is_available

### Type of change

- [x] Refactoring
2025-02-05 18:01:23 +08:00
e1526846da Fixed GPU detection on CPU only environment (#4711)
### What problem does this PR solve?

Fixed GPU detection on CPU only environment. Close #4692

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-02-05 12:02:43 +08:00
c4b1c4e6f4 Fix onnxruntime-gpu marks (#4643)
### What problem does this PR solve?

Fix onnxruntime-gpu marks

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-26 09:37:59 +08:00
3c2c8942d5 Removed onnxruntime (#4632)
### What problem does this PR solve?

Removed onnxruntime. It conflicts with the onnxruntime-gpu.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-24 23:41:52 +08:00
8b49734241 Added onnxruntime-gpu (#4631)
### What problem does this PR solve?

Added onnxruntime-gpu

### Type of change

- [x] Refactoring
2025-01-24 14:33:21 +08:00
dd0ebbea35 Light GraphRAG (#4585)
### What problem does this PR solve?

#4543

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-01-22 19:43:14 +08:00
2962284c79 Bump akshare (#4536)
### What problem does this PR solve?

Bump akshare. Close #4525 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-20 11:17:59 +08:00
57b4e0c464 Bump infinity to v0.6.0-dev2 (#4497)
### What problem does this PR solve?

Bump infinity to v0.6.0-dev2. Close #4477 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-16 10:07:17 +08:00
d3c07794b5 Replace poetry with uv (#4471)
### What problem does this PR solve?

Replace poetry with uv

### Type of change

- [x] Refactoring
2025-01-14 11:49:43 +08:00
95261f17f6 Bump infinity to v0.6.0-dev1 (#4448)
### What problem does this PR solve?

Bump infinity to v0.6.0-dev1 and poetry to 2.0.1

### Type of change

- [x] Refactoring
2025-01-12 19:54:33 +08:00
6acbd374d8 fix duckduckgo search subsection error (#4430)
### What problem does this PR solve?

duckduckgo search 6.3.0 still has error sometimes, need to update to
7.2.0, after updated, it works ok.
this PR is going to fix this issue
https://github.com/infiniflow/ragflow/issues/4396

### Type of change

 Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: xiaohzho <xiaohzho@cisco.com>
2025-01-10 09:34:20 +08:00
ac89a2dda1 [Fix] fix duckduck go search 202 ratelimit failed (#4398)
this PR is going to fix issue
https://github.com/infiniflow/ragflow/issues/4396

### Type of change

 Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: xiaohzho <xiaohzho@cisco.com>
2025-01-07 18:40:46 +08:00
439d20e41f Use LTS polars to resolve some machines don't support AVX CPU flag (#4364)
### What problem does this PR solve?

Some old types of machine or virtual machine doesn't support AVX CPU
flag. This PR is to use lts polars module to avoid this fault.

fix issue: #4349

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2025-01-06 18:52:17 +08:00
1160b58b6e Update pyproject.toml to 0.15.1 (#4222)
### What problem does this PR solve?

Missing update information

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-25 14:06:00 +08:00
b7a7413419 Bump infinity to 0.5.2 (#4207)
### What problem does this PR solve?

Bump infinity to 0.5.2

### Type of change

- [x] Refactoring
2024-12-24 15:17:37 +08:00