Commit Graph

219 Commits

Author SHA1 Message Date
5efb3476f2 turn down es bulk size (#2013)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-08-20 09:59:17 +08:00
6b3a40be5c Format file format from Windows/dos to Unix (#1949)
### What problem does this PR solve?

Related source file is in Windows/DOS format, they are format to Unix
format.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-08-15 09:17:36 +08:00
da8802d010 refine error log while chunking (#1937)
### What problem does this PR solve?



### Type of change

- [x] Refactoring
2024-08-14 11:09:07 +08:00
ede733e130 add support for eml file parser (#1768)
### What problem does this PR solve?

add support for eml file parser
#1363

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-06 16:42:14 +08:00
152072f900 Add graphrag (#1793)
### What problem does this PR solve?

#1594

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-08-02 18:51:14 +08:00
ff8702f7de add support for LocalLLM (#1744)
### What problem does this PR solve?

add support for LocalLLM

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-30 14:07:00 +08:00
H
ac7a0d4fbf Add ParsertType Audio (#1637)
### What problem does this PR solve?

#1514 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-07-22 19:17:30 +08:00
GYH
b85c15cc96 Add file rag/svr/discord_svr.py (#1008)
### What problem does this PR solve?

File rag/svr/discord_svr.py is for discord bot.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-31 13:47:15 +08:00
9ed0e50f6b Update info (#1005)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-05-31 09:53:04 +08:00
6f99bbbb08 add raptor (#899)
### What problem does this PR solve?

#882 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 14:31:16 +08:00
95f809187e add stream chat (#811)
### What problem does this PR solve?

#709 
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-16 20:14:53 +08:00
4d47b2b459 fix a string format error (#781)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-15 13:02:31 +08:00
aa1c915d6e support gpt-4o (#773)
### What problem does this PR solve?
#771 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-15 11:16:08 +08:00
77b1520b66 Refactor message output format (#772)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-05-15 10:48:42 +08:00
ffe5737f7d let index be batchly. (#733)
### What problem does this PR solve?

let index be batchly.

### Type of change


- [x] Refactoring
2024-05-11 19:47:53 +08:00
04a9e95161 let file in knowledgebases visible in file manager (#714)
### What problem does this PR solve?

Let file in knowledgebases visible in file manager.
#162 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-11 16:04:28 +08:00
c28f7b5d38 make sure the error will be recorded. (#672)
### What problem does this PR solve?


### Type of change

- [x] Refactoring
2024-05-08 13:58:41 +08:00
a6e4b74d94 remove unused dependency (#664)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 19:46:17 +08:00
de839fc3f0 optimize srv broker and executor logic (#630)
### What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment
complexity.

### Type of change
- [x] Performance Improvement
- [x] Refactoring

### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-05-07 11:43:33 +08:00
5f03a4de11 remove redis (#629)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-30 19:00:41 +08:00
8c07992b6c refine code (#595)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 19:13:33 +08:00
9d60a84958 refactor code (#583)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 13:19:54 +08:00
944776f207 fix bug about fetching file from minio (#574)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-28 09:57:40 +08:00
66f8d35632 Refactor (#537)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-25 14:14:28 +08:00
0dfc8ddc0f enlarge docker memory usage (#501)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-23 14:41:10 +08:00
b8e58fe27a add redis to accelerate access of minio (#482)
### What problem does this PR solve?

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-22 14:11:09 +08:00
7e41b4bc94 change readme for 0.3.0 release (#459)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2024-04-19 18:19:15 +08:00
ed6081845a Fit a lot of encodings for text file. (#458)
### What problem does this PR solve?

#384

### Type of change

- [x] Performance Improvement
2024-04-19 18:02:53 +08:00
890561703b Add bce-embedding and fastembed (#383)
### What problem does this PR solve?


Issue link:#326

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-16 16:42:19 +08:00
91068edf16 Support Xinference (#320)
### What problem does this PR solve?

Issue link:#299

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-11 18:22:25 +08:00
923cbe488e fix #258 task_executor occupy cpu too much (#288)
### What problem does this PR solve?

Issue link:#285

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-10 10:11:22 +08:00
3708b97db9 Support Ollama (#261)
### What problem does this PR solve?

Issue link:#221

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-08 19:20:57 +08:00
572e5b1ff1 Let task continue dispaching while meeting unexpected doc formats (#199)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

Issue link:#[[Link the issue
here](https://github.com/infiniflow/ragflow/issues/198)]

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Breaking Change (fix or feature that could cause existing
functionality not to work as expected)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Test cases
- [ ] Python SDK impacted, Need to update PyPI
- [ ] Other (please describe):
2024-04-02 11:39:01 +08:00
38e5737067 add base url for OpenAI (#166) 2024-03-28 19:15:16 +08:00
fd7fcb5baf apply pep8 formalize (#155) 2024-03-27 11:33:46 +08:00
979b3a5b4b support snapshot download from local (#153)
* support snapshot download from local

* let snapshot download from local
2024-03-27 09:53:42 +08:00
da21320b88 fix plainPdf bugs (#152) 2024-03-26 15:11:07 +08:00
71fe314955 refine page ranges (#147) 2024-03-25 13:11:57 +08:00
f6aee7f230 add use layout or not option (#145)
* add use layout or not option

* trival
2024-03-22 19:21:09 +08:00
6c6b144de2 refine manual parser (#140) 2024-03-21 18:17:32 +08:00
5875c8ba08 Add 'One' chunk method (#137) 2024-03-20 18:57:22 +08:00
6999598101 refine for English corpus (#135) 2024-03-20 16:56:16 +08:00
675a9f8d9a add dockerfile for cuda envirement. Refine table search strategy, (#123) 2024-03-14 19:45:29 +08:00
0feb085c88 refine table parser (#120) 2024-03-12 18:56:04 +08:00
f1f09df901 add local llm implementation (#119) 2024-03-12 11:57:08 +08:00
b89ac3c4be chage tas execution logic (#103) 2024-03-06 19:16:31 +08:00
602038ac49 fix task cancling bug (#98) 2024-03-05 16:33:47 +08:00
8a57f2afd5 change callback strategy, add timezone to docker (#96) 2024-03-05 12:08:41 +08:00
7bfaf0df29 fix position extraction bug (#93)
* fix position extraction bug

* remove delimiter for naive parser
2024-03-04 17:08:35 +08:00
685b4d8a95 fix table desc bugs, add positions to chunks (#91) 2024-03-04 14:42:26 +08:00