4d47b2b459
fix a string format error ( #781 )
...
### What problem does this PR solve?
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-15 13:02:31 +08:00
aa1c915d6e
support gpt-4o ( #773 )
...
### What problem does this PR solve?
#771
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-05-15 11:16:08 +08:00
77b1520b66
Refactor message output format ( #772 )
...
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2024-05-15 10:48:42 +08:00
ffe5737f7d
let index be batchly. ( #733 )
...
### What problem does this PR solve?
let index be batchly.
### Type of change
- [x] Refactoring
2024-05-11 19:47:53 +08:00
04a9e95161
let file in knowledgebases visible in file manager ( #714 )
...
### What problem does this PR solve?
Let file in knowledgebases visible in file manager.
#162
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-05-11 16:04:28 +08:00
648a2baaa9
fix disabled doc is still retreivalable ( #695 )
...
### What problem does this PR solve?
Fix that disabled doc is still retreivalable
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-09 15:32:24 +08:00
4153a36683
truncate text to fitin embedding model ( #692 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-05-09 11:35:08 +08:00
c28f7b5d38
make sure the error will be recorded. ( #672 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-05-08 13:58:41 +08:00
eb27a4309e
add support for deepseek ( #668 )
...
### What problem does this PR solve?
#666
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-05-08 10:30:02 +08:00
a6e4b74d94
remove unused dependency ( #664 )
...
### What problem does this PR solve?
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 19:46:17 +08:00
7013d7f620
refine text decode ( #657 )
...
### What problem does this PR solve?
#651
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 12:25:47 +08:00
de839fc3f0
optimize srv broker and executor logic ( #630 )
...
### What problem does this PR solve?
Optimize task broker and executor for reduce memory usage and deployment
complexity.
### Type of change
- [x] Performance Improvement
- [x] Refactoring
### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com >
2024-05-07 11:43:33 +08:00
c6b6c748ae
fix file encoding detection bug ( #653 )
...
### What problem does this PR solve?
#651
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 10:01:24 +08:00
5f03a4de11
remove redis ( #629 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-30 19:00:41 +08:00
7d3b68bb1e
refine code ( #626 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-30 17:53:28 +08:00
cab274f560
remove PyMuPDF ( #618 )
...
### What problem does this PR solve?
#613
### Type of change
- [x] Other (please describe):
2024-04-30 12:38:09 +08:00
674b3aeafd
fix disable and enable llm setting in dialog ( #616 )
...
### What problem does this PR solve?
#614
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-30 11:04:14 +08:00
2af74cc494
refine docker layers ( #606 )
...
### What problem does this PR solve?
### Type of change
- [x] Performance Improvement
2024-04-29 17:57:40 +08:00
8acc01a227
refine redis connection ( #599 )
...
### What problem does this PR solve?
#591
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-29 08:52:38 +08:00
8c07992b6c
refine code ( #595 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-28 19:13:33 +08:00
9d60a84958
refactor code ( #583 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-28 13:19:54 +08:00
944776f207
fix bug about fetching file from minio ( #574 )
...
### What problem does this PR solve?
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-28 09:57:40 +08:00
f1c98aad6b
Update version info ( #564 )
...
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Documentation Update
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2024-04-26 20:07:26 +08:00
66f8d35632
Refactor ( #537 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-25 14:14:28 +08:00
369400c483
fix bug of table in docx ( #510 )
...
### What problem does this PR solve?
#509
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-23 19:10:33 +08:00
aa71462a9f
fix bug #502 ( #504 )
...
### What problem does this PR solve?
#502
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-23 16:01:46 +08:00
72384b191d
Add .doc file parser. ( #497 )
...
### What problem does this PR solve?
Add `.doc` file parser, using tika.
```
pip install tika
```
```
from tika import parser
from io import BytesIO
def extract_text_from_doc_bytes(doc_bytes):
file_like_object = BytesIO(doc_bytes)
parsed = parser.from_buffer(file_like_object)
return parsed["content"]
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: chrysanthemum-boy <fannc@qq.com >
2024-04-23 15:31:43 +08:00
0dfc8ddc0f
enlarge docker memory usage ( #501 )
...
### What problem does this PR solve?
### Type of change
- [x] Refactoring
2024-04-23 14:41:10 +08:00
a38e163035
remove doc from supported processing types ( #488 )
...
### What problem does this PR solve?
#474
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-22 15:46:09 +08:00
3610e1e5b4
fix ollama issuet push ( #486 )
...
### What problem does this PR solve?
#477
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-22 15:13:01 +08:00
11949f9f2e
feat: support markdown files ( #483 )
...
parse markdown files as txt
### What problem does this PR solve?
support markdown files
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-22 14:43:36 +08:00
b8e58fe27a
add redis to accelerate access of minio ( #482 )
...
### What problem does this PR solve?
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-22 14:11:09 +08:00
7e41b4bc94
change readme for 0.3.0 release ( #459 )
...
### What problem does this PR solve?
### Type of change
- [x] Documentation Update
2024-04-19 18:19:15 +08:00
ed6081845a
Fit a lot of encodings for text file. ( #458 )
...
### What problem does this PR solve?
#384
### Type of change
- [x] Performance Improvement
2024-04-19 18:02:53 +08:00
453c29170f
make sure the models will not be load twice ( #422 )
...
### What problem does this PR solve?
#381
### Type of change
- [x] Refactoring
2024-04-18 09:37:23 +08:00
e8570da856
Update table.py to convert clmns to string ( #414 )
...
### What problem does this PR solve?
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-17 19:48:11 +08:00
800b5c7aaa
fix bulk error for table method ( #407 )
...
### What problem does this PR solve?
Issue link:#366
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-17 12:17:14 +08:00
d4e0bfc8a5
fix gb2312 encoding issue ( #394 )
...
### What problem does this PR solve?
Issue link:#384
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-16 19:45:14 +08:00
890561703b
Add bce-embedding and fastembed ( #383 )
...
### What problem does this PR solve?
Issue link:#326
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-16 16:42:19 +08:00
826ad6a33a
feat: FastEmbed embedding support ( #291 )
...
### Description
Following up on https://github.com/infiniflow/ragflow/pull/275 , this PR
adds support for FastEmbed model configurations.
The options are not exhaustive. You can find the full list
[here](https://qdrant.github.io/fastembed/examples/Supported_Models/ ).
P.S. I ran into OOM issues when building the image.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: KevinHuSh <kevinhu.sh@gmail.com >
2024-04-15 15:58:06 +08:00
c39b751600
conversation API backend update ( #360 )
...
### What problem does this PR solve?
Issue link:#345
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-15 14:43:44 +08:00
8ffc09cb5c
Support Xinference ( #321 )
...
### What problem does this PR solve?
Issue link:#299
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-11 18:25:37 +08:00
91068edf16
Support Xinference ( #320 )
...
### What problem does this PR solve?
Issue link:#299
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-11 18:22:25 +08:00
f6c7204002
refine log format ( #312 )
...
### What problem does this PR solve?
Issue link:#264
### Type of change
- [x] Documentation Update
- [x] Refactoring
2024-04-11 10:13:43 +08:00
24c15daaed
fix es exception ( #298 )
...
### What problem does this PR solve?
Issue link:#295
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-10 16:00:48 +08:00
a0a480b708
continue add layout model for 'laws' ( #292 )
...
### What problem does this PR solve?
Issue link:#289
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-10 14:06:36 +08:00
243de6ac90
add a new model for 'Laws' ( #290 )
...
### What problem does this PR solve?
Issue link:#289
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-04-10 11:59:00 +08:00
923cbe488e
fix #258 task_executor occupy cpu too much ( #288 )
...
### What problem does this PR solve?
Issue link:#285
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-10 10:11:22 +08:00
653c759d8d
resolve issure to call ZH?IPUAI ( #277 )
...
### What problem does this PR solve?
Issue link:#265
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-09 16:16:10 +08:00
6748872b0b
remove presence_penalty for chatglm ( #268 )
...
### What problem does this PR solve?
Issue link:#265
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-09 09:24:08 +08:00
