### What problem does this PR solve?
#308
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: KevinHuSh <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Support displaying tables in the chunks of pdf file when using QA parser
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
due to security problem, need updagre to fix, see bellow
### Type of change
- [x] Other (please describe):
Name| version | CVE | upgrade version
-- | -- | -- | --
PyMySQL | 1.1.0 | CVE-2024-36039 | 1.1.1
Werkzeug | 3.0.1 | CVE-2024-34069 | 3.0.3
aiohttp | 3.9.3 | CVE-2024-30251 | 3.9.4
pillow | 10.2.0 | CVE-2024-28219 | 10.3.0
### What problem does this PR solve?
Support extracting questions and answers from PDF files
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Knowledge base management supports crawling information from web pages
and generating PDF documents
### Type of change
- [x] New Feature (Support document from web pages)
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Optimize task broker and executor for reduce memory usage and deployment
complexity.
### Type of change
- [x] Performance Improvement
- [x] Refactoring
### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
This commit resolves an issue where the 'ollama' package was
inadvertently omitted from the requirements.txt file. The package has
now been added to ensure all dependencies are correctly installed for
the project.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This commit resolves an issue where the 'redis' package was
inadvertently omitted from the requirements.txt file. The package has
now been added to ensure all dependencies are correctly installed for
the project.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add `.doc` file parser, using tika.
```
pip install tika
```
```
from tika import parser
from io import BytesIO
def extract_text_from_doc_bytes(doc_bytes):
file_like_object = BytesIO(doc_bytes)
parsed = parser.from_buffer(file_like_object)
return parsed["content"]
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: chrysanthemum-boy <fannc@qq.com>
