optimize srv broker and executor logic (#630)

### What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment
complexity.

### Type of change
- [x] Performance Improvement
- [x] Refactoring

### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
This commit is contained in:
Fakai Zhao
2024-05-07 11:43:33 +08:00
committed by GitHub
parent c6b6c748ae
commit de839fc3f0
20 changed files with 414 additions and 300 deletions

View File

@ -12,29 +12,14 @@ function task_exe(){
done
}
function watch_broker(){
while [ 1 -eq 1 ];do
C=`ps aux|grep "task_broker.py"|grep -v grep|wc -l`;
if [ $C -lt 1 ];then
$PY rag/svr/task_broker.py &
fi
sleep 5;
done
}
function task_bro(){
watch_broker;
}
task_bro &
WS=1
for ((i=0;i<WS;i++))
do
task_exe $i $WS &
done
while [ 1 -eq 1 ];do
$PY api/ragflow_server.py
while [ 1 -eq q ];do
$PY api/ragflow_server.py
done
wait;
wait;