mirror of
https://github.com/infiniflow/ragflow.git
synced 2025-12-08 20:42:30 +08:00
optimize srv broker and executor logic (#630)
### What problem does this PR solve? Optimize task broker and executor for reduce memory usage and deployment complexity. ### Type of change - [x] Performance Improvement - [x] Refactoring ### Change Log - Enhance redis utils for message queue(use stream) - Modify task broker logic via message queue (1.get parse event from message queue 2.use ThreadPoolExecutor async executor ) - Modify the table column name of document and task (process_duation -> process_duration maybe just a spelling mistake) - Reformat some code style(just what i see) - Add requirement_dev.txt for developer - Add redis container on docker compose --------- Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
This commit is contained in:
@ -12,29 +12,14 @@ function task_exe(){
|
||||
done
|
||||
}
|
||||
|
||||
function watch_broker(){
|
||||
while [ 1 -eq 1 ];do
|
||||
C=`ps aux|grep "task_broker.py"|grep -v grep|wc -l`;
|
||||
if [ $C -lt 1 ];then
|
||||
$PY rag/svr/task_broker.py &
|
||||
fi
|
||||
sleep 5;
|
||||
done
|
||||
}
|
||||
|
||||
function task_bro(){
|
||||
watch_broker;
|
||||
}
|
||||
|
||||
task_bro &
|
||||
|
||||
WS=1
|
||||
for ((i=0;i<WS;i++))
|
||||
do
|
||||
task_exe $i $WS &
|
||||
done
|
||||
|
||||
while [ 1 -eq 1 ];do
|
||||
$PY api/ragflow_server.py
|
||||
while [ 1 -eq q ];do
|
||||
$PY api/ragflow_server.py
|
||||
done
|
||||
wait;
|
||||
|
||||
wait;
|
||||
|
||||
Reference in New Issue
Block a user