Introduced jemalloc (#5590)

### What problem does this PR solve?

Introduced jemalloc.
Python uses pymalloc (which is an reimplementation of gblibc malloc) to
manage RES. It has pools for small objects to avoid returning memory to
OS aggressively. My experience is: Replacing pymalloc with
[jemalloc](https://github.com/jemalloc/jemalloc) can reduce RES and
speedup task_executor.py.

### Type of change

- [x] Performance Improvement
This commit is contained in:
Zhichang Yu
2025-03-04 12:49:39 +08:00
committed by GitHub
parent 688cb8f19d
commit afe9269534
3 changed files with 5 additions and 2 deletions

View File

@ -15,8 +15,9 @@ CONSUMER_NO_BEG=$1
CONSUMER_NO_END=$2
function task_exe(){
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
while [ 1 -eq 1 ]; do
$PY rag/svr/task_executor.py $1;
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
done
}

View File

@ -17,8 +17,9 @@ if [[ -z "$WS" || $WS -lt 1 ]]; then
fi
function task_exe(){
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
while [ 1 -eq 1 ];do
$PY rag/svr/task_executor.py $1;
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
done
}