Refa: only support MinerU-API now (#11977)

### What problem does this PR solve?

Only support MinerU-API now, still need to complete frontend for
pipeline to allow the configuration of MinerU options.

### Type of change

- [x] Refactoring
This commit is contained in:
Yongteng Lei
2025-12-17 12:58:48 +08:00
committed by GitHub
parent 5e05f43c3d
commit 03f9be7cbb
19 changed files with 273 additions and 624 deletions

View File

@ -69,7 +69,6 @@ from common.signal_utils import start_tracemalloc_and_snapshot, stop_tracemalloc
from common.exceptions import TaskCanceledException
from common import settings
from common.constants import PAGERANK_FLD, TAG_FLD, SVR_CONSUMER_GROUP_NAME
from common.misc_utils import check_and_install_mineru
BATCH_SIZE = 64
@ -1169,7 +1168,6 @@ async def main():
show_configs()
settings.init_settings()
settings.check_and_install_torch()
check_and_install_mineru()
logging.info(f'default embedding config: {settings.EMBEDDING_CFG}')
settings.print_rag_settings()
if sys.platform != "win32":