Dynamically create the service_conf.yaml file by replacing environment variables from .env (#3341)

### What problem does this PR solve?

This pull request implements the feature mentioned in #3322. 

Instead of manually having to edit the `service_conf.yaml` file when
changes have been made to `.env` and mapping it into the docker
container at runtime, a template file is used and the values replaced by
the environment variables from the `.env` file when the container is
started.
 

### Type of change

- [X] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>

docker/.env

@@ -1,6 +1,9 @@
 # The version of Elasticsearch.
 STACK_VERSION=8.11.3
 
+# The hostname where the Elasticsearch service is exposed
+ES_HOST=es01
+
 # The port used to expose the Elasticsearch service to the host machine, 
 # allowing EXTERNAL access to the service running inside the Docker container.
 ES_PORT=1200
@@ -27,10 +30,16 @@ INFINITY_PSQL_PORT=5432
 # The password for MySQL. 
 # When updated, you must revise the `mysql.password` entry in service_conf.yaml.
 MYSQL_PASSWORD=infini_rag_flow
+# The hostname where the MySQL service is exposed
+MYSQL_HOST=mysql
+# The database of the MySQL service to use
+MYSQL_DBNAME=rag_flow
 # The port used to expose the MySQL service to the host machine, 
 # allowing EXTERNAL access to the MySQL database running inside the Docker container. 
 MYSQL_PORT=5455
 
+# The hostname where the MySQL service is exposed
+MINIO_HOST=minio
 # The port used to expose the MinIO console interface to the host machine, 
 # allowing EXTERNAL access to the web-based console running inside the Docker container. 
 MINIO_CONSOLE_PORT=9001
@@ -44,6 +53,8 @@ MINIO_USER=rag_flow
 # When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly.
 MINIO_PASSWORD=infini_rag_flow
 
+# The hostname where the Redis service is exposed
+REDIS_HOST=redis
 # The port used to expose the Redis service to the host machine, 
 # allowing EXTERNAL access to the Redis service running inside the Docker container.
 REDIS_PORT=6379

docker/README.md

@@ -27,7 +27,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 - `ES_PORT`  
   The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container.  Defaults to `1200`.
 - `ELASTIC_PASSWORD`  
-  The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for Elasticsearch. 
 
 ### Kibana
 
@@ -46,7 +46,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 ### MySQL
 
 - `MYSQL_PASSWORD`  
-  The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for MySQL. 
 - `MYSQL_PORT`  
   The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
 
@@ -57,16 +57,16 @@ The [.env](./.env) file contains important environment variables for Docker.
 - `MINIO_PORT`  
   The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
 - `MINIO_USER`  
-  The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The username for MinIO.
 - `MINIO_PASSWORD`  
-  The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for MinIO. 
 
 ### Redis
 
 - `REDIS_PORT`  
   The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
 - `REDIS_PASSWORD`  
-  The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
+  The password for Redis.
 
 ### RAGFlow
 
@@ -119,7 +119,7 @@ The [.env](./.env) file contains important environment variables for Docker.
 
 ## 🐋 Service configuration
 
-[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
+[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. In a dockerized setup, this file is automatically created based on the [service_conf.yaml.template](./service_conf.yaml.template) file (replacing all environment variables by their values).
 
 - `ragflow`
   - `host`: The API server's IP address inside the Docker container. Defaults to `0.0.0.0`.
@@ -139,11 +139,11 @@ The [.env](./.env) file contains important environment variables for Docker.
   - `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`.
 
 - `oauth`  
-  The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
+  The OAuth configuration for signing up or signing in to RAGFlow using a third-party account.  It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
   - `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings page](https://github.com/settings/developers) to obtain your client_id and secret_key.
 
 - `user_default_llm`  
-  The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.  
+  The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.  
   - `factory`: The LLM supplier. Available options:
     - `"OpenAI"`
     - `"DeepSeek"`

docker/docker-compose-gpu.yml

@@ -16,7 +16,6 @@ services:
       - 80:80
       - 443:443
     volumes:
-      - ./service_conf.yaml:/ragflow/conf/service_conf.yaml
       - ./ragflow-logs:/ragflow/logs
       - ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
       - ./nginx/proxy.conf:/etc/nginx/proxy.conf

docker/entrypoint.sh

@@ -1,5 +1,12 @@
 #!/bin/bash
 
+# replace env variables in the service_conf.yaml file
+rm -rf /ragflow/conf/service_conf.yaml
+while IFS= read -r line || [[ -n "$line" ]]; do
+    # Use eval to interpret the variable with default values
+    eval "echo \"$line\"" >> /ragflow/conf/service_conf.yaml
+done < /ragflow/conf/service_conf.yaml.template
+
 # unset http proxy which maybe set by docker daemon
 export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
 

docker/service_conf.yaml.template

@@ -0,0 +1,74 @@
+ragflow:
+  host: ${RAGFLOW_HOST:-0.0.0.0}
+  http_port: 9380
+mysql:
+  name: '${MYSQL_DBNAME:-rag_flow}'
+  user: '${MYSQL_USER:-root}'
+  password: '${MYSQL_PASSWORD:-infini_rag_flow}'
+  host: '${MYSQL_HOST:-mysql}'
+  port: 3306
+  max_connections: 100
+  stale_timeout: 30
+minio:
+  user: '${MINIO_USER:-rag_flow}'
+  password: '${MINIO_PASSWORD:-infini_rag_flow}'
+  host: '${MINIO_HOST:-minio}:9000'
+es:
+  hosts: 'http://${ES_HOST:-es01}:9200'
+  username: '${ES_USER:-elastic}'
+  password: '${ES_PASSWORD:-infini_rag_flow}'
+redis:
+  db: 1
+  password: '${REDIS_PASSWORD:-infini_rag_flow}' 
+  host: '${REDIS_HOST:-redis}:6379'
+
+# postgres:
+#   name: '${POSTGRES_DBNAME:-rag_flow}'
+#   user: '${POSTGRES_USER:-rag_flow}'
+#   password: '${POSTGRES_PASSWORD:-infini_rag_flow}'
+#   host: '${POSTGRES_HOST:-postgres}'
+#   port: 5432
+#   max_connections: 100
+#   stale_timeout: 30
+# s3:
+#   endpoint: 'endpoint'
+#   access_key: 'access_key'
+#   secret_key: 'secret_key'
+#   region: 'region'
+# azure:
+#   auth_type: 'sas'
+#   container_url: 'container_url'
+#   sas_token: 'sas_token'
+# azure:
+#   auth_type: 'spn'
+#   account_url: 'account_url'
+#   client_id: 'client_id'
+#   secret: 'secret'
+#   tenant_id: 'tenant_id'
+#   container_name: 'container_name'
+# user_default_llm:
+#   factory: 'Tongyi-Qianwen'
+#   api_key: 'sk-xxxxxxxxxxxxx'
+#   base_url: ''
+# oauth:
+#   github:
+#     client_id: xxxxxxxxxxxxxxxxxxxxxxxxx
+#     secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
+#     url: https://github.com/login/oauth/access_token
+#   feishu:
+#     app_id: cli_xxxxxxxxxxxxxxxxxxx
+#     app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
+#     app_access_token_url: https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal
+#     user_access_token_url: https://open.feishu.cn/open-apis/authen/v1/oidc/access_token
+#     grant_type: 'authorization_code'
+# authentication:
+#   client:
+#     switch: false
+#     http_app_key:
+#     http_secret_key:
+#   site:
+#     switch: false
+# permission:
+#   switch: false
+#   component: false
+#   dataset: false