feat(observability): self-contained background-execution Grafana/Prometheus/Loki stack

deploy/observability/grafana-loki/README.md

@@ -0,0 +1,92 @@
+# Langflow on Grafana + Loki
+
+Reference stack that ingests Langflow's structured JSON logs into [Loki](https://grafana.com/oss/loki/) and visualizes them with a pre-provisioned Grafana dashboard.
+
+Use this as a starting point. The compose file, Promtail config, and dashboard JSON are independent of the rest of `deploy/` and can be lifted into any environment.
+
+## What you get
+
+- **Loki 3.2** on `:3100`
+- **Promtail 3.2** scraping a directory of Langflow log files
+- **Grafana 11.3** on `:3000` with the Loki datasource and the `Langflow Logs` dashboard already provisioned
+
+## Prerequisites on the Langflow side
+
+The dashboard expects Langflow to be running in JSON mode with service metadata set. At minimum:
+
+```bash
+LANGFLOW_LOG_ENV=container
+LANGFLOW_LOG_FILE=/absolute/path/to/langflow/logs/langflow.log
+LANGFLOW_SERVICE_NAME=langflow
+LANGFLOW_VERSION=1.10.0
+LANGFLOW_ENVIRONMENT=production
+```
+
+Promtail scrapes a directory of `*.log` files, so `LANGFLOW_LOG_FILE` must point at a file inside
+the directory you expose to Promtail as `LANGFLOW_LOG_DIR` (see [Run](#run)). Set both to the same
+directory, otherwise Promtail watches an empty folder and the dashboard stays blank. Use an
+absolute path: `LANGFLOW_LOG_FILE` is resolved against Langflow's working directory, not this one.
+
+In JSON mode the file is a single JSON stream: application logs and third-party stdlib loggers
+(`uvicorn`, `sqlalchemy`, `httpx`, `langchain`) are all rendered as JSON and run through PII
+redaction, so the `json` parse stage and the **Stdlib intercept routing** panel work against it
+directly. This stack scrapes a file, so `LANGFLOW_LOG_FILE` is required. If you instead run
+Langflow as a container, you can drop the file and scrape its stdout by swapping Promtail's
+`static_configs` file target for `docker_sd_configs` (same JSON, same labels).
+
+See [Logs and observability](../../../docs/docs/Develop/observability-grafana-loki.mdx) for the full list of environment variables (per-logger overrides, extra PII redaction keys, trace correlation, etc.).
+
+## Run
+
+From this directory:
+
+```bash
+# Point Promtail at the directory that holds the file you set in
+# LANGFLOW_LOG_FILE above. Must be the same directory. Defaults to the
+# bundled ./logs (used by the quick smoke test below).
+export LANGFLOW_LOG_DIR=/absolute/path/to/langflow/logs
+
+docker compose up -d
+```
+
+Then open [http://localhost:3000/d/langflow-prod-logs](http://localhost:3000/d/langflow-prod-logs). Default credentials are `admin` / `admin` (override with `GF_ADMIN_USER` and `GF_ADMIN_PASSWORD`).
+
+To stop:
+
+```bash
+docker compose down -v
+```
+
+### Quick smoke test (no Langflow required)
+
+To verify the stack end to end without running Langflow, write a sample record into the bundled
+`./logs` directory and query Loki directly:
+
+```bash
+mkdir -p logs
+echo '{"event":"smoke test","level":"info","logger":"langflow.api.run","timestamp":"2026-06-01T00:00:00Z","service":"langflow","environment":"production","version":"1.10.0"}' >> logs/langflow.log
+
+docker compose up -d
+
+# Give Promtail a few seconds to tail the file, then confirm the line reached Loki:
+sleep 5
+curl -sG 'http://localhost:3100/loki/api/v1/query_range' --data-urlencode 'query={job="langflow"}' | grep -q "smoke test" && echo "OK: log reached Loki"
+```
+
+## What each dashboard panel proves
+
+| Panel | LogQL it runs |
+|---|---|
+| **PII leak count (must be 0)** | `sum(count_over_time({job="langflow"} \|~ "sk-do-not-leak\|hunter2\|Bearer xyz" [$__range]))` |
+| **Errors with structured tracebacks** | `{job="langflow", level=~"error\|critical"} \|= "exception" \| json` |
+| **Redaction proof** | `{job="langflow"} \|~ "\\*\\*\\*"` |
+| **Stdlib intercept routing** | `{job="langflow", logger=~"uvicorn.*\|sqlalchemy.*\|httpx.*\|langchain.*"}` |
+| **Service / environment / version coverage** | `sum by (service, environment, version) (count_over_time({job="langflow"}[$__range]))` |
+| **Log rate by level** | `sum by (level) (rate({job="langflow"}[1m]))` |
+| **Log rate by logger** | `topk(10, sum by (logger) (rate({job="langflow"}[1m])))` |
+
+## Notes
+
+- Promtail only promotes `level`, `service`, `environment`, `version`, `logger` to labels. High-cardinality fields (`user_id`, `flow_id`, `trace_id`) stay in the log body — query them with `| json` in LogQL.
+- Replace Promtail with [Grafana Alloy](https://grafana.com/oss/alloy/) if you already standardize on it; the JSON parse stage maps 1:1.
+- If your runtime ships logs through a different transport (Fluent Bit, Vector, OTLP), only the scrape side changes — the dashboard and label schema stay the same.

deploy/observability/grafana-loki/docker-compose.yml

@@ -0,0 +1,56 @@
+services:
+  loki:
+    image: grafana/loki:3.2.0
+    container_name: lf-loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/local-config.yaml
+    healthcheck:
+      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3100/ready"]
+      interval: 5s
+      timeout: 3s
+      retries: 20
+
+  promtail:
+    image: grafana/promtail:3.2.0
+    container_name: lf-promtail
+    volumes:
+      - ./promtail/config.yml:/etc/promtail/config.yml
+      # Mount the directory that holds your langflow JSON log file.
+      # Override LANGFLOW_LOG_DIR to point at your real log location.
+      - ${LANGFLOW_LOG_DIR:-./logs}:/var/log/langflow:ro
+    command: -config.file=/etc/promtail/config.yml
+    depends_on:
+      loki:
+        condition: service_healthy
+
+  prometheus:
+    image: prom/prometheus:v2.55.0
+    container_name: lf-prometheus
+    ports:
+      - "9091:9090"
+    command:
+      - --config.file=/etc/prometheus/config.yml
+    volumes:
+      - ./prometheus/config.yml:/etc/prometheus/config.yml
+    # Reach a Langflow running on the host (metrics on :9090) from inside the container.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  grafana:
+    image: grafana/grafana:11.3.0
+    container_name: lf-grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_USER=${GF_ADMIN_USER:-admin}
+      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD:-admin}
+      - GF_USERS_DEFAULT_THEME=dark
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning
+      - ./grafana/dashboards:/var/lib/grafana/dashboards
+    depends_on:
+      loki:
+        condition: service_healthy
+      prometheus:
+        condition: service_started

deploy/observability/grafana-loki/grafana/dashboards/langflow-background-execution.json

@@ -0,0 +1,115 @@
+{
+  "uid": "langflow-bg-execution",
+  "title": "Langflow Background Execution",
+  "description": "Durable background execution service: queue depth, job states, throughput, durations, worker liveness, failures, and per-job forensics.",
+  "tags": ["langflow", "background-execution", "metrics"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "5s",
+  "time": {
+    "from": "now-15m",
+    "to": "now"
+  },
+  "templating": {
+    "list": []
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Queued",
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "datasource": "Prometheus",
+      "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"steps": [{"color": "green", "value": null}, {"color": "yellow", "value": 50}, {"color": "red", "value": 200}]}}},
+      "targets": [
+        {"expr": "sum(langflow_bg_jobs{status=\"queued\"})", "refId": "A"}
+      ],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "In progress",
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "datasource": "Prometheus",
+      "targets": [
+        {"expr": "sum(langflow_bg_jobs{status=\"in_progress\"})", "refId": "A"}
+      ],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "Alive workers",
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "datasource": "Prometheus",
+      "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}}},
+      "targets": [
+        {"expr": "sum(langflow_bg_alive_workers)", "refId": "A"}
+      ],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Oldest queued (s)",
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "datasource": "Prometheus",
+      "fieldConfig": {"defaults": {"unit": "s", "color": {"mode": "thresholds"}, "thresholds": {"steps": [{"color": "green", "value": null}, {"color": "yellow", "value": 30}, {"color": "red", "value": 120}]}}},
+      "targets": [
+        {"expr": "max(langflow_bg_oldest_queued_seconds)", "refId": "A"}
+      ],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Throughput (jobs/s)",
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
+      "datasource": "Prometheus",
+      "targets": [
+        {"expr": "rate(langflow_bg_jobs_started_total[1m])", "refId": "A", "legendFormat": "started"},
+        {"expr": "rate(langflow_bg_jobs_completed_total[1m])", "refId": "B", "legendFormat": "completed"},
+        {"expr": "rate(langflow_bg_jobs_failed_total[1m])", "refId": "C", "legendFormat": "failed"}
+      ],
+      "options": {"legend": {"displayMode": "table", "placement": "right", "calcs": ["mean", "max"]}}
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Job duration",
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
+      "datasource": "Prometheus",
+      "fieldConfig": {"defaults": {"unit": "s"}},
+      "targets": [
+        {"expr": "histogram_quantile(0.5, sum(rate(langflow_bg_job_duration_seconds_bucket[5m])) by (le))", "refId": "A", "legendFormat": "p50"},
+        {"expr": "histogram_quantile(0.95, sum(rate(langflow_bg_job_duration_seconds_bucket[5m])) by (le))", "refId": "B", "legendFormat": "p95"}
+      ],
+      "options": {"legend": {"displayMode": "table", "placement": "right", "calcs": ["mean", "max"]}}
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "Failures by reason",
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 12},
+      "datasource": "Prometheus",
+      "targets": [
+        {"expr": "sum by (reason) (rate(langflow_bg_jobs_failed_total[5m]))", "refId": "A", "legendFormat": "{{reason}}"}
+      ],
+      "options": {"legend": {"displayMode": "table", "placement": "right", "calcs": ["sum"]}}
+    },
+    {
+      "id": 8,
+      "type": "logs",
+      "title": "Background job lifecycle (per-job forensics)",
+      "description": "One structured loguru line per lifecycle transition. High-cardinality identifiers (job_id, flow_id, user_id) live on logs, never on metrics.",
+      "gridPos": {"h": 12, "w": 24, "x": 0, "y": 20},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "{job=\"langflow\"} | json | event_type=\"bg_job\"", "refId": "A"}
+      ],
+      "options": {"showLabels": true, "showTime": true, "wrapLogMessage": true, "prettifyLogMessage": true, "dedupStrategy": "none"}
+    }
+  ]
+}

deploy/observability/grafana-loki/grafana/dashboards/langflow-production-logs.json

@@ -0,0 +1,163 @@
+{
+  "uid": "langflow-prod-logs",
+  "title": "Langflow Logs",
+  "description": "Structured logs from Langflow and lfx: tracebacks, PII redaction, service labels, stdlib intercept, trace correlation.",
+  "tags": ["langflow", "lfx", "structured-logs"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "5s",
+  "time": {
+    "from": "now-15m",
+    "to": "now"
+  },
+  "templating": {
+    "list": [
+      {
+        "name": "environment",
+        "type": "query",
+        "datasource": "Loki",
+        "query": "label_values({job=\"langflow\"}, environment)",
+        "current": {"text": "All", "value": "$__all"},
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2
+      },
+      {
+        "name": "service",
+        "type": "query",
+        "datasource": "Loki",
+        "query": "label_values({job=\"langflow\"}, service)",
+        "current": {"text": "All", "value": "$__all"},
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Total log lines (window)",
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "sum(count_over_time({job=\"langflow\", service=~\"$service\", environment=~\"$environment\"}[$__range]))", "refId": "A"}
+      ],
+      "options": {"reduceOptions": {"calcs": ["lastNotNull"]}}
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Errors (window)",
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "datasource": "Loki",
+      "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}},
+      "targets": [
+        {"expr": "sum(count_over_time({job=\"langflow\", level=~\"error|critical\", service=~\"$service\", environment=~\"$environment\"}[$__range]))", "refId": "A"}
+      ]
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "Distinct loggers",
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "count(count by (logger)({job=\"langflow\", service=~\"$service\", environment=~\"$environment\"}))", "refId": "A"}
+      ]
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "PII leak count (must be 0)",
+      "description": "Counts log lines that contain literal sensitive values that should have been redacted. Anything non-zero means redaction failed.",
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "datasource": "Loki",
+      "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}},
+      "targets": [
+        {"expr": "sum(count_over_time({job=\"langflow\"} |~ \"sk-do-not-leak|hunter2|Bearer xyz\"[$__range]))", "refId": "A"}
+      ]
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Log rate by level",
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "sum by (level) (rate({job=\"langflow\", service=~\"$service\", environment=~\"$environment\"}[1m]))", "refId": "A", "legendFormat": "{{level}}"}
+      ],
+      "options": {"legend": {"displayMode": "table", "placement": "right", "calcs": ["sum"]}}
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Log rate by logger (top 10)",
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "topk(10, sum by (logger) (rate({job=\"langflow\", service=~\"$service\", environment=~\"$environment\"}[1m])))", "refId": "A", "legendFormat": "{{logger}}"}
+      ],
+      "options": {"legend": {"displayMode": "table", "placement": "right", "calcs": ["sum"]}}
+    },
+    {
+      "id": 7,
+      "type": "logs",
+      "title": "Structured tracebacks (errors with exception array)",
+      "description": "Every error in JSON mode now ships a structured exception array (exc_type, exc_value, frames) instead of being silently dropped. Expand a row to see the structure.",
+      "gridPos": {"h": 12, "w": 24, "x": 0, "y": 12},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "{job=\"langflow\", level=~\"error|critical\", service=~\"$service\", environment=~\"$environment\"} |= \"exception\" | json", "refId": "A"}
+      ],
+      "options": {"showLabels": true, "showTime": true, "wrapLogMessage": true, "prettifyLogMessage": true, "dedupStrategy": "none"}
+    },
+    {
+      "id": 8,
+      "type": "logs",
+      "title": "PII redaction proof (look for *** sentinels, not the raw value)",
+      "description": "Sensitive keys (password, api_key, token, authorization, cookie) appear as *** before the JSON ever leaves the process.",
+      "gridPos": {"h": 10, "w": 12, "x": 0, "y": 24},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "{job=\"langflow\", service=~\"$service\"} |~ \"\\\\*\\\\*\\\\*\"", "refId": "A"}
+      ],
+      "options": {"showLabels": false, "showTime": true, "wrapLogMessage": true, "prettifyLogMessage": true}
+    },
+    {
+      "id": 9,
+      "type": "logs",
+      "title": "Stdlib intercept: third-party loggers routed into JSON",
+      "description": "uvicorn, sqlalchemy, httpx, langchain, asyncio used to emit unstructured text. With LANGFLOW_LOG_ENV=container they come through the same JSON stream with logger=<name>.",
+      "gridPos": {"h": 10, "w": 12, "x": 12, "y": 24},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "{job=\"langflow\", logger=~\"uvicorn.*|sqlalchemy.*|httpx.*|langchain.*|third_party.*\", service=~\"$service\"}", "refId": "A"}
+      ],
+      "options": {"showLabels": true, "showTime": true, "wrapLogMessage": true, "prettifyLogMessage": true}
+    },
+    {
+      "id": 10,
+      "type": "table",
+      "title": "Top error sources",
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 34},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "topk(10, sum by (logger) (count_over_time({job=\"langflow\", level=~\"error|critical\"}[$__range])))", "refId": "A", "instant": true, "format": "table"}
+      ]
+    },
+    {
+      "id": 11,
+      "type": "table",
+      "title": "Service / environment / version coverage",
+      "description": "Confirms every record carries the service metadata Grafana dashboards key off of.",
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 34},
+      "datasource": "Loki",
+      "targets": [
+        {"expr": "sum by (service, environment, version) (count_over_time({job=\"langflow\"}[$__range]))", "refId": "A", "instant": true, "format": "table"}
+      ]
+    }
+  ]
+}

deploy/observability/grafana-loki/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,10 @@
+apiVersion: 1
+providers:
+  - name: langflow
+    folder: ''
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards

deploy/observability/grafana-loki/grafana/provisioning/datasources/loki.yml

@@ -0,0 +1,10 @@
+apiVersion: 1
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    isDefault: true
+    editable: false
+    jsonData:
+      maxLines: 1000

deploy/observability/grafana-loki/grafana/provisioning/datasources/prometheus.yml

@@ -0,0 +1,8 @@
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: false
+    editable: false

deploy/observability/grafana-loki/prometheus/config.yml

@@ -0,0 +1,11 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: langflow
+    static_configs:
+      # Langflow exposes background-execution metrics on :9090 when
+      # LANGFLOW_PROMETHEUS_ENABLED=true. host.docker.internal reaches a
+      # Langflow running on the host from inside the prometheus container.
+      - targets: ["host.docker.internal:9090"]

deploy/observability/grafana-loki/promtail/config.yml

@@ -0,0 +1,35 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: langflow
+    static_configs:
+      - targets:
+          - localhost
+        labels:
+          job: langflow
+          __path__: /var/log/langflow/*.log
+    pipeline_stages:
+      # Parse the structlog JSON line and promote useful fields to labels.
+      # Keep this set small; high-cardinality fields (user_id, flow_id,
+      # trace_id) should stay in the log body and be queried with `| json`.
+      - json:
+          expressions:
+            level: level
+            service: service
+            environment: environment
+            version: version
+            logger: logger
+      - labels:
+          level:
+          service:
+          environment:
+          version:
+          logger: