mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-24 04:13:36 +08:00
9b7486efe84728880eaf2becc23da2d41f089e98
1150 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 9b7486efe8 |
docs: macos support matrix (#13363)
* docs: macos support matrix * docs: fix linking errors and clarify limitations * cleanup |
|||
| 5e86ab6c5f |
feat: accept v2 workflow globals in request body (#13351)
* fix: accept v2 workflow globals in request body * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * review feedback --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 2ddac97c77 |
feat(job_queue): cross-worker cancel + polling watchdog for multi-worker Redis-backed builds (#13084)
* feat: implement Redis job queue and fakeredis support
- Refactored the job queue service to support Redis-backed management for cross-worker scaling.
- Added environment variables for configuration:
- `LANGFLOW_JOB_QUEUE_TYPE=redis`
- `LANGFLOW_REDIS_QUEUE_DB=1`
- Updated job ownership methods to be asynchronous for improved concurrency handling.
- Enhanced Redis cache service with namespacing via key prefixes.
- Introduced `fakeredis` for in-memory Redis simulation in testin>
- Added comprehensive unit tests for Redis job queue components.
* fix: run experimental warning for RedisCache usage only once
- Introduced a mechanism to emit a one-time warning for the RedisCache experimental feature during server runtime.
- The warning is logged only if no other worker has already emitted it, ensuring clarity for users regarding the experimental status of RedisCache.
- The implementation includes a temporary file check to prevent multiple warnings across different processes.
* docs: document environment variables for worker management
- Added documentation for LANGFLOW_GUNICORN_PRELOAD to explain preloading for better performance.
- Detailed the use of LANGFLOW_JOB_QUEUE_TYPE for specifying backends (e.g., Redis).
- Included LANGFLOW_REDIS_QUEUE_DB to define the database index for job queues.
- Updated the "High-Load Environments" guide with these optimal configurations.
* docs: updated 'High-load and multi-worker environments' section
* feat: enhance RedisJobQueueService with consumer wrapper management
- Introduced a caching mechanism for Redis stream consumers to optimize job data retrieval.
- Added methods to manage consumer wrappers, ensuring they are reused across sequential polls.
- Implemented cleanup logic to cancel and clear consumer wrappers during job cleanup and service stop.
- Expanded unit tests to verify consumer wrapper reuse and cleanup behavior.
* fix: ensure Redis keys are deleted during job cleanup even on cancellation
- Updated the cleanup_job method in RedisJobQueueService to guarantee Redis keys are removed even if the job cleanup is interrupted by a CancelledError.
- Added a new unit test to verify that Redis keys are deleted correctly when cleanup is called during task cancellation.
* fix: manage connection check task in RedisJobQueueService
- Added handling for the connection check task in the stop method to ensure it is properly cancelled and awaited if still running.
- This change improves resource management and prevents potential issues during service shutdown.
* fix: handle unpublished sentinel requeue on cancellation in RedisJobQueueService
- Updated the job processing logic to ensure that if a job is cancelled during the xadd operation, the unpublished sentinel is requeued instead of being dropped.
- Introduced a new unit test to verify this behavior, ensuring robustness in job handling during cancellations.
* fix: improve Redis job queue service with enhanced configuration and cleanup
- Added atexit cleanup to remove stale temporary files for RedisCache.
- Refactored Redis job queue service to use shared constants for stream prefixes, improving maintainability.
- Updated type hints for better clarity and consistency in RedisQueueWrapper and RedisJobQueueService.
- Enhanced error handling with configurable backoff for transient read failures.
* fix: enhance Redis job queue service with maxlen configuration for xadd
- Updated the xadd method in RedisJobQueueService to include maxlen and approximate parameters, improving stream management and preventing excessive memory usage.
* fix: enhance job ownership retrieval in RedisJobQueueService
- Updated the get_job_owner method to refresh the Redis key TTL on successful lookups, ensuring long-running jobs maintain their ownership anchor.
- Improved code clarity by extracting the owner key into a variable and adding detailed docstring explanations for better understanding of the TTL management.
* fix: improve job ownership handling in cleanup_job method
- Enhanced the cleanup_job method in RedisJobQueueService to accurately capture job ownership before deleting Redis keys, preventing potential data corruption in multi-worker scenarios.
- Added comments for clarity on ownership logic and its implications during job cleanup.
* fix: optimize TTL management in RedisJobQueueService
- Introduced periodic TTL refresh logic in the _bridge_to_redis method to enhance Redis stream management, reducing round-trips and improving throughput.
- Added constants for TTL refresh events and seconds to maintain clarity and configurability.
- Updated event handling to ensure TTL is refreshed appropriately based on event count and time elapsed.
* fix: improve event handling in flow response management
- Removed unnecessary error handling for missing event tasks in get_flow_events_response, allowing for smoother operation when no task exists.
- Updated create_flow_response to handle optional event_task parameter, ensuring proper cleanup during disconnections.
- Added unit tests to verify behavior when event tasks are missing, enhancing robustness in streaming scenarios.
* fix: enhance RedisQueueWrapper with startup grace period and stream observation
- Added a startup grace period to prevent premature end-of-stream signals when the producer has not yet issued its first XADD.
- Introduced a flag to track whether the stream has been observed, improving the handling of early polling scenarios.
- Updated logic to ensure proper handling of stream existence checks and logging for better debugging during job processing.
* fix: implement cleanup for old cross-worker job queues in RedisJobQueueService
- Added a new method to clean up done cross-worker consumer wrappers that are not owned by the current worker, ensuring proper resource management.
- Enhanced the existing cleanup logic to prevent memory leaks by explicitly pruning stale entries from the consumer wrappers dictionary.
- Improved logging to provide better visibility into the cleanup process for cross-worker jobs.
* fix: enhance job handling in JobQueueService with guarded task execution
- Updated the start_job method to accept a Coroutine type for task_coro, ensuring type safety.
- Introduced a new _guarded_task method to wrap job coroutines, guaranteeing that unhandled exceptions emit an error event and write a sentinel to the Redis Stream, improving reliability in job processing.
- Enhanced documentation to clarify the behavior of the new task handling mechanism and its implications for cross-worker consumers.
* fix: improve client disconnection handling in create_flow_response
- Added logging for scenarios where a client disconnects without an associated event_task, clarifying that the producer will continue running until the build completes.
- Documented the limitation regarding cross-worker passive disconnects and the need for a Redis side-channel for proper cancellation, enhancing observability in the logs.
* fix: enhance RedisQueueWrapper to manage initial read state and buffer behavior
- Introduced a flag to track the completion of the first XREAD call, ensuring proper buffer management during the initial read phase.
- Updated the empty method to reflect the state of the buffer accurately, preventing premature exits from the drain loop until the first read is complete.
- Improved documentation to clarify the behavior of the new flag and its impact on job processing.
* fix: clarify RedisQueueWrapper behavior in tests for buffer state and no-op operations
- Updated test documentation to explain the behavior of the empty() method in relation to the first XREAD completion, ensuring accurate understanding of buffer state during job processing.
- Adjusted assertions in tests to reflect the intended behavior of the RedisQueueWrapper, specifically regarding the no-op nature of put_nowait and its impact on the internal buffer.
* fix: update dependencies in pyproject.toml and uv.lock
- Added fakeredis dependency with a minimum version of 2.0.0 to both pyproject.toml and uv.lock files.
- Ensured proper formatting and comments in pyproject.toml for clarity on onnxruntime version constraints.
* feat(job_queue): cross-worker cancel via Redis pub/sub + configurable startup grace
- Add LANGFLOW_REDIS_QUEUE_STARTUP_GRACE_S (default 30.0) so deployments
with slow producer-worker cold-start can bump the wrapper's early-poll
grace period without editing code.
- Add LANGFLOW_REDIS_QUEUE_CANCEL_CHANNEL_ENABLED (default True) wiring a
per-job Redis pub/sub channel (langflow:cancel:<job_id>) so
POST /build/{job_id}/cancel works cross-worker. The producer subscribes
when the job starts; any worker can publish via
RedisJobQueueService.signal_cancel and the subscriber cancels the local
build task on receipt.
- cancel_flow_build now falls back to signal_cancel when event_task is
None on this worker, replacing the previous no-op for cross-worker
cancellation.
- Subscriber tasks are tracked in _cancel_subscribers and torn down in
cleanup_job/stop alongside the bridge tasks.
- Tests: 3 new unit tests covering cross-worker signal propagation, the
disabled-flag no-op, and the wrapper's per-instance startup_grace_s
override.
* fix(job_queue): close three cross-worker cancel corner cases
Discovered by a deeper local stress pass over the prototype:
1. **Signal-before-subscribe race**: publish from worker A would be lost if
worker B's subscriber had not finished SUBSCRIBE yet. signal_cancel now
also sets a short-lived langflow:cancel-marker:<job_id> key, and the
subscriber checks it immediately after subscribing — so a cancel that
raced the SUBSCRIBE still fires.
2. **Restart subscriber leak**: start_job called twice for the same job_id
silently overwrote the dict entry and orphaned the previous subscriber
(and its Redis pubsub connection). Now cancels the previous subscriber
before storing the new one.
3. **Slow end-of-stream after cross-worker cancel**: cancelling the local
task did not unblock cross-worker consumers — the bridge sat waiting on
local_queue.get() until periodic cleanup ran ~5 min later. The
subscriber now puts a sentinel on the local queue after task.cancel()
so the bridge flushes a clean end-of-stream marker to Redis. Measured
8ms from signal_cancel to consumer end-of-stream against real Redis.
Tests: 3 new unit tests covering each corner case; 9-scenario local
stress harness against real Redis confirms all green.
* refactor(job_queue): address self-review of cross-worker cancel
Addresses the eight points from the PR review of the previous commits:
1. Configurable marker TTL — new LANGFLOW_REDIS_QUEUE_CANCEL_MARKER_TTL
setting (default 60s) replaces the hardcoded constant.
2. Connection-pool footprint is O(1) in active jobs — replaced the per-job
pub/sub subscriber with a single PSUBSCRIBE dispatcher per worker
listening on langflow:cancel:*.
3. Prompt stream cleanup after cancel — the dispatcher now waits for the
bridge to drain the sentinel before triggering cleanup_job, so the
Redis stream + owner keys are deleted within milliseconds instead of
waiting for the 5-minute periodic cleanup grace.
4. signal_cancel raises on Redis failure — publish errors propagate to the
caller instead of silently returning 0. The cancel HTTP endpoint
catches and returns False so the client can retry.
5. Auth note in signal_cancel docstring — explicit note that callers must
verify authorization; the HTTP cancel endpoint already does via
_verify_job_ownership before calling through.
6. Structured cancel logging — INFO-level logs on publish, marker hit,
owned dispatch, foreign dispatch. _cancel_stats counters expose
{published, marker_hit, dispatched_owned, dispatched_foreign,
publish_errors} for ops/metrics.
7. redis-py version compatibility — _close_pubsub falls back from
aclose() to close(), handles both sync and async return values.
8. Fire-and-forget tasks hold strong references — _background_tasks set
keeps marker checks and post-cancel cleanups from being GC'd before
they run; each task self-removes via add_done_callback. stop() drains
the set on shutdown.
Tests: 24 passing (1 skipped due to timing); deep stress harness verified
6 scenarios against real Redis.
* feat(job_queue): propagate client disconnect via signal_cancel for cross-worker builds
Closes the remaining cross-worker passive-disconnect gap. Previously, when a
client closed its streaming connection on a non-owner worker, the producer
worker kept emitting events until the build completed naturally. Now the
disconnect handler in create_flow_response publishes a cross-worker cancel
via signal_cancel so the owning worker stops promptly.
- create_flow_response accepts optional queue_service + job_id kwargs and
uses them in on_disconnect for the cross-worker case (event_task is None).
Both are keyword-only and default to None, preserving the single-worker
contract.
- get_flow_events_response wires both through.
- New unit test test_cross_worker_disconnect_publishes_signal_cancel covers
the full pubsub propagation path with two services sharing a fake Redis.
- Tested end-to-end against real Redis with a two-worker harness: 10/10
scenarios pass, including the new disconnect propagation case.
* feat(job_queue): production-harden cross-worker cancel for multi-worker setups
Five complementary improvements that make the Redis-backed cancel/lifecycle
path safe to leave running unattended under real ops pressure.
A. Dispatcher reconnect loop with exponential backoff. _run_cancel_dispatcher
used to exit silently on any Redis-side error (broker restart, network
blip, listen() hiccup), leaving the worker permanently blind to
cross-worker cancels until process restart. The loop now reconnects with
capped backoff (max 30s), and a dispatcher_reconnects counter exposes the
event for monitoring.
B. Outer timeout on _post_cancel_cleanup. The background cleanup task could
hang indefinitely if cleanup_job stalled (Redis pathology during DELETE);
wrap in asyncio.wait_for and let periodic cleanup retry instead.
C. Public metrics_snapshot() on JobQueueService (memory + Redis backends)
plus GET /monitor/job_queue endpoint behind get_current_active_user.
Surfaces backend, active_jobs, bridge_count, consumer_wrapper_count,
background_task_count, cancel_dispatcher_running, and the full
cancel_stats counter set.
D. Deterministic rewrite of test_redis_service_signal_cancel_flushes_sentinel_to_consumer.
The previous version was skipped roughly half the time due to a fakeredis
timing race; the new version no-ops cleanup_job for the duration of the
assertion so the bridge XADD ordering can be observed reliably. No more
skipped tests in the suite (33 pass, 0 skip).
E. Polling-mode stale-client watchdog. Polling has no persistent connection
for on_disconnect, so abandoned polling builds previously ran to natural
completion even after the client gave up. New flow:
* touch_activity(job_id) writes langflow:activity:<job_id> on every poll
and on streaming-response open; consume_and_yield refreshes it every
10s while the connection is held.
* _run_polling_watchdog scans owned jobs every
LANGFLOW_REDIS_QUEUE_POLLING_WATCHDOG_INTERVAL_S (default 15s) and
publishes signal_cancel for jobs whose activity is older than
LANGFLOW_REDIS_QUEUE_POLLING_STALE_THRESHOLD_S (default 90s).
* Streaming clients are protected by the heartbeat refresh; threshold <=
0 disables the watchdog entirely.
* cleanup_job folds the activity key into the existing DEL round-trip
so successful builds clean up after themselves.
End-to-end coverage: 33 unit tests pass (up from 25, with the previous skip
now passing deterministically); a two-worker real-Redis harness exercises 12
scenarios including dispatcher reconnect, post-cancel timeout, watchdog
reclamation, and metrics_snapshot schema completeness.
* fix(job_queue): address PR review + Codex adversarial review findings
Acts on every must-fix finding from a multi-agent review pass (code-reviewer,
pr-test-analyzer, silent-failure-hunter, comment-analyzer) plus the Codex
adversarial review. No behavior is left to chance under realistic operational
conditions.
Critical fixes:
- Streaming heartbeat now runs as an INDEPENDENT task for the lifetime of the
response, not coupled to event yield cadence. A quiet build (long graph
step, slow LLM, no tokens for a while) previously had no heartbeat path, so
the polling watchdog could reclaim a live streaming client after the
threshold. The new heartbeat fires every streaming_activity_refresh_s
regardless of queue activity, and is cancelled cleanly on both
on_disconnect AND natural stream end.
- Watchdog start-grace window. A new in-memory _job_start_times[job_id]
timestamp is set in start_job() BEFORE super().start_job() so a job can
never appear in self._queues without a corresponding start time. When the
watchdog scans and the Redis activity key is missing, it skips the kill
until (now - start_ts) >= threshold. Without this, a slow background
touch_activity (event-loop pressure, Redis blip) could nuke a brand-new
build on the very first watchdog tick — the prior code's comment promised
the guard but didn't implement it.
- touch_activity errors are now observable. Replaces a blanket
contextlib.suppress(Exception) with an explicit try/except that bumps
activity_touch_errors and emits a debug log. CancelledError is no longer
swallowed. Operators get a signal when the heartbeat is silently failing.
- Watchdog UnboundLocalError closed. Initializes last = 0.0 before the
raw-is-None branch so a malformed activity value (parse failure) can't
crash the entire watchdog task. Adds activity_parse_errors and
activity_get_errors counters.
- Dispatcher except is split: ConnectionError/TimeoutError/OSError stay at
awarning (transient Redis), every other Exception goes to aerror with
exc_info=True AND increments dispatcher_internal_errors. Code bugs in
_handle_cancel no longer look identical to Redis disconnects.
- _post_cancel_cleanup narrows the bridge-wait suppress to TimeoutError
only, with a dedicated awarning naming the consequence ("cross-worker
consumers may see late end-of-stream"). Real bridge failures bubble up
via a separate Exception arm with awarning instead of being silent.
- Polling watchdog uses local _handle_cancel for owned jobs instead of
round-tripping through pubsub. Faster (no Redis RTT), dispatcher-
independent (works during a reconnect backoff), and keeps the
cancel_stats["published"] counter honest as a count of *external* cancels.
- /monitor/job_queue gated to get_current_active_superuser instead of
get_current_active_user — the snapshot exposes process-wide tenant
workload + cancel activity, which matters in multi-tenant deployments.
Docstring corrections:
- Removed the "cross-worker cancel is a known limitation" remnant from
RedisJobQueueService.get_queue_data — the limitation was closed in the
prior commit, and the new docstring tells callers exactly how cross-worker
cancel travels (signal_cancel → dispatcher).
- touch_activity docstring rewritten. The previous text described a
"TTL-based reasoning" fallback that didn't exist; the new text correctly
explains the 4x-threshold TTL and how it interacts with the watchdog's
start-time grace window.
New tests (5 added, total 38 pass):
- test_polling_watchdog_grants_start_grace_window: brand-new job with
deleted activity key survives until the threshold passes, then dies.
- test_polling_watchdog_skips_malformed_activity_value: malformed value
bumps activity_parse_errors and does NOT kill the job.
- test_streaming_heartbeat_runs_independent_of_event_yield: quiet streaming
build is not reclaimed; heartbeat task is cancelled on disconnect.
- test_concurrent_cancels_from_multiple_workers_are_idempotent: two workers
publish cancel simultaneously, no crashes, single observable cancel.
- test_dispatcher_internal_error_logged_at_error_level: bug in
_handle_cancel bumps dispatcher_internal_errors and dispatcher_reconnects;
the dispatcher task is still running afterwards.
Plus: test_metrics_snapshot_exposes_cancel_stats_and_counters now pins the
full cancel_stats key set (using set equality), so adding an increment
without registering the key — or vice versa — fails the test instead of
producing a silent KeyError in production.
End-to-end coverage: 38 unit tests pass; the two-worker real-Redis harness
passes all 12 scenarios on commit HEAD.
* fix: harden redis job queue cancellation
* feat(job_queue): seamless redis setup + actionable errors for unsupported delivery
- RedisQueueWrapper: restore _BUFFER_MAXSIZE bounded buffer and the
_on_fill_done done-callback safety net that release-1.10.0 added in #12588,
so a slow consumer cannot grow the buffer without bound and a crashing or
cancelled _fill_task cannot leave consumers stuck on await get().
- build.get_flow_events_response: explicit exhaustiveness guard. Unknown
EventDeliveryType values now return HTTP 400 with the supported set and a
remediation hint instead of silently falling through to the polling path.
- lfx settings.set_event_delivery: when workers > 1 without a redis queue,
upgrade the warning to name the requested mode, the forced fallback, and
the LANGFLOW_JOB_QUEUE_TYPE env var that would preserve the original mode.
- Tests: port the three RedisQueueWrapper safety tests from release-1.10.0
and add coverage for the new event_delivery guard.
* fix(job_queue): polling watchdog only watches jobs with a registered owner
Surfaced by locust load testing on a Redis-queue, 2-worker setup:
2016 requests / 2016 polling_watchdog_kills (1:1 ratio)
TaskService.launch_task uses JobQueueService.start_job for server-internal
tasks (telemetry, background work) without calling register_job_owner.
These tasks never refresh the activity heartbeat because no polling
client is involved, so the watchdog's start-time fallback was killing
every single one once it crossed polling_stale_threshold_s.
Under fast-completing flows the user-visible result was unaffected (the
build finished before the watchdog struck) — but a long-running internal
task (slow LLM call, retrieval, etc.) would be cancelled mid-flight even
though no one was waiting on it.
Scope the watchdog to jobs in self._job_owners: user-facing flow builds
register an owner; TaskService tasks don't. After this change the same
locust run reports polling_watchdog_kills=0 and dispatched_owned=0.
Existing watchdog tests register an owner explicitly to mirror the real
flow path. Adds a regression test (TaskService-style start_job with no
owner must not be reclaimed).
* test(job_queue): add coverage for fixes surfaced by load testing
Three regression tests for the recent fixes:
- TaskService integration: launch a task through
TaskService.fire_and_forget_task and assert the polling watchdog
leaves it alone (no owner registered, no kill). Catches a future
regression where someone adds register_job_owner inside TaskService.
- Settings multi-worker warning: verify Settings.set_event_delivery
warns with the requested mode AND names LANGFLOW_JOB_QUEUE_TYPE in
the fallback log when workers > 1 and the job queue is in-memory.
An operator seeing missing events needs that env var name to fix it.
- Bounded buffer backpressure: floods the fill task past the maxsize
ceiling and verifies qsize() never exceeds _BUFFER_MAXSIZE. The
previous test only asserted the constant, not the behavior.
Skipped two suggested tests:
- TestClient version of the unknown-event_delivery guard: FastAPI's
enum validation returns 422 before our handler runs, so HTTP cannot
reach the defensive 400. The existing unit test is the right test.
- Exact watchdog kill-count assertion (== 1): the watchdog can tick
again before cleanup_job removes the job from _queues, making any
exact-count check flaky on slow CI. The existing >= 1 is correct.
* fix(job_queue): use asyncio.create_task for trace cleanup on cancel
background_tasks.add_task() is silently dropped after FastAPI drains
the POST /build response queue — by the time any cancel fires (local,
cross-worker pub/sub, or polling watchdog), the task list from the
original HTTP request is already consumed.
Replace the background_tasks call in _run_vertex_build's CancelledError
handler with asyncio.create_task(graph.end_all_traces_in_context()())
so the trace cleanup runs as an independent task regardless of the
background_tasks lifecycle.
Adds a regression test that exercises generate_flow_events end-to-end:
a blocking vertex is cancelled mid-flight and the test asserts that
end_all_traces is eventually called via the spawned task.
* [autofix.ci] apply automated fixes
* fix: addresses a few pr review findings (#13179)
* fix(job_queue): address PR review findings from cross-worker cancel implementation
- Restore _last_id cursor update to after await buffer.put() in _fill_from_redis;
advancing before the await loses the event if the task is cancelled while put()
is blocked on a full buffer
- Restore XREAD persistent-error grace period: track elapsed error time and deliver
an end-of-stream sentinel after _STARTUP_GRACE_S of continuous failures instead
of looping forever, which left consumers stuck on await get()
- Fix cancel_flow_build returning True when no task and no signal_cancel (in-memory
backend); return False to correctly signal that cancellation did not take effect
- Decouple polling watchdog from cancel_channel_enabled; the watchdog uses only
local state and _handle_cancel with no pub/sub dependency, so disabling the
cancel channel must not silently disable stale-client reclamation
- Restore _BUFFER_MAXSIZE to 1000 (reverts unexplained 10x increase to 10_000)
* test(job_queue): add regression coverage for PR review fixes
- test_fill_from_redis_last_id_not_advanced_before_put_completes: verifies
the XREAD cursor (_last_id) stays at the prior message's position when the
fill task is cancelled while put() is blocked on a full buffer
- test_fill_from_redis_persistent_xread_error_delivers_sentinel: verifies a
persistent XREAD failure eventually delivers the end-of-stream sentinel
after _STARTUP_GRACE_S of continuous errors (not an infinite retry loop)
- test_cancel_flow_build_returns_false_for_in_memory_backend_without_task:
verifies cancel_flow_build returns False when no local task exists and the
queue service has no cross-worker cancel path
- test_polling_watchdog_runs_when_cancel_channel_disabled: verifies the
polling watchdog reclaims stale jobs even when cancel_channel_enabled=False
* format and comment updates
* comments
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fix(job_queue): address CodeRabbit review findings
- cancel_flow_build: gate signal_cancel on a new cross_worker_cancel_enabled
property so Redis with cancel_channel_enabled=False is treated the same as
the in-memory backend (signal_cancel short-circuits to 0 without setting
the marker, so the previous unconditional return True was misleading).
- _run_vertex_build: hold a strong reference to the trace cleanup task on
CancelledError (Ruff RUF006). Tasks self-discard via add_done_callback.
- RedisQueueWrapper.finish_with_sentinel: stop awaiting on a bounded buffer
during teardown. Evict + put_nowait mirrors _on_fill_done so shutdown can
no longer hang on a slow or abandoned consumer.
- RedisJobQueueService.cleanup_job: wrap the Redis DEL in try/except so
Redis failures during teardown surface as warnings instead of escaping
stop() and explicit cancel paths.
- Settings (lfx): add Field bounds on the new Redis timing knobs so a
negative startup_grace_s or non-positive cancel_marker_ttl /
watchdog_interval_s fails fast at config load instead of silently
reopening races.
- Streaming heartbeat interval: lift to module-level STREAMING_ACTIVITY_REFRESH_S
so the heartbeat test can monkeypatch it.
Tests:
- _make_service tracks shared_client ownership so the first _stop_service
doesn't aclose the FakeRedis under a sibling worker in cross-worker tests.
- test_polling_watchdog_skips_fresh_activity now registers a job owner;
without it the watchdog skips the job entirely and the test would pass
even with a broken touch_activity.
- test_streaming_heartbeat_runs_independent_of_event_yield patches the
interval down and waits for the spawned task to refresh the activity
timestamp, instead of calling touch_activity manually.
- Fix Ruff ARG001, EM101, TRY003, PT018 in test stubs.
* test(job_queue): cover behavior changes from CodeRabbit review fixes
- test_cancel_flow_build_returns_false_for_redis_with_disabled_cancel_channel:
Redis with cancel_channel_enabled=False has signal_cancel but no marker
delivery; cancel_flow_build must report failure, not false success.
- test_finish_with_sentinel_does_not_hang_on_full_buffer: regression cover for
the await-on-bounded-buffer hang. Fills the buffer to capacity and asserts
finish_with_sentinel returns within a tight budget.
- test_redis_service_cleanup_swallows_redis_delete_error: proxies a FakeRedis
whose delete() raises; cleanup_job must absorb it and complete local
teardown.
- test_redis_queue_bounds (lfx): pydantic ValidationError on negative
startup_grace_s / non-positive cancel_marker_ttl /
polling_watchdog_interval_s; zero remains valid for startup_grace_s and
polling_stale_threshold_s (documented disable switch).
* log updates
* [autofix.ci] apply automated fixes
* test(cors): accept both '*' and ['*'] for Python 3.14 compatibility
pydantic-settings parses LANGFLOW_CORS_ORIGINS='*' as the raw string '*'
on Python 3.13 but as ['*'] (a single-element list) on Python 3.14+. The
`str | list[str]` union on Settings.cors_origins resolves differently
between versions, but both shapes carry the same 'all origins' semantic.
Updates two assertions in test_security_cors.py that previously required
the exact string form and were failing on Group 4 / Python 3.14 in CI:
- test_default_cors_settings_current_behavior
- test_wildcard_with_credentials_allowed_current_behavior
The PR itself (cross-worker cancel + polling watchdog) is unrelated to
CORS handling -- the failure is a Python-3.14-introduced incompatibility
in this pre-existing test that happens to surface against this branch's
CI matrix.
* [autofix.ci] apply automated fixes
---------
Co-authored-by: Arek Mateusiak <arek@a4bits.com>
Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
|
|||
| ec3aea7ff8 |
fix(extensions): discover editable bundles; clean up ticket refs and reload UX (#13219)
* fix(extensions): discover editable installs via langflow.extensions entry-point
`lfx extension list` was silently dropping bundles installed via `uv pip
install -e` / `pip install -e`. Editable installs surface only
`dist-info/` entries in `dist.files`, so the wheel-shaped scan in
`_distribution_manifest_path` finds no `extension.json`. The bundle
pyproject.toml comment already promised an entry-point fallback for this
case, but it was never implemented in discovery.py.
Add `_distribution_manifest_path_via_entry_points` that resolves the
package via `importlib.util.find_spec` (no module body execution) and
locates the manifest under the resulting package directory. The fallback
runs only when `dist.files` produces no manifest, so wheel installs are
unaffected.
Tests:
- editable distribution with dist-info-only files is discovered
- entry-point pointing at an unimportable module yields no record
- wheel-install path never consults find_spec (guards against
re-importing every bundle package on startup)
* chore(extensions): drop internal ticket refs; relax reload --bundle and --all
Two related changes:
1. Strip internal `LE-NNNN` ticket references from the extension source,
bundles, tests, and public docs. The references were not actionable
outside the project and surfaced in user-visible CLI messages and the
author guide.
2. Relax `lfx extension reload` CLI ergonomics now that local discovery
(`discover_all_extensions`) gives us the install map without needing
an HTTP list endpoint:
- `lfx extension reload <ext_id>`: `--bundle` is now optional; when
omitted, the bundle name is resolved from local discovery. Explicit
`--bundle` still wins for cases where the local install isn't
visible to the running server.
- `lfx extension reload --all`: iterates over every locally-discovered
bundle, POSTs reload to each, renders per-bundle status, and exits 1
if any reload fails. Previously hard-errored as "not yet wired".
- `--all` is mutually exclusive with a positional id / `--bundle`
(exit 2 with a clear message).
- Missing both `extension_id` and `--all` exits 2 pointing at both.
Updated `test_reload_requires_explicit_bundle` (which enforced the old
"--bundle is mandatory" behavior) to cover the new resolution paths.
* docs(bundle-api): changelog entries for editable-install discovery + reload CLI
Satisfies the BUNDLE_API surface-change gate for the editable-install
discovery fallback (entry-point lookup in discovery.py) and the relaxed
`lfx extension reload` CLI (`--bundle` optional, `--all` implemented).
No additional behavior change in this commit -- it only documents the
two changes already shipped in this branch.
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Debojit Kaushik <Kaushik.debojit@gmail.com>
|
|||
| cb10f1d10f |
docs: file system component (#13183)
* docs-file-system-component * delete-internal-feature-doc * docs-filesystem-component * fix-links * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * clarify-metadata * fix-merge --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| 09bd68d349 |
docs: text operations component (#13152)
docs-add-text-operations-component |
|||
| feec57e16f |
docs: merge knowledge base components (#13150)
* docs-merge-kb-components * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * use-tabs-for-params-table --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| aefd8acb37 |
docs: directory component is legacy (#13186)
* remove-and-redirect-component-page * update-tutorials-that-used-directory-component * peer-review * update-tutorial-screenshot |
|||
| c78e10e85a |
docs: add directory depth limitation warning to custom components (#13104)
* docs: add directory depth limitation warning to custom components Add a warning admonition to the custom components documentation explaining the MAX_DEPTH=2 limitation for component discovery. The warning clarifies: - Components must be at most 2 levels deep (category/component.py) - Each first-level directory becomes an independent category in the UI - How to modify MAX_DEPTH for advanced use cases This addresses confusion around subdirectory support mentioned in issue #13102. * docs: simplify directory depth warning per review feedback Co-Authored-By: Antonio <antonio@oriontech.me> --------- Co-authored-by: Antonio <antonio@oriontech.me> |
|||
| cc009c9133 |
feat(lfx): Bundle Separation and LFX Extension Framework (#13043)
* feat(lfx): installed-package + seed-directory discovery for production install (LE-1022) Adds the read-only production install path for Modes A, B, and C of the Bundle Separation iteration. Manifest-shipping pip-installed distributions and seed-directory subdirectories are discovered at server startup and registered as Extensions at @official. * discovery.py: walks importlib.metadata.distributions() + the $LANGFLOW_SEED_DIR / /opt/langflow/bundles seed root; produces DiscoveredExtension records and typed errors for malformed manifests / configured-but-missing seed dirs. * registry.py: ExtensionRegistry service with the immutability invariant for installed and seed entries. Mutation verbs (uninstall, disable, enable, install, update_entry) all raise ExtensionImmutableError carrying the typed installed-extension-immutable / seed-directory-immutable code so the invariant is testable today; the CLI uninstall surface ships in B4. * lfx extension list: read-only inspector with text and JSON output for operators inspecting Mode B/C images. * Errors: four new typed codes (installed-extension-immutable, seed-directory-immutable, seed-directory-not-found, duplicate-extension-id) plus snapshot coverage in tests/unit/extension/test_errors.py. * Tests: 165 extension tests pass, including the LE-1022 acceptance cases -- three pip-installed wheels visible at @official, three seed bundles visible at @official, and the parametrized service-layer immutability check across every mutation verb. * Docs: docs/Deployment/deployment-extensions-production.mdx covers the Dockerfile template, k8s deployment notes, the bundle packaging convention (extension.json shipped via package-data), and troubleshooting for the typed error codes. * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (#12967) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * refactor(lfx): clarify broad except, log malformed bundle.json, expand test docstring - Document why _discovery.import_bundle_module catches BaseException (startup-time loader, must surface bad bundles as typed errors rather than abort). - Add debug-level logging when bundle.json is malformed or non-object so a stale-cache footgun is at least observable. - Expand test_skips_re_imported_class docstring so future maintainers don't accidentally weaken the __module__-equality guard if package-style relative imports get added later. Also drops the stale duplicate-distribution claim from the PR description; that code is correctly deferred to LE-1022 along with /all integration. * feat(lfx): wire Extension System into /all, pathsep-split LANGFLOW_COMPONENTS_PATH, emit duplicate-distribution Closes the four AC gaps the previous reviewer flagged on PR #12967: 1. /all integration: get_and_cache_all_types_dict now also calls a new import_extension_components() that loads installed Extensions via load_installed_extensions, loads inline bundles via discover_inline_bundles, and builds frontend-node templates with extension/bundle/extension_version fields stamped on. Failures are logged and skipped per bundle. 2. LANGFLOW_COMPONENTS_PATH is now split on os.pathsep so multi-entry env vars (e.g. /a:/b on POSIX) produce multiple components-path entries instead of one literal non-existent path. Empty segments and missing paths are skipped. 3. duplicate-distribution is a real producer: load_installed_extensions surfaces a typed warning on the winner LoadResult when two distributions share a canonical name, naming every involved manifest path. 4. Manifest-first precedence runtime wiring: new filter_component_entry_points loads each entry-point and only skips ones that resolve to a Component subclass on a manifest-shipping distribution. plugin_routes.load_plugin_routes now applies it so non-component entry-points (route registrars) keep loading per the AC's 'unaffected' promise. Added the previously-missing AC test for same-distribution component+non-component partition. Also makes _distribution_canonical_name defensive against MagicMock test seams. * fix(lfx): register extension components under namespaced ID; promote duplicate-distribution to error Addresses the latest review of PR #12967: P1: /all integration now keys the cache inner dict by LoadedComponent.namespaced_id (ext:<bundle>:<Class>@<slot>) rather than the bare class name. Templates also carry the namespaced_id as an explicit field so consumers that look at the value (not the key) still see the canonical address. This is the form the LE-1020 migration table will rewrite legacy class-name references to. P2: load_installed_extensions now appends duplicate-distribution to result.errors instead of result.warnings, so LoadResult.ok=False when two distributions share a canonical name. The winner's components still appear in result.components so flows already pinned to them keep working; only the conflict status changes. Updated test to assert errors + ok=False; added explicit assertion that the winner's components are still present. * fix(lfx): installed-distribution discovery accepts pyproject.toml manifest form Closes the latest review finding on PR #12967: the installed-distribution scan only looked for extension.json, ignoring distributions whose manifest lives in [tool.langflow.extension] inside pyproject.toml. The AC explicitly treats both as valid manifest forms. _distribution_manifest_path now: - Returns extension.json immediately when present (preserves precedence matching load_manifest's discovery order). - Falls back to pyproject.toml only when extension.json is absent AND the pyproject's [tool.langflow.extension] section is parseable. Validation reuses load_manifest itself so the rule lives in exactly one place; a stray pyproject.toml without the section is correctly ignored. Tests cover: pyproject-only discovery, pyproject-without-section ignored, extension.json wins on collision, end-to-end pyproject load at @official, and pyproject-form manifest-first entry-point suppression. * refactor(lfx): tighten loader invariants, surface silent skips, harden tests Addresses the latest review feedback on PR #12967: Type-level invariants (_types.py): - LoadedComponent.__post_init__ enforces that @extra components must NOT carry a distribution. The reverse (@official without distribution) is permitted because load_extension is also used for dev-mode loads against a working tree before pip install. - LoadResult docstring documents the partial-success contract: components may be non-empty when errors is non-empty (some files imported, others failed). Callers branching on ok get strict success. Silent-failure fixes (_orchestrator.py + settings/base.py): - inline-path-missing: a non-existent / non-dir LANGFLOW_COMPONENTS_PATH entry now produces a typed warning per skipped path so a typo no longer yields zero diagnostics. Settings-layer skip bumped from debug to warning for the same reason. - bundle-json-invalid: a malformed or non-object bundle.json now surfaces a typed warning instead of silently rewriting the user-declared id/version to derived values under the same bundle name. - no-component-subclass gating uses a call-local counter instead of result.errors so the diagnostic stays accurate when a future caller reuses a LoadResult (multi-bundle / batch wrapper scenarios). Test hardening: - test_re_imported_class_is_skipped_via_module_filter rewritten to actually exercise the __module__-equality check via sys.modules injection; previously passed via module-import-failed (relative-import failure), which would silently weaken if package registration changes. - test_user_declared_path_order_is_preserved: AC #8's multi-path order case (distinct bundles in [path_b, path_a]) was unasserted; added. - test_inline_module_import_failure_attributes_identity: AC #10's identity-on-partial-failure was covered for @official but not @extra; added. - test_uses_real_distributions_by_default tightened to assert ep placement (in kept, not in skipped) instead of exact-list equality, so a future Langflow-shipped manifest doesn't silently flip the assertion. bumped 64 -> 175 passing extension tests; 20 backend integration tests still pass. * fix(lfx): malformed pyproject manifests surface manifest-invalid instead of disappearing Closes the latest review finding on PR #12967: a pyproject.toml with a [tool.langflow.extension] section that has missing/invalid required fields was silently dropped because _pyproject_has_extension_section ran full schema validation via load_manifest and returned False on ValueError/TypeError. That conflated 'no section' with 'section malformed'. Fix: detect section presence only. _pyproject_has_extension_section now calls _read_pyproject_extension (TOML parse + key lookup, no schema check). Behavior: - Section absent or pyproject TOML unparseable -> False (treat as regular non-manifest package). - Section present and is a table (valid OR schema-invalid) -> True. - Section present but is not a table -> True; the author intended to declare an extension and load_extension will surface the typed error. This way a typo'd pyproject Extension produces a typed manifest-invalid LoadResult with extension_id attribution, and manifest-first precedence still suppresses its legacy component entry-points -- matching the 'typed load results on success/failure' contract for the supported pyproject manifest form. Tests: two new cases pin the behavior. test_malformed_pyproject_section_ surfaces_manifest_invalid asserts a typed load-failure result with distribution attribution; the second test pins manifest-first suppression for malformed pyproject distributions. * refactor(lfx): emit inline-path-unreadable, document reload contract, trim rot Closes the remaining nits on PR #12967: - inline-path-unreadable (new typed error code): a configured LANGFLOW_COMPONENTS_PATH entry that raises OSError on iterdir (typically permission-denied) now produces a typed LoadResult error carrying str(exc) instead of silently swallowing the message. - duplicate-inline-bundle hint trimmed: removed forward promise about hard-error-in-a-later-release; same actionability, no expiration. - discover_inline_bundles docstring trimmed from three paragraphs to two sentences (per CLAUDE.md anti-multi-paragraph rule). - _distribution_manifest_path docstring trimmed to one-liner. - installed_extension_roots dropped Used-by caller-narration list; manifest_owning_distributions docstring rewritten to call out the shadow-load risk for direct callers and point to load_installed_ extensions for the typed warning surface. - _discovery.import_bundle_module: replaced 'until that lands in a later milestone' rot with a single line ('Absolute imports only between bundle modules; relative imports unsupported.') AND added the single-load-per-process contract note documenting why LE-1018 reload must scrub registry/sys.modules before re-invoking the loader. - load_extension docstring grew a 'Single-load-per-process contract' block telling direct callers not to rely on this function for refresh. Tests: new test_unreadable_path_emits_inline_path_unreadable pins the OSError -> typed-error path. --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) (#12979) * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) Five-stage reload (parallel staging load -> validate -> swap under write lock -> cleanup -> emit) for installed Bundles in Mode A. In-flight flows keep the pre-swap class via existing references; new flows pick up the post-swap class atomically; concurrent reloads on the same Bundle are rejected with reload-in-progress. Adds: * lfx/extension/registry.py -- BundleRegistry with per-bundle reload-in-progress guard and components_index.json writer * lfx/extension/reload.py -- the five-stage pipeline; events emission is stubbed (TODO LE-1017) so the swap mechanics can ship before the events service lands * loader: optional module_namespace param so Stage 1 lands in __reload_staging__.<id> instead of the live _lfx_ext.* namespace * errors: four new typed reload codes (reload-in-progress, reload-bundle-not-installed, reload-bundle-name-mismatch, reload-source-missing) with branch templates and snapshot tests * HTTP: POST /api/v1/extensions/{id}/bundles/{name}/reload, gated by the existing get_current_active_user dependency, returns 409 with a typed body for the in-progress collision case * CLI: lfx extension reload <id> [--bundle <name>] -- HTTP client against the dev server with text/json output and proper exit codes (--all is gated until LE-1019 lands the list endpoint) * tests: 16 reload-pipeline tests covering the AC matrix (rename round-trip, broken-bundle isolation, concurrent readers, in-flight flow, double-reload guard, bundle-name mismatch) plus 9 CLI client tests Mode A only. In Mode B/C bundle changes require a Docker image rebuild and the reload path is not exercised. The events emission in Stage 5 is intentionally stubbed -- the LE-1017 ticket will swap the body of _emit_bundle_reload_event in one place without touching the pipeline core. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * feat(lfx): add `extension init` and `extension dev` CLIs (LE-1016) (#12968) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): add `extension init` and `extension dev` CLIs (LE-1016) The two scaffolding CLIs an Extension author types: - `lfx extension init <target>` writes the basic single-Bundle template (manifest with $schema, README, .gitignore, one Component subclass + a pytest smoke test). AC #1: the generated extension validates clean against LE-1014. AC #2: the generated test file is a valid pytest module that exercises the component's build() method. AC #3: any --template other than 'basic' fails with a typed template-deferred-in-this-milestone error and a non-zero exit. Refuses to scaffold over a non-empty target dir. - `lfx extension dev <target>` validates the local extension, records its absolute path in <config_dir>/extensions/dev_extensions.json, prints reload instructions, and execs `langflow run` (or `python -m langflow` when langflow isn't on PATH). --skip-launch registers without launching (used by tests + external dev-server scripts); --skip-validate lets authors register a known-broken manifest to debug it under the loader. Stack: - Wave 0: error codes (extension-target-exists, extension-target-invalid, local-extension-missing) with format branches and snapshot tests. - Wave 1: lfx.extension.init_template -- pure-data scaffolder, no Typer dependency so the CLI is a thin shell over it. - Wave 1: lfx.extension.dev_registry -- atomic JSON state file under the langflow user-cache dir; helpers for register / list / unregister / load_dev_extensions / dev_extension_component_paths. - Wave 2: lfx.cli._extension_commands gains init/dev subcommands. - Wave 2: langflow.main lifespan hook reads the dev registry after bundle loading and extends components_path with each registered bundle dir, so the existing palette discovery picks up dev extensions. Missing paths surface as local-extension-missing warnings (AC #5) without aborting startup. Tests (84 new, 197 total in tests/unit/extension/): - test_init_template.py: AC scenarios + identifier derivation + deterministic file shape. - test_dev_registry.py: register/list/unregister round-trip, idempotent re-register refreshes timestamp, malformed state file treated as empty, missing-path warning, recovery when path reappears, env-var override precedence. - test_cli.py: AC #1 init->validate, AC #3 deferred templates, --skip-launch registers without launching, --skip-validate short-circuits the pre-flight pass. - test_errors.py: snapshot rows for all three new codes; the every-known-code-has-a-snapshot test enforces parity. LE-1018 (reload) reuses load_dev_extensions; LE-1022 (installed-pkg discovery) shares the components_path extension pattern. AC #4 ("boots Langflow with the extension visible in the palette within 5s") is delivered jointly by `extension dev` (registers + execs) and the lifespan hook (loads on startup). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1016 review feedback Fixes the four HIGH issues + the elevated LOW from the review pass: 1. dev_extension_component_paths now forwards EVERY warning, not just local-extension-missing. Previously a duplicate-component-name (or any future warning code) was silently dropped, hiding real signal from the lifespan hook's logs. 2. Defensive emit when a LoadResult has components but source_path is None. The current loader always sets source_path, but a future hand-built LoadResult could violate that contract; we now surface a typed local-extension-missing error rather than dropping the extension silently. 3. Replaced the fragile ``min(len(parts))`` bundle-root selection with a relative-to-source-path measurement. Handles deep-vs-shallow sibling extensions correctly without depending on absolute path depth. 4. Generated README now documents the langflow/lfx prerequisite under the Develop section so authors know `pytest` requires the lfx environment, not just Python. 5. Forced LANGFLOW_LAZY_LOAD_COMPONENTS=false unconditionally in the `extension dev` exec env (was setdefault, which let a developer's global lazy-loading export silently hide their dev components from the palette and miss AC #4's 5s budget). Plus three MEDIUMs: - sys.modules cleanup in test_generated_test_file_runs_against_generated_component so a later test importing the same dotted path doesn't pick up a stale module from a deleted tmp_path. Component instantiation moved inside the try block because Component.__init__ uses inspect.getsourcefile against self.__class__'s still-live module. - Added regex-drift test that pins the init_template patterns to match manifest.py's so a schema regex change can't quietly produce invalid scaffolded manifests. - Added two new dev_registry tests covering the forward-all-warnings contract and the source_path=None defense. Tests: 201 passing (4 new); ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * fix(lfx): align init template with renamed manifest API After merging feat/extension-loader into this branch, two surfaces fell out of sync with the renamed manifest schema: - init_template generates extension.json with `lfx: {bundle_api: [1]}`, but the validator now requires `lfx: {compat: ["1"]}` per LfxCompat. This made `test_basic_template_validates_clean` fail with manifest-invalid (`lfx.compat: Field required; lfx.bundle_api: Extra inputs are not permitted`). - test_init_template_regexes_match_manifest_schema reads `manifest_mod._BUNDLE_NAME_RE`, but that symbol was promoted to public `BUNDLE_NAME_RE` in |
|||
| 4dc9e28eb0 |
fix(initial_setup): upsert by (user_id, name) when loading flows from disk (#13132)
* fix(initial_setup): upsert by (user_id, name) when loading flows from disk Langflow fails to start with IntegrityError on the unique_flow_name constraint when LANGFLOW_LOAD_FLOWS_PATH is used in CI/CD pipelines and the flow file's id differs from the DB record's id (e.g. nightly upgrade, regenerated UUIDs, fresh DB). Manual DB intervention was the only workaround. Root cause: find_existing_flow only looked up by endpoint_name and id, never by (user_id, name). The unique constraint is on (user_id, name), so when those lookups missed, the loader fell through to INSERT and PostgreSQL rejected it. Fix: - find_existing_flow now takes optional keyword args (user_id, name) and falls back to a (user_id, name) lookup when endpoint_name/id don't match. The endpoint_name branch is also scoped by user_id when supplied so it can't accidentally return another user's flow. - upsert_flow_from_file passes name and user_id to find_existing_flow. When the match is by name (existing.id != file's flow_id), the DB id is preserved during the setattr loop -- rewriting it from under FK referrers was unsafe and was never required by the original logic. Tests: 8 regression tests cover lookup precedence (id wins, endpoint scoped by user, name fallback fires only when id misses), the reporter's scenario (no IntegrityError on repeated import with differing file ids), DB-id preservation on name-match, and a no-regression case where matched-by-id behavior is unchanged. Fixes the CI/CD upgrade failure reported in IBM Cloud Slack (C06SUKEN3LJ thread 1777992833.170099). * fix(initial_setup): fix matched_by_id on SQLite, improve update log message Normalize existing.id to UUID before the matched_by_id comparison. On SQLite, SQLAlchemy can return ids as strings; str == UUID is always False, so matched_by_id would silently be wrong and the file id would be dropped even on a genuine id-match. Improve the update log to include the DB id and flow name alongside the file's UUID so the CI/CD upgrade scenario (name-matched upsert with differing UUIDs) is debuggable from logs. * [autofix.ci] apply automated fixes * fix: Preserve db id * [autofix.ci] apply automated fixes * feat: Add flag to control name matching overwrite * [autofix.ci] apply automated fixes * Flip the default value for the flag * Update test_upsert_flow_from_file.py * fix: Sqlalchemy-based crash on startup --------- Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 5e9f0c37bc |
Merge branch 'release-1.9.3' into release-1.10.0
# Conflicts: # .secrets.baseline # docs/package.json # pyproject.toml # src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json # src/backend/base/pyproject.toml # src/backend/tests/unit/components/files_and_knowledge/test_retrieval.py # src/frontend/package-lock.json # src/frontend/package.json # src/frontend/src/components/core/parameterRenderComponent/components/modelInputComponent/__tests__/ModelInputComponent.test.tsx # src/frontend/src/components/core/parameterRenderComponent/components/modelInputComponent/index.tsx # src/frontend/src/modals/modelProviderModal/hooks/useProviderConfiguration.ts # src/lfx/pyproject.toml # src/lfx/src/lfx/_assets/component_index.json # src/sdk/pyproject.toml # uv.lock |
|||
| 5f1463fb12 |
docs: mcp tool call configurable timeouts (#13051)
* docs-configure-mcp-tool-timeouts * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * docs-peer-review --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| ea29bcc6b9 |
feat: Redis-backed job queue for multi-worker deployments (#12588)
* feat: implement Redis job queue and fakeredis support
- Refactored the job queue service to support Redis-backed management for cross-worker scaling.
- Added environment variables for configuration:
- `LANGFLOW_JOB_QUEUE_TYPE=redis`
- `LANGFLOW_REDIS_QUEUE_DB=1`
- Updated job ownership methods to be asynchronous for improved concurrency handling.
- Enhanced Redis cache service with namespacing via key prefixes.
- Introduced `fakeredis` for in-memory Redis simulation in testin>
- Added comprehensive unit tests for Redis job queue components.
* fix: run experimental warning for RedisCache usage only once
- Introduced a mechanism to emit a one-time warning for the RedisCache experimental feature during server runtime.
- The warning is logged only if no other worker has already emitted it, ensuring clarity for users regarding the experimental status of RedisCache.
- The implementation includes a temporary file check to prevent multiple warnings across different processes.
* docs: document environment variables for worker management
- Added documentation for LANGFLOW_GUNICORN_PRELOAD to explain preloading for better performance.
- Detailed the use of LANGFLOW_JOB_QUEUE_TYPE for specifying backends (e.g., Redis).
- Included LANGFLOW_REDIS_QUEUE_DB to define the database index for job queues.
- Updated the "High-Load Environments" guide with these optimal configurations.
* docs: updated 'High-load and multi-worker environments' section
* feat: enhance RedisJobQueueService with consumer wrapper management
- Introduced a caching mechanism for Redis stream consumers to optimize job data retrieval.
- Added methods to manage consumer wrappers, ensuring they are reused across sequential polls.
- Implemented cleanup logic to cancel and clear consumer wrappers during job cleanup and service stop.
- Expanded unit tests to verify consumer wrapper reuse and cleanup behavior.
* fix: ensure Redis keys are deleted during job cleanup even on cancellation
- Updated the cleanup_job method in RedisJobQueueService to guarantee Redis keys are removed even if the job cleanup is interrupted by a CancelledError.
- Added a new unit test to verify that Redis keys are deleted correctly when cleanup is called during task cancellation.
* fix: manage connection check task in RedisJobQueueService
- Added handling for the connection check task in the stop method to ensure it is properly cancelled and awaited if still running.
- This change improves resource management and prevents potential issues during service shutdown.
* fix: handle unpublished sentinel requeue on cancellation in RedisJobQueueService
- Updated the job processing logic to ensure that if a job is cancelled during the xadd operation, the unpublished sentinel is requeued instead of being dropped.
- Introduced a new unit test to verify this behavior, ensuring robustness in job handling during cancellations.
* fix: improve Redis job queue service with enhanced configuration and cleanup
- Added atexit cleanup to remove stale temporary files for RedisCache.
- Refactored Redis job queue service to use shared constants for stream prefixes, improving maintainability.
- Updated type hints for better clarity and consistency in RedisQueueWrapper and RedisJobQueueService.
- Enhanced error handling with configurable backoff for transient read failures.
* fix: enhance Redis job queue service with maxlen configuration for xadd
- Updated the xadd method in RedisJobQueueService to include maxlen and approximate parameters, improving stream management and preventing excessive memory usage.
* fix: enhance job ownership retrieval in RedisJobQueueService
- Updated the get_job_owner method to refresh the Redis key TTL on successful lookups, ensuring long-running jobs maintain their ownership anchor.
- Improved code clarity by extracting the owner key into a variable and adding detailed docstring explanations for better understanding of the TTL management.
* fix: improve job ownership handling in cleanup_job method
- Enhanced the cleanup_job method in RedisJobQueueService to accurately capture job ownership before deleting Redis keys, preventing potential data corruption in multi-worker scenarios.
- Added comments for clarity on ownership logic and its implications during job cleanup.
* fix: optimize TTL management in RedisJobQueueService
- Introduced periodic TTL refresh logic in the _bridge_to_redis method to enhance Redis stream management, reducing round-trips and improving throughput.
- Added constants for TTL refresh events and seconds to maintain clarity and configurability.
- Updated event handling to ensure TTL is refreshed appropriately based on event count and time elapsed.
* fix: improve event handling in flow response management
- Removed unnecessary error handling for missing event tasks in get_flow_events_response, allowing for smoother operation when no task exists.
- Updated create_flow_response to handle optional event_task parameter, ensuring proper cleanup during disconnections.
- Added unit tests to verify behavior when event tasks are missing, enhancing robustness in streaming scenarios.
* fix: enhance RedisQueueWrapper with startup grace period and stream observation
- Added a startup grace period to prevent premature end-of-stream signals when the producer has not yet issued its first XADD.
- Introduced a flag to track whether the stream has been observed, improving the handling of early polling scenarios.
- Updated logic to ensure proper handling of stream existence checks and logging for better debugging during job processing.
* fix: implement cleanup for old cross-worker job queues in RedisJobQueueService
- Added a new method to clean up done cross-worker consumer wrappers that are not owned by the current worker, ensuring proper resource management.
- Enhanced the existing cleanup logic to prevent memory leaks by explicitly pruning stale entries from the consumer wrappers dictionary.
- Improved logging to provide better visibility into the cleanup process for cross-worker jobs.
* fix: enhance job handling in JobQueueService with guarded task execution
- Updated the start_job method to accept a Coroutine type for task_coro, ensuring type safety.
- Introduced a new _guarded_task method to wrap job coroutines, guaranteeing that unhandled exceptions emit an error event and write a sentinel to the Redis Stream, improving reliability in job processing.
- Enhanced documentation to clarify the behavior of the new task handling mechanism and its implications for cross-worker consumers.
* fix: improve client disconnection handling in create_flow_response
- Added logging for scenarios where a client disconnects without an associated event_task, clarifying that the producer will continue running until the build completes.
- Documented the limitation regarding cross-worker passive disconnects and the need for a Redis side-channel for proper cancellation, enhancing observability in the logs.
* fix: enhance RedisQueueWrapper to manage initial read state and buffer behavior
- Introduced a flag to track the completion of the first XREAD call, ensuring proper buffer management during the initial read phase.
- Updated the empty method to reflect the state of the buffer accurately, preventing premature exits from the drain loop until the first read is complete.
- Improved documentation to clarify the behavior of the new flag and its impact on job processing.
* fix: clarify RedisQueueWrapper behavior in tests for buffer state and no-op operations
- Updated test documentation to explain the behavior of the empty() method in relation to the first XREAD completion, ensuring accurate understanding of buffer state during job processing.
- Adjusted assertions in tests to reflect the intended behavior of the RedisQueueWrapper, specifically regarding the no-op nature of put_nowait and its impact on the internal buffer.
* fix: update dependencies in pyproject.toml and uv.lock
- Added fakeredis dependency with a minimum version of 2.0.0 to both pyproject.toml and uv.lock files.
- Ensured proper formatting and comments in pyproject.toml for clarity on onnxruntime version constraints.
* fix: enhance RedisQueueWrapper and job queue handling for improved cancellation and buffer management
- Introduced a structural protocol `_CancellableQueue` to ensure queues can handle cancellation properly during client disconnects.
- Updated `RedisQueueWrapper` to implement this protocol, allowing for graceful cancellation of background tasks.
- Added a maximum size limit to the internal buffer to prevent unbounded memory usage and ensure backpressure on slow consumers.
- Implemented a done callback to handle unexpected fill task crashes, ensuring consumers are not left hanging indefinitely.
- Enhanced unit tests to verify compliance with the new protocol and the behavior of the buffer under various conditions.
* fix: RedisQueueWrapper sentinel delivery, error time-bound, and cancel response
- Deliver end-of-stream sentinel on fill-task cancellation (_on_fill_done now
handles both cancelled and exception paths so consumers are never left hanging)
- Add _error_start time-bound to xread and exists() error loops: after
_STARTUP_GRACE_S seconds of continuous Redis errors the sentinel is delivered
instead of retrying forever
- Advance _last_id cursor only after buffer.put() succeeds so cancellation mid-put
does not silently skip that message in the Redis cursor
- Return False from cancel_flow_build when event_task is None (cross-worker path)
so the HTTP response correctly reports success=False instead of false success
* [autofix.ci] apply automated fixes
---------
Co-authored-by: ogabrielluiz <gabriel@langflow.org>
Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
|
|||
| bbe2ccdc91 |
docs: fix sidebar and TOC styles for Redocusaurus (#12748)
* docs: improve sidebar and TOC readability + scroll progress - Fix left nav sidebar hugging the left edge (remove margin-left) - Increase sidebar font sizes and lift muted colors for legibility - TOC right-side: white inactive items, pink active, gray for passed sections - Add clientModule (tocProgress.js) to track scroll progress in TOC - Add Redocusaurus API page styles to match site theme Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: fix sidebar/TOC height stability and active item reflow - Force sidebarViewport to 100vh to prevent shrinking during scroll - Replace font-weight on active TOC item with text-shadow to avoid reflow * docs: left-anchor sidebar and increase fonts on large screens * docs: constrain prev/next nav to match content max-width * docs: fix sidebar jump on scroll by moving padding-top inside menu * docs: fix sidebar-ad spacing with transparent padding on li * docs: fix sidebarViewport height and MutationObserver leak * docs: constrain doc content to 75vw and remove dead CSS comments - Cap docMainContainer at 75vw - Set pagination-nav to 100% width to match content column - Remove commented-out .markdown and .ch-scrollycoding blocks * docs: set docMainContainer max-width to 75% for better reading width * docs: center main-wrapper on large screens and center docItemWrapper * fix: revert lodash override to stable 4.17.21 lodash 4.18.0 is a broken release — template.js uses assignWith and arrayEach without importing them, crashing npm start. Pin to 4.17.21 across all nested overrides. * fix: prevent horizontal scroll on mobile docs pages docsWrapper has flex:1 0 auto (flex-shrink:0), so it never shrinks below the natural content width (~570px on iPhone). Cap it to 100vw and add min-width:0 down the flex chain (docRoot, docMainContainer) so the layout stays within viewport on narrow screens. Also remove the overflow:visible override from ch-code-wrapper, which was defeating the code block's own horizontal containment. * feat: responsive navbar search and hide social icons on mobile Shrink search bar as viewport narrows (160px at 1100px, 100px at 996px), collapse to icon-only at 960px. Hide GitHub, X, and Discord icons below 996px where the hamburger menu takes over navigation. * fix: images always respect container width above 996px Use min(100%, 600px) so images are capped at 600px but never overflow their container when the content area is narrower (e.g. sidebar open). * fix: improve TOC progress scroll tracking and bottom-of-page detection Adds scroll handler with RAF throttle, pauses/resumes MutationObserver to avoid loops, and forces last TOC link active when scrolled to page bottom. * fix: translate Portuguese comments to English in custom.css * fix: sidebar viewport fills full screen height * feat: sidebar section icon turns pink with ease transition when active * fix: reduce TOC font size to 0.8rem * fix: remove docMainContainer right padding and add 48px gap to content col * fix: card header responsive alignment and title margin reset * fix: darken card background and active sidebar item bg on light theme * fix: revert TOC colors and add inline code border radius tweak * fix: minimal scrollbar for sidebar and TOC — thumb only on hover * fix: scrollbar fade-in/out transition at 0.25s ease * feat: add rehype plugin for table column wrapping and centering Inserts zero-width space after underscores in inline code cells so long env-var names wrap at underscore boundaries. Also centers Format/Default columns via a col-center class applied at build time. * fix: use --ifm-heading-color for sidebar group labels in both themes * refactor: split docs custom.css into themed partials Breaks the 1,658-line monolithic custom.css into 8 focused files (tokens, layout, navbar, sidebar, typography, components, code, redocusaurus). custom.css is now a thin entry point with @imports. * fix: align sidebar and navbar with shared --docs-gutter token Introduces --docs-gutter: 1rem in tokens.css and applies it to both navbar__inner horizontal padding and sidebar nav.menu padding-left, keeping the Langflow logo and sidebar items visually aligned. * fix: match sidebar ad background to page background color Dark mode uses #111 (same as sidebar bg); light mode uses --ifm-background-color. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
|||
| 994a2e8b94 |
feat: Isolate FileSystem tool sandbox per authenticated user (#13031)
* add user id scope to file sys actions * add security sandbox * fs user isolation doc * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * fix be tests * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * ruff style and checker * ruff style and checker * ruff style and checker * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * [autofix.ci] apply automated fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 330b8acdd8 |
docs: uninstall langflow (#13017)
* docs-uninstall-instructions * remove-cache-folder-from-deletion |
|||
| eb5f83f950 |
fix: pin postgres image to bookworm in docker_example (#13027)
* fix: pin postgres image to bookworm in docker_example to prevent collation mismatch
The postgres:16 tag silently moved its base from Debian Bookworm
(glibc 2.36) to Trixie (glibc 2.41), causing a recurring collation
version mismatch warning on existing langflow-postgres volumes.
Pin to postgres:16-bookworm in both docker-compose files and update
the README so existing data volumes keep matching the OS locale data.
Refs: https://github.com/langflow-ai/langflow/issues/9608
* docs: pin postgres image to bookworm in current docs compose snippets
Mirror the docker_example pin in the four current docs that publish
copyable Compose snippets pairing postgres:16 with a persistent
langflow-postgres volume. Prevents the same Bookworm-to-Trixie
collation version mismatch warning when users follow docs instead
of docker_example.
Versioned historical docs are left as-is.
Refs: https://github.com/langflow-ai/langflow/issues/9608
* fix: switch postgres pin from bookworm to trixie for OS consistency
Aligns the pinned postgres base with the langflow runtime image, which
moved to Debian Trixie in #12990. Keeping postgres on bookworm would
have diverged the stack and locked the database to an aging glibc that
will receive fewer security backports as bookworm ages into oldstable.
The pin itself still solves the original bug — postgres:16 cannot
silently roll its OS underneath an existing volume.
Document the one-time REFRESH COLLATION VERSION step for users
upgrading from a bookworm-initialized volume in docker_example/README.md.
Refs: https://github.com/langflow-ai/langflow/issues/9608
(cherry picked from commit
|
|||
| 00bc383f86 |
docs: SSRF protection enabled by default (#13106)
docs-ssrf-protection-enabled-by-default |
|||
| b54bdf318d |
fix: update security dependencies (#13053)
* chore: update security dependencies - Update brace-expansion to ^5.0.5 in docs - Update picomatch to ^4.0.4 in docs - Update ip-address to ^10.1.1 in frontend - Update GitPython to >=3.1.48 in backend - Add protobuf constraint >=6.33.6,<7.0.0 in backend * chore: address additional CVEs and add security documentation - Update lodash from deprecated 4.18.0 to 4.17.21 - Add docs/SECURITY_OVERRIDES.md documenting all CVEs - Revert mem0ai 2.x upgrade due to compatibility issues * fix: update dependencies to address CVE vulnerabilities - Update langchain-core to >=1.3.3 (fixes CVE-2026-44843) - Update GitPython to >=3.1.50 (partially fixes CVE-2026-44243, CVE-2026-44244, GHSA-mv93-w799-cj2w) - Update pyarrow constraint to >=23.0.1,<24.0.0 (fixes CVE-2026-25087) - Add override-dependencies for transitive packages: - lxml >=6.1.0 (fixes CVE-2026-41066) - mako >=1.3.12 (fixes CVE-2026-44307) - urllib3 >=2.7.0 (fixes CVE-2026-44431, CVE-2026-44432) - python-liquid >=2.2.0 (fixes CVE-2026-45017) Total: 9 CVEs addressed Smoke tests: All imports successful, no breaking changes detected --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> |
|||
| 7504eb4c72 |
fix: pin postgres image to bookworm in docker_example (#13027)
* fix: pin postgres image to bookworm in docker_example to prevent collation mismatch The postgres:16 tag silently moved its base from Debian Bookworm (glibc 2.36) to Trixie (glibc 2.41), causing a recurring collation version mismatch warning on existing langflow-postgres volumes. Pin to postgres:16-bookworm in both docker-compose files and update the README so existing data volumes keep matching the OS locale data. Refs: https://github.com/langflow-ai/langflow/issues/9608 * docs: pin postgres image to bookworm in current docs compose snippets Mirror the docker_example pin in the four current docs that publish copyable Compose snippets pairing postgres:16 with a persistent langflow-postgres volume. Prevents the same Bookworm-to-Trixie collation version mismatch warning when users follow docs instead of docker_example. Versioned historical docs are left as-is. Refs: https://github.com/langflow-ai/langflow/issues/9608 * fix: switch postgres pin from bookworm to trixie for OS consistency Aligns the pinned postgres base with the langflow runtime image, which moved to Debian Trixie in #12990. Keeping postgres on bookworm would have diverged the stack and locked the database to an aging glibc that will receive fewer security backports as bookworm ages into oldstable. The pin itself still solves the original bug — postgres:16 cannot silently roll its OS underneath an existing volume. Document the one-time REFRESH COLLATION VERSION step for users upgrading from a bookworm-initialized volume in docker_example/README.md. Refs: https://github.com/langflow-ai/langflow/issues/9608 |
|||
| b707c9a41d |
docs: opensearch connector feature (#12998)
* docs-add-opensearch-provider-and-adjust-kb-docs * docs-combine-kb-config-sections-and-update-partial * add-release-note * Apply suggestions from code review Co-authored-by: Mendon Kissling <59585235+mendonk@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * docs-clarify-embedding-model-step --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| 7c347a56ea |
docs: release note refactor (#12987)
* docs-1.10-release-notes-and-link-out-to-previous-versions * docs-remove-outdated-windows-upgrade-note |
|||
| 596e98b9b7 |
fix: Respect proxy environment variables in URL Component (#12989)
* fix: URLComponent ignores proxy in async mode (#12285) The URLComponent fails to connect for users behind corporate proxies because its asynchronous mode does not recognize standard system proxy environment variables. The component defaults to use_async=True, which initializes the RecursiveUrlLoader; the underlying async loader does not natively respect system proxy environment variables, so the component attempts a direct connection and fails in restricted network environments. Detect standard proxy environment variables (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY) in URLComponent._create_loader. If a proxy is detected and use_async is enabled, override use_async to False so the loader uses its synchronous implementation, which natively respects system proxies. Empty and whitespace-only values are correctly evaluated as no-proxy and do not trigger the fallback. Closes #10297 * fix(URLComponent): broaden proxy detection and harden tests Address review findings on the proxy fix: - Add ALL_PROXY and all_proxy to the detected env vars. ALL_PROXY is commonly set in corporate and container environments (curl, git, many Unix tools honour it), and is sometimes the only proxy var configured. - Replace the unused proxy_url string with a single boolean any() over the env var keys, since only the presence of a proxy is consulted. - Reorganize the proxy tests under TestURLComponentProxyHandling with a shared default-attributes helper, parametrize over all six env var spellings, and add coverage for multiple-simultaneous-proxies and the use_async=False path (which should not log a warning). * [autofix.ci] apply automated fixes * fix: remove no_proxy="*" macOS startup hack so corporate proxies work Two startup paths set `os.environ["no_proxy"] = "*"` on macOS, which disables proxy use for every HTTP client in the process and every child gunicorn worker (httpx, requests, urllib3 — and therefore the OpenAI, Anthropic, Groq, etc. SDKs that wrap them). For users behind corporate proxies on macOS this made every external LLM call unroutable, even with HTTPS_PROXY properly set. The override traces to commit history with no concrete justification — the only artifact is a Stack Overflow link about a uWSGI segfault, but Langflow uses gunicorn, not uWSGI. Bench-verified that gunicorn boots cleanly and serves requests on macOS without it (1 worker via LangflowApplication, OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES retained, HTTPS_PROXY survives intact in parent and child). Verification: - gunicorn worker spawns and serves /health (200 OK), exits cleanly - httpx, urllib, requests all resolve HTTPS_PROXY after the macOS init (previously: all three returned empty proxy maps because of no_proxy=*) - OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES still set — the actual fork-safety fix is preserved Closes the macOS half of #10297. Companion fix for the async URLComponent half is in #12285 / branch pr-12285-rebased. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * Orjson update --------- Co-authored-by: Diogo Veiga <diogo.veiga@tecnico.ulisboa.pt> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 52ae55b20a |
docs: deprecate voice to voice websocket endpoint (#12970)
* docs-deprecate-voice-to-voice-endpoint * docs-explain-voice-mode-vs-speech-to-text |
|||
| 539316bb67 |
docs: troubleshoot desktop errors from telemetry (#11282)
* add-troubleshooting-items-from-scarf * sort-desktop-errors-by-os * include-err-strings * casing * remove-placeholders * python-version * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * peer-review * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * docs-update-some-error-messages-that-have-changed * docs-check-desktop-errors-against-error-constants-file --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| 6d2e255f5d |
Merge remote-tracking branch 'origin/release-1.9.2' into release-1.10.0
# Conflicts: # .secrets.baseline # pyproject.toml # src/backend/base/langflow/initial_setup/starter_projects/Knowledge Retrieval.json # src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json # src/backend/base/pyproject.toml # src/frontend/package-lock.json # src/frontend/package.json # src/lfx/pyproject.toml # src/lfx/src/lfx/_assets/component_index.json # src/sdk/pyproject.toml # uv.lock |
|||
| 5bf73714e3 | chore: Remove the diskcache dependency (#12953) | |||
| bc16a400d6 |
docs: add telemetry for deployments endpoints (#12922)
* docs-add-telemetry-for-deployments-endpoints * peer-review |
|||
| d965a3ae6c |
docs: wxo deployment is behind feature flag (#12937)
* docs-add-release-note-and-tip-for-wxo-feature-flag * remove-unnecessary-information-from-1.10 * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| 9dcdc704ec |
docs: change upgrade reinstall order (#12911)
* docs-change-upgrade-reinstall-order * docs-make-langflow-oss-version-section |
|||
| a46ae46f9f |
feat: Add an Astra DB Data API Component (#12766)
* feat: Add an Astra DB Data API Component * Run pre-commit hooks to format * Update astradb_data_api.py * Update component index * Merge branch 'release-1.10.0' into feat/astra-db-data-api-component * Updates * Update component_index.json * Update component_index.json |
|||
| 1e9a96f5b9 |
docs: include host field for external db in Helm chart (#12899)
docs-include-host-value-for-external-db |
|||
| b9e81f6edd | Merge remote-tracking branch 'origin/main' into release-1.10.0 | |||
| 0e6d284bc4 |
fix(security): default WEBHOOK_AUTH_ENABLE to True (#12845)
* fix(security): default WEBHOOK_AUTH_ENABLE to True (unauth webhook execution)
POST /api/v1/webhook/{flow_id} previously executed any user's flow
without authentication because WEBHOOK_AUTH_ENABLE defaulted to False.
Change the default to True so webhook endpoints require an API key and
validate that the caller owns the flow being executed. Operators who
need the prior behavior can explicitly opt in with
LANGFLOW_WEBHOOK_AUTH_ENABLE=False.
Docs updated to reflect the new secure-by-default behavior.
* test(security): add regression tests for WEBHOOK_AUTH_ENABLE default
Guard against a regression of the unauthenticated webhook execution fix:
one test pins the class-level default to True, the other confirms the
runtime config rejects an unauthenticated POST with 403 under defaults.
(cherry picked from commit
|
|||
| 849158ed51 |
docs: tip for running local ollama models (#12840)
add-tip-about-running-large-models-locally
(cherry picked from commit
|
|||
| 8524d49db4 |
docs: concatenate and merge options for dataframe operations component (#12835)
concat-and-merge-table-component
(cherry picked from commit
|
|||
| 0ab5f040f8 |
docs: clarify trace storage when multiple providers are enabled (#12816)
clarify-trace-storage-with-multiple-providers
(cherry picked from commit
|
|||
| f78830730d |
docs: wxo feature (#12677)
* deployment-sidebars-and-release-note
* fix-sidebars-name
* manage-deployments
* add-requests
* clarify-endpoint-difference
* improve-anchors
* Apply suggestions from code review
Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Mendon Kissling <59585235+mendonk@users.noreply.github.com>
* clarify-menu
* apply-changes-to-next-and-latest
---------
Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com>
(cherry picked from commit
|
|||
| 4beab56b6b |
docs: add multi-model opensearch component (#12799)
add-opensearch-multi-model
(cherry picked from commit
|
|||
| aced13ef79 |
docs: clarify autosaving and versioned flow saving (#12794)
* add-clarification-to-1.9-and-next
* Apply suggestions from code review
Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com>
---------
Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com>
(cherry picked from commit
|
|||
| 89d1c60bed |
feat: Enhance config loading by applying GUNICORN_CMD_ARGS before programmatic options (#12313)
* feat: Enhance config loading by applying GUNICORN_CMD_ARGS before programmatic options
* docs: Add Gunicorn configuration details to environment variables documentation
* Update src/backend/base/langflow/server.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/backend/base/langflow/server.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add unit test
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai>
Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com>
(cherry picked from commit
|
|||
| 919724523c |
fix: resolve ghosting issues in preload optimization (#12587)
* fix: enable preload_app option in LangflowApplication configuration * feat: Integrate Sentry and Prometheus support in worker lifespan - Initialize Sentry SDK in the worker lifespan if a DSN is provided, allowing for better error tracking. - Start Prometheus HTTP server if enabled in settings, enhancing monitoring capabilities. - Added logging for both Sentry initialization and Prometheus server startup to aid in debugging and monitoring. Refactor existing code to defer Sentry initialization to avoid issues with process forking. This change improves the overall observability of the application. * fix: Correct Prometheus port validation logic in create_app function - Updated the condition for validating the Prometheus port to use 'and' instead of 'or', ensuring that the port is both greater than 0 and less than MAX_PORT. This change enhances the reliability of the Prometheus server configuration. * test: Add unit tests for LangflowApplication.pre_fork method - Introduced a new test suite for the pre_fork method in LangflowApplication, covering various scenarios including warnings for non-main threads and non-LISTEN TCP connections. - Implemented tests to ensure proper handling of psutil import errors and unexpected exceptions. - Verified that garbage collection is always executed during the pre_fork process. - Added a fake server mock to facilitate testing without requiring a real server instance. * refactor: Move telemetry service initialization to lifespan context - Removed the initialization of the telemetry service from the beginning of the `get_lifespan` function and added it within the lifespan context. This change ensures that the telemetry service is only initialized when needed, improving resource management and application performance. * fix: Enhance Sentry integration with error handling and import checks - Added error handling for Sentry SDK initialization to log warnings if the SDK is not installed or if initialization fails. - Updated the `setup_sentry` function to conditionally import `SentryAsgiMiddleware`, logging a warning if the import fails. - This improves the robustness of the application by preventing crashes related to missing Sentry dependencies and providing clearer logging for debugging. * fix: Improve Prometheus server error handling in lifespan context - Enhanced error handling for starting the Prometheus server by adding specific checks for ImportError and OSError. - Added logging for cases where the Prometheus client is not installed or when the port is already in use, improving clarity for debugging and operational monitoring. - This change aims to provide more informative feedback during server startup, enhancing the overall robustness of the application. * refactor: Rename and enhance Sentry middleware integration - Changed the function name from `setup_sentry` to `add_sentry_middleware` for clarity. - Updated the middleware setup to ensure Sentry is attached at request time, deferring SDK initialization to the worker lifespan to avoid ghost transactions. - Adjusted unit tests to mock the new middleware function name, ensuring continued test coverage and functionality. * feat: Enhance pre_fork method to identify benign threads before forking - Introduced a new class-level constant `_BENIGN_THREAD_PREFIXES` to define known benign thread prefixes. - Added a class method `_is_benign_thread` to check if a thread is benign based on its name. - Updated the `pre_fork` method to log warnings for non-benign threads, improving thread management during the forking process. - Added a unit test to ensure no warnings are emitted for benign threads, enhancing test coverage for the `pre_fork` method. * fix: Improve logging for psutil import error handling - Added a debug log statement to indicate when the psutil library is not installed, enhancing visibility into the application's behavior during TCP connection checks. - This change aims to provide clearer feedback for debugging and operational monitoring when the psutil dependency is missing. * fix: Enhance garbage collection handling in pre_fork method - Wrapped the gc.collect() call in a try-except block to prevent crashes if an exception is raised during garbage collection. - Added logging to capture any warnings when gc.collect() fails, improving error visibility during the pre-fork process. - Introduced a new unit test to ensure that the pre_fork method handles gc.collect() exceptions gracefully while still calling gc.freeze(). * docs: LANGFLOW_GUNICORN_PRELOAD environment variable introduced * refactor: Update application setup for Windows and non-Windows environments - Introduced conditional logic to handle application setup differently based on the operating system. - Added a factory pattern for creating the FastAPI application, improving flexibility for non-Windows systems. - Enhanced error handling to provide clear runtime messages when the application cannot be initialized correctly. * refactor: move test_server.py to correct unit/base location The test file exercises langflow.server which belongs to the base package (src/backend/base/). Moving it to src/backend/tests/unit/base/ follows the project convention and aligns with the path expected by CI ruff checks. Made-with: Cursor * feat: Enhance config loading by applying GUNICORN_CMD_ARGS before programmatic options (#12313) * feat: Enhance config loading by applying GUNICORN_CMD_ARGS before programmatic options * docs: Add Gunicorn configuration details to environment variables documentation * Update src/backend/base/langflow/server.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/backend/base/langflow/server.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add unit test --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai> Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com> Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai> Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com> Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com> |
|||
| d274b7b32a |
docs: release note typo (#12690)
release-note-typo |
|||
| 71fbf5ff00 |
docs: cut version 1.9.0 (#12681)
* version-1.9.0-docs * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 19df4606ae |
chore: update deps (#12657)
* chore: update deps update deps due to sec vuln * chore: langchain-core>=1.2.28 langchain-core>=1.2.28 * chore: update pypdf pypdf 6.10 * chore: pyarrow, openai, litellm, mem0ai, toolguard picomatch, pyarrow, openai, litellm, mem0ai, toolguard * chore: override litellm * chore: match base uv.lock to root * chore: update tool.uv position update tool.uv position * chore: langflow-base[complete]>=0.9.0 langflow-base[complete]>=0.9.0 * chore: revert pyarrow revert pyarrow * chore: revert "lfx~=0.4.0", * chore: lfx white space * chore: remove allow-direct-references = true * chore: uv lock --upgrade after merge |
|||
| 83acc4143d |
docs: mcp client and other changes (#12627)
* tools-component * tools-partial-and-release-notes * mcp-astra-changes * tutorial-changes * client-page-and-release-notes * langflow-mcp-client-for-coding-agents * fix-links * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * include-explanation-for-step-4 --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> |
|||
| a425540bdb |
docs: add flow versioning (#12634)
* add-flow-versioning-and-release-note * add-flow-db-location |
|||
| 73a48b5810 |
docs: generate and bump open API spec to 1.9.0 (#12638)
generate-and-bump-openapi-spec-to-1.9.0 |
|||
| da24dfad61 |
docs: components index path env var (#12630)
* env-var * allowlist-for-custom-components * clarify-which-wins |
