mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-23 22:44:41 +08:00
v1.11.0.dev21
948 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| d4c8ffd9ea | Update version for nightly | |||
| ce71c72868 |
chore: restore bundles that are now published (#13825)
* chore: restore bundles that are now published * fix: torchvision * flip torchvision sign |
|||
| b686855a94 |
feat(lfx): add pre-warm functionality (#13682)
* feat(lfx): add `prewarm` command for snapshot/preload warm-up Warm component imports, the graph execution machinery, and specific flows before a Firecracker snapshot or Gunicorn --preload fork, so the first build/run after restore skips the cold lazy-import cost. - prewarm_core_imports(): import core component classes + run a model-free hermetic flow (fork-safe: no network, no threads) - prewarm_flow(flow, run=...): warm a specific flow by building it, or fully executing it for max warmth (drops graph + gc.collect after a run) - freeze_heap(): gc.freeze() for copy-on-write sharing across fork/snapshot - CLI: lfx prewarm with --flow, --skip-run, --freeze, and the gated --unsafe-run-may-leak-connections (Firecracker only; not fork-safe) * feat(lfx): extract shared fork-safety toolkit (lfx.fork) Add lfx.fork with the benign-thread allowlist, ghost-thread / ghost-TCP detectors, a best-effort fork-safety report, and a dispose-in-finally helper. - prewarm_flow now reports fork-hostile state left by a run (ghost_threads / ghost_connections), and the CLI warns when a run leaves connections open (safe for Firecracker restore, unsafe before a Gunicorn/preload fork) - refactor langflow's gunicorn pre_fork hook to use lfx.fork instead of its own duplicated benign-thread allowlist + detection (behavior preserved) * feat(lfx): dispose pluggable services before fork in prewarm path The fork-safe prewarm path (prewarm_core_imports / prewarm_flow) triggers lazy service discovery + instantiation, but never disposed what it instantiated. With only bundled (fork-safe no-op) services this was latent, but once a real lfx.services plugin (a DB pool, external cache socket, or telemetry thread) is registered, the Gunicorn --preload fork would inherit its live fds — the exact hazard Langflow's preload avoids via engine.dispose() / cache teardown() before its master fork. Wire that disposal into lfx: - ServiceManager.teardown(raise_on_error=False): new strict mode that attempts every teardown then re-raises the first failure. Default preserves existing best-effort behavior. - preload.teardown_warm_services(): disposes the global service manager, fatal (PrewarmError) on failure so a half-disposed process is never captured into a fork. - prewarm_core_imports / prewarm_flow gain teardown_services=True (opt-out). prewarm_flow skips teardown on run=True (Firecracker intentionally keeps live connections). Results expose services_torn_down. - The prewarm command centralizes teardown to one pass after all warming (so the per-flow layer doesn't re-instantiate disposed services) and skips it on --unsafe-run. The global manager self-heals after teardown (deps re-registers factories on next access; plugin service_classes survive — only instances drop). * fix(lfx): capture prewarm_flow teardown failure in error, don't raise prewarm_flow promises that one bad flow never aborts a multi-flow warming loop (load/build/run failures go into result.error rather than raising). The service-teardown step ran after the try/finally and could raise PrewarmError, breaking that contract — a direct caller looping over many flows would be aborted mid-batch by a single plugin's teardown failure. Funnel the teardown failure into result.error like the other failures, and document the cross-loop teardown caveat (warming runs in throwaway asyncio loops; a loop-bound plugin resource may fail cross-loop, surfacing as a fail-safe PrewarmError rather than silent corruption). * fix(lfx): deliver input_value to prewarm runs; close fork-safety test gaps Address code-review findings on the prewarm foundation: - preload: wrap input_value in InputValueRequest in _run_flow_once. astep only reads inputs via .model_dump(), so the raw dict was silently dropped and the --unsafe-run flow executed with empty input. - preload: dedup the two async_start loops into _run_flow_once. - server: restore the 'psutil not installed' debug breadcrumb lost when pre_fork moved to the shared lfx.fork helpers. - cli: include failed-import count in the prewarm summary line. Tests (close clean-path-only gaps, same class as the input_value miss): - input_value now reaches the flow output (regression). - find_ghost_connections detects a real open connection (not just the empty path). - prewarm_flow(run=True) surfaces a dirty fork-safety report. - CLI emits the fork-unsafe warning when a run leaves ghost state. - cross-loop service teardown fails loudly as PrewarmError. - idempotent prewarm does not re-import (stable module identity). - tighten warm-vs-cold speedup floor 5x -> 10x; verbose + unsafe-run CLI coverage. - autouse fixture isolates the global ServiceManager between tests. * chore(lfx): trim reviewer-facing comments and dedup prewarm tests Cleanup pass on the prewarm foundation, no behavior change: - tighten multi-sentence 'why' comments/docstrings to one natural line in preload, fork, _prewarm_commands, and manager.teardown. - drop the tautological BENIGN_THREAD_PREFIXES membership test (the behavioral ghost-thread tests already cover it). - reuse the _write_hermetic_flow helper in test_prewarm_flow_build_only instead of inlining the flow construction. * fix(lfx): always run build-only teardown in prewarm_flow, even after a build error Address CodeRabbit review on PR #13682: - preload: drop the 'result.error is None' guard so prewarm_flow(run=False) disposes services even when the build failed — a partial build can leave services live and the process fork-unsafe (the library API was exposed; the CLI already did a final centralized teardown). Teardown failures now append to any existing build error instead of overwriting it. - tests: cover build-failure-still-tears-down and the combined-error message. - cli test: drop the fragile fork-unsafe warning assertion (process-global state incl. harness threads makes it flaky under --timeout-method=thread); connection-cleanliness is asserted at the library level instead. * test(lfx): make prewarm speedup test measure cold-start, fix CI flake The perf test timed build+run only (imports happened before the timer), where the warm benefit is just a few x and machine-dependent — it failed on CI py3.14 at 6x vs the 10x floor. Move imports inside the timed region so the test measures the real value: full cold-start (import + build + run) in a fresh interpreter. Import elimination is worth hundreds of x (observed ~700x locally), so a 10x floor now has an enormous, machine-independent margin. * fix(lfx): disable tracing during the prewarm warm-up run The hermetic warm-up run writes nothing to the DB (message store needs a session_id, vertex-build logging a flow_id — the hermetic graph has neither), but lfx and Langflow share one service manager, so in a Langflow process the graph AND each component independently resolve the real tracer and would ship a junk 'prewarm' trace to a configured provider (LangSmith/Langfuse/...) and open a socket, breaking fork-safety. Wrap the run in a _tracing_disabled() context manager that blanks TRACING_SERVICE in the shared manager so every get_tracing_service() returns None (no-tracing build path), restoring the prior state afterward via an object() sentinel that distinguishes 'absent' from 'present-but-None'. Tests: warm-up run never touches a registered tracer; _tracing_disabled restores all three prior states (absent / real service / present-None). * fix(lfx): make fork-safety connection check reliable; drop dead helpers Address review on PR #13682: - deps: declare psutil in lfx (was only transitive via docling/accelerate extras, so a lean 'lfx serve' --preload install silently skipped the TCP ghost-connection check). find_ghost_connections now logs a warning instead of silently returning [] if psutil is ever missing. - cli: emit the fork-unsafe warning even when an --unsafe-run flow errors mid-run. Teardown is skipped on that path, so dirty state left by a failed run must still be announced; the warning is no longer gated behind the success branch. - fork: remove fork_safe_teardown and ForkSafetyReport.is_clean — no production callers (they belong to the follow-up gunicorn PR), plus their now-dead tests/imports. * fix(lfx): await-free all-types lookup so flows load (un-awaited coroutine) AllTypesDict.get_type_dict called the async get_all_types_dict without awaiting it, so the sync all_types_dict property did {**coroutine} and raised "'coroutine' object is not a mapping". This path is hit during graph build when a vertex falls back to resolve its base_type, so any flow reaching it failed to load (lfx run and lfx prewarm --flow alike). Use the sync build_custom_components instead — it is exactly what get_all_types_dict awaits internally (verified equivalent: identical categories/components/templates; only build-time timestamps differ, even sync-vs-sync). The consumer path is sync, so it must not be a coroutine. Verified: 21/33 starter templates warm via --flow end-to-end; the rest fail only on missing optional component SDKs (trustcall, langchain_chroma, ...), not parsing. |
|||
| ecd7888161 | fix: temporarily disable unpublished bundle deps | |||
| 4a8b47e46a |
Merge remote-tracking branch 'origin/release-1.10.1' into release-1.11.0
# Conflicts: # .secrets.baseline # pyproject.toml # src/backend/base/langflow/initial_setup/starter_projects/Pokédex Agent.json # src/backend/base/langflow/initial_setup/starter_projects/Structured Data Analysis Agent.json # src/backend/base/langflow/services/auth/service.py # src/backend/base/langflow/services/job_queue/service.py # src/backend/base/pyproject.toml # src/backend/tests/unit/components/models_and_agents/test_cuga_agent.py # src/backend/tests/unit/test_redis_job_queue_service.py # src/bundles/lfx-bundles/src/lfx_bundles/deepseek/deepseek.py # src/bundles/lfx-bundles/src/lfx_bundles/glean/glean_search_api.py # src/bundles/lfx-bundles/src/lfx_bundles/homeassistant/home_assistant_control.py # src/bundles/lfx-bundles/src/lfx_bundles/homeassistant/list_home_assistant_states.py # src/bundles/lfx-bundles/src/lfx_bundles/huggingface/huggingface_inference_api.py # src/bundles/lfx-bundles/src/lfx_bundles/lmstudio/lmstudioembeddings.py # src/bundles/lfx-bundles/src/lfx_bundles/xai/xai.py # src/frontend/package-lock.json # src/frontend/package.json # src/lfx/pyproject.toml # src/lfx/src/lfx/_assets/component_index.json # src/lfx/src/lfx/components/files_and_knowledge/filesystem.py # src/lfx/tests/unit/components/tools/test_filesystem_deny_list.py # uv.lock |
|||
| 7896b9f7f1 |
feat(bundles): metapackage split (Phase A) — engine-only lfx, lfx-bundles long tail, 5 graduated partner packages (#13563)
* feat(extension): add manifest-less lfx.bundles discovery + precedence tier Foundation for the bundle metapackage split (1.11). Adds a third @official-slot discovery source: a distribution declaring the [project.entry-points."lfx.bundles"] group ships a package whose immediate subdirectories are each a manifest-less bundle, folder-walked and registered at @official with no extension.json (the langchain-community model). - new loader/_bundles_root.py::load_lfx_bundles_extensions, mirroring discover_inline_bundles but sourcing roots from the lfx.bundles entry-point group via find_spec and reusing _load_bundle_directory - discovery precedence becomes installed > seed > lfx_bundles > dev > inline so a manifest-shipping lfx-<provider> always shadows the same-named provider in the metapackage (lets a provider graduate with no lockstep release), emitting the existing bundle-shadowed warning - new bundle-discovery-malformed warning code, emitted on warnings (never flips ok / aborts startup) for unresolvable declarations and invalid provider folder names - exported via loader/__init__ and the lfx.extension PEP-562 lazy surface 10 new tests in test_load_lfx_bundles.py; full extension unit suite (449) passes; ruff + format clean; mypy clean on the new module. * docs(bundles): changelog entry for the lfx.bundles discovery surface The BUNDLE_API.md changelog gate diffs each PR's branch against main, so the entry covering this PR's surface additions (lfx.bundles discovery, the lfx_bundles precedence tier, bundle-discovery-malformed) must live on this branch, not only further up the stack. Text is verbatim from the graduate-partners commit so the stacked merge dedupes trivially. * fix(extension): reject plain-module lfx.bundles targets; correct validate wording CodeRabbit review fixes on #13563: - _spec_package_dir no longer falls back to a plain module's parent directory -- a single-file entry-point target now emits the bundle-discovery-malformed warning instead of folder-walking unrelated sibling directories as bundles (+ regression test) - BUNDLE_API.md / loader docstring no longer say manifest-less providers are 'exempt from lfx extension validate'; they are not valid validator input (validate requires a manifest and reports manifest-not-found) * fix(extension): review fixes — shadowed metapackage providers never import; typed manifest-less reload refusal Review findings on #13563: - A manifest-less provider whose bundle name is already claimed by a higher-precedence @official source (installed/seed) is now skipped WITHOUT importing: all @official sources share the _lfx_ext.official.<bundle>.* sys.modules namespace, so importing the losing copy overwrote the winner's live modules even though _resolve_bundle_shadowing dropped its components afterwards. The orchestrator passes the claimed-name map (built by the new _claimed_official_bundles helper, mirroring the resolver's winner rule); the skipped copy still emits the typed bundle-shadowed warning. This collision is the expected post-graduation state, not an error. - Manifest-less records now carry a manifestless provenance flag (additive field on LoadResult + BundleRecord). The reload pipeline refuses them with the new typed reload-manifestless-unsupported code instead of routing through load_extension and failing with a misleading manifest-not-found. Hot reload of metapackage providers is a process-restart concern (pip-installed content); a loader-based reload path can follow if QA wants it. BUNDLE_API.md changelog entry extended for both surface changes. * fix(extension): harden lfx.bundles discovery per review — broad find_spec guard, namespace portions, per-mode error codes Review findings from ogabrielluiz on #13563: - find_spec on a dotted declaration imports the PARENT package, whose __init__ can raise anything; a non-Import error escaped the old catch tuple and (through the palette cache's catch-all) wiped every source's components for the boot. Catch Exception and degrade to the malformed sentinel; comment corrected. - Namespace packages: walk ALL submodule_search_locations portions (one root per portion) instead of only locations[0], and dedupe resolved roots by path so duplicate entry-point declarations (or overlapping portions) never walk a directory twice and self-shadow. _spec_package_dir -> _spec_package_dirs. - Split bundle-discovery-malformed into one code per failure mode, mirroring the inline tier: bundles-provider-name-invalid (rename the directory), bundles-root-unreadable (check permissions), keeping bundle-discovery-malformed for unresolvable declarations (fix the entry-point). Same-tier duplicates get duplicate-lfx-bundles-provider instead of overloading bundle-shadowed, whose rendered template (format_extension_error renders templates, not ad-hoc messages) was wrong on both counts for that case. - The claimed-bundles cross-source skip now emits bundle-shadowed on errors (matching _resolve_bundle_shadowing) so filtering by code never mixes severities; bundle-shadowed is already in the CLI warn-only set. 3 new regression tests (raising parent package, namespace portions, duplicate entry points); BUNDLE_API changelog updated. * feat(bundles): create lfx-bundles metapackage skeleton (#13564) * feat(bundles): add lfx-bundles metapackage skeleton Creates the manifest-less lfx-bundles distribution (the langchain-community model) that the long-tail providers will move into. Empty skeleton for now; the bulk move (scripts/migrate/consolidate_bundles.py) populates the provider folders + per-provider extras later. - src/bundles/lfx-bundles: a single pyproject declaring the lfx.bundles entry-point (lfx_bundles = "lfx_bundles"), an lfx>=1.10.0,<2.0.0 pin, and a generated (currently empty) `all` extra; plus a bare lfx_bundles namespace package and a README documenting the model + install stories - wired into the root workspace via the existing bundle marker blocks: dep lfx-bundles[all]>=1.0,<2.0, uv source, and member - hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change Verified: the wheel builds (entry_points.txt carries the lfx.bundles group); load_lfx_bundles_extensions (PR-1) discovers the entry point and the empty skeleton registers zero providers with no error. * fix(ci): nightly bundle rename follows [extras] refs and self-refs Two gaps in update_bundle_versions.py around extras suffixes, both fatal or silently wrong for the first nightly carrying the lfx-bundles metapackage: - update_root_pyproject_for_bundle's dep regex required the version specifier immediately after the name, so "lfx-bundles[all]>=1.0,<2.0" (a MAIN dep) was left unrewritten while the workspace member was renamed to lfx-bundles-nightly; uv lock then tries to resolve stable lfx-bundles from PyPI, where it does not exist. The same gap silently left "lfx-docling[local]>=0.1.0" optional-dep refs pointing at the stable distribution. The regex now tolerates an [extras] group and carries it into the replacement. - rename_bundle_pyproject skipped self-referencing extras, so the metapackage's generated `all` extra kept 45 "lfx-bundles[<provider>]" members after the rename, pulling the stable distribution (same lfx_bundles import package, install collision) once published. Self refs now follow the rename, idempotently. Tests drive the real script module, mirroring test_bundle_lfx_pin.py. This closes the PR-2 audit item flagged when the metapackage was introduced. The canonical-pre-release cutover would retire the nightly rename entirely; until it lands, the rename must be correct. * fix(bundles): address review — drop orphaned nightly bundle-rename script, fix README install stories - Delete scripts/ci/update_bundle_versions.py and its tests: the nightly bundle-rename track it served was retired by the canonical pre-release cutover (src/bundles/NIGHTLY.md) and no workflow or Makefile target on main or any release branch invokes it. Stale doc references in sync_bundle_lfx_pin.py and test_bundle_lfx_pin.py updated to match. - README: split the install section into what works today (langflow, bare lfx) vs what arrives with the bulk move / engine-only split (lfx[bundles], per-provider lfx-bundles[<provider>] extras), and note the generated all extra is empty until the first provider tranche lands. * ci(bundles): cross-bundle test matrix (lfx contract axis) (#13566) * ci(bundles): add cross-bundle test matrix (lfx contract axis) Tests every extracted bundle (the lfx-bundles metapackage + each graduated lfx-<provider>) against the lfx contract surface it depends on -- overdue since 1.10 left 4 independently-versioned bundles depending on lfx. - .github/workflows/cross-bundle-test.yml: discovers bundles via the same src/bundles/*/pyproject.toml glob release.yml uses, then per (bundle x python 3.10/3.13): installs the in-repo lfx + the bundle, imports the declared entry-point package, asserts lfx.bundles discovery is error-free (for the metapackage), runs `lfx extension validate` for manifest bundles, and runs the bundle's own tests/. - Triggers: pull_request (src/bundles, src/lfx), workflow_dispatch, a weekly schedule, and workflow_call. The lfx-minor axis seeds with the in-repo lfx (the 1.10 line is unpublished); when minors publish, add the version dimension (oldest+latest get full tests, every supported minor gets contract-smoke) per the epic's cost-control shape. Verified: the workflow YAML parses and the contract-smoke logic imports a real bundle (lfx_arxiv) cleanly. * fix(ci): cross-bundle smoke tolerates extras-less SDK degradation Review findings on the cross-bundle matrix: - CRITICAL: the contract smoke asserted zero discovery errors, but the CI venv installs lfx-bundles WITHOUT per-provider extras, so providers whose modules import their SDK at top level degrade with module-import-failed -- the expected graceful-degradation contract. The smoke now fails only on structural codes and reports the degraded-module count. (Graduated partner bundles carry their deps directly, not as extras, so their steps are unaffected.) - the scheduled run now actually delivers the exhaustive grid the header promised (all supported Pythons on schedule; oldest+latest on PRs) - concurrency group with cancel-in-progress on PRs so stacked bundle PRs don't queue N-bundles x N-pythons jobs per push * ci(bundles): tomli fallback for the py3.10 cross-bundle smoke tomllib is stdlib only from Python 3.11, so the contract-smoke heredoc failed with ModuleNotFoundError on every py3.10 matrix leg. Install the tomli backport into the smoke venv and import it as a fallback. * fix(test): coherent sys.modules restore in the ibm without_ibm_db fixture The cross-bundle matrix's py3.10 leg exposed test-ordering pollution in the ibm bundle suite: without_ibm_db monkeypatch-deleted three lfx_ibm modules and re-imported them mid-test, but entries created during the test survive teardown (delitem with raising=False records nothing for keys it didn't find) and re-imported parents never regain the submodule attribute bindings that mock.patch target resolution walks on Python 3.10 (3.11+ mock resolves via sys.modules and tolerates the incoherence). Two fixture uses in sequence left the tree inconsistent and 29 later watsonx tests failed with AttributeError on the package. Snapshot, drop, and restore the entire lfx_ibm module tree wholesale instead. Full suite now passes on both 3.10 and 3.13. * feat(lfx): add lfx[bundles] extra and keep lfx engine-only (#13565) * feat(bundles): add lfx-bundles metapackage skeleton Creates the manifest-less lfx-bundles distribution (the langchain-community model) that the long-tail providers will move into. Empty skeleton for now; the bulk move (scripts/migrate/consolidate_bundles.py) populates the provider folders + per-provider extras later. - src/bundles/lfx-bundles: a single pyproject declaring the lfx.bundles entry-point (lfx_bundles = "lfx_bundles"), an lfx>=1.10.0,<2.0.0 pin, and a generated (currently empty) `all` extra; plus a bare lfx_bundles namespace package and a README documenting the model + install stories - wired into the root workspace via the existing bundle marker blocks: dep lfx-bundles[all]>=1.0,<2.0, uv source, and member - hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change Verified: the wheel builds (entry_points.txt carries the lfx.bundles group); load_lfx_bundles_extensions (PR-1) discovers the entry point and the empty skeleton registers zero providers with no error. * feat(lfx): add lfx[bundles] extra and keep lfx engine-only lfx ships no bundles by default. The new optional extra pulls the long-tail metapackage for users who want the provider components without the full langflow server install. - src/lfx/pyproject.toml: [project.optional-dependencies] bundles = ["lfx-bundles[all]>=1.0,<2.0"]; intentionally NO lfx[all] - pip install lfx -> engine only pip install "lfx[bundles]" -> engine + the lfx-bundles long tail (documented as a headless/serverless deployment footnote, not a headline) Verified against the built lfx wheel METADATA: lfx-bundles appears only under `extra == "bundles"`, never in core Requires-Dist, so the engine-only default is preserved. * fix(ci): nightly bundle rename follows [extras] refs and self-refs Two gaps in update_bundle_versions.py around extras suffixes, both fatal or silently wrong for the first nightly carrying the lfx-bundles metapackage: - update_root_pyproject_for_bundle's dep regex required the version specifier immediately after the name, so "lfx-bundles[all]>=1.0,<2.0" (a MAIN dep) was left unrewritten while the workspace member was renamed to lfx-bundles-nightly; uv lock then tries to resolve stable lfx-bundles from PyPI, where it does not exist. The same gap silently left "lfx-docling[local]>=0.1.0" optional-dep refs pointing at the stable distribution. The regex now tolerates an [extras] group and carries it into the replacement. - rename_bundle_pyproject skipped self-referencing extras, so the metapackage's generated `all` extra kept 45 "lfx-bundles[<provider>]" members after the rename, pulling the stable distribution (same lfx_bundles import package, install collision) once published. Self refs now follow the rename, idempotently. Tests drive the real script module, mirroring test_bundle_lfx_pin.py. This closes the PR-2 audit item flagged when the metapackage was introduced. The canonical-pre-release cutover would retire the nightly rename entirely; until it lands, the rename must be correct. * ci(bundles): freeze lfx/components against new top-level providers (#13567) * feat(bundles): add lfx-bundles metapackage skeleton Creates the manifest-less lfx-bundles distribution (the langchain-community model) that the long-tail providers will move into. Empty skeleton for now; the bulk move (scripts/migrate/consolidate_bundles.py) populates the provider folders + per-provider extras later. - src/bundles/lfx-bundles: a single pyproject declaring the lfx.bundles entry-point (lfx_bundles = "lfx_bundles"), an lfx>=1.10.0,<2.0.0 pin, and a generated (currently empty) `all` extra; plus a bare lfx_bundles namespace package and a README documenting the model + install stories - wired into the root workspace via the existing bundle marker blocks: dep lfx-bundles[all]>=1.0,<2.0, uv source, and member - hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change Verified: the wheel builds (entry_points.txt carries the lfx.bundles group); load_lfx_bundles_extensions (PR-1) discovers the entry point and the empty skeleton registers zero providers with no error. * ci(bundles): freeze lfx/components against new top-level providers After the metapackage split, new providers go to lfx-bundles (or a graduated lfx-<provider>), never in-tree. Adds an additions-only CI gate. - scripts/ci/check_components_frozen.py: stdlib gate comparing the live top-level dir listing of src/lfx/src/lfx/components/ against a committed baseline; fails on any directory not in the baseline. Removals are allowed so M4 shim cleanup (which shrinks the set) never trips it. - scripts/ci/frozen_component_dirs.txt: the 111-dir baseline. - .github/workflows/lint-py.yml: new freeze-components job runs the gate (stdlib-only, no uv sync). Verified locally: passes on the baseline, fails (exit 1) on a simulated new provider directory with an actionable message, passes again after cleanup. * fix(ci): freeze gate counts only dirs shipping __init__.py Review finding: a stray directory holding only __pycache__ bytecode (left behind by a branch switch) tripped the additions-only gate locally. A directory without __init__.py is not a provider; ignore it. * fix(ci): nightly bundle rename follows [extras] refs and self-refs Two gaps in update_bundle_versions.py around extras suffixes, both fatal or silently wrong for the first nightly carrying the lfx-bundles metapackage: - update_root_pyproject_for_bundle's dep regex required the version specifier immediately after the name, so "lfx-bundles[all]>=1.0,<2.0" (a MAIN dep) was left unrewritten while the workspace member was renamed to lfx-bundles-nightly; uv lock then tries to resolve stable lfx-bundles from PyPI, where it does not exist. The same gap silently left "lfx-docling[local]>=0.1.0" optional-dep refs pointing at the stable distribution. The regex now tolerates an [extras] group and carries it into the replacement. - rename_bundle_pyproject skipped self-referencing extras, so the metapackage's generated `all` extra kept 45 "lfx-bundles[<provider>]" members after the rename, pulling the stable distribution (same lfx_bundles import package, install collision) once published. Self refs now follow the rename, idempotently. Tests drive the real script module, mirroring test_bundle_lfx_pin.py. This closes the PR-2 audit item flagged when the metapackage was introduced. The canonical-pre-release cutover would retire the nightly rename entirely; until it lands, the rename must be correct. * feat(bundles): move 45 long-tail providers into lfx-bundles + graduate 5 partner packages (#13568) * feat(bundles): add lfx-bundles metapackage skeleton Creates the manifest-less lfx-bundles distribution (the langchain-community model) that the long-tail providers will move into. Empty skeleton for now; the bulk move (scripts/migrate/consolidate_bundles.py) populates the provider folders + per-provider extras later. - src/bundles/lfx-bundles: a single pyproject declaring the lfx.bundles entry-point (lfx_bundles = "lfx_bundles"), an lfx>=1.10.0,<2.0.0 pin, and a generated (currently empty) `all` extra; plus a bare lfx_bundles namespace package and a README documenting the model + install stories - wired into the root workspace via the existing bundle marker blocks: dep lfx-bundles[all]>=1.0,<2.0, uv source, and member - hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change Verified: the wheel builds (entry_points.txt carries the lfx.bundles group); load_lfx_bundles_extensions (PR-1) discovers the entry point and the empty skeleton registers zero providers with no error. * feat(bundles): consolidate first long-tail tranche into lfx-bundles Adds scripts/migrate/consolidate_bundles.py (the inverse of port_bundle.py -- moves in-tree providers into the manifest-less lfx-bundles metapackage) and runs it on a verified 5-provider tranche: tavily, exa, wikipedia, yahoosearch, wolframalpha. Per provider the script: - moves src/lfx/src/lfx/components/<p>/ -> lfx_bundles/<p>/ (lowercase names); - leaves a fail-soft import shim (first line `# lfx-bundles-shim`) so `from lfx.components.<p> import X` keeps working when lfx-bundles is installed, and raises an actionable ImportError otherwise; - merges the provider's third-party deps into a PEP 685-normalized lfx-bundles extra and regenerates the `all` aggregate. Dep parity holds: `uv sync` is a no-op because those deps were already pulled via langflow-base[complete]; - appends the 4-entry migration block per Component class (28 entries) so saved flows referencing lfx.components.<p>.<Class> migrate to ext:<p>:<Class>@official. To avoid double registration, the in-tree component walk (_load_components_dynamically) now skips shimmed provider dirs, and component_index.json is regenerated (355->348 components, 95->90 modules); the moved providers load only at @official via lfx.bundles discovery. Verified: discovery finds all 5 at @official with no errors; shims resolve; `import lfx.components` still works; the index drops the 7 moved component entries (residual `tools`-category name refs resolve via the shim); ruff clean. First tranche proves the engine; the remaining long-tail scales by extending PROVIDER_DEPS (each provider's deps verified individually -- the careful part). * feat(bundles): consolidate 30-provider tranche 2 into lfx-bundles Extends PROVIDER_DEPS with 30 individually-verified providers and runs the consolidation: vector stores (chroma, clickhouse, couchbase, milvus, mongodb, pgvector, pinecone, qdrant, supabase, upstash, weaviate), model providers (groq, mistral, ollama, perplexity, sambanova), and tools/memory/data (apify, assemblyai, confluence, firecrawl, git, glean, icosacomputing, mem0, needle, scrapegraph, serpapi, unstructured, youtube, zep). Dep verification (the careful part): every spec comes from langflow-base's per-provider extras or its direct dependencies; langchain-community providers carry the wrapper plus the SDK the wrapper lazy-imports (e.g. pgvector, atlassian-python-api); requests is declared explicitly where imported (it is only transitive in today's env); pinecone keeps its python_version<'3.14' marker verbatim. Tranche excludes: providers with langflow imports (vlmrun), provider-specific lfx.base dirs (composio/huggingface/langwatch), case- sensitive names (FAISS/Notion), the openai-SDK family (azure/aiml/deepseek/ litellm/lmstudio/novita/openrouter/vllm/xai/cometapi -- cleaner after PR-8), and the partner set. Dep parity verified at the resolution level: the uv.lock diff is +220 lines of lfx-bundles extras metadata with ZERO packages added or removed (`name =` diff empty), so pip install langflow resolves the identical set. Also: 192 append-only migration entries (48 classes x 4, zero bare-name ambiguities); component_index.json regenerated 348->300 components / 90->60 modules (exactly the moved set, no stale standalone entries); mongodb_atlas.py SLF001 per-file-ignore and the mem0/mongodb detect-secrets baseline entries migrated to the new paths (lint/secrets exceptions travel with moved files); 35 bundles now discover at @official with 55 components; shims verified across categories; 449 extension tests pass. * feat(bundles): consolidate openai-SDK family tranche 3 into lfx-bundles 10 providers that ride the langchain-openai wrapper, deferred from tranche 2 until the partner graduation settled the openai-SDK dep story: aiml, azure, cometapi, deepseek, litellm, lmstudio, novita, openrouter, vllm, xai. Dep verification: every provider declares langchain-openai>=1.1.6; the openai SDK is declared only where a component imports it directly (aiml, deepseek, litellm, lmstudio, vllm, xai -- wrapper-transitive elsewhere); requests declared where imported (cometapi, deepseek, novita, xai); lmstudio's lazy NVIDIAEmbeddings path gets langchain-nvidia-ai-endpoints~=1.0.0. The litellm component drives LiteLLM-served endpoints through the OpenAI client and does NOT import the litellm package -- langflow-base's litellm extra stays put. These providers use the lazy _dynamic_imports __init__ shape; it survives the move unchanged (import_mod resolves via __spec__.parent and lfx.components._importing remains a core helper). Dep parity: uv.lock diff has zero package additions/removals (all specs already resolved via langflow-base[complete]). Also: 56 append-only migration entries (14 classes x 4, zero ambiguities); component_index.json regenerated 300->286 components / 60->50 modules (exactly the moved set); detect-secrets baseline re-keyed; 45 bundles now discover at @official with 69 components; shims verified (azure/xai/litellm/deepseek); ruff clean. * fix(ci): nightly bundle rename follows [extras] refs and self-refs Two gaps in update_bundle_versions.py around extras suffixes, both fatal or silently wrong for the first nightly carrying the lfx-bundles metapackage: - update_root_pyproject_for_bundle's dep regex required the version specifier immediately after the name, so "lfx-bundles[all]>=1.0,<2.0" (a MAIN dep) was left unrewritten while the workspace member was renamed to lfx-bundles-nightly; uv lock then tries to resolve stable lfx-bundles from PyPI, where it does not exist. The same gap silently left "lfx-docling[local]>=0.1.0" optional-dep refs pointing at the stable distribution. The regex now tolerates an [extras] group and carries it into the replacement. - rename_bundle_pyproject skipped self-referencing extras, so the metapackage's generated `all` extra kept 45 "lfx-bundles[<provider>]" members after the rename, pulling the stable distribution (same lfx_bundles import package, install collision) once published. Self refs now follow the rename, idempotently. Tests drive the real script module, mirroring test_bundle_lfx_pin.py. This closes the PR-2 audit item flagged when the metapackage was introduced. The canonical-pre-release cutover would retire the nightly rename entirely; until it lands, the rename must be correct. * feat(bundles): graduate partner set to standalone lfx-<provider> packages (#13573) * feat(bundles): graduate partner set to standalone lfx-<provider> packages Extracts the five partner/flagship providers into manifest-shipping distributions: lfx-openai, lfx-anthropic, lfx-amazon, lfx-datastax, lfx-cohere. Each ships extension.json (lfx.compat ["1"]), a langflow.extensions entry-point, an lfx>=1.10.0,<2.0.0 pin, and is wired into the root workspace marker blocks. Zero flow impact by the bundle-name invariant: ext:<provider>:<Class>@official ids are unchanged. Mechanics: - adopts the five-phase scripts/migrate/port_bundle.py from feat/bundle-mass-extraction (handles shared-base moves, consumer rewrites, surgical index updates, migration-table appends) and runs it per partner with --migration-release 1.11.0 - datastax's shared lfx.base.datastax moves into lfx_datastax.base; its backend unit tests move to src/bundles/datastax/tests (74 pass); its ruff per-file-ignore and its check_component_env_writes ALLOWLIST entry travel; 10 repo consumers rewritten (incl. the vector_store_rag starter project) - amazon_bedrock_converse.py legacy langflow.* imports rewritten to the lfx.* equivalents (thin re-export aliases; behavior-identical) pre-extraction - runtime deps pinned from langflow-base's extras / direct deps (wrapper- guaranteed SDKs stay transitive, parity-exact); --remove-base-extra dropped the openai/anthropic/cohere/aws extras from langflow-base[complete] in favor of the bundle pins; the astradb extra stays (also used outside datastax) - in-tree dirs replaced with marker shims pointing at lfx_<p>.components.<p> (skipped by the in-tree walk; legacy from lfx.components.<p> imports keep working); lfx/components/__init__.py entries kept, consistent with the consolidated providers - validate.py: accept classes whose base is a *derived* Component base (LCVectorStoreComponent / LCToolComponent / ...) -- they inherit the class-level outputs declaration and only override the output method, which the AST-only check could not see; bare Component subclasses keep the strict build/outputs requirement (+ regression test) - BUNDLE_API.md changelog entries added for the branch's surface changes: the lfx.bundles manifest-less discovery group + precedence tier + bundle-discovery-malformed code (from the foundation PR) and the validator acceptance above - 100 append-only migration entries (20 classes x 5 partners x 4 shapes); component_index.json surgically updated 300->280 components / 60->55 modules, sha256 recomputed and verified - detect-secrets baseline re-keyed for moved partner files Dep parity: lock diff adds only the five lfx-* package names; no third-party package added or removed. One benign transitive re-resolution: anthropic SDK 0.105.2 -> 0.109.0 (allowed by langchain-anthropic's range on both sides). Verified: all five register at @official via the installed-manifest tier (extension_id/distribution = lfx-<p>, 20 components); manifest-less lfx_bundles unchanged (35 bundles, disjoint); all 5 shims import; engine-safe; `lfx extension validate` passes for all five; 450 extension unit tests, 60 pilot-upgrade migration tests, 74 moved datastax tests pass; ruff clean. * fix(bundles): shim lfx.base.datastax so stored-flow imports keep resolving The datastax graduation moved lfx.base.datastax into lfx_datastax.base, but saved flows and starter templates (Hybrid Search RAG, TemplateAssistant) embed `from lfx.base.datastax.astradb_base import AstraDBBaseComponent` inside their stored component code fields, which is re-executed verbatim at flow build time. Without a shim that import raises ModuleNotFoundError and the flows fail to build (test-starter-projects red). Mirror the lfx.components shim contract: module-aliasing to lfx_datastax.base, narrow except that only translates a missing bundle (transitive dep failures re-raise untouched), marker-tagged single-file dir, removed at M4 together with the components shims. * chore(bundles): bounded version ranges for curated lfx-* packages (#13576) langflow's pyproject is the release-coordination point now that lfx and the lfx-* bundles release independently. Every curated lfx-* dependency declares a BOUNDED range (>=A,<B), never an exact pin -- exact pins in reusable library metadata create resolver conflicts for downstreams that depend on a different version. Exact pins for reproducibility live in uv.lock and the release-build manifests. - the 4 pilots + 5 graduated partners: >=0.1.0 -> >=0.1.0,<1.0.0 (bundles follow their own 0.1.x cadence) - lfx-bundles[all]>=1.0,<2.0 unchanged (versions with the metapackage contract) - the three lfx-docling[...] optional-dependency refs bounded the same way - policy documented inline in the bundle-deps marker block; the cross-bundle CI matrix verifies the ranges resolve together across the supported lfx axis Verified: uv lock resolves with zero package/version changes (bounds are metadata-only today); the nightly rename's dep regex already matches the bounded form. * test(bundles): lock the in-tree shim contract + breakage audit (#13577) * chore(bundles): bounded version ranges for curated lfx-* packages langflow's pyproject is the release-coordination point now that lfx and the lfx-* bundles release independently. Every curated lfx-* dependency declares a BOUNDED range (>=A,<B), never an exact pin -- exact pins in reusable library metadata create resolver conflicts for downstreams that depend on a different version. Exact pins for reproducibility live in uv.lock and the release-build manifests. - the 4 pilots + 5 graduated partners: >=0.1.0 -> >=0.1.0,<1.0.0 (bundles follow their own 0.1.x cadence) - lfx-bundles[all]>=1.0,<2.0 unchanged (versions with the metapackage contract) - the three lfx-docling[...] optional-dependency refs bounded the same way - policy documented inline in the bundle-deps marker block; the cross-bundle CI matrix verifies the ranges resolve together across the supported lfx axis Verified: uv lock resolves with zero package/version changes (bounds are metadata-only today); the nightly rename's dep regex already matches the bounded form. * test(bundles): lock the in-tree shim contract + breakage audit Locks the five-point contract for the 50 lfx-bundles import shims under lfx/components/ (45 metapackage + 5 graduated partners): 1. first line is the `# lfx-bundles-shim` marker (keys the in-tree walk's double-registration skip); 2. a shim dir is exactly one __init__.py -- no impls, no deps; 3. sys.modules module-aliasing (submodule trees resolve); 4. bundle missing -> actionable ModuleNotFoundError whose wording is part of the contract ("pip install lfx-bundles" / "pip install lfx-<p>"); 5. a missing *transitive* dep re-raises untouched. Test shape (env-adaptive by design -- the lfx test env prunes the venv, so nothing here assumes bundles are installed): - source-level sweep of every real shim, parameterized (no imports); - hermetic mechanism tests against a synthetic shim + synthetic target (alias-resolves / locked-message / transitive-reraise); - the walk-skip detector and the marker sweep must agree exactly; - one adaptive live test on lfx.components.tavily exercising whichever branch the running env is in. 106 tests pass in the pruned lfx env (where the live test exercises the bare-engine locked-message branch). Breakage audit (gh code search, recorded in the PR): ~20-25 public repos reference lfx.components.*; the majority are langflow forks (vendored tree, unaffected by packaging). Genuine library consumers exist and the moved providers have real surface (openai: 11 external repos, datastax: 17) -- covered by the shims wherever bundles are co-installed (every `pip install langflow`). Decision: deprecation warnings NOT warranted now; revisit at M4 (shim removal), where one loud minor release before removal is the right shape. * test(bundles): extend the shim contract sweep to lfx.base shims The datastax graduation moved lfx.base.datastax into lfx_datastax.base and left a module-aliasing shim behind (stored flow code fields embed the legacy import and are re-executed verbatim at build time). Sweep lfx/base for marker shims with the same source-level contract as the components sweep: one-file stub, module-aliasing to the bundle's base subpackage, narrow name-checked except, locked install message. * docs(bundles): install shapes, override rule, bundle_api_version (#13578) Documents the deliberately small user-facing rule surface of the metapackage split (1.11): - extensions-overview.mdx: the two install stories (pip install langflow = everything, same as today; pip install lfx = engine only, bring your own bundles); the bundle-name-is-identity invariant; the current package table (lfx-bundles metapackage + the 5 partner packages + the 4 pilots); the legacy-import shim behavior with the locked ModuleNotFoundError wording and the migration recipe for direct lfx users (switch to lfx[bundles] or pin lfx-<provider> packages); the override rule ("ship a manifest -- a manifest always wins", with the bundle-shadowed warning); bundle_api_version as the single compat number (lfx.compat ["1"]; manifest-less packages ride their PEP 508 lfx pin) and the no-version-arithmetic rule. - deployment-lfx-compatibility.mdx: lfx[bundles] as the headless/serverless deployment footnote (engine + lfx-bundles[all]; slimmer per-provider alternative; intentionally no lfx[all]). Both files verified MDX-safe (no unbackticked angle brackets). * fix(bundles): stack-review fixes — symlink containment, idempotency, docs accuracy (#13579) * chore(bundles): bounded version ranges for curated lfx-* packages langflow's pyproject is the release-coordination point now that lfx and the lfx-* bundles release independently. Every curated lfx-* dependency declares a BOUNDED range (>=A,<B), never an exact pin -- exact pins in reusable library metadata create resolver conflicts for downstreams that depend on a different version. Exact pins for reproducibility live in uv.lock and the release-build manifests. - the 4 pilots + 5 graduated partners: >=0.1.0 -> >=0.1.0,<1.0.0 (bundles follow their own 0.1.x cadence) - lfx-bundles[all]>=1.0,<2.0 unchanged (versions with the metapackage contract) - the three lfx-docling[...] optional-dependency refs bounded the same way - policy documented inline in the bundle-deps marker block; the cross-bundle CI matrix verifies the ranges resolve together across the supported lfx axis Verified: uv lock resolves with zero package/version changes (bounds are metadata-only today); the nightly rename's dep regex already matches the bounded form. * test(bundles): lock the in-tree shim contract + breakage audit Locks the five-point contract for the 50 lfx-bundles import shims under lfx/components/ (45 metapackage + 5 graduated partners): 1. first line is the `# lfx-bundles-shim` marker (keys the in-tree walk's double-registration skip); 2. a shim dir is exactly one __init__.py -- no impls, no deps; 3. sys.modules module-aliasing (submodule trees resolve); 4. bundle missing -> actionable ModuleNotFoundError whose wording is part of the contract ("pip install lfx-bundles" / "pip install lfx-<p>"); 5. a missing *transitive* dep re-raises untouched. Test shape (env-adaptive by design -- the lfx test env prunes the venv, so nothing here assumes bundles are installed): - source-level sweep of every real shim, parameterized (no imports); - hermetic mechanism tests against a synthetic shim + synthetic target (alias-resolves / locked-message / transitive-reraise); - the walk-skip detector and the marker sweep must agree exactly; - one adaptive live test on lfx.components.tavily exercising whichever branch the running env is in. 106 tests pass in the pruned lfx env (where the live test exercises the bare-engine locked-message branch). Breakage audit (gh code search, recorded in the PR): ~20-25 public repos reference lfx.components.*; the majority are langflow forks (vendored tree, unaffected by packaging). Genuine library consumers exist and the moved providers have real surface (openai: 11 external repos, datastax: 17) -- covered by the shims wherever bundles are co-installed (every `pip install langflow`). Decision: deprecation warnings NOT warranted now; revisit at M4 (shim removal), where one loud minor release before removal is the right shape. * docs(bundles): install shapes, override rule, bundle_api_version Documents the deliberately small user-facing rule surface of the metapackage split (1.11): - extensions-overview.mdx: the two install stories (pip install langflow = everything, same as today; pip install lfx = engine only, bring your own bundles); the bundle-name-is-identity invariant; the current package table (lfx-bundles metapackage + the 5 partner packages + the 4 pilots); the legacy-import shim behavior with the locked ModuleNotFoundError wording and the migration recipe for direct lfx users (switch to lfx[bundles] or pin lfx-<provider> packages); the override rule ("ship a manifest -- a manifest always wins", with the bundle-shadowed warning); bundle_api_version as the single compat number (lfx.compat ["1"]; manifest-less packages ride their PEP 508 lfx pin) and the no-version-arithmetic rule. - deployment-lfx-compatibility.mdx: lfx[bundles] as the headless/serverless deployment footnote (engine + lfx-bundles[all]; slimmer per-provider alternative; intentionally no lfx[all]). Both files verified MDX-safe (no unbackticked angle brackets). * fix(bundles): review fixes — symlink containment, idempotency, docs accuracy Fixes from the multi-agent stack review (kept as a separate PR so the reviewed PRs' diffs stay frozen for QA's independent pass): - _bundles_root.py: directory-level symlink containment — a provider dir that resolves outside the bundle root is skipped with a typed bundle-discovery-malformed warning, mirroring the seed-directory walk's is_within rule (+ regression test). Anchors the trust boundary to the installed package tree. - consolidate_bundles.py: migration-append idempotency guard — entries are deduped on full content so a partially-failed earlier run cannot duplicate rows on re-run (table stays append-only). - langflow-base pyproject: documented WHY the aws extra is deliberately retained after the lfx-amazon graduation (boto3 also backs the server's own S3 storage backend and lfx's S3 ingestion — review suggested removing it; that would regress S3 storage support). - extensions-overview.mdx: note that `lfx extension list` shows manifest-shipping extensions only; manifest-less packages load at startup but are not listed. 93 loader+migration tests pass (incl. the new symlink test); ruff clean. * refactor(bundles): stabilize import_mod as a public BUNDLE_API utility The lazy-import helper that bundle packages call from their __getattr__-based __init__.py files lived at lfx.components._importing -- an internal path with no stability contract, imported by 35 separately-installed bundle __init__ files (30 lfx-bundles providers + the 5 graduated partners). - canonical home is now lfx.utils.lazy_import.import_mod (code unchanged); lfx.components._importing re-exports it so in-tree callers and any external code on the old path keep working - all 35 bundle-side imports rewritten to the stable path - BUNDLE_API.md: surface table entry + changelog (additive) - contract tests: re-export identity, both call forms, the AttributeError conversion Verified: identity holds across both paths; lazy __init__ loads work through the new path for both bundle families; 110 tests pass; ruff clean. * fix(bundles): tombstone the broken legacy ZepChatMemory build method (#13580) build_message_history targeted the zep-python v1 SDK (ZepClient + zep_python.langchain.ZepChatMessageHistory); both were removed in zep-python 2.x and the zep extra pins 2.0.2, so the method has been unable to run for as long as the pin has existed -- its ImportError guard misleadingly told users to 'pip install zep-python' (already installed). The component is legacy=True with helpers.Memory as its designated replacement, so rather than hand-write a new integration against the 2.x SDK, the method now raises a clear RuntimeError pointing at the Message History component. Flow identity is preserved: class/component name, display_name, description, inputs and the memory output are byte-identical, so saved flows keep loading, i18n locale keys are unchanged, and migration_table.json needs no edits. New bundle tests pin the stub contract (identity, actionable error, no zep_python import). * test(bundles): extend the shim contract sweep to lfx.base shims The datastax graduation moved lfx.base.datastax into lfx_datastax.base and left a module-aliasing shim behind (stored flow code fields embed the legacy import and are re-executed verbatim at build time). Sweep lfx/base for marker shims with the same source-level contract as the components sweep: one-file stub, module-aliasing to the bundle's base subpackage, narrow name-checked except, locked install message. * test(bundles): extras-drift guard for the lfx-bundles metapackage Risk-2 of the metapackage split: the generated `all` extra and the per-provider extras must never drift by hand-edit, or `pip install langflow` silently loses a provider's deps. Guard the four invariants: extras <-> provider dirs (PEP 685-normalized), `all` == the exact self-ref set, normalized keys collision-free, and the metapackage provider set disjoint from the graduated partner distributions. * fix(extension): symlink-escaped providers reject with path-escape, matching the documented contract The lfx.bundles provider containment check (stack-review symlink fix) emitted bundle-discovery-malformed, whose template and hint describe a broken entry-point declaration. The changelog's path-safety entry already documents symlink escapes as path-escape on every other discovery path; use the same code here. Also resolves the semantic merge conflict with the per-mode code split (the removed _malformed_error location kwarg). * fix(tests): repoint lfx tests off moved providers; complete provider fallback map CI fail-fast had been masking these: the LFX test job runs in an engine-only env where openai/anthropic/chroma are now bundle-package shims, so every test that used them as the example category failed on all Python versions (3.14 just reported first), and the backend Group 2 leg failed collecting test_lfx_bundles_extras.py on Python 3.10. - flow_requirements: complete _PROVIDER_PACKAGE_FALLBACKS for the nine moved model providers (OpenAI, Anthropic, Amazon Bedrock, Groq, Google Generative AI, SambaNova, IBM watsonx.ai, Ollama + existing Azure OpenAI). In engine-only installs MODEL_PROVIDERS_DICT registers only in-tree providers, so the dynamic source-inspection path cannot resolve moved providers; the static fallbacks keep lfx run/serve requirements inference working. A dict hit still wins. - test_dynamic_imports / test_import_utils: use composio (still-in-tree lazy category with its SDK absent in the bare env) as the example category instead of openai/anthropic/chroma; patch import_module at its canonical home lfx.utils.lazy_import. - flow-builder tests (build_flow_from_spec, flow_builder_tools, propose_field_edit): specs use LanguageModelComponent (in-tree) instead of OpenAIModel. - test_lfx_bundles_extras: tomli fallback for Python 3.10 (tomllib is stdlib 3.11+; pytest guarantees tomli on <3.11). Full lfx unit suite: 4550 passed. Backend extras+pin tests: 24 passed. * fix(tests): repoint MCP client-server tests off graduated OpenAIModel key Same fail-fast-masked class as |
|||
| 17fd60e929 |
fix: downgrade torchvision until cpu index fix (#13745)
* fix: downgrade torchvision until cpu index fix downgrade torchvision until cpu index fix. currently latest is 0.27.1+df56172 but we want 0.27.1 and 0.27.1+cpu d is viewed as a new version than c * chore: add comment add comment |
|||
| e5d42e54fc |
feat: upgrade Firecrawl to v2 SDK and extract into the lfx-firecrawl extension bundle (#13495)
* Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Upgrade Firecrawl components to firecrawl-py v2 + add Search component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add Firecrawl Search API to the bundle page Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * fix(firecrawl): D205 docstring + defensive .get('data') in crawl Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(firecrawl): D205 docstring in scrape Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(firecrawl): use list.extend in map (PERF401) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: add unit tests for Firecrawl v2 components Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * test: align Firecrawl component tests with lfx _attributes pattern Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * refactor(firecrawl): remove deprecated extract endpoint component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(firecrawl): remove deprecated extract endpoint component Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: drop extract component test (extract endpoint removed) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: remove Firecrawl Extract API section (deprecated) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * fix(firecrawl): map Search component to Firecrawl icon, drop Extract The frontend icon map in styleUtils.ts still referenced the removed FirecrawlExtractApi and lacked the new FirecrawlSearchApi, so the Search node would not render the Firecrawl logo. * chore: regenerate component index for Firecrawl v2 (Search added, Extract removed) * refactor(firecrawl): extract components into the lfx-firecrawl extension bundle Port the firecrawl provider out of lfx.components into a standalone Extension Bundle at src/bundles/firecrawl (distribution: lfx-firecrawl), following src/bundles/PORTING.md: - Move the v2-SDK components (Scrape, Crawl, Map, Search) to src/bundles/firecrawl/src/lfx_firecrawl/components/firecrawl/ and remove the in-tree provider + its lfx.components registration. - Own the firecrawl-py>=4,<5 pin in the bundle; drop it from langflow-base. - Wire the workspace (root pyproject deps/sources/members) and uv.lock. - Append migration-table entries (bare name, both import paths, pre-a slot) for the four classes; FirecrawlExtractApi gets no entries since the v2 SDK removed the extract endpoint and the component was dropped. - Regenerate the component index (firecrawl category removed) and locales/en.json (54 firecrawl keys move out of core). - Bundle-local unit tests + test_pilot_firecrawl_upgrade integration test; update Dockerfile bundle enumeration comments. * fix(lfx): repair migration table entry fused during release-1.11.0 merge The release-1.11.0 back-merge collided the FirecrawlSearchApi legacy_slot entry with the NextPlaidVectorStoreComponent bare_class_name entry, mashing both into a single object with duplicate 'target' keys, populating two of {bare_class_name, import_path, legacy_slot}. That fails the MigrationTable validator (entries.51) and broke the lfx loader tests. Split it into the two intended entries so each populates exactly one key-field. Restores the FirecrawlSearchApi and NextPlaidVectorStoreComponent quads (60 entries total, +16 vs base). Append-only and bare-name guards pass. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Hare <ericrhare@gmail.com> |
|||
| 40d7814739 |
feat: New Bundle support for Multi-vector NextPLAID vector store (#13553)
* nextplaid integrate * doc ids fix * multivec fixes * type check * refactor: ship NextPlaid as the lfx-nextplaid extension bundle Convert the in-tree NextPlaid vector store and its companion vLLM multivector embeddings into a standalone `lfx-nextplaid` Extension Bundle, following src/bundles/PORTING.md (cf. lfx-arxiv, lfx-ibm). The whole feature now ships as an additive bundle with no core modifications. - Move both components into src/bundles/nextplaid (components/nextplaid), dropping src/lfx/.../components/nextplaid and reverting the core vllm __init__ multivector additions. - Bundle declares its own runtime deps (langchain-plaid, pillow) and ships extension.json + the langflow.extensions entry point. - Add migration_table entries mapping the legacy class names / import paths to ext:nextplaid:*@official; wire the workspace (root pyproject, uv.lock). - Add the pilot upgrade integration test and bundle-local unit tests. - Declare explicit outputs on NextPlaidVectorStoreComponent so the extension validator can resolve its output methods. * fix: address CodeRabbit review on NextPlaid bundle - nextplaid: derive stable text IDs from source/page+content (not content alone) and fingerprint raw PIL images by bytes (not batch position) so ingests upsert instead of colliding/overwriting unrelated vectors - vllm impl: validate the /pooling response envelope before nested indexing, raising a clear RuntimeError on malformed text/image responses - icon: use the isDark boolean prop contract instead of the isdark string - test: gate the distribution check on package presence (PackageNotFoundError) so genuine import failures surface instead of being skipped --------- Co-authored-by: Meet <meet@dhcp-9-127-22-227.c4p-in.ibm.com> Co-authored-by: Eric Hare <ericrhare@gmail.com> |
|||
| 7b984c835f | chore: bump version to 1.10.1 | |||
| fb64e45e75 | chore: bump version to 1.11.0 | |||
| 0d9f9112ef |
perf(telemetry): batched off-pool writer for transactions + vertex_builds (#13126)
* perf(telemetry): batched off-pool writer for transactions + vertex_builds
Adds TelemetryWriterService that buffers transaction and vertex_build rows
in memory and drains them in batched INSERTs via a dedicated AsyncEngine
(pool_size=1 for SQLite, 2 for Postgres, max_overflow=0). Retention is
amortized in a 60s sweeper instead of running on every insert.
Producers (log_transaction, log_vertex_build) enqueue instead of opening
a DB session, so telemetry traffic no longer competes with the
request-handling pool. Falls back to the legacy direct-write path via
LANGFLOW_TELEMETRY_WRITER_ENABLED=false.
Durability: in-flight rows spill to a diskcache.Deque per PID on shutdown
and are restored on next startup; orphan PID directories left by crashed
workers are adopted.
* perf(telemetry): tighten error handling and add coverage
Review feedback from pr-review-toolkit + silent-failure-hunter:
- Retention sweep snapshots the dirty-flow sets before commit and only
clears them after it lands; a crashed sweep no longer drops the flows on
the floor, so per-flow caps cannot drift unboundedly.
- Producer fall-through is no longer silent. transaction_service and
log_vertex_build each log a one-shot WARNING when telemetry_writer_enabled
is True but the writer is not running.
- Lifespan startup failure now logs ERROR instead of WARNING — if the user
opted into the writer and it didn't come up, that's an error.
- Shutdown drain timeout no longer suppressed silently: logs a WARNING with
pending row count + a hint to raise telemetry_writer_shutdown_drain_s.
- Writer's flush loop catches asyncio.CancelledError separately and
re-prepends the in-flight batch to the buffer so teardown's disk spill
catches it.
- After 6 consecutive batch failures the writer emits a loud ERROR with
buffer depths so operators see sustained data-loss risk.
Added tests:
- test_sanitization_survives_writer_round_trip
- test_retention_failure_preserves_dirty_flows
- test_in_flight_batch_returned_on_cancel
* perf(telemetry): address copilot review
- _restore_from_disk + _adopt_orphan_outboxes now route through _enqueue so a
large disk-spilled or orphan outbox can't bypass telemetry_writer_max_queue
and OOM the process. Oldest rows are dropped and counted via the existing
dropped_transactions / dropped_vertex_builds counters.
- chmod 0o700 the outbox root + per-PID directory so sanitized-but-still-
sensitive payloads aren't exposed cross-user on multi-tenant hosts.
Suppressed on platforms where chmod is a no-op (Windows).
- Added test_adopt_orphan_outboxes_honors_max_queue.
The remaining two copilot notes (private API access to DatabaseService and
diskcache.Cache.close) were also flagged by the in-tree review; tracking
separately. The "diskcache not declared" note is a false positive — the
dependency is at src/backend/base/pyproject.toml:76.
* perf(telemetry): address coderabbit review
- Sweeper hands off dirty sets via capture-and-clear so concurrent
flushes during a retention pass aren't wiped by the post-commit
subtract; failure path restores the snapshot.
- Stress README uses a concrete Postgres DSN matching the docker
example instead of an unset env var.
- Test PID-probe loops bounded via a shared helper with pytest.fail
fallback.
* perf(telemetry): swap diskcache outbox for stdlib sqlite (CVE-2025-69872)
Replaces the diskcache.Deque-backed spill outbox with a stdlib sqlite3
outbox (WAL mode, JSON payloads in TEXT). diskcache 5.6.3 has
CVE-2025-69872 (pickle deserialization RCE for an attacker with write
access to the cache dir), no fixed version released, and was never
declared as a dependency — import failed on Python 3.10.
The replacement is encapsulated in a small _Outbox helper that owns the
connection, schema, JSON codec, and lifecycle. The codec uses tagged
wrappers for datetime and UUID so SQLAlchemy's typed columns accept
restored payloads on the way back out.
Hardening informed by a survey of OTel collector, Fluent Bit, Vector,
Prometheus remote_write, Datadog agent, Logstash, and Filebeat:
- Per-PID outbox dirs now stamp an owner.json (hostname + Linux boot_id
or time()-monotonic() proxy). Adoption only proceeds when host+boot
match; cross-host or pre-owner-file dirs are logged and skipped so a
recycled PID after a container restart cannot pull in a stranger's
spill data.
- PRAGMA synchronous=FULL on the spill connection so the shutdown
commit hits the platter (NORMAL only fsyncs on WAL checkpoint, which
may never run if the process exits immediately after commit).
- Spill honors telemetry_writer_max_queue with drop-oldest, matching
the producer-side overflow policy so a backlogged buffer at shutdown
can neither stall teardown nor fill disk.
- append_all encodes payloads up front and only clears the deque after
the transaction commits — no more partial-drain on mid-flight SQLite
failure. drain collapses to a single DELETE FROM outbox after the
SELECT.
- Exception handlers at the spill/restore/adopt boundaries narrowed to
(sqlite3.Error, OSError) so genuinely unexpected exceptions
propagate rather than being silently logged.
Tests cover: cross-host orphan skipped, missing-owner orphan skipped,
spill cap drops oldest, and a realistic UUID+datetime payload
surviving spill → restore → SQLAlchemy INSERT.
* [autofix.ci] apply automated fixes
* fix(telemetry): name wait_for inner tasks so pyleak can filter
Integration tests using @pyleak_marker were flagging an inner asyncio
task created by the telemetry writer's `wait_for(Event.wait(), ...)`
loops. The wrapper task is auto-named (Task-N) and lands mid-await
across the test boundary, so pyleak counts it as leaked. The writer
itself shuts down cleanly via teardown; the "leak" is a pattern
mismatch with pyleak's per-test snapshot model, not a real lifecycle
bug.
Wrap Event.wait() in an explicitly named task (`telemetry-writer-tick`,
`telemetry-writer-backoff`, `telemetry-sweeper-tick`) via a small
_wait_or_shutdown helper, and extend pyleak_marker with a task_name_filter
that excludes `telemetry-*` from leak detection. CI was green on earlier
commits in this branch only because the diskcache import error
prevented the writer from starting at all — fixing the import surfaced
this latent pyleak collision.
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* test(telemetry): disable writer in tests so reads see writes synchronously
* perf(telemetry): age out cross-host orphan outboxes on shared volumes
A pod on a shared PV (NFS/RWX) cannot adopt rows from a dead pod on a
different host because the dead pod's hostname doesn't match. Without a
janitor those directories accumulate forever. Add an owner-file mtime
heartbeat from the sweeper plus a prune pass that deletes cross-host
outboxes whose owner file hasn't been touched within
telemetry_writer_orphan_max_age_s (default 1h). Same-host orphans still
flow through the existing adoption path.
* perf(telemetry): add byte-aware flush + drop strategy
Today the writer bounds memory by row count only, so a worker logging fat
vertex_build artifacts can hold tens of MB per row and silently dwarf the
configured max_queue. Add a size_strategy switch ('count' | 'bytes' |
'either', default 'count' for parity) with two byte thresholds:
- batch_size_bytes (256KB) caps per-flush INSERT size when bytes apply
- max_queue_bytes (200MB) drops oldest by bytes when bytes apply
Parallel sizes deques mirror the payload buffers so accounting stays
consistent across drain, spill, restore, and the cancel/retry rollback.
Single rows above the byte budget are still emitted (no row is refused);
operators get dropped_*_bytes counters alongside dropped_*.
* test(telemetry): tighten byte-strategy assertions and cover end-to-end paths
Address gaps in the byte-aware strategy tests:
- pin exact dropped counts and remaining buffer state for the drop-by-bytes
case (was 'greater than zero')
- compute exact drain count from encoded size (was a 1-4 range)
- round-trip bytes through spill+restore to verify size deque is rebuilt
- round-trip bytes through orphan adoption for the same reason
- exercise the sweeper loop end-to-end to confirm heartbeat + prune are
wired in (previously only unit-tested in isolation)
Clarify in the settings docs that the byte caps measure encoded JSON size,
not Python in-memory footprint.
* ref: updates to telemetry writer PR (#13294)
* fix(telemetry-writer): harden error handling and add missing test coverage
Critical:
- teardown() now re-raises CancelledError after awaiting the writer task so the
asyncio cancellation chain propagates correctly on lifespan task kill
- suppress(OSError) on owner file write replaced with explicit error log so
operators know when disk-spilled rows will not be recoverable on restart
High / important:
- Escalation threshold check changed from == to >= so the error log fires on
every failure past the threshold, not just the 6th
- Dead `except OSError` on time.time() - time.monotonic() changed to
`except Exception` since time functions cannot raise OSError
- Removed redundant `from uuid import UUID` inside _run_retention_pass (already
imported at module top)
- Orphan directory cleanup: replaced blanket suppress(OSError) with per-child
suppress so ENOTEMPTY on an individual rmdir doesn't abort the whole loop and
leave the parent directory leaking silently; outer failure now logs at debug
Tests (3 new):
- test_retention_sweep_caps_vertex_builds_per_vertex: inserts 8 builds for a
single vertex with max_per_vertex=3 so the per-vertex DELETE subquery
actually executes (previous test used 8 distinct vertex IDs, bypassing it)
- test_either_strategy_trips_on_bytes_first: verifies bytes can be the first
trigger under 'either' strategy (previous test only covered the count-first path)
- test_writer_retries_on_batch_failure: injects 2 flush failures then success;
confirms failed_batches increments, rows are preserved in the buffer, and
flushed_rows reflects the final successful write
* ci: add stress-tests job to nightly build pipeline
Wires the stress-tests workflow into nightly so telemetry write stress
tests run automatically. Also gates release-nightly-build and
slack-notification on stress-tests result so a stress regression blocks
the nightly release and surfaces in Slack.
* ci: run stress tests in nightly without blocking release
Stress tests are informational for now — a failure is visible in the
workflow run and Slack but does not gate the nightly release or build.
* fix(telemetry-writer): address PR review on cancelled teardown and nightly stress tests
- Add the stress-tests.yml reusable workflow (was untracked, so the
nightly stress-tests job could never resolve)
- Notify Slack on stress-tests failure (add to slack-notification gate
and FAILED_JOB detection as non-blocking)
- Run sweeper cancel, disk spill, and engine dispose in a finally so a
cancelled teardown() still persists the in-memory buffer
- Add tests for the cancelled-teardown spill path and the >= escalation
threshold; drop the misleading wait-loop in the retry test
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
|
|||
| 1ab6251164 |
feat: sync langflow and lfx versions (#13176)
* feat(lfx): synchronize LFX onto Langflow major.minor version line (Phase 1) - Bump src/lfx/pyproject.toml from 0.5.0 to 1.10.0 - Tighten langflow-base lfx pin: ~=0.5.0 → ~=1.10.0 - Fix release.yml pre-release boundary from next-major to next-minor (<X.(Y+1).dev0) - Add minor-parity soft check to scripts/release-lfx.sh - Extend make patch to sync lfx version alongside langflow/base/frontend - Document LFX compatibility contract in RELEASE.md Contract: LFX X.Y.N is compatible with any Flow from Langflow X.Y.M. * feat(lfx/upgrade): add compatibility checker (Phase 2) * fix(lfx/upgrade): fix registry_code guard and types order-sensitivity in checker * feat(lfx/upgrade): implement safe-upgrade applier (Phase 2) * feat(lfx): add lfx upgrade command (Phase 2) * feat(lfx/run): add --upgrade-flow option (Phase 2) * feat(lfx/serve): add --upgrade-flow option (Phase 2) * update lfx pyproject version * [autofix.ci] apply automated fixes * test(lfx/upgrade): add v1.9.0 starter flow fixtures for upgrade integration tests * test(lfx/upgrade): add v1.9.0 starter flow fixtures and real-flow integration tests * [autofix.ci] apply automated fixes * fix(lfx/upgrade): use alias-aware registry lookup so renamed components are not falsely blocked * fix(lfx/upgrade): handle outer flow envelope and file-path inputs in upgrade checks * fix(lfx/upgrade): fail-fast on upgrade_flow errors; add regression tests for Bugs 1-3 * chore(tests): remove bug-number labels from test comments * [autofix.ci] apply automated fixes * fix(lfx/upgrade): address PR review comments - Reject .py files early in serve --upgrade-flow with a clear error message - Preserve outer flow envelope metadata (name, description, etc.) when lfx upgrade --write rewrites a file - Move Mapping import under TYPE_CHECKING to satisfy Ruff TC003 - Assert fixture flows and registry are non-empty so parametrized tests cannot pass vacuously - Remove unused capsys arg and replace print with sys.stderr.write (Ruff T201/ARG001) - Add upgrade_flow param to run command docstring * fix(lfx/upgrade): preserve envelope on run, apply nested upgrades (#13200) * fix(lfx/upgrade): preserve envelope on run, apply nested upgrades Three follow-ups on top of the upgrade tooling: - run --upgrade-flow: re-attach the inner graph to the outer envelope before handing the flow to aload_flow_from_json. Previously the upgrade path unwrapped {"data": ...} and passed the inner dict to the loader, which raised KeyError: 'data'. Adds happy-path tests for envelope and flat file inputs. - applier: recurse into one level of nested flow nodes (node.data.node.flow.data.nodes), matching the checker. Without this, outdated_safe nodes inside grouped components were reported but never written. Adds a regression test. - upgrade --write test: assert the envelope is preserved (name, description, endpoint_name, etc.) instead of the previous unwrapped-output expectation, so the test matches the actual fix in this PR. - Drive-by ruff cleanup: D417 docstring, SIM103, SIM108, E501, RUF059. * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix(lfx/upgrade): fix loader KeyError, outer-envelope checker gap, and nested-node applier Three bugs from ogabrielluiz's PR review: 1. run/base.py loader KeyError: file-path branch unwrapped the outer envelope for the checker but passed the inner dict to aload_flow_from_json, which does flow_graph["data"] unconditionally. Re-wrap as {"data": flow_dict} before the loader call. Applied consistently to all three input paths (--flow-json, --stdin, file-path). 2. run/base.py zero-node checker for --flow-json/--stdin: those paths stored the raw parsed JSON without unwrapping, so a caller passing an exported flow {"name":..., "data":{...}} caused the checker to see zero nodes and silently pass. Now unwrap with raw.get("data", raw) matching the file-path branch. 3. applier.py nested flows never upgraded: the early `continue` on top-level nodes not in safe_ids prevented the nested-node loop from running. Restructured so the nested check fires unconditionally for every top-level node, mirroring checker.py:160-165. * fix(lfx/run): unify outer-envelope unwrap across all --upgrade-flow input paths File-path reads now call raw.get("data", raw) matching --flow-json and --stdin, so the upgrade checker always sees the inner graph. This also removes the has_envelope/re-wrap machinery that was double-wrapping outer-envelope files when passing to aload_flow_from_json. Fix two test assertions that described the old double-wrapped shape. * fix(lfx/upgrade): inject registry into upgrade_command; fix Makefile lfx pin regex - upgrade_command now accepts an optional registry parameter so tests can pass the dict directly instead of mocking load_registry_from_index - Makefile sed regex broadened from \"lfx~=.*\" to match both ~= and >= forms so make patch works after release.yml rewrites the pin - Echo label changed to LFX (synced) to clarify the variable is the shared Langflow version, not a separate LFX-specific value * fix(make/patch): fix langflow-base sed regex and validation grep The dependency in pyproject.toml is "langflow-base[complete]>=X.Y.Z", not the "langflow-base==X" form the original regex expected, so make patch silently left the pin unchanged and the validation step always failed. The sed pattern now matches any extras/operator combination and rewrites to the canonical [complete]>= form; the grep uses -F so the [complete] brackets are treated literally. * test(lfx): add patch regex tests and symmetric safe-mode envelope tests test_patch_regexes.py — 15 tests covering the Python regexes embedded in the Makefile patch target. Exercises all three substitutions (langflow-base pin, lfx pin, version field) against every realistic pin format including the >=X.Y.Z,<dev0 form that release.yml writes. Would have caught the langflow-base==.* vs [complete]>= mismatch before manual testing. test_base.py — two new TestUpgradeFlowOption tests: test_upgrade_flow_safe_envelope_inline_json_loads_successfully test_upgrade_flow_safe_envelope_stdin_loads_successfully Symmetric to the existing file-path envelope test; verifies that --flow-json and --stdin with an outer-envelope flow also pass {"data": inner} to the loader after safe upgrades, not a double-wrapped {"data": outer_envelope}. * fix(lfx/upgrade): address review: compat checker, shared gate, fail-fast registry Checker correctness: - _outputs_are_compatible: drop cosmetic display_name from the breaking check; treat widened output types as safe (flow types must be a subset of registry types), only narrowing breaks downstream edges. - _input_types_contained: stop flagging widened input_types as breaking; keep narrowing as the only breaking case; fix misleading comments. - check_flow_compatibility now recurses fully into nested grouped components (symmetric with the applier) and accepts a pre-built registry lookup. CLI/run/serve: - New lfx.upgrade.cli_gate (UpgradeFlowMode enum, UpgradeFlowError, apply_upgrade_gate) shared by run_flow and serve_command so the two --upgrade-flow paths can't diverge. - run_flow: extract _materialize_flow_dict and route gating through the shared helper. - run/serve --upgrade-flow options typed as UpgradeFlowMode (check|safe choices). - lfx upgrade: load_registry_from_index fails fast when the bundled registry is empty/missing instead of silently marking every node blocked; ASCII-only report output; new --strict flag; build the registry lookup once and reuse it. Docs/tests: - RELEASE.md: migration note for the lfx 0.5.0 -> 1.10.0 version jump. - Regression tests for the checker fixes, the shared gate, fail-fast registry, --strict, and serve --upgrade-flow parity. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * fix(lfx): consolidate flow-envelope handling and fix version-ceiling regex Extract the outer-envelope unwrap/rewrap logic (previously hand-rolled as raw.get("data", raw) with subtly different rules across serve, run, and upgrade) into a single lfx.utils.flow_envelope module with split/merge helpers. This fixes a serve bug where an enveloped flow had its inner graph written bare to the temp file, making the loader's flow_graph["data"] raise KeyError. Also fix release.yml's major.minor extraction: the greedy sed grabbed the upper bound from a range pin (>=1.10.0,<1.11.dev0), drifting the version ceiling up one minor each release cycle. Anchor to the first version instead. * fix(lfx): upgrade-flow gate reads bundled component index, not empty cache The --upgrade-flow=check|safe gate on lfx run and lfx serve rejected every flow as 'blocked'. Both call sites passed component_cache.all_types_dict to apply_upgrade_gate, but that cache is populated lazily after services start, so at gate time it is empty -- an empty registry classifies every node as blocked. The standalone lfx upgrade command was unaffected because it reads the bundled _assets/component_index.json instead. Make the gate own registry loading: apply_upgrade_gate now defaults all_types_dict to None and loads the bundled index (the same source lfx upgrade uses) via a new _load_bundled_registry helper, raising UpgradeFlowError on a missing/empty index so a broken install fails loudly instead of silently blocking every component. Both call sites pass mode= and let the gate load the registry. Existing gate tests mocked component_cache with a populated registry, which is exactly what hid the bug; repoint them at the new _load_bundled_registry seam and add regression tests that do not mock the registry, including an end-to-end run_flow check against a real clean v1.9.0 starter flow. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai> |
|||
| 71cd5ce376 |
feat: improve lfx serve production-readiness (#13121)
* docs: add design spec for lfx serve multi-flow startup and dynamic upload
* docs: add implementation plan for lfx serve multi-flow startup and dynamic upload
* feat(serve): add FlowRegistry for mutable in-process flow management
* feat(serve): add flow_id_from_content for content-based flow ID generation
* fix(test): update streaming tests to use FlowRegistry API
* test(serve): add upload endpoint tests
* feat(serve): add build_registry_from_directory and build_registry_from_paths helpers
* refactor(serve): remove dead _analyze_graph_structure and _generate_dynamic_run_description helpers
* feat(serve): accept directory and multiple file paths at startup; wire registry to serve_command
* [autofix.ci] apply automated fixes
* fix(serve): move traceback import to module level; remove planning docs from branch
* [autofix.ci] apply automated fixes
* fix(serve): deepcopy graph in stream_flow; use json.dumps for error stream payload
* test(serve): verify stream setup error payload is valid JSON with special characters
* [autofix.ci] apply automated fixes
* fix(serve): use load_flow_from_json directly in _load_graph_and_meta for clear error messages
* docs(serve): document multi-worker and persistence limitations on FlowRegistry
* feat(serve): allow starting server with no flows; accept full Langflow flow JSON on upload
- Remove the requirement for a flow path/JSON/stdin at startup; lfx serve
now starts with an empty FlowRegistry when no input is provided, ready
to accept flows via POST /flows/upload/
- UploadFlowRequest now accepts the full Langflow export JSON directly
(extra="allow", id field added) so callers can do -d @flow.json without
wrapping; load_flow_from_json receives the full dict and the flow's own
stable id is used when present
* fix(serve): harden upload endpoint and remove unused verbose_print param
- Pass body.model_dump(exclude={"replace"}) to load_flow_from_json instead of
body.model_dump(); keeps the top-level "data" key the function expects while
excluding the internal replace flag
- Set graph.flow_id after upload so it matches the registry key
- Return 409 Conflict on duplicate flow ID; add replace=true field to allow
explicit overwrite
- Remove unused verbose_print param from _load_graph_and_meta and update callers
- Clarify directory scan is top-level only (non-recursive) in CLI help text
- Add test_upload_full_export_as_body regression test: sends a complete Langflow
export JSON as the request body (curl -d @flow.json pattern) where body.data
has no nested "data" key — would have caught the KeyError('data') regression
* fix(serve): address PR review comments
- Restore .py script support: _load_graph_and_meta now dispatches to
load_graph_from_script for .py paths; CLI validation accepts .json and .py;
help text updated in both _running_commands.py and commands.py
- Fix flow ID collision in build_registry_from_paths: compute a shared common
root from all supplied paths so same-named files in different directories
get distinct UUIDv5 IDs instead of silently overwriting each other
- Harden FlowRegistry.add(): add overwrite=False parameter that raises
ValueError on duplicate IDs; startup paths keep default (collision = error),
upload endpoint passes overwrite=body.replace
- Add regression test: test_same_filename_in_different_dirs_gets_distinct_ids
- Update test_duplicate_add_replaces_graph to reflect new overwrite semantics
* test(serve): add unit tests for .py script path in serve command
- test_load_graph_and_meta_dispatches_to_script_loader_for_py: verifies
_load_graph_and_meta calls load_graph_from_script (not load_flow_from_json)
for .py paths, and that meta.title and relative_path are set correctly
- test_serve_command_accepts_py_file: end-to-end check that lfx serve my.py
reaches the server (exit_code == 0)
- test_serve_command_rejects_unsupported_extension: .txt and other unknown
extensions must exit non-zero with a helpful message
* [autofix.ci] apply automated fixes
* Add --no-env-fallback option to support scoped request_vars
* [autofix.ci] apply automated fixes
* feat(serve): filesystem-backed flow registry for multi-worker sharing
Adds a FlowStore abstraction with two implementations:
- NullFlowStore (default): in-memory only, existing behavior unchanged
- FilesystemFlowStore: persists each flow as {id}.json; writes are
atomic (tmp→rename); reads and deletes are TOCTOU-safe; path
traversal guard rejects unsafe flow IDs
FlowRegistry now delegates persistence to FlowStore:
- add(raw_json=...) writes to store when raw JSON is available
- get() falls through to store on cache miss, reconstructs + caches
- list_metas() merges in-memory and store-side entries
- remove() propagates deletion to store
- warm_from_store() pre-warms the in-memory cache at startup
New --flow-dir option on lfx serve activates the filesystem store.
All uvicorn workers pointing at the same directory serve the same
flows — use /tmp/lfx-flows for single-pod sharing or a PVC mount
for cross-pod sharing. Raw JSON is plumbed through _load_graph_and_meta
and both builder functions so startup-loaded .json flows are persisted
automatically; .py flows remain in-memory only.
* feat(serve): add --workers option for multi-worker uvicorn support
Adds --workers N to lfx serve, enabling multiple uvicorn processes to
share flows via --flow-dir.
serve_command is restructured from @syncify async to plain sync: all
async registry-building is extracted into _build_serve_registry() and
called via asyncio.run(), then uvicorn.run() handles the server.
For workers > 1, uvicorn requires an import string rather than an app
object. The factory create_serve_app() in serve_app.py reads
LFX_SERVE_FLOW_DIR and LFX_SERVE_NO_ENV_FALLBACK from env, pre-warms
from the shared FilesystemFlowStore, and returns a ready FastAPI app.
The parent sets those env vars unconditionally before dispatching.
For workers == 1, the pre-built app object is used directly (no change
to existing single-worker behavior). A warning is emitted when
--workers > 1 is used without --flow-dir since each worker would have
an isolated in-memory registry.
The startup panel flow count is now accurate for single-worker +
--flow-dir by calling warm_from_store() before printing.
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* [autofix.ci] apply automated fixes (attempt 3/3)
* fix(serve): remove no_args_is_help, drop verbose_print param, harden _safe_path
- Remove no_args_is_help=True from serve command so lfx serve with no
arguments starts with an empty registry instead of showing help
- Remove unused verbose_print parameter from create_multi_serve_app and
update all callers (14 call sites across tests)
- Wrap _safe_path's relative_to() check in try/except for a clear error
message; restructure delete() to use try/except/else (TRY300)
- Add TYPE_CHECKING guard for FlowStore annotation in commands.py; wrap
long --workers help string; restore FBT003 noqa on no_env_fallback
- Fix test_env_fallback_skipped_when_flag_true to allow logger's getenv
calls while still asserting the credential variable is never looked up
- Fix ARG002/D205/RUF059 lint issues in test files
* [autofix.ci] apply automated fixes
* fix(serve): resolve merge conflicts and clean up test section comments
* ruff - test updates
* fix(serve): cross-worker delete propagation and HTTP 500 for execution failures
Cross-worker stale cache (multi-worker DELETE bug):
- Add FlowStore.is_persistent to distinguish real stores from NullFlowStore
- Track _store_sourced in FlowRegistry for flows written to a persistent store
- get() re-reads the store for store-sourced flows to detect cross-worker deletes;
evicts and returns None if the file is gone
- list_metas() skips store-sourced flows whose store key is no longer present
- Add _evict() helper to clean up _flows, _store_meta_cache, _store_keys, _store_sourced
HTTP status for execution failures:
- POST /flows/{id}/run now returns HTTP 500 (not 200) when execution raises or
when the flow returns success=false; body is still RunResponse for debug detail
- Updated OpenAPI responses declaration: 500 now maps to RunResponse model
* fix(serve): harden multi-worker registry correctness and safety
- Add FlowAlreadyRegisteredError (ValueError subclass) so upload_flow
catches only the specific race-condition exception rather than any
ValueError from unrelated internals.
- Add _SERVE_ENV_PREFIX constant; rebuild LFX_SERVE_* env var names from
it and add try/finally cleanup so those vars are always deleted after
uvicorn.run() returns in multi-worker mode.
- Reject .py startup files when --workers > 1: Python graphs cannot be
serialized to the filesystem store so workers would silently start with
empty registries for those flows.
- Remove warm_from_store() from build_registry_from_paths/directory; the
builders no longer pre-warm to avoid double-loading when the startup dir
and flow-dir overlap. serve_command still warms for single-worker mode.
- Fix FlowRegistry.__len__ to delegate to list_metas() so store-only
flows (not yet cache-loaded) are counted; this makes the startup panel
and /health flow_count accurate for multi-worker + pre-placed files.
- Fix FlowRegistry.remove() to delete both the primary store key file and
any alternate-keyed file (UUID or stem), ensuring cross-worker DELETE
propagation: after a delete, every worker's next request to that flow
triggers the per-request stale check, reads None from the store, and
returns 404 regardless of which key its cache entry uses.
- Document the per-request stale check in FlowRegistry docstring with
latency guidance for network volumes vs. local tmpfs.
- Add tests for all of the above, including a cross-worker DELETE
propagation scenario validated by a live 2-worker integration test.
* fix(serve): load startup flows in every uvicorn worker via LFX_SERVE_STARTUP_PATHS
Previously, lfx serve flow.json --workers 4 silently dropped startup flows:
the parent loaded them into a NullFlowStore registry that workers never
inherited. Each worker called create_serve_app() fresh and started empty,
returning flow_count: 0 from /health and 404 for the startup flow.
Fix: before calling uvicorn.run() in multi-worker mode, set
LFX_SERVE_STARTUP_PATHS to the JSON-encoded list of resolved startup file
paths (or the inline-JSON temp file path). create_serve_app() reads this env
var and re-loads the startup flows independently in each worker process using
a ThreadPoolExecutor to avoid RuntimeError from asyncio.run() being called
inside uvicorn's already-running event loop.
Startup flows loaded this way live in each worker's in-memory registry.
Uploaded flows still require --flow-dir for cross-worker sharing (existing
warning preserved). The LFX_SERVE_* env vars are cleaned up after
uvicorn.run() returns via the existing prefix-sweep finally block.
Tests added:
- test_serve_command_sets_startup_paths_env_for_multi_worker: verifies
LFX_SERVE_STARTUP_PATHS is set before uvicorn.run() and cleaned up after
- TestCreateServeAppFactory: unit tests for create_serve_app() in all three
modes (empty start, with startup paths, empty paths list)
* fix(serve): fix 3 worker/store consistency issues
Issue 1 — redundant store writes with --flow-dir + startup files + workers > 1:
When flow_dir is set the parent already persisted startup flows to the store.
create_serve_app() was re-loading and re-writing them in every worker.
Fix: create_serve_app() now skips LFX_SERVE_STARTUP_PATHS when flow_dir is
set and relies solely on warm_from_store(). Similarly, serve_command now only
populates LFX_SERVE_STARTUP_PATHS when flow_dir is NOT set.
Issue 2 — .py restriction was too broad:
The error blocked .py + --workers > 1 even without --flow-dir, where each
worker can safely re-execute the .py via LFX_SERVE_STARTUP_PATHS. The error
now only fires when --flow-dir is also set (workers would skip startup paths
and warm_from_store, missing the .py flow entirely since it can't be stored).
Issue 3 — parent built FastAPI app for multi-worker then discarded it:
create_multi_serve_app() was called before the workers > 1 / else branch,
building a full FastAPI app that was never served in multi-worker mode.
Moved into the else branch so it's only built for single-worker.
Tests:
- test_create_serve_app_with_flow_dir_skips_startup_paths_uses_store:
asserts build_registry_from_paths NOT called when flow_dir is set;
flow comes from store via warm_from_store instead.
- test_create_serve_app_without_flow_dir_loads_startup_paths_from_files:
asserts flow IS loaded when no flow_dir.
- test_serve_command_does_not_set_startup_paths_when_flow_dir_set:
asserts LFX_SERVE_STARTUP_PATHS is [] when flow_dir is set.
- test_serve_command_allows_py_with_multiple_workers_no_flow_dir:
asserts .py + workers>1 + no flow_dir is allowed and path appears in env.
* test(serve): remove duplicate test and fix mislabeled test name
- Remove test_create_serve_app_loads_startup_paths: functionally identical
to test_create_serve_app_without_flow_dir_loads_startup_paths_from_files
(same setup, same assertions, added in a prior session).
- Rename test_verify_api_key_header_takes_precedence to
test_verify_api_key_query_param_takes_precedence: the test passes the
correct key as the query_param argument and the wrong key as header_param,
then asserts the query param value is returned — query param has precedence,
not header. The old name was the opposite of what the test actually checks.
* [autofix.ci] apply automated fixes
* fix(serve): fix concurrent request isolation and startup UX
Three bug fixes discovered via live QA:
1. Component.__deepcopy__ produced shallow copies of _outputs_map,
_edges, and _components, causing concurrent requests to share the
same Output objects across graph copies. Output.cache=True by
default: the first request's output.value was served to all
subsequent concurrent requests, producing identical responses for
different inputs. Fixed by deep-copying all three fields with memo.
2. POST /flows/upload/ without an explicit id used uuid5(content),
which is deterministic — two uploads of any flow with the same
nodes/edges structure always generated the same UUID and the second
always returned 409. Fixed by using uuid4() (random).
3. Startup CLI emitted emoji and exclamation-point decoration in the
serve banner and verbose output. Replaced with plain text.
* [autofix.ci] apply automated fixes
* fix(serve): address PR review comments
- Move 409 conflict check before load_flow_from_json+prepare so callers
with an explicit id don't pay the parse cost for a flow that will be
rejected anyway
- Remove unreachable FlowAlreadyRegisteredError catch: no await exists
between registry.get() and registry.add(), so asyncio cannot
interleave another request in that window
- Update FlowMeta.id and UploadFlowResponse.id field descriptions from
stale "Deterministic UUIDv5" to plain "Flow identifier (UUID)"
* test(serve): fix upload duplicate/replace tests after uuid4 change
Both tests relied on uuid5(content) determinism to generate the same
flow ID from identical data. With uuid4, each upload gets a fresh ID,
so duplicate detection was never triggered. Tests now supply an
explicit id field to exercise the 409/replace paths directly.
* fix(serve): address code review suggestions
- Fix test_write_is_atomic to glob for any *.tmp files rather than
checking a hardcoded filename that could never exist
- Remove incorrect noqa: B904 from validate_id_is_uuid (not in an
except block); use `from None` instead
- Add TTL cache (1 s) for FlowStore.list_ids() in FlowRegistry to
avoid one filesystem glob per /health request; mutations invalidate
immediately, warm_from_store and delete-scan bypass it
- Document the UUID/stem dual-key alias invariant on _flows
- Add test verifying create_serve_app() propagates load errors from
the ThreadPoolExecutor startup path (fail fast, not silent empty registry)
* [autofix.ci] apply automated fixes
* fix: lfx serve no_env_fallback, deepcopy cycle, cross-worker delete (#13297)
fix: address concurrency and isolation issues in lfx serve
- Re-apply no_env_fallback to the per-request graph copy in the run and
stream endpoints; deepcopy() drops graph.context so the flag was lost.
- Register the copy in memo before recursive deepcopy calls in
Component.__deepcopy__ so reference cycles are preserved.
- Check the store directly in list_metas() so cross-worker deletes are
reflected immediately instead of after the TTL cache window.
- Remove the redundant upload-conflict test (already covered by
test_upload_duplicate_without_replace_returns_409).
Also mark test secrets with allowlist pragmas and fix two docstring
lint warnings in the touched test file.
* [autofix.ci] apply automated fixes
* feat(lfx): WXO/TRM request-scoped variables for lfx serve (child of #13121) (#13327)
* feat(lfx): WXO/TRM request-scoped variables for lfx serve global_vars
Align lfx serve with the TRM/WXO credential contract on top of #13121:
- Parse LANGFLOW_REQUEST_VARIABLES from global_vars payloads
- ContextVar-backed lookup in VariableService (no os.environ mutation)
- Activate request scope during execute_graph_with_capture
Child of add-workers-and-upload-lfx (PR #13121).
* [autofix.ci] apply automated fixes
* fix(lfx): harden request-scoped variable resolution + add isolation t… (#13347)
fix(lfx): harden request-scoped variable resolution + add isolation tests
Addresses PR review of WXO/TRM request-scoped variables for lfx serve:
- service: honor no_env_fallback in VariableService so model/KB credential
resolution no longer falls back to os.environ under --no-env-fallback;
correct the get_variable precedence docstring (5-step interleaved order)
- request_scope: add a per-request no_env_fallback ContextVar; use precise
Token type; document the None-vs-empty scope contract
- common: activate/reset the no_env_fallback scope from graph.context
- runtime_variables: log (instead of silently swallowing) a malformed or
non-object LANGFLOW_REQUEST_VARIABLES blob; anchor the override comment to
the TRM contract
Tests:
- add request-scope concurrency isolation + no-env-fallback suite
- add execute_graph_with_capture wiring test (graph.context -> ContextVars)
- give TestGraphExecution mock graphs a real context, fixing 11 tests that
broke once the helper began reading graph.context
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
* Update use of stem alias tracking to ensure proper deletes
* fix(lfx): enforce no_env_fallback across all credential resolution paths
Closes credential-isolation gaps in lfx serve and hardens request-scoped
variable handling:
- VariableService: skip the process-wide LANGFLOW_REQUEST_VARIABLES env read
when env fallback is disabled (previously leaked process credentials for an
empty-global_vars request despite --no-env-fallback).
- Unified models: gate os.environ reads in get_api_key_for_provider and the
connection-config fallbacks in instantiation.py (WatsonX URL/project-id,
Ollama base-url, OpenRouter headers) on the no-env-fallback flag.
- run_until_complete: propagate the caller's contextvars into the worker
thread so request-scoped vars and the flag survive the thread hop.
- Request variables: serialize structured (dict/list) values as valid JSON and
drop nulls instead of producing lossy Python reprs or a truthy "None".
- FlowRegistry: extract the duplicated store-alias scan shared by add/remove.
Adds tests for request-scope isolation, credential/env gating, and contextvar
propagation.
* fix(lfx): close no_env_fallback credential leaks and harden serve isolation
- credentials: honor no_env_fallback in get_all_variables_for_provider so served
flows no longer leak process env (was defeating the _env_if_allowed guards in
instantiation.py via pre-filled provider_vars)
- loading: apply the no_env_fallback contract to table load_from_db column
resolution (sibling of load_from_env_vars, previously still read os.getenv)
- variable service: resolve request-scoped vars (exact name + x-langflow-global-var
alias) before env fallback so a caller's per-request credential is never shadowed
by a same-named process env var
- runtime_variables: flatten the LANGFLOW_REQUEST_VARIABLES blob into graph context
for parity between the ContextVar and load_from_db resolution paths
- flow_store: clean up the partial temp file when an atomic write fails
- commands: on multi-worker shutdown remove only the serve env vars we set, not the
whole LFX_SERVE_* namespace
- components: add check_component_env_writes lint banning os.environ writes; stop the
mem0 component from writing the OpenAI key into the process environment
* fix(lfx): contain corrupt flow-store files so one bad file can't crash worker startup
FilesystemFlowStore.read() only caught FileNotFoundError, so a corrupted/partial
{id}.json in a shared store (PVC corruption, an external writer killed mid-write)
raised JSONDecodeError. warm_from_store() calls get() for every store id at startup
with no per-file guard, so a single bad file would crash every worker's startup and
500 GET /flows.
- read(): catch JSONDecodeError/OSError/UnicodeDecodeError, log, return None (treat
as absent) so warm_from_store/list_metas/get skip it and other flows keep serving.
- warm_from_store(): skip+log a flow that fails to reconstruct (corrupt JSON, or valid
JSON referencing a component not available in this build) instead of aborting startup.
- tests: corrupt-JSON read returns None; warm_from_store skips unloadable flows.
* fix test -- request wins over env now
* fix test. mock graph context correctly
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai>
Co-authored-by: Himavarsha <40851462+HimavarshaVS@users.noreply.github.com>
|
|||
| f2d49e5098 | fix: tidy up docling extra reqs | |||
| 5120b7705a |
fix: Move Docling components into bundle (#13442)
* feat: move docling components to bundle * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * chore: wake CI * fix: trim docling chunking extra * Update test_endpoints.py * Update test_pilot_docling_upgrade.py * fix: show friendly component label in build banner for bundle nodes Extension-bundle components have namespaced node ids of the form ext:<bundle>:<ClassName>@<slot>-<uuid>. The flow-build status banner rendered that raw id, which overflowed the fixed-width (530px) container and collided with the elapsed-time counter. Collapse namespaced ids to <ComponentName>-<uuid> (matching the built-in ComponentName-UUID label) via a new getRunningNodeLabel helper; built-in node ids are returned unchanged. Covered by unit tests. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
|||
| e65ced99cc | Merge release-1.9.6 into release-1.10.0 | |||
| d70757976c |
chore: bump toml and package.json versions
bump toml and package.json versions |
|||
| 3e694305f9 |
fix: separate docling chunking dependencies (#13411)
* separate docling chunking * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * fix nitpicks * [autofix.ci] apply automated fixes * remove langchain docling dependency * add docling image description as an optional dep * [autofix.ci] apply automated fixes * fix line too long * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| dbe482c800 |
fix(ci): lockstep langflow-nightly and langflow-base-nightly versions (#13413)
* fix(ci): lockstep langflow-nightly and langflow-base-nightly versions
The nightly pipeline computed each package's .devN number independently
(pypi_nightly_tag.py queried each package's own PyPI history and
incremented separately), while langflow-nightly pins an exact
langflow-base-nightly[complete]==X.Y.Z.devN dependency. Because
langflow-nightly publishes on more nights than langflow-base-nightly,
the two counters drift (live: dev54 vs dev48). Each time base lags a
publish while main succeeds, the newest langflow-nightly pins a base
dev that does not exist yet, so 'uv pip install langflow-nightly'
becomes unsatisfiable and silently falls back to a weeks-old version.
Fix: version the two packages in lockstep. pypi_nightly_tag.py now
derives a single shared dev number from max(dev across BOTH packages'
PyPI histories) + 1 (restricted to the root base_version), so main and
base always get the identical tag and main's exact pin always
references a base version built and published in the same run. The tag
is computed in a single invocation ('both' mode) so the release and
base tags cannot drift across separate calls.
Also hardens tag computation: 404 / network / malformed responses for
either package now contribute nothing instead of crashing, a
base-version bump resets the dev counter cleanly, and non-dev/final
releases never advance it.
Adds scripts/ci/test_pypi_nightly_tag.py (unit tests) and a
ci-scripts-test.yml workflow to run scripts/ci tests on PRs.
* fix(ci): fail closed on non-404 PyPI errors in nightly tag computation
_all_dev_numbers() previously returned [] for ANY failure (network error,
non-404 HTTP status, malformed 200), which is unsafe: _shared_nightly_version()
takes max(dev)+1 across both packages, so a transient lookup failure on the
higher-versioned package silently LOWERS the next tag. E.g. if the langflow-nightly
lookup fails while langflow-base-nightly reports dev48, the script would emit
v1.10.0.dev49 even though langflow-nightly already published through dev54 — the
workflow then force-recreates an old git tag and fails publishing an already-existing
PyPI version.
Fail closed: only a true 404 (package has no releases) contributes []; network
errors, 5xx/non-404 statuses, and malformed 200s now raise so the nightly job aborts
before mutating tags. Adds tests for malformed/5xx/network failures and a direct
regression test for the higher-package-lookup-failure scenario.
* Update scripts/ci/test_pypi_nightly_tag.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Update test_pypi_nightly_tag.py
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
|||
| 39bb3a12ca |
ci: isolate bundle installs from the core release gate; lazy-import ibm-db (#13402)
* ci: isolate bundle installs from the core release gate; lazy-import ibm-db
Two follow-ups to wiring lfx-ibm (and future native-dep bundles) into the
nightly/release pipeline.
1. cross-platform-test.yml: split the monolithic "Install packages from
wheel" step (in both the stable and experimental jobs) into three:
- core install (SDK/LFX + langflow-base) -- release gate
- per-bundle bundle install -- NON-BLOCKING (continue-on-error), reports
per-bundle pass/fail to the GitHub step summary
- langflow main install -- release gate
A bundle whose dependency has no wheel on a tested platform is now reported
as "failed (non-blocking)" instead of failing the whole nightly/release
publish. Local bundle wheels are still installed before main, so langflow
resolves its bundle deps from the freshly-built wheels rather than PyPI.
NOTE: a bundle that langflow hard-depends on is still pulled (and thus
gated) by the main install; to make a bundle optional on a platform, also
add a PEP 508 marker to its dependency line in langflow's pyproject.toml.
2. lfx-ibm: import ibm_db_dbi lazily inside DB2VS.__init__ (db2vs.py) instead
of at module top level, mirroring db2_vector.build_vector_store. ibm-db
ships no linux/aarch64 wheel, so the bundle now imports on that arch (only
the DB2 vector store is inert there; the watsonx components keep working).
Guard the driver-dependent test modules (test_db2vs, test_db2_vector) with
a module-level skip when ibm_db_dbi is unavailable, matching the existing
watsonx test pattern so they no longer error at collection on linux/aarch64.
Verified locally: 59 DB2 tests pass with ibm-db present; with ibm-db absent
the bundle imports cleanly, DB2 use raises a clean ImportError, and the
guarded test modules skip (9 passed, 2 skipped, 0 errors). actionlint adds no
new shellcheck findings vs HEAD; ruff check/format clean.
* fix: Remove long comments for bundle deps (#13403)
* [autofix.ci] apply automated fixes
* ci: revert bundle-install restructure to blocking; add ibm-db import regression test
Addresses two review findings on the lfx-ibm bundle changes.
1. Revert the cross-platform-test.yml bundle-install restructure. The
non-blocking per-bundle step could let the main install satisfy a hard-dep
bundle from PyPI (the index was not disabled), masking a broken local build,
and it never actually protected hard-dep bundles -- main re-gates them
anyway. Restore the original blocking, install-by-path step (local bundle
wheels install before main, so the gate is reliable and PyPI cannot
substitute). Document the real isolation lever for a genuinely
platform-optional bundle -- a PEP 508 marker on its dependency line in the
root pyproject -- in src/bundles/PORTING.md.
2. The module-level skip hid the importability regression the lazy import was
meant to guard. Add test_optional_dependency.py: it simulates ibm-db's
absence and asserts db2vs + the bundle package import and that DB2 use
raises a clean ImportError -- it runs on every platform, including the
x86_64 CI runners. Replace the module-level skips in test_db2vs.py /
test_db2_vector.py with class-level skipif on the DB2-driver test classes
only, so the helper tests and the module-level imports still run (and assert
importability) where ibm-db is absent.
Verified: with ibm-db present, 112 bundle tests pass. With ibm-db uninstalled
(the real linux/aarch64 case, find_spec -> None), 67 pass / 45 skip / 0 errors
-- the regression tests and helper tests run, only the driver-dependent classes
skip. actionlint reports no new findings; ruff check/format clean.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
|
|||
| f8ada15e2a |
feat(components): Add IBM DB2 Vector Store component (#13237)
* feat(ibm): add DB2 Vector Store component Add comprehensive IBM DB2 Vector Store integration for Langflow. Components Added: - DB2VectorStoreComponent: Main vector store component - DB2VS: LangChain-compatible vector store implementation - DB2 security validators: Input validation and sanitization Features: - Vector similarity search (Similarity, MMR, Similarity Score Threshold) - Secure connection handling with input validation - Metadata filtering for complex data types - Duplicate detection support - Batch document ingestion Test Coverage: - 41 unit tests (38 passing, 3 skipped) - Component integration tests - Security validation tests - DB2VS class tests Documentation: - Comprehensive developer guidelines * refactor(db2): simplify component to match Chroma patterns - Remove complex data ingestion (CSV, DataFrame, dict, Message handling) - Add _add_documents_to_vector_store() method matching Chroma - Add similarity_score_threshold search type - Simplify data ingestion from 220 to 45 lines (80% reduction) - Add 5 new tests (similarity_search, mmr_search, search_with_different_types, duplicate_handling, metadata_filtering) - All 41 tests passing (38 passed, 3 skipped) BREAKING CHANGE: Component now only accepts Data objects for ingestion. * ci: fix CI failures - update component index and starter projects - Regenerate component_index.json with DB2 component - Update all starter projects with new component registry - Fix ruff formatting issues * ci: trigger CI re-run to fix merge issue * ci: add merge conflict resolution for component_index.json in autofix workflow Handle merge conflicts in the Update Component Index job by: - Automatically resolving component_index.json conflicts using --theirs strategy - Failing if non-generated files have conflicts (requires manual resolution) - Completing the merge after resolving generated file conflicts This prevents the workflow from failing when component_index.json has conflicts during PR updates, as this file is auto-generated and can be safely regenerated. * fix: Remove DB2SQLComponent reference from IBM components - Removed DB2SQLComponent from __init__.py as the db2_sql.py module doesn't exist - This fixes the test_all_modules_importable test failure - Fixes CI failure in Unit Tests - Python 3.14 - Group 5 * revert: Remove unrelated starter project JSON changes * revert: Remove unrelated CI workflow and formatting changes * docs: add DB2 Vector Store .mdx documentation following Chroma DB structure * fix: address CodeRabbit review comments - add __init__.py docstring and implement score threshold filtering - Added docstring to src/backend/tests/unit/components/__init__.py to fix Ruff INP001 error - Added FloatInput score_threshold parameter (default 0.5) to DB2 Vector Store component - Implemented threshold filtering in similarity_score_threshold search mode - Filters results to only include documents with relevance scores >= threshold * chore: remove auto-generated and unnecessary files from PR - Remove uv.lock (dependency lock file - auto-generated) - Remove src/frontend/package-lock.json (frontend lock file - auto-generated) - Remove src/lfx/src/lfx/_assets/component_index.json (auto-generated by CI) - Remove .secrets.baseline (security baseline - auto-generated) - Remove starter_projects JSON (auto-generated example) - Remove tweaks_builder.py (unrelated test helper) Per DEVELOPMENT.md guidelines, these files should not be committed by contributors. * chore: restore auto-generated files to keep PR focused on DB2 component * fix(ci): update auto-generated files and fix docs build - Update component index with new IBM DB2 components (363 components, 97 modules) - Update frontend package-lock.json - Fix docs build by removing reference to missing image in bundles-db2.mdx Fixes CI failures: - Update Component Index - Update Starter Projects - Test Docs Build * feat(db2): add SSL/TLS support to DB2 Vector Store component - Add SSL/TLS encryption support for DB2 database connections - Add SSL certificate validation and download functionality - Support local certificate files (.crt, .pem, .cer) and URLs - Add optional certificate password support for encrypted keystores - Implement automatic cleanup of temporary downloaded certificates - Add comprehensive error handling and logging for SSL connections - Update component_index.json with new SSL configuration inputs - Update package-lock.json dependencies Security improvements: - Validate certificate paths and file permissions - Support system default CA certificates (recommended for IBM Cloud DB2) - Redact sensitive information in error messages - Clean up temporary files on connection failure This enhancement enables secure encrypted connections to DB2 databases, which is recommended for production environments. * [autofix.ci] apply automated fixes * refactor(db2): minimize metadata storage in DB2 vector store - Clear metadata before storage to reduce unnecessary data (file_path, filename, etc.) - Return only text content during retrieval for cleaner results - Optimize list comprehension for better performance (PERF401, RET504) * test: remove skipped tests from DB2 vector store test suite - Removed version compatibility tests for versions where component didn't exist - Changed file_names_mapping to return empty list for new component - Overrode base class version tests to prevent skips - Fixed linting issues (hardcoded passwords, nested with statements) - All 40 tests now pass with 0 skipped tests * fix(tests): update DB2 vector store test description format Fixes test_component_metadata assertion to match the actual component description format. The test was failing due to line wrapping differences in the multi-line description string. Fixes CI failure in PR #13163 * [autofix.ci] apply automated fixes * chore: trigger CI re-run to resolve flaky test failures * fix(tests): mark OpenAI-dependent test as api_key_required to prevent CI failures * restructure as new bundle package * [autofix.ci] apply automated fixes * fix(ibm): improve DB2 Vector Store component reliability and validation * fix: implement SSL/TLS toggle functionality for DB2 Vector Store - Add update_build_config method to dynamically show/hide SSL certificate fields - SSL certificate path and password fields now only visible when use_ssl is enabled - Improves UX by hiding irrelevant fields when SSL is disabled - Update component_index.json with new component configuration - All 38 existing tests pass --------- Co-authored-by: Dhruv Chaturvedi <dhruv_insights@mac.lkw-in.ibm.com> Co-authored-by: Dhruv Chaturvedi <dhruv_insights@Dhruvs-MacBook-Pro.local> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: priyanshu-krishnan1 <priyanshu.krishnan1@ibm.com> Co-authored-by: Eric Hare <ericrhare@gmail.com> |
|||
| 8c7fe0ac3a |
chore: bump versions
bump package versions |
|||
| 0a5f25f32b |
chore: port Remove the diskcache dependency
port of https://github.com/langflow-ai/langflow/pull/12953 |
|||
| bf7e494260 | chore: add back docling-core extras | |||
| fca26ff79f | chore: address node and jquery issues | |||
| 7f8bd2d432 | chore: update versions and revert dockerfile npm | |||
| 0ba4c3725d | chore: mass dep update | |||
| ec3aea7ff8 |
fix(extensions): discover editable bundles; clean up ticket refs and reload UX (#13219)
* fix(extensions): discover editable installs via langflow.extensions entry-point
`lfx extension list` was silently dropping bundles installed via `uv pip
install -e` / `pip install -e`. Editable installs surface only
`dist-info/` entries in `dist.files`, so the wheel-shaped scan in
`_distribution_manifest_path` finds no `extension.json`. The bundle
pyproject.toml comment already promised an entry-point fallback for this
case, but it was never implemented in discovery.py.
Add `_distribution_manifest_path_via_entry_points` that resolves the
package via `importlib.util.find_spec` (no module body execution) and
locates the manifest under the resulting package directory. The fallback
runs only when `dist.files` produces no manifest, so wheel installs are
unaffected.
Tests:
- editable distribution with dist-info-only files is discovered
- entry-point pointing at an unimportable module yields no record
- wheel-install path never consults find_spec (guards against
re-importing every bundle package on startup)
* chore(extensions): drop internal ticket refs; relax reload --bundle and --all
Two related changes:
1. Strip internal `LE-NNNN` ticket references from the extension source,
bundles, tests, and public docs. The references were not actionable
outside the project and surfaced in user-visible CLI messages and the
author guide.
2. Relax `lfx extension reload` CLI ergonomics now that local discovery
(`discover_all_extensions`) gives us the install map without needing
an HTTP list endpoint:
- `lfx extension reload <ext_id>`: `--bundle` is now optional; when
omitted, the bundle name is resolved from local discovery. Explicit
`--bundle` still wins for cases where the local install isn't
visible to the running server.
- `lfx extension reload --all`: iterates over every locally-discovered
bundle, POSTs reload to each, renders per-bundle status, and exits 1
if any reload fails. Previously hard-errored as "not yet wired".
- `--all` is mutually exclusive with a positional id / `--bundle`
(exit 2 with a clear message).
- Missing both `extension_id` and `--all` exits 2 pointing at both.
Updated `test_reload_requires_explicit_bundle` (which enforced the old
"--bundle is mandatory" behavior) to cover the new resolution paths.
* docs(bundle-api): changelog entries for editable-install discovery + reload CLI
Satisfies the BUNDLE_API surface-change gate for the editable-install
discovery fallback (entry-point lookup in discovery.py) and the relaxed
`lfx extension reload` CLI (`--bundle` optional, `--all` implemented).
No additional behavior change in this commit -- it only documents the
two changes already shipped in this branch.
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Debojit Kaushik <Kaushik.debojit@gmail.com>
|
|||
| e08080507b |
feat: Add langflow-stepflow package (#12015)
* feat: Add langflow-stepflow package Introduces a new workspace package `src/langflow-stepflow/` that ports the Stepflow integration from the Stepflow repository into the Langflow codebase. The package has two submodules: - `translation/`: translates Langflow flow JSON to Stepflow flow definitions (ported from integrations/langflow/converter/) - `worker/`: a Stepflow worker that executes individual Langflow components via lfx (ported from integrations/langflow/executor/) Entry point: `python -m langflow_stepflow.worker` starts the HTTP worker server for use by a Stepflow orchestrator. No changes to existing Langflow code in this commit. * fix: Address review feedback on langflow-stepflow package - Widen Python version to >=3.10,<3.14 to match langflow-base - Fix import sorting (ruff I001) and unused variable (ruff F841) - Replace sk- prefixed test strings to avoid Gitleaks false positives - Remove placeholder step reference resolution in component_tool (orchestrator resolves these before reaching the worker) - Let exceptions propagate from component_tool_executor instead of returning error dicts that look like successful results - Skip secret/password field defaults in tool input schemas - Use LRU-style bounded cache (128) for compiled components - Use asyncio.to_thread for sync component methods to avoid blocking - Fix mutable NamedTuple default (list→tuple) in PlaceholderGraph - Warn and skip deps with missing field mappings instead of silently falling back to "input" (which overwrites on multiple deps) - Tighten _is_data_list to check __class_name__ == "Data" specifically - Add comment explaining intentional teardown-before-init sequence - Add integration test for example flows * chore: Upgrade to Stepflow SDK 0.12.0 and adapt to lfx 0.3+ renames - Bump stepflow-py and stepflow-orchestrator from >=0.10.0 to >=0.12.0 - Remove Python 3.11 environment markers (SDK now supports 3.10+) - Replace server.run() with gRPC pull transport (run_grpc_worker) matching the upstream stepflow-langflow integration pattern - Update Flow serialization from Pydantic model_dump to msgspec.to_builtins - Remove Pydantic ValueExpr/actual_instance unwrapping (now plain dicts) - Add _langflow_type_name() to map lfx 0.3+ class renames (JSON→Data, Table→DataFrame) back to canonical Langflow type names - Fix DataFrame isinstance checks for lfx Table subclass - Add missing README.md, tests/__init__.py, helpers package - Add skip conditions for missing fixture files * change to lfx schema * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * chore: Align langflow-stepflow ruff config with repo and format - Set line-length = 120 to match root pyproject.toml (was 88, causing conflicts between pre-commit format hook and check hook) - Run ruff format across all source and test files - Add pragma: allowlist secret on test fixtures with fake API keys * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * chore: Rebuild component index * [autofix.ci] apply automated fixes * fix(langflow-stepflow): allow Python 3.14 in requires-python Lift requires-python from <3.14 to <3.15 to match the rest of the workspace and the lockfile. Without this, uv sync on Python 3.14 fails the Unit/LFX/Integration test jobs and the Docker build. * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: ogabrielluiz <gabriel@langflow.org> Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com> |
|||
| cc009c9133 |
feat(lfx): Bundle Separation and LFX Extension Framework (#13043)
* feat(lfx): installed-package + seed-directory discovery for production install (LE-1022) Adds the read-only production install path for Modes A, B, and C of the Bundle Separation iteration. Manifest-shipping pip-installed distributions and seed-directory subdirectories are discovered at server startup and registered as Extensions at @official. * discovery.py: walks importlib.metadata.distributions() + the $LANGFLOW_SEED_DIR / /opt/langflow/bundles seed root; produces DiscoveredExtension records and typed errors for malformed manifests / configured-but-missing seed dirs. * registry.py: ExtensionRegistry service with the immutability invariant for installed and seed entries. Mutation verbs (uninstall, disable, enable, install, update_entry) all raise ExtensionImmutableError carrying the typed installed-extension-immutable / seed-directory-immutable code so the invariant is testable today; the CLI uninstall surface ships in B4. * lfx extension list: read-only inspector with text and JSON output for operators inspecting Mode B/C images. * Errors: four new typed codes (installed-extension-immutable, seed-directory-immutable, seed-directory-not-found, duplicate-extension-id) plus snapshot coverage in tests/unit/extension/test_errors.py. * Tests: 165 extension tests pass, including the LE-1022 acceptance cases -- three pip-installed wheels visible at @official, three seed bundles visible at @official, and the parametrized service-layer immutability check across every mutation verb. * Docs: docs/Deployment/deployment-extensions-production.mdx covers the Dockerfile template, k8s deployment notes, the bundle packaging convention (extension.json shipped via package-data), and troubleshooting for the typed error codes. * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (#12967) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * refactor(lfx): clarify broad except, log malformed bundle.json, expand test docstring - Document why _discovery.import_bundle_module catches BaseException (startup-time loader, must surface bad bundles as typed errors rather than abort). - Add debug-level logging when bundle.json is malformed or non-object so a stale-cache footgun is at least observable. - Expand test_skips_re_imported_class docstring so future maintainers don't accidentally weaken the __module__-equality guard if package-style relative imports get added later. Also drops the stale duplicate-distribution claim from the PR description; that code is correctly deferred to LE-1022 along with /all integration. * feat(lfx): wire Extension System into /all, pathsep-split LANGFLOW_COMPONENTS_PATH, emit duplicate-distribution Closes the four AC gaps the previous reviewer flagged on PR #12967: 1. /all integration: get_and_cache_all_types_dict now also calls a new import_extension_components() that loads installed Extensions via load_installed_extensions, loads inline bundles via discover_inline_bundles, and builds frontend-node templates with extension/bundle/extension_version fields stamped on. Failures are logged and skipped per bundle. 2. LANGFLOW_COMPONENTS_PATH is now split on os.pathsep so multi-entry env vars (e.g. /a:/b on POSIX) produce multiple components-path entries instead of one literal non-existent path. Empty segments and missing paths are skipped. 3. duplicate-distribution is a real producer: load_installed_extensions surfaces a typed warning on the winner LoadResult when two distributions share a canonical name, naming every involved manifest path. 4. Manifest-first precedence runtime wiring: new filter_component_entry_points loads each entry-point and only skips ones that resolve to a Component subclass on a manifest-shipping distribution. plugin_routes.load_plugin_routes now applies it so non-component entry-points (route registrars) keep loading per the AC's 'unaffected' promise. Added the previously-missing AC test for same-distribution component+non-component partition. Also makes _distribution_canonical_name defensive against MagicMock test seams. * fix(lfx): register extension components under namespaced ID; promote duplicate-distribution to error Addresses the latest review of PR #12967: P1: /all integration now keys the cache inner dict by LoadedComponent.namespaced_id (ext:<bundle>:<Class>@<slot>) rather than the bare class name. Templates also carry the namespaced_id as an explicit field so consumers that look at the value (not the key) still see the canonical address. This is the form the LE-1020 migration table will rewrite legacy class-name references to. P2: load_installed_extensions now appends duplicate-distribution to result.errors instead of result.warnings, so LoadResult.ok=False when two distributions share a canonical name. The winner's components still appear in result.components so flows already pinned to them keep working; only the conflict status changes. Updated test to assert errors + ok=False; added explicit assertion that the winner's components are still present. * fix(lfx): installed-distribution discovery accepts pyproject.toml manifest form Closes the latest review finding on PR #12967: the installed-distribution scan only looked for extension.json, ignoring distributions whose manifest lives in [tool.langflow.extension] inside pyproject.toml. The AC explicitly treats both as valid manifest forms. _distribution_manifest_path now: - Returns extension.json immediately when present (preserves precedence matching load_manifest's discovery order). - Falls back to pyproject.toml only when extension.json is absent AND the pyproject's [tool.langflow.extension] section is parseable. Validation reuses load_manifest itself so the rule lives in exactly one place; a stray pyproject.toml without the section is correctly ignored. Tests cover: pyproject-only discovery, pyproject-without-section ignored, extension.json wins on collision, end-to-end pyproject load at @official, and pyproject-form manifest-first entry-point suppression. * refactor(lfx): tighten loader invariants, surface silent skips, harden tests Addresses the latest review feedback on PR #12967: Type-level invariants (_types.py): - LoadedComponent.__post_init__ enforces that @extra components must NOT carry a distribution. The reverse (@official without distribution) is permitted because load_extension is also used for dev-mode loads against a working tree before pip install. - LoadResult docstring documents the partial-success contract: components may be non-empty when errors is non-empty (some files imported, others failed). Callers branching on ok get strict success. Silent-failure fixes (_orchestrator.py + settings/base.py): - inline-path-missing: a non-existent / non-dir LANGFLOW_COMPONENTS_PATH entry now produces a typed warning per skipped path so a typo no longer yields zero diagnostics. Settings-layer skip bumped from debug to warning for the same reason. - bundle-json-invalid: a malformed or non-object bundle.json now surfaces a typed warning instead of silently rewriting the user-declared id/version to derived values under the same bundle name. - no-component-subclass gating uses a call-local counter instead of result.errors so the diagnostic stays accurate when a future caller reuses a LoadResult (multi-bundle / batch wrapper scenarios). Test hardening: - test_re_imported_class_is_skipped_via_module_filter rewritten to actually exercise the __module__-equality check via sys.modules injection; previously passed via module-import-failed (relative-import failure), which would silently weaken if package registration changes. - test_user_declared_path_order_is_preserved: AC #8's multi-path order case (distinct bundles in [path_b, path_a]) was unasserted; added. - test_inline_module_import_failure_attributes_identity: AC #10's identity-on-partial-failure was covered for @official but not @extra; added. - test_uses_real_distributions_by_default tightened to assert ep placement (in kept, not in skipped) instead of exact-list equality, so a future Langflow-shipped manifest doesn't silently flip the assertion. bumped 64 -> 175 passing extension tests; 20 backend integration tests still pass. * fix(lfx): malformed pyproject manifests surface manifest-invalid instead of disappearing Closes the latest review finding on PR #12967: a pyproject.toml with a [tool.langflow.extension] section that has missing/invalid required fields was silently dropped because _pyproject_has_extension_section ran full schema validation via load_manifest and returned False on ValueError/TypeError. That conflated 'no section' with 'section malformed'. Fix: detect section presence only. _pyproject_has_extension_section now calls _read_pyproject_extension (TOML parse + key lookup, no schema check). Behavior: - Section absent or pyproject TOML unparseable -> False (treat as regular non-manifest package). - Section present and is a table (valid OR schema-invalid) -> True. - Section present but is not a table -> True; the author intended to declare an extension and load_extension will surface the typed error. This way a typo'd pyproject Extension produces a typed manifest-invalid LoadResult with extension_id attribution, and manifest-first precedence still suppresses its legacy component entry-points -- matching the 'typed load results on success/failure' contract for the supported pyproject manifest form. Tests: two new cases pin the behavior. test_malformed_pyproject_section_ surfaces_manifest_invalid asserts a typed load-failure result with distribution attribution; the second test pins manifest-first suppression for malformed pyproject distributions. * refactor(lfx): emit inline-path-unreadable, document reload contract, trim rot Closes the remaining nits on PR #12967: - inline-path-unreadable (new typed error code): a configured LANGFLOW_COMPONENTS_PATH entry that raises OSError on iterdir (typically permission-denied) now produces a typed LoadResult error carrying str(exc) instead of silently swallowing the message. - duplicate-inline-bundle hint trimmed: removed forward promise about hard-error-in-a-later-release; same actionability, no expiration. - discover_inline_bundles docstring trimmed from three paragraphs to two sentences (per CLAUDE.md anti-multi-paragraph rule). - _distribution_manifest_path docstring trimmed to one-liner. - installed_extension_roots dropped Used-by caller-narration list; manifest_owning_distributions docstring rewritten to call out the shadow-load risk for direct callers and point to load_installed_ extensions for the typed warning surface. - _discovery.import_bundle_module: replaced 'until that lands in a later milestone' rot with a single line ('Absolute imports only between bundle modules; relative imports unsupported.') AND added the single-load-per-process contract note documenting why LE-1018 reload must scrub registry/sys.modules before re-invoking the loader. - load_extension docstring grew a 'Single-load-per-process contract' block telling direct callers not to rely on this function for refresh. Tests: new test_unreadable_path_emits_inline_path_unreadable pins the OSError -> typed-error path. --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) (#12979) * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) Five-stage reload (parallel staging load -> validate -> swap under write lock -> cleanup -> emit) for installed Bundles in Mode A. In-flight flows keep the pre-swap class via existing references; new flows pick up the post-swap class atomically; concurrent reloads on the same Bundle are rejected with reload-in-progress. Adds: * lfx/extension/registry.py -- BundleRegistry with per-bundle reload-in-progress guard and components_index.json writer * lfx/extension/reload.py -- the five-stage pipeline; events emission is stubbed (TODO LE-1017) so the swap mechanics can ship before the events service lands * loader: optional module_namespace param so Stage 1 lands in __reload_staging__.<id> instead of the live _lfx_ext.* namespace * errors: four new typed reload codes (reload-in-progress, reload-bundle-not-installed, reload-bundle-name-mismatch, reload-source-missing) with branch templates and snapshot tests * HTTP: POST /api/v1/extensions/{id}/bundles/{name}/reload, gated by the existing get_current_active_user dependency, returns 409 with a typed body for the in-progress collision case * CLI: lfx extension reload <id> [--bundle <name>] -- HTTP client against the dev server with text/json output and proper exit codes (--all is gated until LE-1019 lands the list endpoint) * tests: 16 reload-pipeline tests covering the AC matrix (rename round-trip, broken-bundle isolation, concurrent readers, in-flight flow, double-reload guard, bundle-name mismatch) plus 9 CLI client tests Mode A only. In Mode B/C bundle changes require a Docker image rebuild and the reload path is not exercised. The events emission in Stage 5 is intentionally stubbed -- the LE-1017 ticket will swap the body of _emit_bundle_reload_event in one place without touching the pipeline core. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [autofix.ci] apply automated fixes * feat(lfx): add `extension init` and `extension dev` CLIs (LE-1016) (#12968) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): add `extension init` and `extension dev` CLIs (LE-1016) The two scaffolding CLIs an Extension author types: - `lfx extension init <target>` writes the basic single-Bundle template (manifest with $schema, README, .gitignore, one Component subclass + a pytest smoke test). AC #1: the generated extension validates clean against LE-1014. AC #2: the generated test file is a valid pytest module that exercises the component's build() method. AC #3: any --template other than 'basic' fails with a typed template-deferred-in-this-milestone error and a non-zero exit. Refuses to scaffold over a non-empty target dir. - `lfx extension dev <target>` validates the local extension, records its absolute path in <config_dir>/extensions/dev_extensions.json, prints reload instructions, and execs `langflow run` (or `python -m langflow` when langflow isn't on PATH). --skip-launch registers without launching (used by tests + external dev-server scripts); --skip-validate lets authors register a known-broken manifest to debug it under the loader. Stack: - Wave 0: error codes (extension-target-exists, extension-target-invalid, local-extension-missing) with format branches and snapshot tests. - Wave 1: lfx.extension.init_template -- pure-data scaffolder, no Typer dependency so the CLI is a thin shell over it. - Wave 1: lfx.extension.dev_registry -- atomic JSON state file under the langflow user-cache dir; helpers for register / list / unregister / load_dev_extensions / dev_extension_component_paths. - Wave 2: lfx.cli._extension_commands gains init/dev subcommands. - Wave 2: langflow.main lifespan hook reads the dev registry after bundle loading and extends components_path with each registered bundle dir, so the existing palette discovery picks up dev extensions. Missing paths surface as local-extension-missing warnings (AC #5) without aborting startup. Tests (84 new, 197 total in tests/unit/extension/): - test_init_template.py: AC scenarios + identifier derivation + deterministic file shape. - test_dev_registry.py: register/list/unregister round-trip, idempotent re-register refreshes timestamp, malformed state file treated as empty, missing-path warning, recovery when path reappears, env-var override precedence. - test_cli.py: AC #1 init->validate, AC #3 deferred templates, --skip-launch registers without launching, --skip-validate short-circuits the pre-flight pass. - test_errors.py: snapshot rows for all three new codes; the every-known-code-has-a-snapshot test enforces parity. LE-1018 (reload) reuses load_dev_extensions; LE-1022 (installed-pkg discovery) shares the components_path extension pattern. AC #4 ("boots Langflow with the extension visible in the palette within 5s") is delivered jointly by `extension dev` (registers + execs) and the lifespan hook (loads on startup). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1016 review feedback Fixes the four HIGH issues + the elevated LOW from the review pass: 1. dev_extension_component_paths now forwards EVERY warning, not just local-extension-missing. Previously a duplicate-component-name (or any future warning code) was silently dropped, hiding real signal from the lifespan hook's logs. 2. Defensive emit when a LoadResult has components but source_path is None. The current loader always sets source_path, but a future hand-built LoadResult could violate that contract; we now surface a typed local-extension-missing error rather than dropping the extension silently. 3. Replaced the fragile ``min(len(parts))`` bundle-root selection with a relative-to-source-path measurement. Handles deep-vs-shallow sibling extensions correctly without depending on absolute path depth. 4. Generated README now documents the langflow/lfx prerequisite under the Develop section so authors know `pytest` requires the lfx environment, not just Python. 5. Forced LANGFLOW_LAZY_LOAD_COMPONENTS=false unconditionally in the `extension dev` exec env (was setdefault, which let a developer's global lazy-loading export silently hide their dev components from the palette and miss AC #4's 5s budget). Plus three MEDIUMs: - sys.modules cleanup in test_generated_test_file_runs_against_generated_component so a later test importing the same dotted path doesn't pick up a stale module from a deleted tmp_path. Component instantiation moved inside the try block because Component.__init__ uses inspect.getsourcefile against self.__class__'s still-live module. - Added regex-drift test that pins the init_template patterns to match manifest.py's so a schema regex change can't quietly produce invalid scaffolded manifests. - Added two new dev_registry tests covering the forward-all-warnings contract and the source_path=None defense. Tests: 201 passing (4 new); ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * fix(lfx): align init template with renamed manifest API After merging feat/extension-loader into this branch, two surfaces fell out of sync with the renamed manifest schema: - init_template generates extension.json with `lfx: {bundle_api: [1]}`, but the validator now requires `lfx: {compat: ["1"]}` per LfxCompat. This made `test_basic_template_validates_clean` fail with manifest-invalid (`lfx.compat: Field required; lfx.bundle_api: Extra inputs are not permitted`). - test_init_template_regexes_match_manifest_schema reads `manifest_mod._BUNDLE_NAME_RE`, but that symbol was promoted to public `BUNDLE_NAME_RE` in |
|||
| 5e9f0c37bc |
Merge branch 'release-1.9.3' into release-1.10.0
# Conflicts: # .secrets.baseline # docs/package.json # pyproject.toml # src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json # src/backend/base/pyproject.toml # src/backend/tests/unit/components/files_and_knowledge/test_retrieval.py # src/frontend/package-lock.json # src/frontend/package.json # src/frontend/src/components/core/parameterRenderComponent/components/modelInputComponent/__tests__/ModelInputComponent.test.tsx # src/frontend/src/components/core/parameterRenderComponent/components/modelInputComponent/index.tsx # src/frontend/src/modals/modelProviderModal/hooks/useProviderConfiguration.ts # src/lfx/pyproject.toml # src/lfx/src/lfx/_assets/component_index.json # src/sdk/pyproject.toml # uv.lock |
|||
| d5ef588e42 | chore: bump versions to 1.9.3 | |||
| a80cf1c649 |
chore: support Python 3.14 (#13085)
* feat: upgrade Docker images to Python 3.14 (experimental)
- Update pyproject.toml: requires-python from <3.14 to <3.15
- Update all 5 Dockerfiles to use Python 3.14:
- Builder stage: python3.12-trixie-slim → python3.14-trixie-slim
- Runtime stage: python:3.12-slim-trixie → python:3.14-slim-trixie
- src/backend/base/pyproject.toml
- docker/build_and_push_base.Dockerfile
- docker/build_and_push.Dockerfile
- docker/build_and_push_backend.Dockerfile
- docker/build_and_push_with_extras.Dockerfile
- docker/build_and_push_ep.Dockerfile
Python 3.14.5 was released on May 10, 2026. Upgrading to the latest
Python version should help reduce CVE vulnerabilities in Docker images.
This is an experimental change to test Python 3.14 compatibility:
- Docker images will use Python 3.14
- CI/CD workflows still test on Python 3.10-3.13
- Nightly build will validate if dependencies work with 3.14
- Can be reverted quickly if issues are found
- Monitor nightly build for failures
- If successful, update CI/CD workflows to add Python 3.14 to test matrix
- If failures occur, revert and investigate compatibility issues
* chore: support Python 3.14
Bump requires-python upper bound to <3.15 across langflow, langflow-base,
lfx, and langflow-sdk, and add 3.14 to CI test matrices so PRs are gated
on 3.14 compatibility.
Conditional pins for transitive deps without 3.14 wheels at the existing
caps:
- onnxruntime: >=1.26 on 3.14 (existing <1.24 cap retained for 3.10)
- faiss-cpu: >=1.13.2 on 3.14 (existing ==1.9.0.post1 retained for <3.14)
* chore: marker-gate IBM watsonx packages for Python 3.14
ibm-watsonx-ai 1.3.x cannot import on Python 3.14: its StrEnum subclasses
override __init__ in a way that conflicts with 3.14's reworked enum
__set_name__ path (TypeError on KnowledgeBaseFieldRole class creation).
Cap ibm-watsonx-ai, langchain-ibm, and the ibm-watsonx-orchestrate-*
extras at python_version<'3.14' until upstream adapts. Watsonx component
imports are already lazy, so this surfaces only at component-access time
on 3.14, where the existing ImportError handler in
lfx.components.ibm.__init__ degrades it gracefully.
test_model_utils.py imports ChatWatsonx at module top to verify
get_model_name resolves model_id; guard that import so the test
module skips on 3.14 instead of breaking collection.
* [autofix.ci] apply automated fixes
* chore: upgrade remaining Docker images to Python 3.14
build_and_push*.Dockerfile already moved to 3.14 in the merge from
feat/upgrade-python-3.14-docker-images. Apply the same bump to the
dev, devcontainer, and lfx Docker images so all in-repo Dockerfiles
share one Python version.
* [autofix.ci] apply automated fixes
* chore: gate 3.14-broken extras to python_version<'3.14'
cuga (and its transitive fastembed -> py-rust-stemmers source build),
altk/agent-lifecycle-toolkit (re-pulls ibm-watsonx-ai which was already
gated), langchain-pinecone, langwatch, ragstack-ai-knowledge-store, and
OpenDsStar all pin themselves below 3.14 upstream. Without a marker
guard, the workspace lock still tries to install them and a Docker
`uv sync` then attempts source builds that require Rust (CI Docker
test failure with py-rust-stemmers 0.1.5).
Add python_version<'3.14' (or <'3.13' for ragstack) to each pin so
those packages are simply omitted from the 3.14 install set until
upstream catches up. uv pip check is now clean on 3.14.2.
* test: skip test_altk_agent on Python 3.14
altk (agent-lifecycle-toolkit) is gated to python_version<'3.14'
upstream and now via the langflow-base [altk] extra marker. The test
imports altk at module top to exercise ALTKAgentComponent; guard the
import so collection skips on 3.14 instead of failing.
* test: skip remaining altk tests on Python 3.14
Three more test modules import lfx.base.agents.altk_* at module top,
which transitively imports altk. Add the same module-level skip guard
as test_altk_agent.py:
- test_altk_agent_logic.py
- test_altk_agent_tool_conversion.py
- test_conversation_context_ordering.py
* fix(calculator): replace removed ast.Num with ast.Constant for Python 3.14
ast.Num was deprecated in Python 3.8 in favor of ast.Constant and
removed entirely in 3.14. The calculator tool and its core walker
crashed on 3.14 with 'module ast has no attribute Num'.
Switch the isinstance checks to ast.Constant + isinstance(node.value,
(int, float)), and drop the now-dead ast.Num backwards-compat branch
in calculator_core.py (the ast.Constant branch already handles every
input it would have matched). Update the parser unit-test fixtures
to construct ast.Constant nodes.
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* test: skip LangWatch HTTP instrumentation tests on Python 3.14
langwatch is gated to python_version<'3.14' upstream. The
TestLangWatchHttpInstrumentation class patches langwatch.setup and
langwatch.trace inside a fixture, which requires langwatch to be
importable; on 3.14 the test errors with ModuleNotFoundError.
Guard the class with a skipif on langwatch availability.
* test: allow IBM and altk components to be missing on Python 3.14
test_all_modules_importable enforces that every component in __all__
imports cleanly. On 3.14 the ibm-watsonx-ai, langchain-ibm, and altk
packages are gated upstream and intentionally not installed, so the
WatsonxAI, WatsonxEmbeddings, and ALTKAgent components fail to import
as designed.
- test_all_components_in_categories_importable: accept a known-gated
deny-list on 3.14 instead of failing.
- test_all_lfx_component_modules_directly_importable: extend the
existing 'missing optional dependency' allowlist with altk,
langchain_ibm, and ibm_watsonx_ai.
* fix(lfx): pass ensure_exists to user_cache_dir for Python 3.14
Python 3.14 tightened PurePath.__init__ to reject unknown keyword
arguments. The KeyedWorkerLockManager constructor was passing
ensure_exists=True to Path() instead of user_cache_dir(), which
silently worked on 3.10-3.13 but raises TypeError on 3.14 and also
meant the cache directory was never actually being created.
Move the kwarg to user_cache_dir() where it belongs and update the
two unit tests that asserted the buggy call shape.
* [autofix.ci] apply automated fixes
* test: accept either ZIP or JSON error path on Python 3.14
Python 3.14's zipfile.is_zipfile() now validates the central-directory
signature in addition to EOCD, so the garbage+EOCD payload no longer
passes the is_zipfile() dispatch in the /flows/upload/ route. The
endpoint still returns 400 with a descriptive detail (now from the JSON
branch); only the message wording differs, which the test was asserting
verbatim.
* fix(lfx): normalize cors_origins ['*'] back to '*' on Python 3.14
Pydantic-settings on Python 3.14 parses the env var '*' into ['*']
before the cors_origins field validator runs (the list[str] | str union
resolves differently than on 3.10-3.13). Collapse that back to the
bare-string wildcard so downstream consumers — including
warn_about_future_cors_changes and the test suite — see the same shape
on every supported Python version.
---------
Co-authored-by: vijay kumar katuri <vijay.katuri@ibm.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
(cherry picked from commit
|
|||
| 1955f8fae5 |
chore: support Python 3.14 (#13085)
* feat: upgrade Docker images to Python 3.14 (experimental) ## Changes - Update pyproject.toml: requires-python from <3.14 to <3.15 - Update all 5 Dockerfiles to use Python 3.14: - Builder stage: python3.12-trixie-slim → python3.14-trixie-slim - Runtime stage: python:3.12-slim-trixie → python:3.14-slim-trixie ## Files Updated - src/backend/base/pyproject.toml - docker/build_and_push_base.Dockerfile - docker/build_and_push.Dockerfile - docker/build_and_push_backend.Dockerfile - docker/build_and_push_with_extras.Dockerfile - docker/build_and_push_ep.Dockerfile ## Rationale Python 3.14.5 was released on May 10, 2026. Upgrading to the latest Python version should help reduce CVE vulnerabilities in Docker images. ## Testing Strategy This is an experimental change to test Python 3.14 compatibility: - Docker images will use Python 3.14 - CI/CD workflows still test on Python 3.10-3.13 - Nightly build will validate if dependencies work with 3.14 - Can be reverted quickly if issues are found ## Next Steps - Monitor nightly build for failures - If successful, update CI/CD workflows to add Python 3.14 to test matrix - If failures occur, revert and investigate compatibility issues * chore: support Python 3.14 Bump requires-python upper bound to <3.15 across langflow, langflow-base, lfx, and langflow-sdk, and add 3.14 to CI test matrices so PRs are gated on 3.14 compatibility. Conditional pins for transitive deps without 3.14 wheels at the existing caps: - onnxruntime: >=1.26 on 3.14 (existing <1.24 cap retained for 3.10) - faiss-cpu: >=1.13.2 on 3.14 (existing ==1.9.0.post1 retained for <3.14) * chore: marker-gate IBM watsonx packages for Python 3.14 ibm-watsonx-ai 1.3.x cannot import on Python 3.14: its StrEnum subclasses override __init__ in a way that conflicts with 3.14's reworked enum __set_name__ path (TypeError on KnowledgeBaseFieldRole class creation). Cap ibm-watsonx-ai, langchain-ibm, and the ibm-watsonx-orchestrate-* extras at python_version<'3.14' until upstream adapts. Watsonx component imports are already lazy, so this surfaces only at component-access time on 3.14, where the existing ImportError handler in lfx.components.ibm.__init__ degrades it gracefully. test_model_utils.py imports ChatWatsonx at module top to verify get_model_name resolves model_id; guard that import so the test module skips on 3.14 instead of breaking collection. * [autofix.ci] apply automated fixes * chore: upgrade remaining Docker images to Python 3.14 build_and_push*.Dockerfile already moved to 3.14 in the merge from feat/upgrade-python-3.14-docker-images. Apply the same bump to the dev, devcontainer, and lfx Docker images so all in-repo Dockerfiles share one Python version. * [autofix.ci] apply automated fixes * chore: gate 3.14-broken extras to python_version<'3.14' cuga (and its transitive fastembed -> py-rust-stemmers source build), altk/agent-lifecycle-toolkit (re-pulls ibm-watsonx-ai which was already gated), langchain-pinecone, langwatch, ragstack-ai-knowledge-store, and OpenDsStar all pin themselves below 3.14 upstream. Without a marker guard, the workspace lock still tries to install them and a Docker `uv sync` then attempts source builds that require Rust (CI Docker test failure with py-rust-stemmers 0.1.5). Add python_version<'3.14' (or <'3.13' for ragstack) to each pin so those packages are simply omitted from the 3.14 install set until upstream catches up. uv pip check is now clean on 3.14.2. * test: skip test_altk_agent on Python 3.14 altk (agent-lifecycle-toolkit) is gated to python_version<'3.14' upstream and now via the langflow-base [altk] extra marker. The test imports altk at module top to exercise ALTKAgentComponent; guard the import so collection skips on 3.14 instead of failing. * test: skip remaining altk tests on Python 3.14 Three more test modules import lfx.base.agents.altk_* at module top, which transitively imports altk. Add the same module-level skip guard as test_altk_agent.py: - test_altk_agent_logic.py - test_altk_agent_tool_conversion.py - test_conversation_context_ordering.py * fix(calculator): replace removed ast.Num with ast.Constant for Python 3.14 ast.Num was deprecated in Python 3.8 in favor of ast.Constant and removed entirely in 3.14. The calculator tool and its core walker crashed on 3.14 with 'module ast has no attribute Num'. Switch the isinstance checks to ast.Constant + isinstance(node.value, (int, float)), and drop the now-dead ast.Num backwards-compat branch in calculator_core.py (the ast.Constant branch already handles every input it would have matched). Update the parser unit-test fixtures to construct ast.Constant nodes. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * test: skip LangWatch HTTP instrumentation tests on Python 3.14 langwatch is gated to python_version<'3.14' upstream. The TestLangWatchHttpInstrumentation class patches langwatch.setup and langwatch.trace inside a fixture, which requires langwatch to be importable; on 3.14 the test errors with ModuleNotFoundError. Guard the class with a skipif on langwatch availability. * test: allow IBM and altk components to be missing on Python 3.14 test_all_modules_importable enforces that every component in __all__ imports cleanly. On 3.14 the ibm-watsonx-ai, langchain-ibm, and altk packages are gated upstream and intentionally not installed, so the WatsonxAI, WatsonxEmbeddings, and ALTKAgent components fail to import as designed. - test_all_components_in_categories_importable: accept a known-gated deny-list on 3.14 instead of failing. - test_all_lfx_component_modules_directly_importable: extend the existing 'missing optional dependency' allowlist with altk, langchain_ibm, and ibm_watsonx_ai. * fix(lfx): pass ensure_exists to user_cache_dir for Python 3.14 Python 3.14 tightened PurePath.__init__ to reject unknown keyword arguments. The KeyedWorkerLockManager constructor was passing ensure_exists=True to Path() instead of user_cache_dir(), which silently worked on 3.10-3.13 but raises TypeError on 3.14 and also meant the cache directory was never actually being created. Move the kwarg to user_cache_dir() where it belongs and update the two unit tests that asserted the buggy call shape. * [autofix.ci] apply automated fixes * test: accept either ZIP or JSON error path on Python 3.14 Python 3.14's zipfile.is_zipfile() now validates the central-directory signature in addition to EOCD, so the garbage+EOCD payload no longer passes the is_zipfile() dispatch in the /flows/upload/ route. The endpoint still returns 400 with a descriptive detail (now from the JSON branch); only the message wording differs, which the test was asserting verbatim. * fix(lfx): normalize cors_origins ['*'] back to '*' on Python 3.14 Pydantic-settings on Python 3.14 parses the env var '*' into ['*'] before the cors_origins field validator runs (the list[str] | str union resolves differently than on 3.10-3.13). Collapse that back to the bare-string wildcard so downstream consumers — including warn_about_future_cors_changes and the test suite — see the same shape on every supported Python version. --------- Co-authored-by: vijay kumar katuri <vijay.katuri@ibm.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| ea29bcc6b9 |
feat: Redis-backed job queue for multi-worker deployments (#12588)
* feat: implement Redis job queue and fakeredis support
- Refactored the job queue service to support Redis-backed management for cross-worker scaling.
- Added environment variables for configuration:
- `LANGFLOW_JOB_QUEUE_TYPE=redis`
- `LANGFLOW_REDIS_QUEUE_DB=1`
- Updated job ownership methods to be asynchronous for improved concurrency handling.
- Enhanced Redis cache service with namespacing via key prefixes.
- Introduced `fakeredis` for in-memory Redis simulation in testin>
- Added comprehensive unit tests for Redis job queue components.
* fix: run experimental warning for RedisCache usage only once
- Introduced a mechanism to emit a one-time warning for the RedisCache experimental feature during server runtime.
- The warning is logged only if no other worker has already emitted it, ensuring clarity for users regarding the experimental status of RedisCache.
- The implementation includes a temporary file check to prevent multiple warnings across different processes.
* docs: document environment variables for worker management
- Added documentation for LANGFLOW_GUNICORN_PRELOAD to explain preloading for better performance.
- Detailed the use of LANGFLOW_JOB_QUEUE_TYPE for specifying backends (e.g., Redis).
- Included LANGFLOW_REDIS_QUEUE_DB to define the database index for job queues.
- Updated the "High-Load Environments" guide with these optimal configurations.
* docs: updated 'High-load and multi-worker environments' section
* feat: enhance RedisJobQueueService with consumer wrapper management
- Introduced a caching mechanism for Redis stream consumers to optimize job data retrieval.
- Added methods to manage consumer wrappers, ensuring they are reused across sequential polls.
- Implemented cleanup logic to cancel and clear consumer wrappers during job cleanup and service stop.
- Expanded unit tests to verify consumer wrapper reuse and cleanup behavior.
* fix: ensure Redis keys are deleted during job cleanup even on cancellation
- Updated the cleanup_job method in RedisJobQueueService to guarantee Redis keys are removed even if the job cleanup is interrupted by a CancelledError.
- Added a new unit test to verify that Redis keys are deleted correctly when cleanup is called during task cancellation.
* fix: manage connection check task in RedisJobQueueService
- Added handling for the connection check task in the stop method to ensure it is properly cancelled and awaited if still running.
- This change improves resource management and prevents potential issues during service shutdown.
* fix: handle unpublished sentinel requeue on cancellation in RedisJobQueueService
- Updated the job processing logic to ensure that if a job is cancelled during the xadd operation, the unpublished sentinel is requeued instead of being dropped.
- Introduced a new unit test to verify this behavior, ensuring robustness in job handling during cancellations.
* fix: improve Redis job queue service with enhanced configuration and cleanup
- Added atexit cleanup to remove stale temporary files for RedisCache.
- Refactored Redis job queue service to use shared constants for stream prefixes, improving maintainability.
- Updated type hints for better clarity and consistency in RedisQueueWrapper and RedisJobQueueService.
- Enhanced error handling with configurable backoff for transient read failures.
* fix: enhance Redis job queue service with maxlen configuration for xadd
- Updated the xadd method in RedisJobQueueService to include maxlen and approximate parameters, improving stream management and preventing excessive memory usage.
* fix: enhance job ownership retrieval in RedisJobQueueService
- Updated the get_job_owner method to refresh the Redis key TTL on successful lookups, ensuring long-running jobs maintain their ownership anchor.
- Improved code clarity by extracting the owner key into a variable and adding detailed docstring explanations for better understanding of the TTL management.
* fix: improve job ownership handling in cleanup_job method
- Enhanced the cleanup_job method in RedisJobQueueService to accurately capture job ownership before deleting Redis keys, preventing potential data corruption in multi-worker scenarios.
- Added comments for clarity on ownership logic and its implications during job cleanup.
* fix: optimize TTL management in RedisJobQueueService
- Introduced periodic TTL refresh logic in the _bridge_to_redis method to enhance Redis stream management, reducing round-trips and improving throughput.
- Added constants for TTL refresh events and seconds to maintain clarity and configurability.
- Updated event handling to ensure TTL is refreshed appropriately based on event count and time elapsed.
* fix: improve event handling in flow response management
- Removed unnecessary error handling for missing event tasks in get_flow_events_response, allowing for smoother operation when no task exists.
- Updated create_flow_response to handle optional event_task parameter, ensuring proper cleanup during disconnections.
- Added unit tests to verify behavior when event tasks are missing, enhancing robustness in streaming scenarios.
* fix: enhance RedisQueueWrapper with startup grace period and stream observation
- Added a startup grace period to prevent premature end-of-stream signals when the producer has not yet issued its first XADD.
- Introduced a flag to track whether the stream has been observed, improving the handling of early polling scenarios.
- Updated logic to ensure proper handling of stream existence checks and logging for better debugging during job processing.
* fix: implement cleanup for old cross-worker job queues in RedisJobQueueService
- Added a new method to clean up done cross-worker consumer wrappers that are not owned by the current worker, ensuring proper resource management.
- Enhanced the existing cleanup logic to prevent memory leaks by explicitly pruning stale entries from the consumer wrappers dictionary.
- Improved logging to provide better visibility into the cleanup process for cross-worker jobs.
* fix: enhance job handling in JobQueueService with guarded task execution
- Updated the start_job method to accept a Coroutine type for task_coro, ensuring type safety.
- Introduced a new _guarded_task method to wrap job coroutines, guaranteeing that unhandled exceptions emit an error event and write a sentinel to the Redis Stream, improving reliability in job processing.
- Enhanced documentation to clarify the behavior of the new task handling mechanism and its implications for cross-worker consumers.
* fix: improve client disconnection handling in create_flow_response
- Added logging for scenarios where a client disconnects without an associated event_task, clarifying that the producer will continue running until the build completes.
- Documented the limitation regarding cross-worker passive disconnects and the need for a Redis side-channel for proper cancellation, enhancing observability in the logs.
* fix: enhance RedisQueueWrapper to manage initial read state and buffer behavior
- Introduced a flag to track the completion of the first XREAD call, ensuring proper buffer management during the initial read phase.
- Updated the empty method to reflect the state of the buffer accurately, preventing premature exits from the drain loop until the first read is complete.
- Improved documentation to clarify the behavior of the new flag and its impact on job processing.
* fix: clarify RedisQueueWrapper behavior in tests for buffer state and no-op operations
- Updated test documentation to explain the behavior of the empty() method in relation to the first XREAD completion, ensuring accurate understanding of buffer state during job processing.
- Adjusted assertions in tests to reflect the intended behavior of the RedisQueueWrapper, specifically regarding the no-op nature of put_nowait and its impact on the internal buffer.
* fix: update dependencies in pyproject.toml and uv.lock
- Added fakeredis dependency with a minimum version of 2.0.0 to both pyproject.toml and uv.lock files.
- Ensured proper formatting and comments in pyproject.toml for clarity on onnxruntime version constraints.
* fix: enhance RedisQueueWrapper and job queue handling for improved cancellation and buffer management
- Introduced a structural protocol `_CancellableQueue` to ensure queues can handle cancellation properly during client disconnects.
- Updated `RedisQueueWrapper` to implement this protocol, allowing for graceful cancellation of background tasks.
- Added a maximum size limit to the internal buffer to prevent unbounded memory usage and ensure backpressure on slow consumers.
- Implemented a done callback to handle unexpected fill task crashes, ensuring consumers are not left hanging indefinitely.
- Enhanced unit tests to verify compliance with the new protocol and the behavior of the buffer under various conditions.
* fix: RedisQueueWrapper sentinel delivery, error time-bound, and cancel response
- Deliver end-of-stream sentinel on fill-task cancellation (_on_fill_done now
handles both cancelled and exception paths so consumers are never left hanging)
- Add _error_start time-bound to xread and exists() error loops: after
_STARTUP_GRACE_S seconds of continuous Redis errors the sentinel is delivered
instead of retrying forever
- Advance _last_id cursor only after buffer.put() succeeds so cancellation mid-put
does not silently skip that message in the Redis cursor
- Return False from cancel_flow_build when event_task is None (cross-worker path)
so the HTTP response correctly reports success=False instead of false success
* [autofix.ci] apply automated fixes
---------
Co-authored-by: ogabrielluiz <gabriel@langflow.org>
Co-authored-by: Jordan Frazier <jordan.frazier@datastax.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
|
|||
| b54bdf318d |
fix: update security dependencies (#13053)
* chore: update security dependencies - Update brace-expansion to ^5.0.5 in docs - Update picomatch to ^4.0.4 in docs - Update ip-address to ^10.1.1 in frontend - Update GitPython to >=3.1.48 in backend - Add protobuf constraint >=6.33.6,<7.0.0 in backend * chore: address additional CVEs and add security documentation - Update lodash from deprecated 4.18.0 to 4.17.21 - Add docs/SECURITY_OVERRIDES.md documenting all CVEs - Revert mem0ai 2.x upgrade due to compatibility issues * fix: update dependencies to address CVE vulnerabilities - Update langchain-core to >=1.3.3 (fixes CVE-2026-44843) - Update GitPython to >=3.1.50 (partially fixes CVE-2026-44243, CVE-2026-44244, GHSA-mv93-w799-cj2w) - Update pyarrow constraint to >=23.0.1,<24.0.0 (fixes CVE-2026-25087) - Add override-dependencies for transitive packages: - lxml >=6.1.0 (fixes CVE-2026-41066) - mako >=1.3.12 (fixes CVE-2026-44307) - urllib3 >=2.7.0 (fixes CVE-2026-44431, CVE-2026-44432) - python-liquid >=2.2.0 (fixes CVE-2026-45017) Total: 9 CVEs addressed Smoke tests: All imports successful, no breaking changes detected --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> |
|||
| 6d2e255f5d |
Merge remote-tracking branch 'origin/release-1.9.2' into release-1.10.0
# Conflicts: # .secrets.baseline # pyproject.toml # src/backend/base/langflow/initial_setup/starter_projects/Knowledge Retrieval.json # src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json # src/backend/base/pyproject.toml # src/frontend/package-lock.json # src/frontend/package.json # src/lfx/pyproject.toml # src/lfx/src/lfx/_assets/component_index.json # src/sdk/pyproject.toml # uv.lock |
|||
| 5bf73714e3 | chore: Remove the diskcache dependency (#12953) | |||
| 9d94f9b7d0 |
fix: update playwright to 1.59.0 (#12947)
chore: update playwright to 1.59.0 - Update playwright dependency from 1.58.0 to 1.59.0 - Sync uv.lock with updated dependencies Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> |
|||
| 7d4d3e13a3 | chore: bump versions to 1.9.2 (#12898) | |||
| b9e81f6edd | Merge remote-tracking branch 'origin/main' into release-1.10.0 | |||
| 0732423f27 |
chore: security patch (#12725)
* chore: security patch security patch * chore: upgrade package-lock.json * chore: smolagents and transformer update * chore: redis upgrade * chore: litellm upgrade * fix: Pin click to avoid lower versions ## Root cause **litellm 1.83.5+** introduced an exact pin `click==8.1.8` in its `requires_dist` (upstream bug [BerriAI/litellm#26154](https://github.com/BerriAI/litellm/issues/26154) — "Pinning exact dependency versions breaks downstream consumers"). When this branch bumped litellm to 1.83.11, uv was forced to downgrade click from 8.3.2 → 8.1.8. Click 8.2+ captures stderr separately by default; 8.1 merges it — hence `ValueError: stderr not separately captured`. ## Fix - [pyproject.toml:155](pyproject.toml:155) — added `"click>=8.3.0"` to `[tool.uv] override-dependencies` with comment linking the upstream issue. - [src/backend/base/pyproject.toml:112](src/backend/base/pyproject.toml:112) — same override so `langflow-base` Docker builds pick it up too. - Regenerated `uv.lock` (click: 8.1.8 → 8.3.3). - Patched the `click` entry in `src/backend/base/uv.lock` to 8.3.3 (that lockfile is used by `docker/build_and_push_base.Dockerfile` via `uv sync --frozen`; uv treats backend/base as a workspace member so I updated its click block directly — no other references to 8.1.8 remain). - Reverted [src/lfx/tests/unit/cli/test_run_real_flows.py](src/lfx/tests/unit/cli/test_run_real_flows.py) back to its original `result.stderr` usage — now passes cleanly with click 8.3.3. All 19 tests in the file pass. * Update build_and_push_backend.Dockerfile The `uv pip install` command doesn't read `[tool.uv] override-dependencies` from the target packages' pyproject.toml — those overrides only apply during `uv lock`/`uv sync`. The fix is to pass the overrides explicitly via `--overrides`. Now let me verify it works by running the same install locally. Resolution succeeded — click==8.3.3, aiohttp==3.13.5, litellm==1.83.11 all coexist. Let me also test without `--overrides` to confirm I reproduce the original failure. Reproduced the Docker failure locally. Now test with the overrides file: Fix verified — resolution succeeds with the overrides file, failing without. Let me clean up and show the final Dockerfile change. ## Summary **Root cause:** The Dockerfile uses `uv pip install ./src/...` which **does not read** `[tool.uv] override-dependencies` from the target packages' `pyproject.toml` — those overrides only apply during `uv lock`/`uv sync`. On Linux, `cuga>=0.2.20` requires `aiohttp>=3.13.5`, but `litellm==1.83.11` exact-pins `aiohttp==3.13.3` (same upstream bug as the click pin, [BerriAI/litellm#26154](https://github.com/BerriAI/litellm/issues/26154)). Without an active override, resolution fails. **Fix:** Write a `/tmp/uv-overrides.txt` file mirroring the workspace's `override-dependencies` (litellm, python-dotenv, openai, aiohttp, click) and pass `--overrides /tmp/uv-overrides.txt` to `uv pip install`. Verified locally by reproducing the exact failure in an isolated tmp directory (to escape the workspace's pyproject.toml auto-apply), then confirming the overrides file resolves it: - aiohttp==3.13.5 ✓ - click==8.3.3 ✓ - litellm==1.83.11 ✓ - cuga==0.2.22 ✓ * Templates version update * Update .secrets.baseline * chore: update overrides * chore: bump version * chore: bump main version * chore: bump SDK version to 0.1.1 * chore: run uv lock and uv sync after SDK version bump * chore: read the overrides from a single source security patch * chore: add pip check toggle add pip check toggle * chore: litellm base uv.lock update * fix(chore): Pin litellm back to last working release * fix(chore): Pin to 1.83 and higher litellm * Update build_and_push_backend.Dockerfile * chore: update base uv.lock * fix: lazyload toolguard since its an optional extra * Update .secrets.baseline * Update component_index.json * Rebuild component index * Update component_index.json --------- Co-authored-by: Eric Hare <ericrhare@gmail.com> Co-authored-by: vijay kumar katuri <vijay.katuri@ibm.com> (cherry picked from commit 60a8f76c3fde17124057626c671ea8162872837a) |
|||
| b398f63a09 |
test: add missing test coverage for wxo deployments (#12689)
* test(deployments): add comprehensive test coverage for WXO deployment schemas, hooks, and components
Frontend (11 new/expanded test files, ~170 tests):
- use-connection-panel-state: tests for all handlers, ID sanitization, env var management
- step-review: tests for summary display, flows, connections, masking, edit mode
- global variable hooks: tests for useGetGlobalVariables auth guard and Zustand hydration
- use-patch-deployment: fix incorrect spec field assertion, test URL/body destructuring
- deployment-expanded-row, deployment-info-grid, deployment-flow-list: component tests
- flow-list-panel, version-panel, connection-search-list: panel component tests
- deployment-stepper-payload-builders: tests for buildConnectionPayloads,
buildDeploymentUpdatePayload, buildDeploymentPayload, buildProviderAccountPayload
Backend (2 expanded test files, ~21 new tests):
- test_deployments_response_mapping: tests for update/list/execution/config response shapes
- test_deployment_schemas: tests for pagination constraints, execution schemas, snapshot schemas
* Fix nested button issue; remove more unnecessary tests
* update fe tests
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* [autofix.ci] apply automated fixes
* fix tests; filter sql warnings
* ruff
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Viktor Avelino <64113566+viktoravelino@users.noreply.github.com>
(cherry picked from commit
|
|||
| 9d84ec14d9 |
chore: version bump 1.10.0
version bump langflow: 1.10.0 langflow-base: 0.10.0 lfx: 0.5.0 langflow-sdk: 0.2.0 |
|||
| da516b7045 | Merge release branch | |||
| 71fbf5ff00 |
docs: cut version 1.9.0 (#12681)
* version-1.9.0-docs * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> |
|||
| 19df4606ae |
chore: update deps (#12657)
* chore: update deps update deps due to sec vuln * chore: langchain-core>=1.2.28 langchain-core>=1.2.28 * chore: update pypdf pypdf 6.10 * chore: pyarrow, openai, litellm, mem0ai, toolguard picomatch, pyarrow, openai, litellm, mem0ai, toolguard * chore: override litellm * chore: match base uv.lock to root * chore: update tool.uv position update tool.uv position * chore: langflow-base[complete]>=0.9.0 langflow-base[complete]>=0.9.0 * chore: revert pyarrow revert pyarrow * chore: revert "lfx~=0.4.0", * chore: lfx white space * chore: remove allow-direct-references = true * chore: uv lock --upgrade after merge |
|||
| 95d4c94ef9 |
fix: upgrade playwright to 1.58.0 to address Chromium CVEs (#12668)
* fix: upgrade playwright to 1.58.0 to address Chromium CVEs - Add playwright>=1.58.0 to override-dependencies in pyproject.toml - Update uv.lock: playwright 1.49.0 -> 1.58.0, pyee 12.0.0 -> 13.0.1 - Fixes CVE-2026-2313, CVE-2026-2314, CVE-2026-2315, CVE-2026-2319, CVE-2026-2321, CVE-2026-2441, CVE-2026-2648, CVE-2026-2649 - Ensures Docker builds download updated Chromium with security patches * fix: update npm to latest version to address brace-expansion CVE-2026-33750 - Add npm update after Node.js installation in Dockerfile - Fixes CVE-2026-33750 in system npm's brace-expansion dependency - System npm had brace-expansion 2.0.2, update gets 5.0.5+ - Low risk change: npm is backward compatible, only affects CLI tool * revert: remove npm update from Dockerfile - npm update attempts were causing CI build failures - Bundled npm has issues but updating it is proving problematic - Focus on playwright CVE fix which is the primary concern - brace-expansion CVE-2026-33750 is lower priority (DoS only) --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> |
|||
| 34886de048 |
fix: Updated Pillow minimum version (#12609)
* fix(frontend): enforce brace-expansion override * fix: enforce handlebars version >=4.7.9 in frontend overrides * fix: update Pillow minimum version to 12.1.1 --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> |
