A partial add_message re-fire (the agent path emits these at tool
start/end for a message it is still streaming) was treated as the
finalizer: it closed and tombstoned the id, so the post-tool answer
was dropped. Only a non-partial add_message finalizes now; state
defaults to complete, so payloads without properties are unchanged.
remove_message now purges a buffered message and tombstones its id,
so text the backend retracted is not flushed to the client later.
Parallel components stream tokens for different message ids interleaved.
The translator tracked a single open message: the first foreign token
closed the open message and tombstoned its id, so every later event for
it was dropped and its remaining text never reached the client.
Tokens for a message that cannot take the wire now buffer until the open
message genuinely ends (its add_message finalizer), then flush in arrival
order; complete messages landing mid-stream buffer the same way instead
of interleaving a second START. end/error drain all buffers before the
terminal event. The wire still carries at most one open text message, so
the stream stays AG-UI-conformant.
Port the 6 background-execution settings and background_backend_is_scaled into 1.11.0's RuntimeSettings group mixin; union the BACKGROUND_EXECUTION_SERVICE / TELEMETRY_WRITER_SERVICE additions in schema.py and deps.py.
Adopt 1.11.0's composed group-mixin Settings for lfx settings/base.py (all v2 settings already present in the new groups); port the Python 3.14 cors_origins ['*']->'*' fix into groups/security.py.
fix(ci): allow pre-releases when pinning the nightly in migration validation
Migration Test: pip/venv (stable -> nightly) failed deterministically on
nightly run 27260425158 (twice, including a rerun):
hint: langflow-base was requested with a pre-release marker (e.g.,
langflow-base==1.11.0.dev1), but pre-releases weren't enabled
(try: --prerelease=allow)
This is the first nightly publishing as a canonical .devN pre-release
of the langflow distribution (nightly -> stable bundle cutover). The
'latest' branch of the upgrade step already passes --prerelease=allow,
but the pinned-version branches do not. uv implicitly allows the
pre-release for the directly requested ==X.Y.Z.devN pin, yet langflow's
metadata pins langflow-base==X.Y.Z.devN transitively, and uv rejects
transitive pre-releases unless they are enabled - so the install fails
after the stable uninstall, sinking the migration test.
Add --prerelease=allow to both pinned-version install lines.
Integration tests failed twice in nightly run 27260425158 with pyleak
EventLoopBlockError - first Integration Tests 3.14, then 3.12 on the
rerun, each time in a different test. The blocking stack points at
refresh_models_dev_periodically: every app boot unconditionally starts
a lifespan task that immediately fetches https://models.dev/api.json,
so the request lands mid-test in whatever test happens to be running.
Under pyleak's asyncio debug instrumentation the fetch blocked the loop
0.797s against a 0.2s threshold. Whichever test draws the short straw
flakes - which is why it looked transient and moved between versions.
Add a LANGFLOW_MODELS_DEV_REFRESH env gate (default unchanged: enabled)
and disable it session-wide in the backend test conftest. Tests fall
back to the bundled static model lists, which is also deterministic.
Verified: with the gate set, app boot makes zero models.dev requests;
the previously failing integration test passes.
fix(ci): anchor langflow-base version extraction in nightly docker build
The first nightly on the 1.11 workspace (tag v1.11.0.dev0, run
27253229568) failed in Build Nightly Base Package within seconds:
'Base version format is incorrect'. The extraction (uv tree, grep
langflow-base unanchored, awk field 3, first line) broke because uv
tree now prints the langflow-base workspace-root line
('langflow-base v1.11.0.dev0', two fields) before any dependency line
('langflow-base[complete] v1.11.0.dev0' under the langflow root, three
fields), so the first match yields an empty field 3 and the format
check exits 1. uv tree's stderr is discarded, which made the real
cause invisible in CI.
Anchor to the root line and take field 2 instead - the exact pattern
the langflow (main package) extraction in this same file already uses,
which is why the main-package builds passed on the same tag. Verified
both extractions print 1.11.0.dev0 against a local checkout of
v1.11.0.dev0.
The test spawns a child via multiprocessing spawn context, which
cold-imports the full langflow package (plus coverage's multiprocessing
hooks in CI) before appending a single event. The 10s join timeout is
routinely exceeded on a loaded CI runner sharing 4 vCPUs with a second
xdist worker: in nightly run 27253229568 (Unit Tests - Python 3.12 -
Group 5) the test failed all 12 executions (5 reruns x 2 step attempts),
each rerun exactly 10s apart - the join deadline, not a product bug.
Raise the liveness bound to 60s (join returns immediately when the
child exits, so the passing case is unaffected) and kill the child on
timeout so a hung spawn can't leak into later tests.
Unit Tests - Python 3.12 - Group 3 has been failing at 98-99% on the
nightly: the job hits the nick-fields/retry 40-minute per-attempt timeout
and pytest is SIGTERM'd mid-test, so it looks like one flaky last test
when it is actually a deterministic timeout (and the internal retry plus
run-level retries can never succeed).
Root cause is twofold:
1. .test_durations was last regenerated ~May 2025 and covered only 2,219
of ~9,500 current unit tests, so pytest-split weighted 77% of the
suite at the 0.73s average. The expensive client-fixture tests
(test_webhook.py, test_login.py - each pays a full create_app +
lifespan boot per test, 60-120s late in a CI run) clustered into
group 3's tail, making it ~8-10 minutes slower than its siblings
(33:15 on 3.13, >40min on 3.12 which is additionally slowed by
astrapy disabling SSL connection reuse on Python 3.12.0-11).
The weekly store_pytest_durations workflow that should refresh the
file has been dying at the 6-hour GitHub job limit every week (serial
full-suite run no longer fits), so the file silently froze.
This regenerates the file from real per-test wall-clock measured in
the passing Python 3.13 nightly jobs (run 27246228762 attempt 1, all
5 groups, parsed from the -vv xdist logs: per-worker start-to-start
deltas). 9,508 tests now have measured durations; old entries are
kept where no new measurement exists. Simulated least_duration
split goes from one outlier group to 5 even groups (~24.6 min each)
with the >50s tests spread 1-2 per group instead of 7 in one.
2. timeout_minutes 40 -> 50 gives headroom for matrix-cell variance
(3.12 runs ~20% slower than 3.13) so a slightly slow cell degrades
gracefully instead of burning 2x40min and failing the whole run.
Follow-ups (not in this PR): fix store_pytest_durations to run with
xdist or split groups so it fits the 6h limit; investigate the in-worker
degradation that makes client-fixture boots cost 8-12s early in a run
but 60-120s after ~30 minutes.
The first 1.11.0.dev0 nightly failed its "Test Langflow Main CLI" step:
the fork bump's sync_bundle_lfx_pin.py re-synced every bundle floor to
lfx>=1.11.0, and PEP 440 sorts the nightly's 1.11.0.dev0 BELOW 1.11.0,
so the workspace-built bundles (whose metadata shadows the satisfiable
PyPI lfx-arxiv 0.1.1 during `uv pip install dist/*.whl`) reject the
branch's own lfx while langflow-base pins it exactly — unresolvable.
Floor at lfx>=X.Y.0.dev0,<(X+1).0.0 instead: X.Y.0.dev0 is the lowest
version PEP 440 admits in the minor line, so every devN / rcN / final
satisfies the floor while older lines and the next major stay excluded.
This closes the NIGHTLY.md activation-gate hole structurally — future
minor forks re-sync to a floor their own nightlies already satisfy.
- sync_bundle_lfx_pin.py: lfx_floor_spec emits the .dev0 floor
- port_bundle.py: mirrored _current_lfx_floor kept in step
- bundle pyprojects restamped via the script (arxiv/docling/duckduckgo/ibm)
- test_bundle_lfx_pin.py expectations updated (20/20 passing)
- NIGHTLY.md gate section annotated with the post-activation fix
uv.lock is unaffected (workspace lfx is recorded as an editable source
with no specifier — which is also why uv sync/lock passed in the same
job). The release.yml RC floor-relax sed still matches the new form;
now redundant but harmless. No bundle version bump needed: published
0.1.1 floors >=1.10.0.rc0, satisfiable by the whole 1.11 line.
The release-1.11.0 fork bumped lfx to 1.11.0 but left component_index.json's version field at 1.10.0. _read_component_index fails closed on exact version mismatch, so the bundled registry loads as None and the upgrade-gate tests fail (UpgradeFlowError: 'bundled component registry is empty or missing') across the LFX test matrix. That failure blocks the nightly's release-nightly-build job, so no canonical 1.11.0.devN is published and check-nightly-status blocks CI everywhere.
Surgical restamp: version -> 1.11.0 and recomputed sha256 (same orjson OPT_SORT_KEYS hashing and OPT_SORT_KEYS|OPT_INDENT_2 serialization as scripts/build_component_index.py). Entries unchanged. Verified the reader validation passes and the three failing tests go green.
* docs: record nightly→stable bundle cutover plan (gated on lfx 1.10.0)
Add src/bundles/NIGHTLY.md documenting why langflow-nightly currently
renames the bundles (lfx and lfx-nightly ship the same lfx/ import, so a
stable bundle would co-install both and collide) and the deferred cutover
(Approach A: canonical pre-releases; B: lfx as a bundle extra), gated on
stable lfx 1.10.0 being published to PyPI.
Also expand two docstrings in scripts/ci/update_lfx_version.py to state
the deeper install-conflict reason, not just the resolve failure. No
behavior change.
* feat(ci): nightly Approach A — canonical pre-releases, drop nightly bundles [DRAFT/gated] (#13529)
feat(ci): nightly Approach A — canonical pre-releases, drop nightly bundles
DRAFT reference implementation of the nightly→stable-bundle cutover documented
in src/bundles/NIGHTLY.md. Publishes the nightly under CANONICAL package names as
.devN pre-releases instead of separate *-nightly distributions, so the stable
lfx-* bundles resolve against a single canonical lfx (no dual-lfx install
collision) and no nightly bundle packages are produced.
- tag scripts (pypi/lfx/sdk_nightly_tag.py): count .devN against the canonical
PyPI histories instead of the *-nightly projects
- update scripts: stop renaming to *-nightly; set .devN versions; re-pin
inter-package deps to exact canonical dev versions; delete the bundle
rename/repin (update_lfx_dep_in_bundles, rename_bundles_for_nightly)
- release_nightly.yml: publish canonical pre-releases; remove bundle build,
dist-nightly-bundles artifact, publish-nightly-bundles job + its gate; verify
canonical names; main wheel glob dist/langflow-*.whl
- nightly_build.yml: drop the bundle git-add in the tag commit
- NIGHTLY.md: Approach A marked implemented + activation gate + A1/A2 follow-ups
Held as DRAFT: do not activate until stable lfx 1.10.0 is published AND the
nightly base is the next minor (release-1.11.0). A 1.10.0.devN core sorts below
1.10.0 and would fail the bundles' >=1.10.0 floor. Stacked on #13528.
(.secrets.baseline: incidental line-number shifts for the two workflows + prune
of pre-existing stale Pokédex Agent.json entries.)
* docs(ci): drop internal 'Approach A' label from nightly cutover comments
Comment- and docstring-only change across scripts/ci/* and the two nightly
workflows; no logic change. The two workflow edits stay single-line so
.secrets.baseline line numbers are unaffected. src/bundles/NIGHTLY.md keeps
its A/B decision-record framing intentionally.
* feat(ci): make nightly consumers work with canonical pre-releases
Follow-ups from the nightly cutover that are part of its blast radius (the
nightly now publishes canonical `.devN` pre-releases, not `*-nightly`
distributions):
- version.py: derive the "Nightly" label from the `.dev` version marker, since
the canonical `langflow`/`langflow-base` distribution matches first in the
lookup. Keeps the startup banner and telemetry `package` field identifying
nightlies. Adds a canonical-dev test; updates the base-dev assertion.
- ci.yml check-nightly-status: query the canonical `langflow` project and pick
the latest `.devN` release date instead of `langflow-nightly`'s `.info.version`.
- db-migration-validation.yml: install the nightly as `langflow[postgresql]==<dev>`
(pre-release) instead of `langflow-nightly[...]`; verify via version("langflow").
- src/lfx/README.md: nightly install is `uv pip install --pre lfx`.
- NIGHTLY.md: rewrite the follow-ups section (these are addressed; Docker image,
A2 meta-package, and website docs remain deferred by design).
The `langflowai/langflow-nightly` Docker image name is intentionally unchanged.
* fix(ci): correct nightly verify uv tree parsing + stale base-dep regex
Addresses review of #13528:
- release_nightly.yml LFX verify: `uv tree | grep lfx | head -n1` matches the
bundle `lfx-ibm` first → 'Name lfx-ibm does not match lfx'. Root the tree with
`uv tree --package lfx` so the first line is the lfx package itself.
- release_nightly.yml base verify: under canonical naming `langflow-base` prints
as a top-level `langflow-base v<ver>` line, so the old $2/$3 field parse read
name="v0.10.0" and version="". Use `uv tree --package langflow-base` and $1/$2.
- update_lf_base_dependency.py update_base_dep: regex only accepted ~=/==, so its
CLI entry point couldn't match the current root dep `langflow-base[complete]>=0.10.0`.
Add >= (parity with update_uv_dependency.py). The active nightly path uses
update_uv_dependency.py and was unaffected.
* docs(nightly): point NIGHTLY.md status at the activation gate, not draft state
Per review of #13528: this file ships inside #13528 (#13529 was folded in), so
the 'stacked on prep #13528' + 'held as a draft' framing is stale and misleading.
Reword the status block to state the real guard is the activation gate (stable
lfx 1.10.0 published AND next-minor base), not merge/draft state. Also reword the
follow-ups heading 'decide before un-drafting' -> 'decide before activating'.
The reference stack README pointed LANGFLOW_LOG_FILE and LANGFLOW_LOG_DIR at
different directories, so following it produced an empty Grafana dashboard:
Langflow wrote to one path while Promtail scraped another. It also documented a
stdout-scrape alternative the shipped Promtail config does not implement.
Make the two paths consistent, require an absolute log path, correct the stdout
note, and add a no-Langflow smoke test that writes a sample record and confirms
it reaches Loki.
* fix(security): remove the disabled Python Code Structured tool component
Follow-up to #13538, which neutered PythonCodeStructuredTool to a
non-executable stub "for one release cycle, full removal later." This
completes that removal.
- Delete the component and its registration in lfx.components.tools.
- Drop its entry from the component index (num_components 355 -> 354,
sha256 recomputed surgically) and from stable_hash_history.json.
- Remove its 18 i18n keys from every locale file.
- Replace the dedicated stub unit test with a removal test in
test_dynamic_import_integration.py.
- Add a regressions entry to regressions/1.10.x.yaml.
The unauthenticated public-build RCE fix (report H1-3754930) is
unaffected: PythonCodeStructuredTool stays in
CODE_EXECUTION_COMPONENT_TYPES, so build_public_tmp still rejects any
saved or crafted flow that carries the type. instantiate_class execs the
node's stored `code` field regardless of whether the class still exists,
so the type-name block -- not the class -- is what closes the path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document removal
* docs: typo
* Apply suggestion from @mendonk
Co-authored-by: Mendon Kissling <59585235+mendonk@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Mendon Kissling <59585235+mendonk@users.noreply.github.com>
When ci.yml is called from release.yml (inputs.release == true), a
Biome failure was blocking all downstream publish and docker jobs via
needs.ci.result == 'success'. Biome was already excluded from the
ci_success gate for PR merges, but the reusable CI workflow itself
could still fail when the lint job errored.
Add continue-on-error: ${{ inputs.release == true }} to lint-frontend
so Biome failures are informational-only during release runs, while
remaining visible (and blocking ci_success) on regular PRs.
* fix(bundles): bump lfx-* bundles to 0.1.1 to republish with corrected lfx pin
The published 0.1.0 artifacts on PyPI carry stale lfx pins from before the lfx 0.5.0->1.10.0 version realignment:
- lfx-arxiv, lfx-duckduckgo: lfx>=0.5.0,<0.6.0 (hard-broken; the <0.6.0 cap can never resolve lfx 1.10.0)
- lfx-docling, lfx-ibm: lfx>=0.5.0 (uncapped floor/cap mismatch)
The source pin was already corrected to lfx>=1.10.0,<2.0.0 in #13516, but the bundles were never re-published. PyPI versions are immutable, so shipping the fix requires a version bump. Bump all four to 0.1.1 so the Release Bundles workflow cuts fresh wheels carrying the correct pin. Root pyproject keeps its >=0.1.0 floors (satisfied by 0.1.1); only the bundle dist versions and their uv.lock stamps change.
* fix(release): relax bundle lfx floor for pre-release builds + idempotent bundle publish
The RC pre-release run builds lfx as 1.10.0rc0, but bundles floor lfx at >=1.10.0. Under PEP 440 a pre-release sorts below the final, so 1.10.0rc0 fails >=1.10.0 and the cross-platform install test cannot resolve the bundle wheels against the RC lfx wheel.
build-base/build-main/build-lfx already rewrite their inter-package deps to the pre-release version when pre_release=true; build-bundles was the only release artifact missing that step. Add it: when pre_release=true, rewrite each bundle's lfx floor to the exact pre-release version, keeping the wide <2.0.0 BUNDLE_API cap. Stable source stays at >=1.10.0 -- only RC wheels are relaxed, at build time, so no source churn or re-tag.
Also make publish-bundles tolerate 'already exists' duplicate wheels on rerun, matching release_bundles.yml.
* fix(security): block code-execution components on unauthenticated public flow builds (H1-3754930)
PythonCodeStructuredTool exec()'d its attacker-controlled `tool_code` field at
flow-build time. Because a PUBLIC flow can be built with no authentication via
POST /api/v1/build_public_tmp/{flow_id}/flow, that sink was reachable as an
unauthenticated server-side RCE — and other components (Python Interpreter/REPL,
Smart Transform) execute code on the same path, so removing one component alone
would not close the gap.
- Harden the public build path: build_public_tmp now rejects flows containing
code-execution components via validate_public_flow_no_code_execution(). The
check keys on the node `type`, so it holds regardless of the stored component
`code`, and is enforced ONLY on the unauthenticated public path —
authenticated /build is unchanged.
- Neuter PythonCodeStructuredTool to a non-executable compatibility stub: all
exec()/eval() sinks removed; build_tool returns a tool that raises a
deprecation error. The component stays registered with identical
display_name/inputs so saved flows still load and locale keys don't change.
It will be fully removed in a future release.
component_index.json (code_hash) is regenerated by autofix.ci.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [autofix.ci] apply automated fixes
* fix: validate_public_flow_no_code_execution
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fix: serialize DataFrames in tool mode to prevent pandas truncation
When Memory Base or Knowledge components are wired as agent tools, the DataFrame
result was returned directly without serialization. LangChain would then stringify
this for the agent observation using pandas' default repr, which truncates all
cells to 50 characters (display.max_colwidth=50), inserting "...".
This breaks the F3 agent use case where agents need to reliably recall facts from
memory. The agent receives truncated content and cannot see the full text.
The fix serializes DataFrames through the existing serialize() function, which
converts them to list[dict] format with full, untruncated content. This maintains
consistency with how other result types (Message, Data, etc.) are handled in tool mode.
- Affects: Memory Base and Knowledge components used as agent tools
- Does not affect: Component-to-component wiring or normal (non-tool) execution
- Testing: Added tests verifying DataFrames serialize to list[dict] with complete content
Improved MB GUI component description
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* [autofix.ci] apply automated fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
The hard_proof marker name was a vibe word that said nothing about what the
tests need. Rename it to real_services everywhere: the pytest marker
registration, the *_hard_proof.py test files, the Makefile target
(real_services_tests), the -m selector in migration-validation.yml, and the
CI job. real_services says what these tests require: real Postgres + Redis +
worker subprocesses. (integration was already taken for the external-API
suite under tests/integration.)
* refactor: remove getenvvar component and update locales
* test: add test for GetEnvVar component removal and update component index
---------
Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local>
fix: use shadcn canvas colors for playground code blocks and chat-bg for the container
Standardize the playground tool-call surfaces on shadcn theme tokens so
they swap correctly between light and dark mode:
- Code blocks (SimplifiedCodeTabComponent) use bg-canvas: light grey
(#F4F4F5) / dark black (#000) so they stand out as the code surface.
- The 'Called tool' container (ContentBlockDisplay) uses bg-background so
it matches the chat background in both themes instead of inverting to
solid black in dark mode (it was bg-primary-foreground).
Previously the container and code blocks were both bg-primary-foreground,
which rendered the whole section black in dark mode with no contrast.
* perf(telemetry): batched off-pool writer for transactions + vertex_builds
Adds TelemetryWriterService that buffers transaction and vertex_build rows
in memory and drains them in batched INSERTs via a dedicated AsyncEngine
(pool_size=1 for SQLite, 2 for Postgres, max_overflow=0). Retention is
amortized in a 60s sweeper instead of running on every insert.
Producers (log_transaction, log_vertex_build) enqueue instead of opening
a DB session, so telemetry traffic no longer competes with the
request-handling pool. Falls back to the legacy direct-write path via
LANGFLOW_TELEMETRY_WRITER_ENABLED=false.
Durability: in-flight rows spill to a diskcache.Deque per PID on shutdown
and are restored on next startup; orphan PID directories left by crashed
workers are adopted.
* perf(telemetry): tighten error handling and add coverage
Review feedback from pr-review-toolkit + silent-failure-hunter:
- Retention sweep snapshots the dirty-flow sets before commit and only
clears them after it lands; a crashed sweep no longer drops the flows on
the floor, so per-flow caps cannot drift unboundedly.
- Producer fall-through is no longer silent. transaction_service and
log_vertex_build each log a one-shot WARNING when telemetry_writer_enabled
is True but the writer is not running.
- Lifespan startup failure now logs ERROR instead of WARNING — if the user
opted into the writer and it didn't come up, that's an error.
- Shutdown drain timeout no longer suppressed silently: logs a WARNING with
pending row count + a hint to raise telemetry_writer_shutdown_drain_s.
- Writer's flush loop catches asyncio.CancelledError separately and
re-prepends the in-flight batch to the buffer so teardown's disk spill
catches it.
- After 6 consecutive batch failures the writer emits a loud ERROR with
buffer depths so operators see sustained data-loss risk.
Added tests:
- test_sanitization_survives_writer_round_trip
- test_retention_failure_preserves_dirty_flows
- test_in_flight_batch_returned_on_cancel
* perf(telemetry): address copilot review
- _restore_from_disk + _adopt_orphan_outboxes now route through _enqueue so a
large disk-spilled or orphan outbox can't bypass telemetry_writer_max_queue
and OOM the process. Oldest rows are dropped and counted via the existing
dropped_transactions / dropped_vertex_builds counters.
- chmod 0o700 the outbox root + per-PID directory so sanitized-but-still-
sensitive payloads aren't exposed cross-user on multi-tenant hosts.
Suppressed on platforms where chmod is a no-op (Windows).
- Added test_adopt_orphan_outboxes_honors_max_queue.
The remaining two copilot notes (private API access to DatabaseService and
diskcache.Cache.close) were also flagged by the in-tree review; tracking
separately. The "diskcache not declared" note is a false positive — the
dependency is at src/backend/base/pyproject.toml:76.
* perf(telemetry): address coderabbit review
- Sweeper hands off dirty sets via capture-and-clear so concurrent
flushes during a retention pass aren't wiped by the post-commit
subtract; failure path restores the snapshot.
- Stress README uses a concrete Postgres DSN matching the docker
example instead of an unset env var.
- Test PID-probe loops bounded via a shared helper with pytest.fail
fallback.
* perf(telemetry): swap diskcache outbox for stdlib sqlite (CVE-2025-69872)
Replaces the diskcache.Deque-backed spill outbox with a stdlib sqlite3
outbox (WAL mode, JSON payloads in TEXT). diskcache 5.6.3 has
CVE-2025-69872 (pickle deserialization RCE for an attacker with write
access to the cache dir), no fixed version released, and was never
declared as a dependency — import failed on Python 3.10.
The replacement is encapsulated in a small _Outbox helper that owns the
connection, schema, JSON codec, and lifecycle. The codec uses tagged
wrappers for datetime and UUID so SQLAlchemy's typed columns accept
restored payloads on the way back out.
Hardening informed by a survey of OTel collector, Fluent Bit, Vector,
Prometheus remote_write, Datadog agent, Logstash, and Filebeat:
- Per-PID outbox dirs now stamp an owner.json (hostname + Linux boot_id
or time()-monotonic() proxy). Adoption only proceeds when host+boot
match; cross-host or pre-owner-file dirs are logged and skipped so a
recycled PID after a container restart cannot pull in a stranger's
spill data.
- PRAGMA synchronous=FULL on the spill connection so the shutdown
commit hits the platter (NORMAL only fsyncs on WAL checkpoint, which
may never run if the process exits immediately after commit).
- Spill honors telemetry_writer_max_queue with drop-oldest, matching
the producer-side overflow policy so a backlogged buffer at shutdown
can neither stall teardown nor fill disk.
- append_all encodes payloads up front and only clears the deque after
the transaction commits — no more partial-drain on mid-flight SQLite
failure. drain collapses to a single DELETE FROM outbox after the
SELECT.
- Exception handlers at the spill/restore/adopt boundaries narrowed to
(sqlite3.Error, OSError) so genuinely unexpected exceptions
propagate rather than being silently logged.
Tests cover: cross-host orphan skipped, missing-owner orphan skipped,
spill cap drops oldest, and a realistic UUID+datetime payload
surviving spill → restore → SQLAlchemy INSERT.
* [autofix.ci] apply automated fixes
* fix(telemetry): name wait_for inner tasks so pyleak can filter
Integration tests using @pyleak_marker were flagging an inner asyncio
task created by the telemetry writer's `wait_for(Event.wait(), ...)`
loops. The wrapper task is auto-named (Task-N) and lands mid-await
across the test boundary, so pyleak counts it as leaked. The writer
itself shuts down cleanly via teardown; the "leak" is a pattern
mismatch with pyleak's per-test snapshot model, not a real lifecycle
bug.
Wrap Event.wait() in an explicitly named task (`telemetry-writer-tick`,
`telemetry-writer-backoff`, `telemetry-sweeper-tick`) via a small
_wait_or_shutdown helper, and extend pyleak_marker with a task_name_filter
that excludes `telemetry-*` from leak detection. CI was green on earlier
commits in this branch only because the diskcache import error
prevented the writer from starting at all — fixing the import surfaced
this latent pyleak collision.
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* test(telemetry): disable writer in tests so reads see writes synchronously
* perf(telemetry): age out cross-host orphan outboxes on shared volumes
A pod on a shared PV (NFS/RWX) cannot adopt rows from a dead pod on a
different host because the dead pod's hostname doesn't match. Without a
janitor those directories accumulate forever. Add an owner-file mtime
heartbeat from the sweeper plus a prune pass that deletes cross-host
outboxes whose owner file hasn't been touched within
telemetry_writer_orphan_max_age_s (default 1h). Same-host orphans still
flow through the existing adoption path.
* perf(telemetry): add byte-aware flush + drop strategy
Today the writer bounds memory by row count only, so a worker logging fat
vertex_build artifacts can hold tens of MB per row and silently dwarf the
configured max_queue. Add a size_strategy switch ('count' | 'bytes' |
'either', default 'count' for parity) with two byte thresholds:
- batch_size_bytes (256KB) caps per-flush INSERT size when bytes apply
- max_queue_bytes (200MB) drops oldest by bytes when bytes apply
Parallel sizes deques mirror the payload buffers so accounting stays
consistent across drain, spill, restore, and the cancel/retry rollback.
Single rows above the byte budget are still emitted (no row is refused);
operators get dropped_*_bytes counters alongside dropped_*.
* test(telemetry): tighten byte-strategy assertions and cover end-to-end paths
Address gaps in the byte-aware strategy tests:
- pin exact dropped counts and remaining buffer state for the drop-by-bytes
case (was 'greater than zero')
- compute exact drain count from encoded size (was a 1-4 range)
- round-trip bytes through spill+restore to verify size deque is rebuilt
- round-trip bytes through orphan adoption for the same reason
- exercise the sweeper loop end-to-end to confirm heartbeat + prune are
wired in (previously only unit-tested in isolation)
Clarify in the settings docs that the byte caps measure encoded JSON size,
not Python in-memory footprint.
* ref: updates to telemetry writer PR (#13294)
* fix(telemetry-writer): harden error handling and add missing test coverage
Critical:
- teardown() now re-raises CancelledError after awaiting the writer task so the
asyncio cancellation chain propagates correctly on lifespan task kill
- suppress(OSError) on owner file write replaced with explicit error log so
operators know when disk-spilled rows will not be recoverable on restart
High / important:
- Escalation threshold check changed from == to >= so the error log fires on
every failure past the threshold, not just the 6th
- Dead `except OSError` on time.time() - time.monotonic() changed to
`except Exception` since time functions cannot raise OSError
- Removed redundant `from uuid import UUID` inside _run_retention_pass (already
imported at module top)
- Orphan directory cleanup: replaced blanket suppress(OSError) with per-child
suppress so ENOTEMPTY on an individual rmdir doesn't abort the whole loop and
leave the parent directory leaking silently; outer failure now logs at debug
Tests (3 new):
- test_retention_sweep_caps_vertex_builds_per_vertex: inserts 8 builds for a
single vertex with max_per_vertex=3 so the per-vertex DELETE subquery
actually executes (previous test used 8 distinct vertex IDs, bypassing it)
- test_either_strategy_trips_on_bytes_first: verifies bytes can be the first
trigger under 'either' strategy (previous test only covered the count-first path)
- test_writer_retries_on_batch_failure: injects 2 flush failures then success;
confirms failed_batches increments, rows are preserved in the buffer, and
flushed_rows reflects the final successful write
* ci: add stress-tests job to nightly build pipeline
Wires the stress-tests workflow into nightly so telemetry write stress
tests run automatically. Also gates release-nightly-build and
slack-notification on stress-tests result so a stress regression blocks
the nightly release and surfaces in Slack.
* ci: run stress tests in nightly without blocking release
Stress tests are informational for now — a failure is visible in the
workflow run and Slack but does not gate the nightly release or build.
* fix(telemetry-writer): address PR review on cancelled teardown and nightly stress tests
- Add the stress-tests.yml reusable workflow (was untracked, so the
nightly stress-tests job could never resolve)
- Notify Slack on stress-tests failure (add to slack-notification gate
and FAILED_JOB detection as non-blocking)
- Run sweeper cancel, disk spill, and engine dispose in a finally so a
cancelled teardown() still persists the in-memory buffer
- Add tests for the cancelled-teardown spill path and the >= escalation
threshold; drop the misleading wait-loop in the retry test
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
* fix: commit slider value before node selection consumes the interaction
Sliders inside React Flow nodes (e.g. the URL component's Depth field) lost
the value the user set when the node was not yet selected. React Flow selects
a node on click and pans/drags it on pointer down, while Radix drives the
slider with the same pointer events. The interactive SliderPrimitive.Root had
no React Flow isolation, so the first interaction on an unselected node was
consumed by node selection: the slider reacted visually but the chosen value
reverted or snapped to wherever the pointer landed.
Stop pointer/click propagation on the slider root and add React Flow's
nodrag/nopan/noflow/nowheel opt-out classes (matching the slider's value-text
input). Radix composes the handlers, so value setting is unaffected.
Adds a regression test asserting slider pointer-down/click do not bubble to
the node wrapper while unrelated children still do.
* Update src/frontend/src/components/core/parameterRenderComponent/components/sliderComponent/__tests__/slider-node-selection.test.tsx
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [autofix.ci] apply automated fixes
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
