mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-24 20:14:58 +08:00
7f181e9f16
* Add Memory Base API: models, migrations, service, endpoints, and tests
Introduces Memory Base (MB) — a per-flow knowledge base that auto-captures
conversation history and ingests it into a Chroma vector store on configurable
thresholds.
Backend changes:
- MemoryBase + MemoryBaseSession DB models with full CRUD
- Three Alembic migrations: base tables, merge head, phase-2 fields
(embedding_model, preprocessing, preproc_model, preproc_instructions)
- MemoryBaseService: create/list/get/update/delete, session tracking,
pending-message cursor logic, mismatch detection, regenerate
- ingest_memory_task: async Chroma ingestion with cursor advance on success
- REST API (/api/v1/memories): CRUD, flush, sessions, mismatch, regenerate
- Flow output hook: on_flow_output() triggers auto-capture after each run
- Plumbing in build.py, endpoints.py, workflow.py to call on_flow_output
- deps.py: expose get_memory_base_service()
- kb_helpers.py: FS/metadata helpers used by MB service
Authored-By: Debojit Kaushik <kaushik.debojit@gmail.com>
Add dedupe_key idempotency enforcement for MB ingestion jobs
Centralizes idempotency into JobService.create_job() with a null-safe check,
removing the redundant pre-flight logic from MemoryBaseService.
Key changes:
- services/jobs/exceptions.py: new DuplicateJobError(RuntimeError) —
raised when a QUEUED/IN_PROGRESS/COMPLETED job with the same dedupe_key
exists; FAILED/CANCELLED are retryable and are excluded
- services/jobs/__init__.py: exports DuplicateJobError
- services/jobs/service.py: null-guarded dedup query inside create_job()
within the same session_scope as the insert (minimizes TOCTOU window)
- services/database/models/jobs/model.py: dedupe_key field -> index=True
- alembic/versions/36aa87831162: adds dedupe_key column + ix_job_dedupe_key
index to job table with checkfirst guards
- services/memory_base/service.py: updated key format to
"ingestion:{mb_id}:{session_id}:{first_msg_id}" for namespace isolation;
removed _has_non_retryable_job_for_dedupe_key and _has_active_job methods
and all call sites; DuplicateJobError catch in _maybe_trigger() for
silent skip on auto-capture; split regenerate() catch clauses
- api/v1/memories.py: explicit DuplicateJobError catch before RuntimeError
in flush_memory_base() for semantic clarity (both return 409)
Co-Authored-By: Debojit Kaushik <kaushik.debojit@gmail.com>
Checkpointing working version of MBs. TODO: User separation, Get messages endpoint, MB resumption midway through a chat for a session, tests.
Added messages endpoint for Memory Bases. Added pagination to sessions endpoint. Modifed messages model to include ingestion related attributes.
Added unit tests, fixed linting issues and formatting issues.
Aligned Workflows API, /run endpoint, playground to all work with Memory Bases. Created DB models asociated with tracking memory base state with sessions and jobs. Added tests, created unit tests for service and task files related to MemoryBases.
Improved concurrency of jobs, added (memory_base_id, session_id) locking to serialize jobs in case the job creation cadence moves ahead of ingestion jobs. Improved concurrency handling and moved the pending check to be live inside the ingestion job rather than a snapshot before triggering the job.
Consolidated all migrations related to memory_bases into one idempotent version and serialized all migrations form release along with memory_bases for cleanliness and maintainability.
Introduced advisory locking to address multi worker environment, added unique constraint to MB creation, added sanitization check to KB pathnames to avoid illegal directory creation.
Added partial write rollback for ChromaDB, aligned same session is used for each job to avoid dangling advisory locks.
* feat(kb): Knowledge Bases Infrastructure Overhaul
* refactor(kb): reuse Memory Base helpers to reduce duplication (#12878)
* refactor(kb): reuse Memory Base helpers to reduce duplication
Stacked on top of #12802 — consolidates the overlap between this KB
infra PR and the Memory Base work in #12417 so the two PRs land
cleanly together.
1. Chain migrations linearly
``72df732be86b_add_ingestion_run_table`` now points at
``mb00a1b2c3d4`` (the Memory Base head) instead of
``d306e5c17c41``. Both PRs had branched from the same release
ancestor, which would have required an Alembic merge migration at
integration time; with the stack in place, a single linear history
lands: ``d306e5c17c41 ← mb00a1b2c3d4 ← 72df732be86b ←
15fe9304bca7 ← e728126476a8``.
2. Delegate the path-traversal guard to the shared helper
``_validate_kb_path_containment`` now wraps
``kb_path_helpers.validate_kb_path`` instead of duplicating its
``is_relative_to`` check — the HTTPException translation and
structured log line stay here because they're route-specific, but
the traversal rule itself is defined once.
3. Reuse ``infer_embedding_provider`` during KB backfill
``knowledge_base_service.backfill_from_disk`` now falls back to
the pure-string inference helper when legacy metadata has
``embedding_model`` set but ``embedding_provider`` is
``"Unknown"``. Matches the inference the Memory Base service uses
and avoids shipping backfilled rows with
``embedding_provider="Unknown"`` when the model name is
recognizable.
4. Use ``dedupe_key`` for connector ingestions
``POST /knowledge_bases/{kb_name}/ingest/connector`` now builds a
stable SHA-256 key over (user, kb, source_type,
canonicalized-source-config) and passes it to
``JobService.create_job``. A double-click on "Ingest" or a retry
racing with an in-flight job now returns HTTP 409 instead of
spawning a duplicate run. Only the hash lands on the job row, so
credentials in ``source_config`` don't leak through ``dedupe_key``.
5. Extract ``KBIngestionHelper.write_documents_to_backend``
Backend-agnostic counterpart to
``write_documents_to_chroma`` (which is Chroma-only and used by
Memory Base). The inline retry/cancellation loop inside
``perform_ingestion`` collapses into a single helper call so
Mongo/Astra/Postgres/OpenSearch paths share identical batching,
cancellation, and exponential-backoff behaviour with the
file-upload/folder paths.
* Update templates
* Update .secrets.baseline
* Update component_index.json
* Update starter projects
* Update .secrets.baseline
* refactor(kb): migrate ingestion-source registry to AdapterRegistry (#12879)
Port the hand-rolled ``ingestion_sources/registry.py`` onto the
existing ``lfx.services.adapters.registry.AdapterRegistry`` so
third parties can publish ingestion sources as
pip-installable plugins without modifying core.
Plugin authors can now register sources two new ways in addition to
the in-tree ``register_source(SourceType.X, Y)`` call:
* Entry point group ``lfx.ingestion_source.adapters``
* ``[ingestion_source.adapters]`` in ``lfx.toml`` (or
``[tool.lfx.ingestion_source.adapters]`` in ``pyproject.toml``)
What stays the same:
* ``SourceType`` enum remains the source of truth for built-in
identifiers; API boundaries (``ConnectorIngestRequest``) still
validate against it.
* Every existing caller (``register_source`` / ``get_source_class``
/ ``create_source`` / ``registered_sources``) keeps its contract
— the six built-ins continue to self-register at import time via
``ingestion_sources/__init__``.
* ``create_source`` still returns a fresh instance per call. Sources
carry request-scoped ``user_id``/``source_config`` so
``AdapterRegistry.get_instance`` (which caches singletons) is
bypassed in favour of ``get_class``.
* Collision semantics preserved: re-registering a different class
under an existing key raises ``ValueError`` at import time.
* Error-message split preserved: unknown strings (call-site typo) →
``"Unknown ingestion source"``; known-but-unregistered →
``"not registered"``. API routes key HTTP 400 vs 500 off this.
New public helper: ``registered_source_keys()`` returns the full
set of registered keys (including third-party plugins) for callers
that want to surface plugins in a picker; ``registered_sources()``
still returns only the built-in ``SourceType`` enum members for
backward compatibility.
Tests:
* Existing 49 ingestion-source tests pass unmodified (behavior
parity with the old registry).
* 4 new tests cover the plugin surface: entry-point registration,
TOML registration, built-ins not shadowed by entry points, and
the Unknown-vs-not-registered error-message split.
* fix(kb): unstick remote-backend deletes + hydrate stale embedding metadata (#12880)
* Update component_index.json
* Template update
* fix(kb): delete opensearch metadata matches (#12873)
* scope: Chroma and OpenSearch backends
* Ruff cleanup
* format
* [autofix.ci] apply automated fixes
* test(kb): cover backend registry ingestion routing (#12870)
* [codex] Add Knowledge Backends settings (#12909)
feat: add knowledge backend settings
* feat(kb): New KB backend settings
* [autofix.ci] apply automated fixes
* Update component_index.json
* chore(logs): Improve the logging display in KB
* feat(kb): Full support for OpenSearch backend
* Update index.tsx
* test(kb): tighten ingestion run auth assertion (#12872)
* test(kb): tighten ingestion run auth assertion
* [autofix.ci] apply automated fixes
* fix(kb): surface job_id in run detail modal and report skipped runs accurately
Addresses two QA findings on PR #12872:
- BUG-1: the Ingestion Run Detail modal hid the run's job_id even
though the API returned it. Add a Run ID + Job ID block to the
modal so operators can correlate runs with the underlying job
service entry.
- BUG-2: a run that produced 0 successes but >=1 skipped item (e.g.
empty file / no extractable text) was finalized as SUCCEEDED and
surfaced via the cumulative-KB-chunks toast as "ingestion complete
- N chunks ready". Treat any run with skipped or failed items
(without a clean full success) as PARTIAL, and replace the toast
with a per-run breakdown (succeeded/skipped/failed + chunks
ingested by THIS run) shown as a notice instead of success.
Also adds a regression test for the skipped-only -> PARTIAL path in
KBIngestionHelper.perform_ingestion, and tightens types in the
existing KnowledgeBasesTab test (removes pre-existing any usage so
the staged biome no-any hook stays green).
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* test(kb): cover folder ingest allow-list wiring (#12869)
* test(kb): cover folder ingest allow-list wiring
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fix(kb): allow clearing persisted separator (#12871)
* fix(kb): allow clearing persisted separator
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Update component_index.json
* Update test_cometapi_component.py
* Revert "Update test_cometapi_component.py"
This reverts commit e789ef6bed.
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* feat: Add test connection option for backends
* Biome check
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* chore: rebake starter projects after release-1.10.0 merge
The Validate Starter Project Templates hook regenerated the embedded
component source in Knowledge Retrieval.json and Vector Store RAG.json
to match the merged retrieval.py (now imports path/metadata helpers
from _kb_paths).
* [autofix.ci] apply automated fixes
* [autofix.ci] apply automated fixes (attempt 2/3)
* feat(jobs): add job_metadata mirror for KB ingestion runs
Step 1 of unifying KB ingestion tracking onto the canonical 'job'
table per @dkaushik94's review feedback on PR 12935.
Adds a nullable 'job_metadata' JSON column on the 'job' table and
mirrors the 'ingestion_run' lifecycle data onto it: create_run,
mark_running and finalize_run all dual-write the same counters,
items list, status, and source config. The 'ingestion_run' table
remains the canonical source of truth for read paths in this PR;
that's deferred to the contract phase once the API/UI migrate.
Component-path ingestions (no parent Job row) skip the mirror
without raising. JobService gains an 'update_job_metadata' helper
for shallow-merge / replace semantics.
Tests: 9 new tests cover create/mark/finalize dual-write,
component-path no-op, ghost-job tolerance, and the JobService
helper's merge/replace/missing-row branches. Existing 10
ingestion-run endpoint tests still pass.
Migration: da7f6b9b638a (EXPAND, nullable column, fully reversible)
* chore: rebake starter projects after retrieval.py TYPE_CHECKING change
* feat(jobs): unify KB ingestion tracking onto job table; drop ingestion_run
Switches both reads and writes off the legacy 'ingestion_run' table
and onto 'Job.job_metadata'. The full sequence in this PR:
da7f6b9b638a (prev commit) — add Job.job_metadata column (EXPAND)
927dd37c3ea0 — backfill existing ingestion_run rows (MIGRATE)
fc7393328808 — drop the ingestion_run table (CONTRACT, reversible)
Reads/writes now go through 'Job.job_metadata'; the response shape
(IngestionRunInfo / IngestionRunDetail) is unchanged so no frontend
changes are needed. 76 tests pass.
* chore(jobs): collapse ingestion_run lifecycle migrations
The ingestion_run table never shipped to any tagged release —
verified by checking every tag from v1.9.x through v1.10.0.dev9. The
table only existed on feature branches (this one and
feat/kb-ingest-metadata). So the 'create table → backfill → drop
table' migration dance was theatre for a table no production user
ever had.
Collapses 5 migrations to 1 by deleting:
72df732be86b — add ingestion_run table
e728126476a8 — add kb_id FK to ingestion_run
927dd37c3ea0 — backfill ingestion_run → job.job_metadata
fc7393328808 — drop ingestion_run table
Repoints the chain so 15fe9304bca7 (knowledge_base) attaches directly
to mb00a1b2c3d4, and da7f6b9b638a (Job.job_metadata) attaches after
15fe9304bca7. Net schema change for production users: a single
nullable JSON column added to the job table.
Devs running this feature branch locally and carrying an
ingestion_run table in their dev DB will need to drop it manually or
nuke + re-migrate. There are no production users of this branch by
construction.
Tests: all 22 KB-related tests still pass.
* refactor(kb): drop unused VectorStoreBackend Protocol
The Protocol was exported as part of the public surface but nothing
in the codebase actually used it as a type annotation, isinstance
narrowing target, or registry constraint:
- Every concrete backend (Chroma/OpenSearch/Mongo/Astra/Postgres)
inherits from the BaseVectorStoreBackend ABC
- The registry signature requires type[BaseVectorStoreBackend]
- No call site used '\: VectorStoreBackend' or
'isinstance(x, VectorStoreBackend)'
- The only test reference (a runtime-check assertion) duplicated
the ABC isinstance check on the line above
Per @dkaushik94's review feedback: the Protocol was dead weight.
Folded its contract-level guidance ('lightweight to construct, heavy
resources released in teardown') into the ABC docstring so the
documentation isn't lost, then deleted the Protocol class and
trimmed exports.
Test surface: replaced the Protocol runtime-check assertion in
test_backends_registry.py with a BaseVectorStoreBackend isinstance
check (same intent, no Protocol needed). 9 registry tests pass.
* refactor(kb): standardize Job.asset_id on KnowledgeBaseRecord.id
Per @dkaushik94's review on list_runs_for_kb: switches the read-side
filter from a JSON-extract on Job.job_metadata.kb_name to an indexed
btree lookup on Job.asset_id. Required standardizing what asset_id
means for KB ingestion jobs.
Before — Job.asset_id was set to metadata['id'] (a UUID stored in
embedding_metadata.json), independent of the knowledge_base table's
primary key. Two id-spaces in flight, with legacy generation logic
inlined at three create_job sites.
After — Job.asset_id is set to KnowledgeBaseRecord.id everywhere.
When the KB has a DB record (the common case post-startup-backfill),
the row's primary key flows directly onto Job.asset_id. Legacy KBs
on disk only fall back to metadata['id'] (or a generated UUID) so
backwards-compat survives.
* New _resolve_kb_asset_id helper centralizes the resolution logic.
Three ingest endpoints (file upload / folder / connector) and the
cancel endpoint now go through it instead of inlining metadata['id']
reads with their own fallback dance.
* list_runs_for_kb resolves the kb_record up front and filters by
Job.user_id + asset_type='knowledge_base' + asset_id=kb_record.id
when present. Falls back to JSON-extract on kb_name only for KBs
without a DB record.
* Tests: _insert_run helper auto-provisions a KnowledgeBaseRecord so
the indexed path is exercised by default. New unit test pins the
indexed-path behavior: two Jobs with the same kb_name in metadata
but different asset_ids must not cross-contaminate the result set.
23 KB tests pass (13 service + 10 endpoint).
* refactor(kb): drop embedding_provider/model columns; model_selection is the single source
Per @dkaushik94's review: the flat embedding_provider /
embedding_model columns on KnowledgeBaseRecord were redundant
with model_selection, which already carried the same data inside
its dict. Drops the columns and routes everything through two new
helpers.
* New helpers get_embedding_provider(model_selection) and
get_embedding_model(model_selection) in
langflow.api.utils.knowledge_base_service. Both extract the
field from a model_selection payload (single-dict or list-wrapped),
with sensible fallbacks ("Unknown" / empty string).
* record_to_metadata_dict continues to emit the flat keys in the
legacy JSON shape (the API response and on-disk metadata.json
consumers still want them) — but they're now derived views over
model_selection via the helpers, not separate columns.
* KnowledgeBaseRecord model: removed the two columns. Migration
15fe9304bca7_add_knowledge_base_table.py edited to not create
them in the first place (branch has no users — same approach as the
ingestion_run lifecycle collapse).
* create_record signature drops embedding_provider /
embedding_model params. Three callers updated:
- knowledge_bases.py:create_knowledge_base folds the request's
flat fields into model_selection when the request didn't
carry one of its own.
- lfx/components/files_and_knowledge/ingestion.py:_persist_kb_record
passes model_selection directly.
- backfill_from_disk synthesizes model_selection from the
legacy on-disk flat fields (preserving the
infer_embedding_provider fallback for KBs that only have a
model name).
* On-disk embedding_metadata.json writes (in
lfx/components/files_and_knowledge/ingestion.py) and
KBAnalysisHelper.get_metadata reads of those flat keys are
unchanged — the file format keeps the flat fields for back-compat
with retrieval / memory_retrieval readers.
* Tests: ~10 sites updated to pass model_selection={...} instead
of the two flat params. test_backfill_inserts_missing_rows now
asserts on record.model_selection.get('provider').
API surface: CreateKnowledgeBaseRequest and KnowledgeBaseInfo
keep the flat fields (frontend back-compat is preserved). The
columns are gone, the helpers are the single source of truth.
31 tests pass (21 service + 10 endpoint).
* [autofix.ci] apply automated fixes
* fix(kb): standardize perform_ingestion on model_selection
CI surfaced a TypeError after the previous commit's global rename
swept up a perform_ingestion() test call too — that function still
took flat embedding_provider / embedding_model args while the test
was now passing model_selection.
Aligned the helper with the rest of the standardization: drops the
two flat params, accepts model_selection (dict | list) and derives
provider / model internally via the existing
get_embedding_provider / get_embedding_model helpers when calling
build_embeddings.
Updated all three call sites in knowledge_bases.py
(file-upload / folder / connector ingest paths) to pass
model_selection — synthesized from the legacy flat metadata fields
when the on-disk metadata predates model_selection. Reverted the one
test that the original sweep missed (test_perform_ingestion_rollback)
to the new shape.
4/4 TestPerformIngestionTask tests pass.
* Update Vector Store RAG.json
* feat(kb): user metadata on ingestion + retrieval (#12921)
* fix(frontend): rename KB upload labels and open ingest section by default
Rename "Configure Sources" heading to "Ingest Content" and the dropdown
trigger from "Add Sources" to "Add Files". Drop the "Hide Configuration"
footer toggle so the ingest section is always visible. Chunk size,
overlap, and separator inputs render disabled until at least one source
is added. Match the Add Files focus-visible ring to border-input so the
Radix-managed focus return no longer flashes a black outline.
* fix(models): mark Google + IBM embedding models deprecated
Flag the Google Generative AI and IBM WatsonX embedding model entries in
the unified model catalog as deprecated. They are not natively supported
by Knowledge Base ingestion, so users were attempting invalid
configurations. The KB upload picker and Model Providers modal now fetch
with include_deprecated=true and render a "Deprecated" badge so the
models stay visible but are clearly flagged as unsupported.
* fix(models): keep gemini-embedding-001 active
Only mark text-embedding-004 and embedding-001 as deprecated; the
gemini-embedding-001 model is still served by the Google Generative AI
v1beta endpoint, so leave it visible without the badge.
* feat(kb): user metadata on ingestion + retrieval
Surface the per-chunk user-metadata pipeline that already lives on the KB
infra branch. Adds the API + UI to collect tags at ingest time, persist
them on the ingestion_run row, filter the chunks browser by them, and
narrow KnowledgeBaseComponent retrieval through them.
Backend
- POST /{kb}/ingest gains optional metadata + per_file_metadata multipart
fields. POST /{kb}/ingest/folder gains the same fields on its JSON body.
- New parse_user_metadata / parse_per_file_metadata helpers enforce ≤16
keys, lowercase ^[a-z0-9_]{1,32}$ keys, ≤256-char values, ≤16-item
string arrays, and reject the reserved chunk-internal keys.
- FileUploadSource and FolderSource merge per-file overrides into each
IngestionItem.source_metadata; perform_ingestion now lets per-item
values win over run-level on key collision.
- ingestion_run grows a user_metadata JSON column (alembic
16a290ab1332, EXPAND, server_default '{}').
- GET /{kb}/chunks accepts repeating meta_<key>=<value> params; values
AND across keys, OR within each key. Sidesteps the global comma-split
query middleware that breaks JSON-blob query params.
- KnowledgeBaseComponent gains a metadata_filter input that post-filters
the similarity_search results client-side until per-backend
translators land.
Frontend
- New MetadataEditor renders inside the KB upload modal under Chunking
Settings for run-level tags, and inside FilesPanel as an expandable
per-file override.
- Chunks browser gets ChunksMetadataFilter (popover + chips) wired into
useGetKnowledgeBaseChunks via the new metadata_filter param map.
Tests
- 29 validator tests for the rule set, 4 ingestion-source tests for
per-file propagation + describe counts, integration tests for the
chunks endpoint AND/OR filter, 9 retrieval-side helper tests for
parse + match, 8 frontend MetadataEditor tests.
* [autofix.ci] apply automated fixes
* feat(kb): show metadata tags in upload review summary
Surface run-level tags + per-file override count in the Step 2 Review
summary so users can confirm what is about to ship onto each chunk
before clicking Create. Empty metadata stays invisible to keep the
summary tight on the common no-tag path.
* [autofix.ci] apply automated fixes
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* fix(kb): restore IngestionRun model after merge wipe
The merge from fix/kb-ingest-content-naming (based off release-1.10.0)
silently dropped the IngestionRun SQLModel + its registration in the
models package __init__. Without the model, alembic's autogenerate
guard saw the live DB column-set drift from the empty target_metadata
and raised AutogenerateDiffsDetected at startup, blocking the backend
from booting.
Restore the model files and re-add the import + __all__ entry so the
table is visible to SQLModel's MetaData on boot.
* [autofix.ci] apply automated fixes
* fix(kb): restore ingestion_run migrations after merge wipe
Same root cause as the previous fix: the merge from
fix/kb-ingest-content-naming dropped not just the IngestionRun model
but the three migrations that build the ingestion_run schema:
- 72df732be86b — create ingestion_run table
- e728126476a8 — add kb_id FK column
- 16a290ab1332 — add user_metadata JSON column
Without these, the model says the table exists while no migration
creates it, so test_no_phantom_migrations flags a phantom add_table
diff and the model/migration consistency CI step fails.
Restore the migration files from the original feature commit. Schema
is unchanged from that commit so this is byte-for-byte the state CI
last saw green on this branch.
* fix(alembic): collapse ingestion_run + job_metadata heads to a single tip
Two issues blocked test_no_phantom_migrations after the previous
restoration:
1. e728126476a8 (add kb_id to ingestion_run) chained off
15fe9304bca7 (creates knowledge_base) which left the
ingestion_run-creation migration 72df732be86b dangling as its own
head. Re-chain e728126476a8 onto 72df732be86b so the ingestion_run
chain is linear; the FK target dependency on knowledge_base is
declared via depends_on instead of down_revision.
2. After step 1 the graph still had two independent tips —
16a290ab1332 (ingestion_run.user_metadata) and da7f6b9b638a
(job.job_metadata) — neither referencing the other. Add an empty
merge migration 5238aab36810 unifying both so 'alembic upgrade
head' resolves to a single revision.
`alembic heads` now reports ['5238aab36810'] and
test_migration_execution passes locally (8 passed, 2 sqlite-only
skips).
* [autofix.ci] apply automated fixes
* fix(alembic): mark merge migration with EXPAND phase
The migration validator only accepts EXPAND / MIGRATE / CONTRACT.
Empty merge migrations are pure graph nodes (no DDL) so they belong
in EXPAND — they never drop or rewrite data.
* Update component_index.json
* fix(frontend): align KB upload modal test text with i18n strings
CI surfaced a TestingLibraryElementError — the test was looking for
'Ingest Content' / 'Add Files', but the merge from
feat/kb-v1-db-connectors switched those to i18n strings
('Configure Sources' / 'Add Sources' via t('knowledge.configureSources')
etc.).
Updates the two assertions and the surrounding test names to match
the current rendered copy. 71/71 KnowledgeBaseUploadModal tests pass.
* feat(kb): metadata input on flow KnowledgeIngestionComponent
Adds a Metadata advanced input accepting a JSON object. The decoded
dict is JSON-encoded once per run and stamped onto every chunk's
source_metadata key — same shape the API path writes — so the chunks
browser metadata filter and KnowledgeBaseComponent.metadata_filter
work uniformly across upload, folder, and flow-driven ingestion.
Malformed JSON or non-object payloads log a warning and skip the
metadata stamp rather than failing the run.
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Hare <ericrhare@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* chore: rename knowledge backends to db providers
* Linting cleanup
* Update dbProviderConstants.ts
* fix: icon assets
* Update index.tsx
* fix: Ensure selection of default provider
* Update indices
* chore: Consolidate database migrations
* Update kb1a2b3c4d5e_add_knowledge_base_schema.py
* fix: Ingestions from component behave same
* Ruff fixes
* More ruff fixes
* Update Vector Store RAG.json
* [autofix.ci] apply automated fixes
* Starter project update
* fix: Address two P1s for folder access
* Regen other deps
* fix(knowledge-base): pin review modal height, poll active ingestion runs, rename labels (#12977)
* fix(knowledge-base): pin review modal height and poll active ingestion runs
- Lock the upload modal to a fixed `h-[690px]` on the Review step so the
envelope no longer resizes as users page through chunk previews of
varying length. Step 1 keeps its `min-h-*` flex behavior.
- Add drawer-scoped polling to the Ingestion Runs section: refetch every
5s while any run is non-terminal (`pending`/`running`), refetch on
drawer open, and stop entirely once all rows are terminal or the
drawer closes. Background tabs are skipped via
`refetchIntervalInBackground: false`.
* fix(knowledge-base): rename Configure Sources to Ingest Content and Add Sources to Add Files
Update the English locale to align the upload modal labels with the
agreed wording:
- `knowledge.configureSources` / `knowledge.helpConfigureSources`:
"Configure Sources" -> "Ingest Content"
- `knowledge.addSources` / `knowledge.addSourcesTitle` /
`knowledge.submitAddSources`: "Add Sources" -> "Add Files"
`knowledge.sourcesLabel` ("Sources") is intentionally left as-is. Other
locales remain untouched and will be picked up via the existing
translation flow.
* Update component_index.json
* Update KnowledgeBaseUploadModal.test.tsx
* Update component_index.json
* fix(knowledge-base): surface metadata on chunk cards and make filter self-documenting (#12993)
Closes the QA-flagged UX gap where the chunks-browser metadata feature
was effectively only usable by API users. Two fixes bundled:
- Chunk cards now render the source file name + user-supplied tags as
chips, so the metadata applied at ingestion time is visible without
hitting the API. Reserved ingestion-internal keys (chunk_index,
job_id, etc.) stay hidden via a frontend mirror of
KB_METADATA_RESERVED_KEYS.
- The "Filter by metadata" popover no longer asks the user to type
blind. A new GET /knowledge_bases/{kb}/metadata/keys endpoint scans
chunks server-side and returns distinct user keys + a sample of
values (capped at 50/key with a truncated flag). The popover binds
both inputs to a <datalist>, so users get native suggestions on
click while keeping free-text typing for keys not yet ingested.
Refetches on every popover open so freshly-ingested metadata shows
up without a hard page refresh.
Test coverage: 41 backend tests in TestKnowledgeBaseAPI (3 new for
the keys endpoint), 30 frontend tests in sourceChunksPage (file_name
rendering, user-tag chips, reserved-key hiding, array-valued tags,
malformed JSON, datalist population, validation, refetch-on-open).
* feat(knowledge-base): KB row settings shortcut + ingestion history in Update modal (#13012)
Two QA-flagged UX gaps on the Knowledge Bases tab:
- "Update Knowledge" was buried inside the row's overflow menu — promoted
to a one-click settings icon directly on the row, with the dropdown
retained for the rest of the row actions. Disabled while ingesting,
matching the dropdown rule.
- The "Update Knowledge" modal gave no visibility into what a KB
already contained, so users couldn't tell if a file had been ingested
before. New collapsible "Previously ingested" panel above the upload
fields shows each prior run with its user-typed source name, type,
date, status, and counts. Empty/loading/error states covered.
Backend additions
- IngestionRunInfo gains a ``source_name`` field, populated in
``_run_row_to_info`` from ``RunRow.source_config["source_name"]``.
Whitespace-only values collapse to ``None`` so the frontend renders
the type label as a clean fallback.
Frontend
- IngestionHistoryPanel uses ``useGetIngestionRuns`` with
``staleTime: 0`` + ``refetchOnMount: "always"`` so the list is fresh
immediately after an ingest, mirroring the cache fix from #12993.
- StepConfiguration receives ``kbName`` and renders the panel only when
``isAddSourcesMode && kbName``.
- Run row prefers ``source_name`` (e.g. "Test6") with the type label
("File Upload") as a small subtitle when both are present.
Tests
- Backend: new ``test_exposes_source_name_from_source_config`` covers
populated, missing, and whitespace-only cases.
- Frontend: 6 IngestionHistoryPanel tests (empty/loading/error/render
states, source_name preference, whitespace handling, collapse,
hook args). 3 new column tests for the row settings icon. Existing
column tests migrated to the new ``kb-row-actions-trigger`` testid.
* Update Vector Store RAG.json
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* Update IngestionHistoryPanel.tsx
* fix(memory-base): resolve embeddings from static registry, fix Google Gemini provider inference (#13023)
Memory Base retrieval failed with "Embedding model X for provider Y not
found." for every supported embedding model. Two interacting bugs:
1. KBIngestionHelper.build_embeddings looked the model up in the
user-filtered catalog returned by get_embedding_model_options. That
catalog is empty whenever the per-user credential lookup silently
returns an empty enabled-providers set — which happens when the call
bridges from an async event loop into a worker thread. Fall back to
the static EMBEDDING_PROVIDER_CLASS_MAPPING / EMBEDDING_PARAM_MAPPINGS
registry instead: the metadata persisted in embedding_metadata.json
is the source of truth, and the runtime API key is still resolved by
get_embeddings.
2. infer_embedding_provider mis-classified models/gemini-embedding-001
as "OpenAI" (no Google pattern matched the gemini-embedding name) and
the Google patterns returned "Google" instead of the canonical
"Google Generative AI" registered in EMBEDDING_PROVIDER_CLASS_MAPPING.
Look the model up in the unified catalog first, and update the
pattern table to use the canonical provider name plus the
"models/..." prefix Google uses.
Adds regression tests covering OpenAI text-embedding-3-*, Google
gemini-embedding-001, and the previously-empty user-filtered catalog
path.
* feat: Adding Chroma Cloud as Vector DB provider (#13030)
* Adding Chroma Cloud support for Knowledge Bases.
Added Chroma cloud support. Users can configure Chroma Cloud and switch between providers especially between ChromaDB local and Cloud.
* [autofix.ci] apply automated fixes
* Update Vector Store RAG.json
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Hare <ericrhare@gmail.com>
* fix(kb): align OpenSearch KB default vector field with canvas component (#13009)
* fix(kb): align OpenSearch KB default vector field with canvas component
The KB OpenSearch backend defaulted to 'vector_field' while the
canvas OpenSearch component defaults to 'chunk_embedding'.
Combining KB ingestion with the canvas component on the same index
failed with "Field 'chunk_embedding' is not knn_vector type".
Switch the KB backend (and the corresponding frontend defaults /
fallbacks in the DB Providers settings page) to 'chunk_embedding'
so both layers agree out of the box. Operators who already
ingested into 'vector_field' can still override via backend_config
or the global variable.
* test(kb): pin OpenSearch backend default vector_field to 'chunk_embedding'
Adds unit coverage for ``OpenSearchBackend`` so the canvas / KB
field-name alignment can't silently regress:
* asserts ``DEFAULT_VECTOR_FIELD == 'chunk_embedding'``,
* asserts ``_build_vector_store`` passes ``vector_field='chunk_embedding'``
to the LangChain wrapper when no override is provided,
* asserts an explicit ``backend_config['vector_field']`` still wins,
* parametrized table for None / empty / truthy resolution.
* fix(kb): cancel in-flight ingestion jobs on KB delete (#13003)
* fix(kb): cancel in-flight ingestion jobs on KB delete
Without this, deleting a KB while an ingestion job is running would
leave the deleted KB reappearing in the list a few seconds later. The
background job kept polling `is_job_cancelled` (it never tripped) and
its open backend client would auto-recreate the KB directory on the
next chunk write — Chroma's PersistentClient creates its sqlite parent
on demand. The ingestion then rewrote `embedding_metadata.json` into
the recreated dir, and the list endpoint's disk-fallback path
re-discovered it as a valid KB.
Add `JobService.cancel_in_flight_jobs_by_asset` and call it from the
single + bulk KB delete endpoints before storage teardown. The
ingestion's existing cancelled-handler then runs cleanup, so no
further chunks are written.
* fix(kb): DB-first delete + .kb_deleted sentinel hides locked-disk dirs
Two related bugs around KB delete on Windows when Chroma still holds an
exclusive SQLite lock:
1. delete_storage() ran before the DB row delete, so a locked rmtree
turned into HTTP 500 and left the row dangling. The dir was
partially cleaned (non-locked files removed) but the row plus the
SQLite stub kept the KB visible to listings.
2. The disk-scan fallback in get_knowledge_bases() and the API list
endpoint had no way to tell a "really still here" dir from a
"we tried to delete this and rmtree was blocked" dir, so a deleted
KB kept reappearing in the canvas component dropdown.
Fix:
* KBStorageHelper.delete_storage() now drops a .kb_deleted sentinel
file inside any directory it could not physically remove. Replaces
the previous .deleted_<name>_<timestamp> rename fallback (which
failed under the same Windows lock conditions on the rename itself).
* delete_knowledge_base() and delete_knowledge_bases_bulk() reorder
to DB-first (Option A). When the DB delete succeeds and storage
cleanup fails, the endpoint returns 200 with a warning instead of
500: the user no longer sees the KB, and the operator gets a clear
follow-up message.
* lfx.base.knowledge_bases.knowledge_base_utils.get_knowledge_bases()
skips dirs carrying the sentinel. The API-side list endpoint's
disk-scan fallback does the same via KBStorageHelper.is_kb_dir_deleted.
* create_knowledge_base() detects sentinel-marked dirs and surfaces a
clearer 409 ("previously deleted; awaiting restart cleanup") instead
of the bare "already exists", and clears any stale sentinel after a
successful mkdir as a defensive no-op.
* Cross-package contract: the lfx layer cannot import from the langflow
API package, so it inlines the sentinel filename string. A unit
test in test_kb_storage_deletion.py asserts the inlined value matches
KB_DELETED_SENTINEL.
Tests: 88 KB-related tests pass (test_kb_storage_deletion.py +
test_knowledge_bases_api.py). Two existing tests updated to assert the
new 200-with-warning shape; the previous 500-on-storage-failure shape
was the buggy behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: Stay deleted when sentinel shows up
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: enforce session isolation in MemoryBase retrieval and purge chunks on session delete (#13037)
* fix: enforce session isolation in MemoryBase retrieval and purge chunks on session delete
Two related bugs in the MemoryBase feature (#12903):
1. Filter by Session: switch the Chroma where-clause to the canonical
`{"session_id": {"$eq": ...}}` form and force-coerce the toggle to
a real bool. Both shapes are accepted by chromadb today, but the
explicit form removes any ambiguity if a non-bool slips into the
attribute and would otherwise be truthy (e.g. the literal string
"false").
2. Ghost chunks on session delete: the message-session delete
endpoints only wiped MessageTable rows, leaving every embedded
chunk behind in Chroma. New sessions then pulled the deleted
data back through the retrieval component. Wire the delete
endpoints to a new `MemoryBaseService.purge_session_data` that
walks the user's Memory Bases, deletes chunks where
`session_id == X`, and clears the corresponding tracking rows
(MemoryBaseSession + MemoryBaseWorkflowRun). Chunk-delete
failures are logged but never abort the user-visible message
delete.
* Update ingestion.py
* fix: exclude component error messages from MemoryBase ingestion (#13039)
Component errors (e.g. an embedding API failure) were persisted to the
message table with `error=True` / `category='error'` and the MemoryBase
ingestion task pulled them in like any other message, embedding the error
text into Chroma. Retrieval then surfaced these chunks as context.
Filter both signals at the SQL fetch boundary so errors never enter the
document-build / Chroma write path. The cursor still advances past any
later non-error message, so skipped error rows are not reconsidered.
Includes regression tests covering the error flag, the error category,
and cross-session/cross-flow isolation.
* fix(kb): require OpenSearch username/password in DB Provider settings (#13005)
The DB Providers settings panel marked Username and Password as
optional, but the OpenSearch runtime components (canvas vector-store +
KB backend) default to basic auth and raise "Auth Mode is 'basic' but
username/password are missing" when the corresponding global variables
aren't set. Operators who filled in only the cluster URL hit a
confusing error far downstream from the settings screen.
Mark the credential fields required so the existing canSave / save
validation in DBProvidersPage forces them up-front. Operators with
auth-less clusters (sigv4 / upstream proxy) can still satisfy the
form with a placeholder value — OpenSearch ignores credentials when
auth is not enforced.
Also assert the asymmetric (one-field-only) case in tests, since the
backend's tuple-based http_auth silently treats partial creds as
no-auth.
* Update component_index.json
* Update component_index.json
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* Update kb1a2b3c4d5e_add_knowledge_base_schema.py
* [autofix.ci] apply automated fixes
* fix: templates
* fix: Session filtering for memory bases
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* Update component_index.json
* chore: auto-bake note keys and regenerate backend locales/en.json [skip ci]
* Update test_memory_retrieval.py
* test: pin OpenSearch backend filter=None drop behavior
The OpenSearch backend's similarity_search override drops `filter` when
None/empty so LangChain's OpenSearchVectorSearch wrapper never injects
`"filter": null` into the k-NN query body. If that regresses, every KB
retrieval against OpenSearch fails with x_content_parse_exception and the
upstream flow loses both KB and any downstream context — which is the most
likely OpenSearch-only symptom shape behind Rafael's report.
These tests pin:
- filter=None: kwarg is dropped from both asimilarity_search and
asimilarity_search_with_score
- filter={}: also dropped (k-NN parser rejects empty objects)
- filter={truthy}: forwarded unchanged
* fix(knowledge-base): KB UI polish — drawer + row dropdown (#13069)
* fix(knowledge-base): keep drawer open when interacting with row dropdown
The KnowledgePage outside-click handler treats clicks on Radix portal
content (dropdown menus, popovers, dialogs, tooltips) as outside the
drawer and closes it. Because dropdown content lives on document.body,
clicking a menu item triggered closeDrawer, the layout reflowed (mr-80
toggled off), AG-Grid resized, and the row re-rendered—tearing down the
open dropdown before the click event reached the menu item. Result:
"View Chunks" and other row actions required two clicks.
Skip the dismissal when the mousedown target is inside any Radix popper
portal, menu, dialog, or tooltip.
* refactor(knowledge-base): rename row action to "Ingest Files" with FileUp icon
The row shortcut and dropdown item previously used a gear icon and the
label "Update Knowledge". The gear implies settings, but the only action
behind it is adding/ingesting files. Switch both the icon (Settings →
FileUp, RefreshCw → FileUp) and the label ("Update Knowledge" → "Ingest
Files") so the affordance matches the action.
Tooltip, aria-label, and dropdown item label updated. Tests adjusted
accordingly.
* fix(knowledge-base): stop clipping stacked Layers icon in Name column
The Name cell renderer applied "-mx-[3px]" to the row icon, pulling it
3px outside its layout box. For KBs with multiple source types the icon
becomes "Layers" (stacked), which has visible content right up to its
bounding box. With the negative margin, the icon's left edge crossed
AG-Grid's cell overflow boundary and got clipped.
Drop the negative margin, give the cell wrapper a 4px left padding for
breathing room, and tighten the icon→text gap from 4 to 3 so total
horizontal layout stays close to the original.
* fix(knowledge-base): give ingestion history rows room to breathe
Rows in the Previously Ingested panel packed the status badge, source
name, type label, and time onto a single line. Long source names + long
type labels (e.g. "Chroma Cloud Collection") clipped against the time
stamp and made the row hard to scan.
Restructure each row into three stacked sections:
1. status badge (left) · relative time (right)
2. source name (sm/medium) with the type label as a quiet subtitle
3. succeeded / failed / skipped counts · chunks
Row padding bumped from p-2 to p-3 and inter-section gap from gap-1 to
gap-2 to match the new hierarchy.
* refactor(knowledge-base): drop Source Files / Linked Flows placeholders
Both sections in the KB inspection drawer always rendered the same empty
state ("No source files available." / "No linked flows available.")
because neither field is wired up on the backend yet. Showing empty
placeholders adds noise without giving the user information.
Remove the two sections from the drawer. Update the mock + assertions
in KnowledgeBaseDrawer.test.tsx so the test stops claiming the removed
strings should render.
* fix(knowledge-base): give drawer ingestion runs the same room to breathe
The drawer's Ingestion Runs rows had the same cramped single-line layout
the modal panel had before the previous commit: status badge + source
type + time crammed onto one row, counts beneath. Match the new pattern
so the two surfaces feel consistent.
Each row now stacks:
1. status badge (left) · relative time (right)
2. source type label (sm/medium)
3. succeeded / failed / skipped counts · chunks · bytes
Row padding bumped p-2 → p-3 and inter-section gap gap-1 → gap-2.
---------
Co-authored-by: Eric Hare <ericrhare@gmail.com>
---------
Co-authored-by: Debojit Kaushik <kaushik.debojit@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: keval shah <kevalvirat@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Getting Started with Create React App
This project was bootstrapped with Create React App.
Available Scripts
In the project directory, you can run:
npm start
Runs the app in the development mode.
Open http://localhost:3000 to view it in the browser.
The page will reload if you make edits.
You will also see any lint errors in the console.
npm test
Launches the test runner in the interactive watch mode.
See the section about running tests for more information.
npm run build
Builds the app for production to the build folder.
It correctly bundles React in production mode and optimizes the build for the best performance.
The build is minified and the filenames include the hashes.
Your app is ready to be deployed!
See the section about deployment for more information.
npm run eject
Note: this is a one-way operation. Once you eject, you can’t go back!
If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project.
Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.
You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.
Learn More
You can learn more in the Create React App documentation.
To learn React, check out the React documentation.