diff --git a/src/bundles/PORTING.md b/src/bundles/PORTING.md new file mode 100644 index 0000000000..5bc422259b --- /dev/null +++ b/src/bundles/PORTING.md @@ -0,0 +1,379 @@ +# Porting a component from `lfx.components.` to `src/bundles/` + +This is the step-by-step recipe for extracting a provider package from the +in-tree `src/lfx/src/lfx/components//` directory into a standalone +Extension Bundle distribution under `src/bundles//`. The DuckDuckGo +bundle ([`src/bundles/duckduckgo`](duckduckgo/)) is the reference +implementation; every section below maps to a single, copy-pasteable change +and a verification command. + +If you'd rather automate the mechanical steps, use +[`scripts/migrate/port_bundle.py`](../../scripts/migrate/port_bundle.py) +(see [§ Automation](#automation)). It writes the same files this doc +describes — review the diff before committing. + +--- + +## 0. Prerequisites — pick a candidate + +Before you start, confirm the component is a good fit: + +- [ ] The provider directory `src/lfx/src/lfx/components//` exists + and contains one or more `Component` subclasses. +- [ ] Imports from `lfx.*` only (no `from langflow...`); the bundle is + installed against the public `BUNDLE_API` surface, not Langflow internals. + Check with: `grep -r "from langflow" src/lfx/src/lfx/components//` +- [ ] No deactivated / legacy duplicate exists under `src/lfx/src/lfx/components/deactivated//`. +- [ ] The runtime dependencies the component pulls in (e.g. `langchain-community`, + a vendor SDK) can be declared in the bundle's `pyproject.toml` without + cycling through `lfx` or `langflow-base`. + +Pick the **bundle name** (snake_case, lowercase, matches the directory name — +e.g. `duckduckgo`, `arxiv`, `wikipedia`) and the **distribution name** +(`lfx-`, e.g. `lfx-duckduckgo`). These two strings are the only +identifiers you'll repeat throughout the port. + +--- + +## 1. Lay out the bundle directory + +Create `src/bundles//` with this exact tree (mirrors +`src/bundles/duckduckgo/`): + +``` +src/bundles// +├── README.md +├── pyproject.toml +└── src/ + └── lfx_/ + ├── __init__.py + ├── extension.json + └── components/ + └── / + ├── __init__.py + └── .py # one file per component class +``` + +> **Why nested `src/lfx_/components//`?** The outer +> `lfx_` is the importable Python package (matches the wheel layout +> `importlib.metadata.files()` walks). The inner `components//` +> is the path declared in `extension.json:bundles[].path` — keeping it as +> `components/` means saved flows that referenced +> `lfx.components...` migrate cleanly via a single +> import-path entry in the migration table. + +### 1a. `pyproject.toml` + +Copy [`src/bundles/duckduckgo/pyproject.toml`](duckduckgo/pyproject.toml) and +substitute names + the runtime-dep block. The non-obvious bits: + +- `dependencies` lists every runtime dep the component imports. Pin `lfx` + to a range that covers the `BUNDLE_API_VERSION` you target + (currently `>=0.5.0,<0.6.0`). +- `[project.entry-points."langflow.extensions"]`: ` = "lfx_"`. + This is what `lfx.extension.loader._plugins._manifest_via_entry_point` + reads to find the manifest; an editable install with no `dist.files` + visibility falls back to this entry point. +- `[tool.hatch.build.targets.wheel]` MUST include + `src/lfx_/extension.json` and the components glob — wheel + installs read the manifest via `dist.files` and skip the bundle if the + file isn't packaged. + +### 1b. `src/lfx_/extension.json` + +```json +{ + "$schema": "https://schemas.langflow.org/extension/v1.json", + "id": "lfx-", + "version": "0.1.0", + "name": "", + "description": ".", + "lfx": { "compat": ["1"] }, + "bundles": [ + { "name": "", "path": "components/" } + ] +} +``` + +The `id` is the distribution name (with a hyphen). The `bundles[0].name` +is the snake_case bundle name used in saved-flow IDs +(`ext::@official`). They differ by one character (`-` vs +`_`); don't mix them up. + +### 1c. `src/lfx_/__init__.py` + +Re-export the component class(es) from the package root so +`lfx_.` resolves. The migration table's `bare_class_name` +entry depends on this import working. + +```python +"""lfx-: .""" + +from lfx_.components.. import + +__all__ = [""] +``` + +### 1d. `src/lfx_/components//__init__.py` + +```python +from . import + +__all__ = [""] +``` + +### 1e. `src/lfx_/components//.py` + +This is the **moved** file. Copy it byte-for-byte from +`src/lfx/src/lfx/components//.py` — do **not** rewrite +imports. The component's `from lfx.*` imports work unchanged because +`lfx` is a runtime dep of the bundle. + +### 1f. `README.md` + +A short page explaining what the bundle ships, how to install it, and how +to develop against it. Use [`duckduckgo/README.md`](duckduckgo/README.md) +as the template. + +--- + +## 2. Remove the in-tree component + +Delete the whole legacy directory: + +```bash +git rm -r src/lfx/src/lfx/components// +``` + +Then surgically remove the three references in +[`src/lfx/src/lfx/components/__init__.py`](../lfx/src/lfx/components/__init__.py): + +1. The `,` line in the import block (around line 10). +2. The `"": "__module__",` entry in the type-mapping dict. +3. The `"",` string in the `__all__`-style list. + +> **Sanity check:** after the edit, +> `grep -n "" src/lfx/src/lfx/components/__init__.py` returns +> nothing. + +--- + +## 3. Wire the workspace + +### 3a. Root [`pyproject.toml`](../../pyproject.toml) + +Three edits — all mechanical: + +```toml +# 1. Add to [project] dependencies (regular dep so `pip install langflow` +# still pulls the component in -- no user-visible change at install time). +dependencies = [ + "langflow-base[complete]>=0.10.0", + "lfx-duckduckgo>=0.1.0", + "lfx->=0.1.0", # <-- add this line +] + +# 2. Add to [tool.uv.sources] +lfx- = { workspace = true } + +# 3. Add to [tool.uv.workspace] members +members = [ + "src/backend/base", + ".", + "src/lfx", + "src/sdk", + "src/bundles/duckduckgo", + "src/bundles/", # <-- add this line +] +``` + +### 3b. `src/backend/base/pyproject.toml` (optional) + +Only touch this if the component had a `langflow-base[]` extra. +Remove the extra and any `langflow-base[]` reference from +`complete`. The duckduckgo port did this; if the component had no extras +(e.g. arxiv), skip this section entirely. + +### 3c. Lockfile + +```bash +uv lock +git add uv.lock src/backend/base/uv.lock 2>/dev/null # whichever changed +``` + +--- + +## 4. Add migration entries + +Append to +[`src/lfx/src/lfx/extension/migration/migration_table.json`](../lfx/src/lfx/extension/migration/migration_table.json). +The schema requires **three** legacy forms covering every shape a saved +flow may have used: + +```json +{ + "bare_class_name": "", + "target": "ext::@official", + "added_in": "" +}, +{ + "import_path": "lfx.components...", + "target": "ext::@official", + "added_in": "" +}, +{ + "import_path": "lfx.components..", + "target": "ext::@official", + "added_in": "" +}, +{ + "legacy_slot": "ext::@official-pre-a", + "target": "ext::@official", + "added_in": "" +} +``` + +If the component declared multiple classes, repeat the four-entry block +per class (the bare-name entry is only added when the class name is +globally unique across every Bundle in the release; +`scripts/migrate/check_bare_names.py` enforces this). + +The migration table is **append-only**. Never remove or rewrite an +existing entry — CI rejects removals so a flow saved years ago against a +long-extracted bundle still loads. + +--- + +## 5. Regenerate the component index + +The pre-built component index drives lazy loading; the moved component's +old entry must be removed. + +```bash +cd src/lfx +LFX_DEV=1 uv run python -m lfx._dev.build_component_index +``` + +(Or whichever one-shot is in your tree — see `scripts/`. The diff should +only delete the `` block; if it touches anything else, your local +checkout has unrelated drift.) + +--- + +## 6. Add an integration test + +Create +`src/lfx/tests/integration/extension/test_pilot__upgrade.py`, +modelled on +[`test_pilot_duckduckgo_upgrade.py`](../lfx/tests/integration/extension/test_pilot_duckduckgo_upgrade.py). +The four test cases that matter: + +1. Bare class name → canonical ID. +2. Full import path → canonical ID. +3. Package-level import path → canonical ID. +4. The `lfx-` distribution is importable AND ships + `extension.json` in a location `importlib.metadata.files` can discover + (or, for editable installs, that `direct_url.json` resolves). + +The integration test is the only place the saved-flow contract is +exercised end-to-end; do **not** skip it. + +--- + +## 7. Verify + +Run, in order, the smallest commands that will fail loudly when a step is +wrong: + +```bash +# 1. The bundle's manifest is structurally valid. +uv run lfx extension validate src/bundles/ + +# 2. Workspace resolves and the bundle is importable. +uv sync +uv run python -c "from lfx_ import ; print(.__name__)" + +# 3. Migration table parses and the new entries are visible. +uv run pytest src/lfx/tests/unit/extension/migration -q + +# 4. Loader discovers the editable install via direct_url.json. +uv run python -c " +from lfx.extension.loader._plugins import installed_extension_roots +roots = installed_extension_roots() +assert 'lfx-' in roots, roots +print('discovered:', roots['lfx-']) +" + +# 5. The integration test passes. +uv run pytest src/lfx/tests/integration/extension/test_pilot__upgrade.py -q + +# 6. Ruff is clean across the touched files. +uv run ruff check src/bundles/ src/lfx/src/lfx/components/__init__.py src/lfx/src/lfx/extension/migration/migration_table.json +``` + +**End-to-end smoke test** (optional but cheap): start a dev server with +the bundle on the palette and click Reload. + +```bash +uv run lfx extension dev src/bundles/ +# In a browser at http://localhost:7860: +# - Confirm appears under the bundle group. +# - Right-click the header -> Reload. No errors. +``` + +--- + +## 8. Docker images (only if shipping a new bundle to the runtime image) + +The four `docker/build_and_push*.Dockerfile` images already `COPY +./src/bundles` into the build context, so a new bundle directory is +picked up automatically by the workspace `uv sync`. The two **non**-uv-sync +Dockerfiles need an extra line: + +- [`docker/build_and_push_backend.Dockerfile`](../../docker/build_and_push_backend.Dockerfile): + add `./src/bundles/` to the explicit `uv pip install` line. +- [`docker/build_and_push_base.Dockerfile`](../../docker/build_and_push_base.Dockerfile): + add a `uv pip install --no-deps /app/src/bundles/` step after + the workspace sync, alongside the bundle's runtime deps if they aren't + already in the base lock. + +If your bundle has no extras and its deps are already in +`langflow-base[complete]`, the `--no-deps` install is enough. + +--- + +## Common pitfalls + +- **Component imports `from langflow...`**: the bundle is installed against + `lfx`, not `langflow`. Either rewrite the import to use the public + `BUNDLE_API` surface or leave the component in-tree. +- **`extension.json` not in the wheel**: `dist.files` doesn't surface it, + so non-editable installs skip the bundle. Confirm the `[tool.hatch.build.targets.wheel] include` glob picks it up. +- **Bundle name has a hyphen**: only the *distribution* name uses + hyphens (`lfx-duckduckgo`); the *bundle* name is snake_case + (`duckduckgo`). The schema rejects hyphens in `bundles[].name`. +- **Forgot the `langflow.extensions` entry-point**: editable installs + fail discovery silently — `installed_extension_roots()` returns an + empty dict and the bundle never enters the registry. +- **Migration entries missing**: saved flows still validate, but the + palette can't render the legacy node — the user sees a "component not + found" toast. The four-entry block in step 4 covers every shape Langflow + has serialized in the past. + +--- + +## Automation + +[`scripts/migrate/port_bundle.py`](../../scripts/migrate/port_bundle.py) +generates the bundle skeleton, removes the in-tree directory, edits the +two `__init__.py` references, and patches the root `pyproject.toml`. It +**does not** edit the migration table or the integration test — those +require human judgement (release version, class-name uniqueness check). + +```bash +# Dry run -- prints the planned changes; reviewer signs off before --apply. +uv run python scripts/migrate/port_bundle.py --bundle arxiv --apply +``` + +After running the script, work through this doc's verification block (§7); +if anything fails, the script's diff is the single artefact to review.