From 43253011ec124e7b96df2d4e9758a24650d11279 Mon Sep 17 00:00:00 2001 From: Hamza Rashid <74062092+HzaRashid@users.noreply.github.com> Date: Tue, 17 Mar 2026 12:51:44 -0400 Subject: [PATCH] feat: add wxO deployment adapter (#12079) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * checkout wxo deployment adapter implementation * checkout wxo deps and flow reqs impl * clean up / minor refactor with updated tests * major refactor: split up the implementation into folders and files * clean up logic * refactor and clean up of execution logic, improve type safety of "retry_create" helper, and remove dead code * remove unused wxo service file * remove "provider_name" arg and references * fix: harden watsonx orchestrate deployment adapter for PR readiness - Fix rollback AttributeError when agent_create_response is None - Fix NoneType access on params in list() when called without params - Fix inconsistent error types: use DeploymentNotFoundError in get/update - Fix typo "occured" -> "occurred" in all error prefix messages - Fix variable shadowing of fastapi.status in get_status() - Fix pre-existing test bugs (wrong exception types, stale method refs) - Fix e2e monkey-patching of non-existent service methods - Add structured logging to create, delete, retry, and rollback flows - Add jitter to retry backoff to avoid thundering herd - Add __repr__ to WxOCredentials that fully masks the API key - Extract hard-coded LLM model string to DEFAULT_WXO_AGENT_LLM constant - Remove commented-out snapshot update code - Expand test coverage from 24 to 75 tests covering retry logic, service methods, client auth, utilities, execution/status helpers, and artifact roundtrip * fix(deployment): fix bugs and harden watsonx orchestrate adapter - Fix update method silently discarding changes instead of sending them to the WXO API - Wrap all synchronous SDK calls in asyncio.to_thread to avoid blocking the event loop - Add error handling to get_status (was propagating raw exceptions) - Use DeploymentType enum values instead of raw strings for SUPPORTED_ADAPTER_DEPLOYMENT_TYPES - Fix type annotations in list method (list -> set) - Fix typo in comment (reccomend -> recommend) - Remove dead code: extract_agent_connection_ids, require_exclusive_resource, require_non_empty_string, resolve_health_environment_id, fetch_agent_release_status, normalize_release_status, resolve_lfx_runner_requirement, _pin_requirement_name, sync_langflow_tool_connections, create_langflow_flow_tool, resolve_snapshot_connections, and unused constants - Make assert_create_resources_available and validate_connection async - Make create_agent_run and get_agent_run async - Add tests for list_types, get_status error handling, and update side-effects - Update existing tests for async function signatures * actually remove the dead code from tools.py * properly await "validate_connection" * update e2e test file * add more scenarios to e2e test runner * refactor(watsonx-orchestrate): improve WxOClient encapsulation and error messages Encapsulate SDK private method access and endpoint paths inside WxOClient wrappers so callers never handle raw paths or touch internal _get/_post methods. Route all private HTTP calls through a single _base client auto-created in __post_init__, removing the externally constructed `base` field. Additionally fix double-period and redundant text in error messages produced by the ErrorPrefix + handler pattern, and update E2E/unit assertions to match the sanitised error output. Changes: - types.py: bake endpoint paths into wrapper signatures (post_run, get_run, upload_tool_artifact, get_agents_raw); auto-create _base via __post_init__; remove base from constructor - client.py: drop BaseWXOClient import and base= constructor arg - core/execution.py: pass run_id / query_suffix instead of raw paths - core/tools.py: pass tool_id instead of raw path - service.py: fix double-period in ErrorPrefix interpolation; remove redundant restatements in generic except handlers - tests: update _with_wxo_wrappers helper and test doubles to use _base; update FakeBaseClient._get to accept params - e2e: update rollback scenario detail_contains to match sanitised msg * skip import and tests of wxo adapter if current env is running python 3.10 * clarify random prefix / retry behavior * implement comments * fix: align watsonx adapter with updated deployment schema - Add `deployment_type` parameter to `get`, `update`, `redeploy`, `duplicate`, `delete`, `get_status`, and `undeploy_deployment` in WatsonxOrchestrateDeploymentService to match the updated BaseDeploymentService ABC - Update e2e script to use renamed `add_ids` field on SnapshotDeploymentBindingUpdate * remove non-interface method undeploy_deployment * implement listing configs and snapshots * add update implementation and improve http->deployment error translation and add tests * improve exception handling * checkout payload slot work * custom payload schema for update * new update implementation * stop passing client cache to helpers * add docs for ordereduniquests * improve import patterns and document future risks for wxo dependencies * remove global-variable prefixing of flows * ref: harden typing, DRY helpers, and correctness fixes - Replace `db: Any` with `db: AsyncSession` across client, config, and update_helpers modules - Extract `_ensure_dict` / `ensure_langflow_connections_binding` helpers to eliminate repeated nested-dict safety logic in tools.py and update_helpers.py, with documentation explaining why malformed API payloads are silently corrected rather than rejected - Use `PayloadSlot.parse()` for update payload validation instead of standalone helper; remove `parse_provider_update_payload` - Fix lambda late-binding with default-argument captures in service.py - Use `zip(strict=True)` in update_helpers for defensive mismatch detection - Replace O(n²) duplicate detection with `collections.Counter` in payloads.py - Simplify `dedupe_list` to `list(dict.fromkeys(...))` - Move type-annotation-only imports into TYPE_CHECKING blocks in types.py and config.py - Introduce `UPDATE_MAX_RETRIES` as a distinct constant from `CREATE_MAX_RETRIES` - Fix "watsonX" → "watsonx" casing in error messages - Clarify `get_status` docstring and `validate_connection` error message - Add TODO(deployments-cache) for client cache invalidation on credential updates * ref: drop client cache and adopt typed deployment context across the wxo adapter path, add request-context memoization with strict mixed-context guards, and lazily initialize wxo SDK clients. Add safer credential handling by storing only authenticators in WxOCredentials. Also improve provider error-detail extraction/messages and make the direct wxo E2E conflict scenario diagnostics and expectations more resilient. * use explicit naming for context class; providerId * fix error handling, retry safety, and exception diagnostics in wxo adapter - Raise DeploymentError on empty API responses instead of fabricating fake success in create_agent_run_result and get_agent_run - Elevate rollback failure logging from WARNING to ERROR for alerting - Apply retryable filter to retry_rollback to skip 401/403/409/422 - Preserve exception chains (from exc) instead of suppressing (from None) across service.py, utils.py, execution.py, and update_helpers.py - Broaden credential resolution catch to handle arbitrary DB exceptions - Separate status code dispatch from string heuristics in raise_for_status_and_detail to prevent misclassification - Add tests for all behavioral changes * [autofix.ci] apply automated fixes * fix: harden wxO adapter safety, immutability, and error diagnostics - Make WxOClient and WxOCredentials frozen dataclasses with eager SDK client initialization to eliminate thread-safety races from asyncio.to_thread workers and prevent post-construction mutation - Move instance_url validation and normalization into type __post_init__ - Raise DeploymentError on missing run_id instead of returning partial success with execution_id=None - Preserve exception chains (from exc) instead of suppressing (from None) across create, update, and delete service methods - Add warning logs when _ensure_dict replaces non-dict binding values and when _resolve_lfx_requirement falls back to minimum version - Initialize derived_spec before try block to prevent potential NameError - Log all ToolUploadBatchError errors before re-raising the first - Change SUPPORTED_ADAPTER_DEPLOYMENT_TYPES from mutable set to frozenset - Add tests for 409/422 error mapping in create, unsupported deployment type rejection, empty update rejection, zip artifact extraction paths, validate_connection negative paths, missing run_id, multiple deployment ID rejection, exception chain preservation, and _ensure_dict warning * [autofix.ci] apply automated fixes * fix ruff errors and and todo for status method * fix mypy errors * [autofix.ci] apply automated fixes --------- Co-authored-by: Jordan Frazier Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> --- .../deployment_watsonx_adapter_e2e.py | 2562 +++++++++++++ .../base/langflow/api/v1/deployments.py | 9 + .../langflow/services/adapters/__init__.py | 1 + .../services/adapters/deployment/__init__.py | 1 + .../services/adapters/deployment/context.py | 43 + .../watsonx_orchestrate/__init__.py | 20 + .../deployment/watsonx_orchestrate/client.py | 275 ++ .../watsonx_orchestrate/constants.py | 50 + .../watsonx_orchestrate/core/__init__.py | 1 + .../watsonx_orchestrate/core/config.py | 152 + .../watsonx_orchestrate/core/execution.py | 161 + .../watsonx_orchestrate/core/retry.py | 205 + .../watsonx_orchestrate/core/status.py | 72 + .../watsonx_orchestrate/core/tools.py | 447 +++ .../watsonx_orchestrate/payloads.py | 248 ++ .../deployment/watsonx_orchestrate/service.py | 799 ++++ .../deployment/watsonx_orchestrate/types.py | 81 + .../watsonx_orchestrate/update_helpers.py | 495 +++ .../deployment/watsonx_orchestrate/utils.py | 207 + src/backend/base/langflow/services/utils.py | 22 + src/backend/base/pyproject.toml | 21 + .../unit/services/deployment/__init__.py | 1 + .../deployment/test_watsonx_orchestrate.py | 3407 +++++++++++++++++ .../test_watsonx_orchestrate_update_schema.py | 249 ++ src/lfx/src/lfx/_assets/component_index.json | 4 +- .../adapters/deployment/exceptions.py | 151 +- .../deployment/test_deployment_exceptions.py | 86 + uv.lock | 65 +- 28 files changed, 9829 insertions(+), 6 deletions(-) create mode 100644 scripts/langflow_deployments_api/adapters/deployment_watsonx_adapter_e2e.py create mode 100644 src/backend/base/langflow/services/adapters/__init__.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/__init__.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/context.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/__init__.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/client.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/constants.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/__init__.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/config.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/execution.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/retry.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/status.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/tools.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/payloads.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/service.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/types.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/update_helpers.py create mode 100644 src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/utils.py create mode 100644 src/backend/tests/unit/services/deployment/__init__.py create mode 100644 src/backend/tests/unit/services/deployment/test_watsonx_orchestrate.py create mode 100644 src/backend/tests/unit/services/deployment/test_watsonx_orchestrate_update_schema.py diff --git a/scripts/langflow_deployments_api/adapters/deployment_watsonx_adapter_e2e.py b/scripts/langflow_deployments_api/adapters/deployment_watsonx_adapter_e2e.py new file mode 100644 index 0000000000..fa8a4c64b0 --- /dev/null +++ b/scripts/langflow_deployments_api/adapters/deployment_watsonx_adapter_e2e.py @@ -0,0 +1,2562 @@ +"""Direct Watsonx adapter scenario runner. + +Runs scenario matrices against `WatsonxOrchestrateDeploymentService` directly +(no `/api/v1/deployments` calls). + +Warning: +-------- +This script performs live integration calls and creates real resources in +Watsonx Orchestrate (agents, snapshots/tools, and configs/connections). +By default, cleanup runs at the end of execution, but cleanup is best-effort: +if the process is interrupted or provider deletes fail, resources may remain. +Use `--keep-resources` only when you intentionally want to inspect leftovers. + +Scenario catalog +---------------- +Live create scenarios: +- `live_create_success`: creates config + snapshot + agent successfully (expects Success). +- `live_invalid_config_reference`: rejects config reference binding at create time + (expects InvalidDeploymentOperationError). +- `live_duplicate_snapshot_names_conflict`: duplicate snapshot names collide in wxO (expects DeploymentConflictError). + +Live lifecycle scenarios: +- `live_lifecycle_create_seed`: creates a seed deployment for lifecycle checks (expects Success). +- `live_list_contains_seed`: verifies list-by-id includes the seed deployment (expects Success). +- `live_get_seed`: fetches deployment details by id (expects Success). +- `live_update_seed_name_description`: updates deployment name/description (expects Success). +- `live_get_after_update_reflects_name`: confirms updated name is persisted (expects Success). +- `live_get_status_connected`: confirms status endpoint reports connected deployment (expects Success). +- `live_create_execution_success`: starts an execution run with valid message payload (expects Success). +- `live_get_execution_success`: fetches execution by returned execution id (expects Success). +- `live_delete_seed`: deletes seed deployment agent (expects Success). +- `live_get_after_delete_not_found`: confirms deleted deployment is no longer fetchable + (expects DeploymentNotFoundError). +- `live_status_after_delete_not_found_state`: confirms status on deleted deployment returns not found + (expects DeploymentNotFoundError). + +Live negative scenarios: +- `live_negative_create_seed`: creates a second seed deployment for negative-path checks (expects Success). +- `live_create_execution_rejects_empty_input`: rejects empty execution input payload (expects InvalidContentError). +- `live_delete_missing_not_found`: delete on unknown deployment id returns not found (expects DeploymentNotFoundError). +- `live_negative_delete_seed`: cleans up negative-path seed deployment (expects Success). + +Live update-matrix scenarios: +- Contract note: in provider_data operations, `app_ids` are unprefixed operation ids. + `resource_name_prefix` is applied only when raw resources are created in the provider. +- `upd_spec_only_name_desc`: updates deployment metadata only (expects Success). +- `upd_snapshot_remove_only_no_config`: removes an attached snapshot via provider_data operation + (expects Success). +- `upd_config_only_existing_tools_with_config_id`: rebinds existing attached snapshots + to an explicit existing app id via provider_data (expects Success). +- `upd_snapshot_add_ids_with_config_id`: binds existing snapshot ids using provider_data + operations (expects Success). +- `upd_snapshot_add_raw_with_config_id`: creates and binds raw snapshot payloads using + provider_data tools/connections/operations (expects Success). +- `upd_mixed_add_remove_raw_with_config`: mixed add/remove/raw snapshot update with provider_data + operations (expects Success). +- `upd_reject_bind_with_undeclared_app_id`: rejects bind operation with undeclared app id + in provider_data (expects InvalidContentError). +- `upd_reject_raw_bind_with_undeclared_app_id`: rejects raw-tool bind operation with undeclared app id + in provider_data (expects InvalidContentError). +- `upd_reject_unbind_with_undeclared_app_id`: rejects unbind operation with undeclared app id + in provider_data (expects InvalidContentError). +- `upd_reject_unbind_unknown_tool_id`: rejects unbind on unknown tool id + in provider_data (expects InvalidContentError). +- `upd_missing_add_id_fails`: rejects unknown bind.tool.reference_id in provider_data (expects InvalidContentError). +- `upd_config_raw_payload_conflict`: detects conflict when creating duplicate provider_data + raw connection app id (expects DeploymentConflictError). +- `upd_not_found_deployment`: update unknown deployment id returns not found (expects DeploymentNotFoundError). + +Failpoint scenarios: +- `fp_retry_create_config_then_success`: injects transient config-create failures; retries + then succeeds (expects Success). +- `fp_non_retryable_create_agent_conflict`: injects non-retryable agent conflict (expects DeploymentConflictError). +- `fp_create_agent_failure_triggers_rollback`: injects repeated agent-create failure and + checks rollback (expects DeploymentError). +- `fp_update_bindings_failure_triggers_rollback`: injects update-stage binding failure + and validates rollback path (expects DeploymentError). +- `fp_update_bindings_failure_with_rollback_failure`: injects update failure + rollback + failure and expects terminal error (expects DeploymentError). +""" + +from __future__ import annotations + +import argparse +import asyncio +import os +import re +import textwrap +import types +from contextlib import suppress +from dataclasses import dataclass +from datetime import UTC, datetime +from types import MethodType, SimpleNamespace +from typing import TYPE_CHECKING, Any +from uuid import UUID, uuid4 + +import langflow.services.adapters.deployment.watsonx_orchestrate.core.retry as retry_module +import langflow.services.adapters.deployment.watsonx_orchestrate.service as service_module +import langflow.services.adapters.deployment.watsonx_orchestrate.update_helpers as update_helpers_module +from dotenv import load_dotenv +from fastapi import HTTPException +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from langflow.services.adapters.deployment.context import ( + DeploymentAdapterContext, + DeploymentProviderIDContext, +) +from langflow.services.adapters.deployment.watsonx_orchestrate import ( + WatsonxOrchestrateDeploymentService, + WxOCredentials, +) +from lfx.services.adapters.deployment.exceptions import ( + DeploymentConflictError, + DeploymentError, + DeploymentNotFoundError, + InvalidContentError, + InvalidDeploymentOperationError, + InvalidDeploymentTypeError, +) +from lfx.services.adapters.deployment.schema import ( + BaseDeploymentData, + BaseDeploymentDataUpdate, + BaseFlowArtifact, + ConfigItem, + ConfigListParams, + DeploymentConfig, + DeploymentCreate, + DeploymentListParams, + DeploymentType, + DeploymentUpdate, + ExecutionCreate, + SnapshotItems, + SnapshotListParams, +) + +OUTCOME_SUCCESS = "Success" +OUTCOME_INVALID_OPERATION = "InvalidDeploymentOperationError" +OUTCOME_CONFLICT = "DeploymentConflictError" +OUTCOME_INVALID_CONTENT = "InvalidContentError" +OUTCOME_FAILURE = "DeploymentError" +OUTCOME_NOT_FOUND = "DeploymentNotFoundError" +HTTP_STATUS_NOT_FOUND = 404 +HTTP_STATUS_CONFLICT = 409 +MIN_MIXED_SNAPSHOT_IDS = 2 +DEFAULT_CONCURRENCY_ITERATIONS = 1 + +_INVALID_WXO_NAME_CHARS = re.compile(r"[^A-Za-z0-9_]") + +if TYPE_CHECKING: + from collections.abc import Awaitable, Callable + + +class DummySettingsService: + def __init__(self) -> None: + self.settings = SimpleNamespace() + + +@dataclass(slots=True) +class ScenarioResult: + name: str + expected_outcomes: set[str] + actual_outcome: str + ok: bool + detail: str + + +class WatsonxAdapterDirectE2E: + def __init__( + self, + *, + provider_backend_url: str, + provider_api_key: str, + project_id: str, + mode: str, + keep_resources: bool, + ) -> None: + self.provider_backend_url = provider_backend_url + self.provider_api_key = provider_api_key + self.project_id = project_id + self.mode = mode + self.keep_resources = keep_resources + self.run_suffix = datetime.now(UTC).strftime("%Y%m%d%H%M%S") + "-" + uuid4().hex[:8] + + self.user_id = str(uuid4()) + self.db = object() + self.provider_id = uuid4() + self.service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + import langflow.services.adapters.deployment.watsonx_orchestrate.client as _client_mod + + self._client_mod = _client_mod + + deployment_context = DeploymentAdapterContext(provider_id=self.provider_id) + self._deployment_context_token = DeploymentProviderIDContext.set_current(deployment_context) + + self._original_resolve_wxo_client_credentials = _client_mod.resolve_wxo_client_credentials + + async def _resolve_credentials(*, user_id, db, provider_id): # noqa: ARG001 + authenticator = _client_mod.get_authenticator( + instance_url=self.provider_backend_url, + api_key=self.provider_api_key, + ) + return WxOCredentials(instance_url=self.provider_backend_url, authenticator=authenticator) + + _client_mod.resolve_wxo_client_credentials = _resolve_credentials # type: ignore[assignment] + + self.created_deployment_ids: set[str] = set() + self.created_snapshot_ids: set[str] = set() + self.created_config_ids: set[str] = set() + + async def run(self) -> int: + print("Starting watsonx direct adapter runner...") + print(f"mode={self.mode} project_id={self.project_id} keep_resources={self.keep_resources}") + try: + results: list[ScenarioResult] = [] + if self.mode in {"live", "both"}: + results.extend(await self._run_live_scenarios()) + if self.mode in {"failpoint", "both"}: + results.extend(await self._run_failpoint_scenarios()) + + self._print_summary(results) + if not self.keep_resources: + await self._cleanup_resources() + return 1 if any(not result.ok for result in results) else 0 + finally: + self._client_mod.resolve_wxo_client_credentials = self._original_resolve_wxo_client_credentials + self._client_mod.clear_provider_clients_request_context() + DeploymentProviderIDContext.reset_current(self._deployment_context_token) + + async def _run_live_scenarios(self) -> list[ScenarioResult]: + duplicate_name = self._mk_name("dup_snapshot") + scenarios = [ + { + "name": "live_create_success", + "expected": {OUTCOME_SUCCESS}, + "payload": self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_live_success")], + config=self._build_config_payload(label="cfg_live_success"), + ), + }, + { + "name": "live_invalid_config_reference", + "expected": {OUTCOME_INVALID_OPERATION}, + "payload": self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_live_invalid_ref")], + config_reference_id="cfg_ref_not_supported", + ), + }, + { + "name": "live_duplicate_snapshot_names_conflict", + "expected": {OUTCOME_CONFLICT}, + "payload": self._build_create_payload( + snapshots=[ + self._build_flow_payload(label="snap_dup_a", name_override=duplicate_name), + self._build_flow_payload(label="snap_dup_b", name_override=duplicate_name), + ], + config=self._build_config_payload(label="cfg_live_dup"), + ), + }, + ] + results = await self._run_scenarios(scenarios) + results.extend(await self._run_live_lifecycle_scenarios()) + results.extend(await self._run_live_update_matrix_scenarios()) + results.extend(await self._run_live_concurrency_scenarios()) + results.extend(await self._run_live_negative_scenarios()) + return results + + async def _run_failpoint_scenarios(self) -> list[ScenarioResult]: + scenarios = [ + { + "name": "fp_retry_create_config_then_success", + "expected": {OUTCOME_SUCCESS}, + "payload": self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_fp_retry")], + config=self._build_config_payload(label="cfg_fp_retry"), + ), + "inject": { + "create_config": {"fail_first_n": 2, "error_type": "runtime", "message": "fp_create_config_retry"} + }, + }, + { + "name": "fp_non_retryable_create_agent_conflict", + "expected": {OUTCOME_CONFLICT}, + "payload": self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_fp_conflict")], + config=self._build_config_payload(label="cfg_fp_conflict"), + ), + "inject": { + "create_agent": { + "fail_first_n": 1, + "error_type": "domain_conflict", + "message": "fp_create_agent_conflict", + } + }, + }, + { + "name": "fp_create_agent_failure_triggers_rollback", + "expected": {OUTCOME_FAILURE}, + "detail_contains": "Please check server logs for details", + "payload": self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_fp_rollback")], + config=self._build_config_payload(label="cfg_fp_rollback"), + ), + "inject": { + "create_agent": { + "fail_first_n": 3, + "error_type": "runtime", + "message": "fp_create_agent_final", + }, + "rollback_delete_config": { + "fail_first_n": 2, + "error_type": "runtime", + "message": "fp_rollback_delete_config", + }, + }, + }, + ] + results = await self._run_scenarios(scenarios) + results.extend(await self._run_update_failpoint_scenarios()) + return results + + async def _run_scenarios(self, scenarios: list[dict[str, Any]]) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + for index, scenario in enumerate(scenarios, start=1): + print(f"\n[{index}/{len(scenarios)}] {scenario['name']}") + status_code, detail, created = await self._run_create(scenario["payload"], inject=scenario.get("inject")) + detail_contains = str(scenario.get("detail_contains") or "").strip() + detail_ok = not detail_contains or detail_contains in detail + ok = status_code in scenario["expected"] and detail_ok + + if created: + self.created_deployment_ids.add(created["deployment_id"]) + self.created_snapshot_ids.update(created["snapshot_ids"]) + if created["config_id"]: + self.created_config_ids.add(created["config_id"]) + + results.append( + ScenarioResult( + name=scenario["name"], + expected_outcomes=set(scenario["expected"]), + actual_outcome=status_code, + ok=ok, + detail=detail[:600], + ) + ) + return results + + async def _run_create( + self, + payload: DeploymentCreate, + *, + inject: dict[str, dict[str, Any]] | None = None, + ) -> tuple[str, str, dict[str, Any] | None]: + originals: list[tuple[Any, str, Any]] = [] + try: + if inject: + self._apply_injections(inject, originals) + + result = await self.service.create(user_id=self.user_id, payload=payload, db=self.db) + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + created = { + "deployment_id": str(result.id), + "config_id": str(result.config_id) if result.config_id else None, + "snapshot_ids": {str(item) for item in (result.snapshot_ids or [])}, + } + return OUTCOME_SUCCESS, "created", created + finally: + for target, attr_name, original in originals: + setattr(target, attr_name, original) + + async def _run_list(self, *, params: DeploymentListParams | None = None) -> tuple[str, str, Any | None]: + try: + result = await self.service.list(user_id=self.user_id, db=self.db, params=params) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "listed", result + + async def _run_get(self, deployment_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.get(user_id=self.user_id, deployment_id=deployment_id, db=self.db) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "fetched", result + + async def _run_update( + self, + deployment_id: str, + payload: DeploymentUpdate, + *, + inject: dict[str, dict[str, Any]] | None = None, + ) -> tuple[str, str, Any | None]: + originals: list[tuple[Any, str, Any]] = [] + try: + if inject: + self._apply_injections(inject, originals) + result = await self.service.update( + user_id=self.user_id, + deployment_id=deployment_id, + payload=payload, + db=self.db, + ) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "updated", result + finally: + for target, attr_name, original in originals: + setattr(target, attr_name, original) + + async def _run_list_snapshots(self, deployment_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.list_snapshots( + user_id=self.user_id, + params=SnapshotListParams(deployment_ids=[deployment_id]), + db=self.db, + ) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "snapshots_listed", result + + async def _run_list_configs(self, deployment_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.list_configs( + user_id=self.user_id, + params=ConfigListParams(deployment_ids=[deployment_id]), + db=self.db, + ) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "configs_listed", result + + async def _run_status(self, deployment_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.get_status(user_id=self.user_id, deployment_id=deployment_id, db=self.db) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "status", result + + async def _run_create_execution( + self, + deployment_id: str, + *, + provider_data: dict[str, Any], + ) -> tuple[str, str, Any | None]: + try: + result = await self.service.create_execution( + user_id=self.user_id, + payload=ExecutionCreate(deployment_id=deployment_id, provider_data=provider_data), + db=self.db, + ) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "execution_created", result + + async def _run_get_execution(self, execution_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.get_execution(user_id=self.user_id, execution_id=execution_id, db=self.db) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "execution_fetched", result + + async def _run_delete(self, deployment_id: str) -> tuple[str, str, Any | None]: + try: + result = await self.service.delete(user_id=self.user_id, deployment_id=deployment_id, db=self.db) + except DeploymentNotFoundError as exc: + return OUTCOME_NOT_FOUND, str(exc), None + except DeploymentConflictError as exc: + return OUTCOME_CONFLICT, exc.message, None + except InvalidContentError as exc: + return OUTCOME_INVALID_CONTENT, exc.message, None + except (InvalidDeploymentOperationError, InvalidDeploymentTypeError) as exc: + return OUTCOME_INVALID_OPERATION, exc.message, None + except DeploymentError as exc: + return OUTCOME_FAILURE, exc.message, None + except HTTPException as exc: + return self._outcome_from_http_exception(exc), str(exc.detail), None + except Exception as exc: # noqa: BLE001 + return OUTCOME_FAILURE, str(exc), None + else: + return OUTCOME_SUCCESS, "deleted", result + + def _build_result( + self, + *, + name: str, + expected: set[str], + actual_outcome: str, + detail: str, + ok: bool, + ) -> ScenarioResult: + return ScenarioResult( + name=name, + expected_outcomes=expected, + actual_outcome=actual_outcome, + ok=ok, + detail=detail[:600], + ) + + def _outcome_from_http_exception(self, exc: HTTPException) -> str: + status_code = int(exc.status_code) + if status_code == HTTP_STATUS_NOT_FOUND: + return OUTCOME_NOT_FOUND + if status_code == HTTP_STATUS_CONFLICT: + return OUTCOME_CONFLICT + if status_code in {400, 405}: + return OUTCOME_INVALID_OPERATION + if status_code in {413, 415, 422}: + return OUTCOME_INVALID_CONTENT + return OUTCOME_FAILURE + + async def _run_live_lifecycle_scenarios(self) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + print("\n[life/1] live_lifecycle_create_seed") + status_code, detail, created = await self._run_create( + self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_live_lifecycle_seed")], + config=self._build_config_payload(label="cfg_live_lifecycle_seed"), + ) + ) + create_ok = status_code == OUTCOME_SUCCESS and created is not None + results.append( + self._build_result( + name="live_lifecycle_create_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=create_ok, + ) + ) + if not create_ok or created is None: + return results + + deployment_id = str(created["deployment_id"]) + self.created_deployment_ids.add(deployment_id) + self.created_snapshot_ids.update(created["snapshot_ids"]) + if created["config_id"]: + self.created_config_ids.add(str(created["config_id"])) + + print("[life/2] live_list_contains_seed") + status_code, detail, list_result = await self._run_list( + params=DeploymentListParams(deployment_ids=[deployment_id]) + ) + list_contains_seed = bool( + list_result + and any(str(deployment.id) == deployment_id for deployment in getattr(list_result, "deployments", [])) + ) + results.append( + self._build_result( + name="live_list_contains_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and list_contains_seed, + ) + ) + + print("[life/3] live_get_seed") + status_code, detail, get_result = await self._run_get(deployment_id) + got_seed = bool(get_result and str(get_result.id) == deployment_id) + results.append( + self._build_result( + name="live_get_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and got_seed, + ) + ) + + updated_name = self._mk_name("dep_agent_updated") + print("[life/4] live_update_seed_name_description") + status_code, detail, _ = await self._run_update( + deployment_id, + DeploymentUpdate( + spec=BaseDeploymentDataUpdate( + name=updated_name, + description="updated by direct adapter e2e", + ) + ), + ) + results.append( + self._build_result( + name="live_update_seed_name_description", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS, + ) + ) + + print("[life/5] live_get_after_update_reflects_name") + status_code, detail, get_after_update = await self._run_get(deployment_id) + updated_name_ok = bool(get_after_update and getattr(get_after_update, "name", None) == updated_name) + results.append( + self._build_result( + name="live_get_after_update_reflects_name", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and updated_name_ok, + ) + ) + + print("[life/6] live_get_status_connected") + status_code, detail, status_result = await self._run_status(deployment_id) + connected_ok = bool(status_result and getattr(status_result, "provider_data", {}).get("status") == "connected") + results.append( + self._build_result( + name="live_get_status_connected", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and connected_ok, + ) + ) + + print("[life/7] live_create_execution_success") + status_code, detail, execution_create_result = await self._run_create_execution( + deployment_id, + provider_data={"message": {"role": "user", "content": "ping from direct adapter e2e"}}, + ) + has_execution_id = bool(execution_create_result and getattr(execution_create_result, "execution_id", None)) + results.append( + self._build_result( + name="live_create_execution_success", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and has_execution_id, + ) + ) + + execution_id_value = ( + execution_create_result.execution_id + if execution_create_result and getattr(execution_create_result, "execution_id", None) + else None + ) + execution_id = str(execution_id_value) if execution_id_value else None + + if execution_id: + print("[life/8] live_get_execution_success") + status_code, detail, execution_status_result = await self._run_get_execution(execution_id) + execution_ok = bool( + execution_status_result and str(getattr(execution_status_result, "execution_id", "")) == execution_id + ) + results.append( + self._build_result( + name="live_get_execution_success", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_SUCCESS and execution_ok, + ) + ) + + print("[life/9] live_delete_seed") + status_code, detail, _ = await self._run_delete(deployment_id) + delete_ok = status_code == OUTCOME_SUCCESS + if delete_ok: + self.created_deployment_ids.discard(deployment_id) + results.append( + self._build_result( + name="live_delete_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=delete_ok, + ) + ) + + print("[life/10] live_get_after_delete_not_found") + status_code, detail, _ = await self._run_get(deployment_id) + results.append( + self._build_result( + name="live_get_after_delete_not_found", + expected={OUTCOME_NOT_FOUND}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_NOT_FOUND, + ) + ) + + print("[life/11] live_status_after_delete_not_found_state") + status_code, detail, _ = await self._run_status(deployment_id) + results.append( + self._build_result( + name="live_status_after_delete_not_found_state", + expected={OUTCOME_NOT_FOUND}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_NOT_FOUND, + ) + ) + + return results + + async def _run_live_negative_scenarios(self) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + print("\n[neg/1] live_negative_create_seed") + status_code, detail, created = await self._run_create( + self._build_create_payload( + snapshots=[self._build_flow_payload(label="snap_live_negative_seed")], + config=self._build_config_payload(label="cfg_live_negative_seed"), + ) + ) + create_ok = status_code == OUTCOME_SUCCESS and created is not None + results.append( + self._build_result( + name="live_negative_create_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=create_ok, + ) + ) + if not create_ok or created is None: + return results + + deployment_id = str(created["deployment_id"]) + self.created_deployment_ids.add(deployment_id) + self.created_snapshot_ids.update(created["snapshot_ids"]) + if created["config_id"]: + self.created_config_ids.add(str(created["config_id"])) + + print("[neg/2] live_create_execution_rejects_empty_input") + status_code, detail, _ = await self._run_create_execution( + deployment_id, + provider_data={"input": " "}, + ) + results.append( + self._build_result( + name="live_create_execution_rejects_empty_input", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[neg/3] live_delete_missing_not_found") + status_code, detail, _ = await self._run_delete(str(uuid4())) + results.append( + self._build_result( + name="live_delete_missing_not_found", + expected={OUTCOME_NOT_FOUND}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_NOT_FOUND, + ) + ) + + print("[neg/4] live_negative_delete_seed") + status_code, detail, _ = await self._run_delete(deployment_id) + delete_ok = status_code == OUTCOME_SUCCESS + if delete_ok: + self.created_deployment_ids.discard(deployment_id) + results.append( + self._build_result( + name="live_negative_delete_seed", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=delete_ok, + ) + ) + + return results + + def _extract_snapshot_ids(self, snapshot_result: Any) -> set[str]: + snapshots = getattr(snapshot_result, "snapshots", []) if snapshot_result else [] + return {str(snapshot.id) for snapshot in snapshots if snapshot and snapshot.id} + + async def _create_update_seed( + self, + *, + label: str, + snapshot_count: int = 2, + ) -> tuple[str, str | None, set[str], str]: + snapshots = [self._build_flow_payload(label=f"{label}_snap_{idx}") for idx in range(snapshot_count)] + status_code, detail, created = await self._run_create( + self._build_create_payload( + snapshots=snapshots, + config=self._build_config_payload(label=f"{label}_cfg"), + ) + ) + if status_code != OUTCOME_SUCCESS or not created: + msg = f"Failed to create seed deployment '{label}': status={status_code}, detail={detail}" + raise RuntimeError(msg) + deployment_id = str(created["deployment_id"]) + self.created_deployment_ids.add(deployment_id) + self.created_snapshot_ids.update(created["snapshot_ids"]) + if created["config_id"]: + self.created_config_ids.add(str(created["config_id"])) + return deployment_id, created["config_id"], set(created["snapshot_ids"]), detail + + async def _run_live_update_matrix_scenarios(self) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + print("\n[upd] building update matrix seed resources") + ( + primary_deployment_id, + _primary_config_id, + primary_snapshot_ids, + _, + ) = await self._create_update_seed(label="upd_primary", snapshot_count=2) + donor_deployment_id, donor_config_id, donor_snapshot_ids, _ = await self._create_update_seed( + label="upd_donor", + snapshot_count=1, + ) + mixed_donor_deployment_id, _mixed_donor_cfg, mixed_donor_snapshot_ids, _ = await self._create_update_seed( + label="upd_mixed_donor", + snapshot_count=1, + ) + + donor_snapshot_id = next(iter(donor_snapshot_ids), "") + mixed_donor_snapshot_id = next(iter(mixed_donor_snapshot_ids), "") + removable_snapshot_id = next(iter(primary_snapshot_ids), "") + retained_snapshot_ids = set(primary_snapshot_ids) + retained_snapshot_ids.discard(removable_snapshot_id) + if not donor_config_id: + results.append( + self._build_result( + name="upd_seed_missing_donor_config", + expected={OUTCOME_SUCCESS}, + actual_outcome=OUTCOME_FAILURE, + detail="donor deployment config id is missing", + ok=False, + ) + ) + return results + + print("[upd/1] upd_spec_only_name_desc") + updated_name = self._mk_name("dep_upd_spec_only") + status_code, detail, update_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + spec=BaseDeploymentDataUpdate( + name=updated_name, + description="updated by update matrix spec-only", + ) + ), + ) + get_status, _get_detail, get_after_update = await self._run_get(primary_deployment_id) + spec_ok = bool(get_after_update and getattr(get_after_update, "name", None) == updated_name) + spec_snapshot_ids_ok = bool(update_result and not getattr(update_result, "snapshot_ids", [])) + results.append( + self._build_result( + name="upd_spec_only_name_desc", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and get_status == OUTCOME_SUCCESS + and spec_ok + and spec_snapshot_ids_ok + ), + ) + ) + + print("[upd/2] upd_snapshot_remove_only_no_config") + status_code, detail, remove_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [removable_snapshot_id]}, + "operations": [{"op": "remove_tool", "tool_id": removable_snapshot_id}], + } + ), + ) + list_status, _list_detail, list_after_remove = await self._run_list_snapshots(primary_deployment_id) + attached_after_remove = self._extract_snapshot_ids(list_after_remove) + remove_snapshot_ids_ok = bool(remove_result and not getattr(remove_result, "snapshot_ids", [])) + results.append( + self._build_result( + name="upd_snapshot_remove_only_no_config", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and list_status == OUTCOME_SUCCESS + and removable_snapshot_id not in attached_after_remove + and retained_snapshot_ids.issubset(attached_after_remove) + and remove_snapshot_ids_ok + ), + ) + ) + + print("[upd/3] upd_config_only_existing_tools_with_config_id") + retained_snapshot_ids_sorted = sorted(retained_snapshot_ids) + status_code, detail, config_only_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": retained_snapshot_ids_sorted}, + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": tool_id}, + "app_ids": [str(donor_config_id)], + } + for tool_id in retained_snapshot_ids_sorted + ], + } + ), + ) + list_status, _list_detail, list_after_config_only = await self._run_list_snapshots(primary_deployment_id) + attached_after_config_only = self._extract_snapshot_ids(list_after_config_only) + config_only_snapshot_ids = ( + {str(item) for item in getattr(config_only_result, "snapshot_ids", [])} if config_only_result else set() + ) + config_only_snapshot_ids_ok = retained_snapshot_ids.issubset(config_only_snapshot_ids) + results.append( + self._build_result( + name="upd_config_only_existing_tools_with_config_id", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and list_status == OUTCOME_SUCCESS + and retained_snapshot_ids.issubset(attached_after_config_only) + and config_only_snapshot_ids_ok + ), + ) + ) + + print("[upd/4] upd_snapshot_add_ids_with_config_id") + status_code, detail, add_id_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [donor_snapshot_id]}, + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": donor_snapshot_id}, + "app_ids": [str(donor_config_id)], + } + ], + } + ), + ) + list_status, _list_detail, list_after_add_id = await self._run_list_snapshots(primary_deployment_id) + attached_after_add_id = self._extract_snapshot_ids(list_after_add_id) + add_id_snapshot_ids = ( + {str(item) for item in getattr(add_id_result, "snapshot_ids", [])} if add_id_result else set() + ) + results.append( + self._build_result( + name="upd_snapshot_add_ids_with_config_id", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and list_status == OUTCOME_SUCCESS + and donor_snapshot_id in attached_after_add_id + and donor_snapshot_id in add_id_snapshot_ids + ), + ) + ) + + print("[upd/5] upd_snapshot_add_raw_with_config_id") + raw_payload = self._build_flow_payload(label="upd_add_raw") + status_code, detail, add_raw_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "resource_name_prefix": f"e2e_upd_{uuid4().hex[:6]}_", + "tools": {"raw_payloads": [raw_payload.model_dump(mode="json")]}, + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": raw_payload.name}, + "app_ids": [str(donor_config_id)], + } + ], + } + ), + ) + add_raw_snapshot_ids = ( + {str(item) for item in getattr(add_raw_result, "snapshot_ids", [])} if add_raw_result else set() + ) + self.created_snapshot_ids.update(add_raw_snapshot_ids) + list_status, _list_detail, list_after_add_raw = await self._run_list_snapshots(primary_deployment_id) + attached_after_add_raw = self._extract_snapshot_ids(list_after_add_raw) + add_raw_created_ids = add_raw_snapshot_ids + results.append( + self._build_result( + name="upd_snapshot_add_raw_with_config_id", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and list_status == OUTCOME_SUCCESS + and bool(add_raw_created_ids) + and add_raw_created_ids.issubset(attached_after_add_raw) + ), + ) + ) + + print("[upd/6] upd_mixed_add_remove_raw_with_config") + mixed_raw_payload = self._build_flow_payload(label="upd_mixed_raw") + mixed_remove_id = next(iter(retained_snapshot_ids), "") + status_code, detail, mixed_result = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "resource_name_prefix": f"e2e_upd_mix_{uuid4().hex[:6]}_", + "tools": { + "existing_ids": [mixed_donor_snapshot_id, mixed_remove_id], + "raw_payloads": [mixed_raw_payload.model_dump(mode="json")], + }, + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": mixed_donor_snapshot_id}, + "app_ids": [str(donor_config_id)], + }, + {"op": "remove_tool", "tool_id": mixed_remove_id}, + { + "op": "bind", + "tool": {"name_of_raw": mixed_raw_payload.name}, + "app_ids": [str(donor_config_id)], + }, + ], + } + ), + ) + mixed_snapshot_ids = ( + {str(item) for item in getattr(mixed_result, "snapshot_ids", [])} if mixed_result else set() + ) + self.created_snapshot_ids.update(mixed_snapshot_ids) + list_status, _list_detail, list_after_mixed = await self._run_list_snapshots(primary_deployment_id) + attached_after_mixed = self._extract_snapshot_ids(list_after_mixed) + results.append( + self._build_result( + name="upd_mixed_add_remove_raw_with_config", + expected={OUTCOME_SUCCESS}, + actual_outcome=status_code, + detail=detail, + ok=( + status_code == OUTCOME_SUCCESS + and list_status == OUTCOME_SUCCESS + and mixed_remove_id not in attached_after_mixed + and mixed_donor_snapshot_id in attached_after_mixed + and mixed_donor_snapshot_id in mixed_snapshot_ids + and len(mixed_snapshot_ids) >= MIN_MIXED_SNAPSHOT_IDS + ), + ) + ) + + print("[upd/7] upd_reject_bind_with_undeclared_app_id") + status_code, detail, _ = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [donor_snapshot_id]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": donor_snapshot_id}, + "app_ids": ["undeclared_app_for_bind"], + } + ], + } + ), + ) + results.append( + self._build_result( + name="upd_reject_bind_with_undeclared_app_id", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[upd/8] upd_reject_raw_bind_with_undeclared_app_id") + missing_cfg_raw_payload = self._build_flow_payload(label="upd_no_cfg_raw") + status_code, detail, _ = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"raw_payloads": [missing_cfg_raw_payload.model_dump(mode="json")]}, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": missing_cfg_raw_payload.name}, + "app_ids": ["undeclared_app_for_raw_bind"], + } + ], + } + ), + ) + results.append( + self._build_result( + name="upd_reject_raw_bind_with_undeclared_app_id", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[upd/9] upd_reject_unbind_with_undeclared_app_id") + status_code, detail, _ = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [donor_snapshot_id]}, + "operations": [ + { + "op": "unbind", + "tool_id": donor_snapshot_id, + "app_ids": ["undeclared_app_for_unbind"], + } + ], + } + ), + ) + results.append( + self._build_result( + name="upd_reject_unbind_with_undeclared_app_id", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[upd/10] upd_reject_unbind_unknown_tool_id") + status_code, detail, _ = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "unbind", + "tool_id": str(uuid4()), + "app_ids": [str(donor_config_id)], + } + ], + } + ), + ) + results.append( + self._build_result( + name="upd_reject_unbind_unknown_tool_id", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[upd/11] upd_missing_add_id_fails") + status_code, detail, _ = await self._run_update( + primary_deployment_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [donor_snapshot_id]}, + "connections": {"existing_app_ids": [str(donor_config_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": str(uuid4())}, + "app_ids": [str(donor_config_id)], + } + ], + } + ), + ) + results.append( + self._build_result( + name="upd_missing_add_id_fails", + expected={OUTCOME_INVALID_CONTENT}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_INVALID_CONTENT, + ) + ) + + print("[upd/12] upd_config_raw_payload_conflict") + conflict_seed_deployment_id, _conflict_cfg_id, _conflict_snapshot_ids, _ = await self._create_update_seed( + label="upd_conflict_seed", + snapshot_count=1, + ) + conflict_suffix = uuid4().hex[:8] + conflict_prefix = f"e2e_upd_conflict_{conflict_suffix}_" + conflict_name = f"dup_cfg_{conflict_suffix}" + conflict_tool_id = next(iter(_conflict_snapshot_ids), "") + if not conflict_tool_id: + results.append( + self._build_result( + name="upd_config_raw_payload_conflict", + expected={OUTCOME_CONFLICT}, + actual_outcome=OUTCOME_FAILURE, + detail="conflict seed snapshot id is missing", + ok=False, + ) + ) + return results + conflict_payload = DeploymentUpdate( + provider_data={ + "resource_name_prefix": conflict_prefix, + "tools": {"existing_ids": [conflict_tool_id]}, + "connections": {"raw_payloads": [{"app_id": conflict_name, "environment_variables": {}}]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": conflict_tool_id}, + "app_ids": [conflict_name], + } + ], + } + ) + setup_status, _setup_detail, _ = await self._run_update(conflict_seed_deployment_id, conflict_payload) + if setup_status == OUTCOME_SUCCESS: + self.created_config_ids.add(f"{conflict_prefix}{conflict_name}") + status_code, detail, _ = await self._run_update(conflict_seed_deployment_id, conflict_payload) + results.append( + self._build_result( + name="upd_config_raw_payload_conflict", + expected={OUTCOME_CONFLICT}, + actual_outcome=status_code, + detail=(f"setup={setup_status}:{_setup_detail} conflict={status_code}:{detail}"), + ok=setup_status == OUTCOME_SUCCESS and status_code == OUTCOME_CONFLICT, + ) + ) + + print("[upd/13] upd_not_found_deployment") + status_code, detail, _ = await self._run_update( + str(uuid4()), + DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="not found update")), + ) + results.append( + self._build_result( + name="upd_not_found_deployment", + expected={OUTCOME_NOT_FOUND}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_NOT_FOUND, + ) + ) + + # keep seed deployments tracked for shared final cleanup + self.created_deployment_ids.add(primary_deployment_id) + self.created_deployment_ids.add(donor_deployment_id) + self.created_deployment_ids.add(mixed_donor_deployment_id) + return results + + async def _run_live_concurrency_scenarios(self) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + iterations_raw = os.getenv("WXO_CONCURRENCY_REPEAT", str(DEFAULT_CONCURRENCY_ITERATIONS)).strip() + try: + iterations = max(1, int(iterations_raw)) + except ValueError: + iterations = DEFAULT_CONCURRENCY_ITERATIONS + + print(f"\n[concurrency] running iterations={iterations}") + for iteration in range(1, iterations + 1): + print(f"\n[concurrency] iteration {iteration}/{iterations}") + results.extend(await self._run_live_concurrency_iteration(iteration=iteration)) + return results + + async def _run_live_concurrency_iteration(self, *, iteration: int) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + + print(f"[cc/{iteration}.1] cc_create_same_prefix_race") + shared_prefix = f"e2e_cc_shared_{uuid4().hex[:6]}_" + shared_dep_name = self._mk_name("dep_cc_shared") + shared_cfg_name = self._mk_name("cfg_cc_shared") + shared_snap_name = self._mk_name("snap_cc_shared") + shared_payload = self._build_create_payload( + snapshots=[self._build_flow_payload(label="cc_shared_snap", name_override=shared_snap_name)], + config=DeploymentConfig( + name=shared_cfg_name, + description="concurrency create collision", + environment_variables={}, + ), + resource_name_prefix=shared_prefix, + ) + shared_payload.spec = shared_payload.spec.model_copy(update={"name": shared_dep_name}, deep=True) + create_race = await self._run_parallel_calls( + { + "left": lambda: self._run_create(shared_payload.model_copy(deep=True)), + "right": lambda: self._run_create(shared_payload.model_copy(deep=True)), + } + ) + left_status, left_detail, left_created = create_race["left"] + right_status, right_detail, right_created = create_race["right"] + self._track_created_result(left_created) + self._track_created_result(right_created) + create_pair = (left_status, right_status) + create_pair_ok = create_pair in { + (OUTCOME_SUCCESS, OUTCOME_CONFLICT), + (OUTCOME_CONFLICT, OUTCOME_SUCCESS), + (OUTCOME_SUCCESS, OUTCOME_SUCCESS), + } + create_no_internal = OUTCOME_FAILURE not in {left_status, right_status} + results.append( + self._build_result( + name="cc_create_same_prefix_race", + expected={OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=max(left_status, right_status), + detail=f"left={left_status}:{left_detail} right={right_status}:{right_detail}", + ok=create_pair_ok and create_no_internal, + ) + ) + + print(f"[cc/{iteration}.2] cc_update_spec_vs_snapshot_race") + primary_id, _primary_cfg_id, _primary_snaps, _ = await self._create_update_seed( + label=f"cc_upd_primary_{iteration}", + snapshot_count=2, + ) + _donor_id, donor_cfg_id, donor_snapshot_ids, _ = await self._create_update_seed( + label=f"cc_upd_donor_{iteration}", + snapshot_count=1, + ) + donor_snapshot_id = next(iter(donor_snapshot_ids), "") + update_race = await self._run_parallel_calls( + { + "spec": lambda: self._run_update( + primary_id, + DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="cc concurrent spec update")), + ), + "snapshot": lambda: self._run_update( + primary_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": [donor_snapshot_id]}, + "connections": {"existing_app_ids": [str(donor_cfg_id)]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": donor_snapshot_id}, + "app_ids": [str(donor_cfg_id)], + } + ], + } + ), + ), + } + ) + spec_status, spec_detail, _ = update_race["spec"] + snapshot_status, snapshot_detail, _ = update_race["snapshot"] + list_status, _list_detail, list_after = await self._run_list_snapshots(primary_id) + attached_after = self._extract_snapshot_ids(list_after) + no_internal_race = OUTCOME_FAILURE not in {spec_status, snapshot_status} + results.append( + self._build_result( + name="cc_update_spec_vs_snapshot_race", + expected={ + OUTCOME_SUCCESS, + OUTCOME_INVALID_OPERATION, + OUTCOME_CONFLICT, + OUTCOME_INVALID_CONTENT, + OUTCOME_NOT_FOUND, + }, + actual_outcome=max(spec_status, snapshot_status), + detail=f"spec={spec_status}:{spec_detail} snapshot={snapshot_status}:{snapshot_detail}", + ok=( + no_internal_race + and list_status == OUTCOME_SUCCESS + and self._has_unique_snapshot_ids(attached_after) + ), + ) + ) + + print(f"[cc/{iteration}.3] cc_update_vs_delete_deployment_race") + race_delete_id, _race_delete_cfg, _race_delete_snaps, _ = await self._create_update_seed( + label=f"cc_upd_del_{iteration}", + snapshot_count=1, + ) + update_delete_race = await self._run_parallel_calls( + { + "update": lambda: self._run_update( + race_delete_id, + DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="cc update while delete")), + ), + "delete": lambda: self._run_delete(race_delete_id), + } + ) + upd_status, upd_detail, _ = update_delete_race["update"] + del_status, del_detail, _ = update_delete_race["delete"] + status_after_delete_race, _status_detail, _status_payload = await self._run_status(race_delete_id) + allowed_pairs = { + (OUTCOME_SUCCESS, OUTCOME_SUCCESS), + (OUTCOME_NOT_FOUND, OUTCOME_SUCCESS), + (OUTCOME_SUCCESS, OUTCOME_NOT_FOUND), + (OUTCOME_NOT_FOUND, OUTCOME_NOT_FOUND), + } + results.append( + self._build_result( + name="cc_update_vs_delete_deployment_race", + expected={OUTCOME_SUCCESS, OUTCOME_NOT_FOUND, OUTCOME_FAILURE}, + actual_outcome=max(upd_status, del_status), + detail=( + f"update={upd_status}:{upd_detail} " + f"delete={del_status}:{del_detail} status={status_after_delete_race}" + ), + ok=(upd_status, del_status) in allowed_pairs + or ( + OUTCOME_FAILURE in {upd_status, del_status} and "not found" in f"{upd_detail} {del_detail}".lower() + ), + ) + ) + self.created_deployment_ids.discard(race_delete_id) + + print(f"[cc/{iteration}.4] cc_execution_vs_delete_deployment_race") + exec_delete_id, _exec_delete_cfg, _exec_delete_snapshots, _ = await self._create_update_seed( + label=f"cc_exec_del_{iteration}", + snapshot_count=1, + ) + exec_delete_race = await self._run_parallel_calls( + { + "execution": lambda: self._run_create_execution( + exec_delete_id, + provider_data={"message": {"role": "user", "content": "cc execution while delete"}}, + ), + "delete": lambda: self._run_delete(exec_delete_id), + } + ) + exec_status, exec_detail, _ = exec_delete_race["execution"] + del_exec_status, del_exec_detail, _ = exec_delete_race["delete"] + results.append( + self._build_result( + name="cc_execution_vs_delete_deployment_race", + expected={OUTCOME_SUCCESS, OUTCOME_NOT_FOUND, OUTCOME_SUCCESS, OUTCOME_FAILURE}, + actual_outcome=max(exec_status, del_exec_status), + detail=f"execution={exec_status}:{exec_detail} delete={del_exec_status}:{del_exec_detail}", + ok=OUTCOME_FAILURE not in {exec_status, del_exec_status} + or "not found" in f"{exec_detail} {del_exec_detail}".lower(), + ) + ) + self.created_deployment_ids.discard(exec_delete_id) + + print(f"[cc/{iteration}.5] cc_delete_snapshot_during_update_bindings") + delete_bind_id, bind_cfg_id, bind_snapshot_ids, _ = await self._create_update_seed( + label=f"cc_del_bind_{iteration}", + snapshot_count=2, + ) + delete_target_snapshot_id = next(iter(bind_snapshot_ids), "") + + async def _delete_target_before_bind(*args: Any, **kwargs: Any) -> None: # noqa: ARG001 + clients = kwargs.get("clients") + existing_tool_deltas = kwargs.get("existing_tool_deltas") or {} + target_ids = list(existing_tool_deltas.keys()) + tool_id = str(target_ids[0]) if target_ids else delete_target_snapshot_id + if clients and tool_id: + await self._safe_delete_snapshot(clients=clients, snapshot_id=tool_id) + + del_bind_status, del_bind_detail, _ = await self._run_with_stage_hook( + stage="update_bindings", + operation=lambda: self._run_update( + delete_bind_id, + DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": sorted(bind_snapshot_ids)}, + "connections": {"existing_app_ids": [str(bind_cfg_id)]}, + "operations": [ + { + "op": "unbind", + "tool_id": tool_id, + "app_ids": [str(bind_cfg_id)], + } + for tool_id in sorted(bind_snapshot_ids) + ], + } + ), + ), + hook_before=_delete_target_before_bind, + ) + results.append( + self._build_result( + name="cc_delete_snapshot_during_update_bindings", + expected={OUTCOME_INVALID_CONTENT, OUTCOME_INVALID_OPERATION, OUTCOME_SUCCESS}, + actual_outcome=del_bind_status, + detail=del_bind_detail, + ok=del_bind_status in {OUTCOME_SUCCESS, OUTCOME_INVALID_OPERATION, OUTCOME_INVALID_CONTENT}, + ) + ) + + print(f"[cc/{iteration}.6] cc_delete_config_after_update_raw_create") + delete_cfg_id, _delete_cfg_base_id, delete_cfg_snapshots, _ = await self._create_update_seed( + label=f"cc_del_cfg_{iteration}", + snapshot_count=1, + ) + target_tool_id = next(iter(delete_cfg_snapshots), "") + if not target_tool_id: + results.append( + self._build_result( + name="cc_delete_config_after_update_raw_create", + expected={OUTCOME_INVALID_OPERATION, OUTCOME_INVALID_CONTENT, OUTCOME_CONFLICT, OUTCOME_FAILURE}, + actual_outcome=OUTCOME_FAILURE, + detail="seed snapshot id missing for update raw config stage", + ok=False, + ) + ) + return results + cfg_prefix = f"e2e_cc_del_cfg_{uuid4().hex[:6]}_" + raw_cfg_name = self._mk_name("cc_raw_cfg") + + async def _delete_created_app_after_config_create(created_app_id: Any, **kwargs: Any) -> None: + if not created_app_id: + created_app_id = kwargs.get("app_id") + app_id = str(created_app_id or "").strip() + clients = await self.service._get_provider_clients(user_id=self.user_id, db=self.db) # noqa: SLF001 + if app_id: + await self._safe_delete_config(clients=clients, config_id=app_id) + + del_cfg_status, del_cfg_detail, _ = await self._run_with_stage_hook( + stage="update_create_config", + operation=lambda: self._run_update( + delete_cfg_id, + DeploymentUpdate( + provider_data={ + "resource_name_prefix": cfg_prefix, + "tools": {"existing_ids": [target_tool_id]}, + "connections": {"raw_payloads": [{"app_id": raw_cfg_name, "environment_variables": {}}]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": target_tool_id}, + "app_ids": [raw_cfg_name], + } + ], + } + ), + ), + hook_after=_delete_created_app_after_config_create, + ) + results.append( + self._build_result( + name="cc_delete_config_after_update_raw_create", + expected={OUTCOME_INVALID_OPERATION, OUTCOME_INVALID_CONTENT, OUTCOME_CONFLICT, OUTCOME_FAILURE}, + actual_outcome=del_cfg_status, + detail=del_cfg_detail, + ok=del_cfg_status + in { + OUTCOME_INVALID_OPERATION, + OUTCOME_INVALID_CONTENT, + OUTCOME_CONFLICT, + OUTCOME_FAILURE, + }, + ) + ) + + print(f"[cc/{iteration}.7] cc_create_during_create_snapshots_stage") + create_race_prefix = f"e2e_cc_create_stage_{uuid4().hex[:6]}_" + create_race_dep = self._mk_name("cc_stage_dep") + create_race_cfg = self._mk_name("cc_stage_cfg") + create_race_snap = self._mk_name("cc_stage_snap") + race_payload = self._build_create_payload( + snapshots=[self._build_flow_payload(label="cc_stage_snap", name_override=create_race_snap)], + config=DeploymentConfig( + name=create_race_cfg, + description="cc competing create", + environment_variables={}, + ), + resource_name_prefix=create_race_prefix, + ) + race_payload.spec = race_payload.spec.model_copy(update={"name": create_race_dep}, deep=True) + competing_create_task: asyncio.Task[tuple[str, str, dict[str, Any] | None]] | None = None + + async def _launch_competing_create(*args: Any, **kwargs: Any) -> None: # noqa: ARG001 + nonlocal competing_create_task + if competing_create_task is None: + competing_create_task = asyncio.create_task(self._run_create(race_payload.model_copy(deep=True))) + await asyncio.sleep(0) + + staged_create_status, staged_create_detail, staged_create_created = await self._run_with_stage_hook( + stage="create_snapshots", + operation=lambda: self._run_create(race_payload.model_copy(deep=True)), + hook_before=_launch_competing_create, + ) + competing_create_result = (OUTCOME_FAILURE, "competing create did not start", None) + if competing_create_task is not None: + competing_create_result = await competing_create_task + comp_status, comp_detail, comp_created = competing_create_result + self._track_created_result(staged_create_created) + self._track_created_result(comp_created) + staged_pair = (staged_create_status, comp_status) + results.append( + self._build_result( + name="cc_create_during_create_snapshots_stage", + expected={OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=max(staged_create_status, comp_status), + detail=f"main={staged_create_status}:{staged_create_detail} competing={comp_status}:{comp_detail}", + ok=( + staged_pair + in { + (OUTCOME_SUCCESS, OUTCOME_CONFLICT), + (OUTCOME_CONFLICT, OUTCOME_SUCCESS), + (OUTCOME_SUCCESS, OUTCOME_SUCCESS), + } + and OUTCOME_FAILURE not in staged_pair + ), + ) + ) + + print(f"[cc/{iteration}.8] cc_create_during_update_raw_config_stage") + update_create_id, _update_create_cfg, update_create_snaps, _ = await self._create_update_seed( + label=f"cc_create_update_cfg_{iteration}", + snapshot_count=1, + ) + update_target_tool_id = next(iter(update_create_snaps), "") + if not update_target_tool_id: + results.append( + self._build_result( + name="cc_create_during_update_raw_config_stage", + expected={OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=OUTCOME_FAILURE, + detail="seed snapshot id missing for competing update", + ok=False, + ) + ) + return results + update_cfg_prefix = f"e2e_cc_upd_cfg_create_{uuid4().hex[:6]}_" + update_cfg_name = self._mk_name("cc_upd_cfg_create") + competing_update_task: asyncio.Task[tuple[str, str, Any | None]] | None = None + + competing_update_payload = DeploymentUpdate( + provider_data={ + "resource_name_prefix": update_cfg_prefix, + "tools": {"existing_ids": [update_target_tool_id]}, + "connections": {"raw_payloads": [{"app_id": update_cfg_name, "environment_variables": {}}]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": update_target_tool_id}, + "app_ids": [update_cfg_name], + } + ], + }, + ) + + async def _launch_competing_update_create(*args: Any, **kwargs: Any) -> None: # noqa: ARG001 + nonlocal competing_update_task + if competing_update_task is None: + competing_update_task = asyncio.create_task( + self._run_update(update_create_id, competing_update_payload.model_copy(deep=True)) + ) + await asyncio.sleep(0) + + update_cfg_status, update_cfg_detail, _ = await self._run_with_stage_hook( + stage="update_create_config", + operation=lambda: self._run_update(update_create_id, competing_update_payload.model_copy(deep=True)), + hook_before=_launch_competing_update_create, + ) + competing_update_result = (OUTCOME_FAILURE, "competing update did not start", None) + if competing_update_task is not None: + competing_update_result = await competing_update_task + competing_upd_status, competing_upd_detail, _ = competing_update_result + results.append( + self._build_result( + name="cc_create_during_update_raw_config_stage", + expected={OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=max(update_cfg_status, competing_upd_status), + detail=( + f"main={update_cfg_status}:{update_cfg_detail} " + f"competing={competing_upd_status}:{competing_upd_detail}" + ), + ok=OUTCOME_FAILURE not in {update_cfg_status, competing_upd_status}, + ) + ) + + print(f"[cc/{iteration}.9] cc_delete_resources_during_update_rollback") + rollback_id, rollback_cfg_id, rollback_seed_snapshots, _ = await self._create_update_seed( + label=f"cc_rollback_delete_{iteration}", + snapshot_count=1, + ) + if not rollback_cfg_id: + results.append( + self._build_result( + name="cc_delete_resources_during_update_rollback", + expected={OUTCOME_SUCCESS}, + actual_outcome=OUTCOME_FAILURE, + detail="seed rollback config id missing", + ok=False, + ) + ) + return results + rollback_seed_tool_id = next(iter(rollback_seed_snapshots), "") + if not rollback_seed_tool_id: + results.append( + self._build_result( + name="cc_delete_resources_during_update_rollback", + expected={OUTCOME_FAILURE}, + actual_outcome=OUTCOME_FAILURE, + detail="seed snapshot id missing for rollback race", + ok=False, + ) + ) + return results + rollback_prefix = f"e2e_cc_rollback_{uuid4().hex[:6]}_" + rollback_raw_flow = self._build_flow_payload(label=f"cc_rb_raw_{iteration}") + rollback_raw_cfg_name = self._mk_name("cc_rb_cfg") + rollback_status, rollback_detail, _ = await self._run_with_stage_hook( + stage="update_rollback_resources", + operation=lambda: self._run_update( + rollback_id, + DeploymentUpdate( + spec=BaseDeploymentDataUpdate(description="cc rollback delete race"), + provider_data={ + "resource_name_prefix": rollback_prefix, + "tools": { + "existing_ids": [rollback_seed_tool_id], + "raw_payloads": [rollback_raw_flow.model_dump(mode="json")], + }, + "connections": { + "existing_app_ids": [str(rollback_cfg_id)], + "raw_payloads": [{"app_id": rollback_raw_cfg_name, "environment_variables": {}}], + }, + "operations": [ + { + "op": "unbind", + "tool_id": rollback_seed_tool_id, + "app_ids": [str(rollback_cfg_id)], + }, + { + "op": "bind", + "tool": {"name_of_raw": rollback_raw_flow.name}, + "app_ids": [rollback_raw_cfg_name], + }, + ], + }, + ), + inject={ + "update_bindings": { + "fail_first_n": 1, + "error_type": "runtime", + "message": "cc_rollback_trigger", + } + }, + ), + hook_before=self._delete_resources_before_rollback_hook, + ) + results.append( + self._build_result( + name="cc_delete_resources_during_update_rollback", + expected={OUTCOME_FAILURE}, + actual_outcome=rollback_status, + detail=rollback_detail, + ok=rollback_status == OUTCOME_FAILURE, + ) + ) + + print(f"[cc/{iteration}.10] cc_create_during_update_rollback") + rollback_create_id, rollback_create_cfg_id, rollback_create_snaps, _ = await self._create_update_seed( + label=f"cc_rollback_create_{iteration}", + snapshot_count=1, + ) + if not rollback_create_cfg_id: + results.append( + self._build_result( + name="cc_create_during_update_rollback", + expected={OUTCOME_SUCCESS}, + actual_outcome=OUTCOME_FAILURE, + detail="seed rollback-create config id missing", + ok=False, + ) + ) + return results + rollback_create_seed_tool_id = next(iter(rollback_create_snaps), "") + if not rollback_create_seed_tool_id: + results.append( + self._build_result( + name="cc_create_during_update_rollback", + expected={OUTCOME_FAILURE, OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=OUTCOME_FAILURE, + detail="seed snapshot id missing for rollback create race", + ok=False, + ) + ) + return results + rollback_create_prefix = f"e2e_cc_rb_create_{uuid4().hex[:6]}_" + rollback_create_cfg_name = self._mk_name("cc_rb_create_cfg") + competing_rollback_create_task: asyncio.Task[tuple[str, str, Any | None]] | None = None + + async def _launch_competing_create_before_rollback(*args: Any, **kwargs: Any) -> None: # noqa: ARG001 + nonlocal competing_rollback_create_task + if competing_rollback_create_task is not None: + return + competing_rollback_create_task = asyncio.create_task( + self._run_update( + rollback_create_id, + DeploymentUpdate( + provider_data={ + "resource_name_prefix": rollback_create_prefix, + "tools": {"existing_ids": [rollback_create_seed_tool_id]}, + "connections": { + "raw_payloads": [{"app_id": rollback_create_cfg_name, "environment_variables": {}}] + }, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": rollback_create_seed_tool_id}, + "app_ids": [rollback_create_cfg_name], + } + ], + }, + ), + ) + ) + await asyncio.sleep(0) + + rollback_create_raw_flow = self._build_flow_payload(label=f"cc_rb_create_raw_{iteration}") + rollback_create_status, rollback_create_detail, _ = await self._run_with_stage_hook( + stage="update_rollback_resources", + operation=lambda: self._run_update( + rollback_create_id, + DeploymentUpdate( + spec=BaseDeploymentDataUpdate(description="cc rollback create race"), + provider_data={ + "resource_name_prefix": rollback_create_prefix, + "tools": { + "existing_ids": [rollback_create_seed_tool_id], + "raw_payloads": [rollback_create_raw_flow.model_dump(mode="json")], + }, + "connections": { + "existing_app_ids": [str(rollback_create_cfg_id)], + "raw_payloads": [{"app_id": rollback_create_cfg_name, "environment_variables": {}}], + }, + "operations": [ + { + "op": "unbind", + "tool_id": rollback_create_seed_tool_id, + "app_ids": [str(rollback_create_cfg_id)], + }, + { + "op": "bind", + "tool": {"name_of_raw": rollback_create_raw_flow.name}, + "app_ids": [rollback_create_cfg_name], + }, + ], + }, + ), + inject={ + "update_bindings": { + "fail_first_n": 1, + "error_type": "runtime", + "message": "cc_rollback_create_trigger", + } + }, + ), + hook_before=_launch_competing_create_before_rollback, + ) + competing_rollback_create_result = (OUTCOME_FAILURE, "competing rollback create missing", None) + if competing_rollback_create_task is not None: + competing_rollback_create_result = await competing_rollback_create_task + comp_rb_status, comp_rb_detail, _ = competing_rollback_create_result + results.append( + self._build_result( + name="cc_create_during_update_rollback", + expected={OUTCOME_FAILURE, OUTCOME_SUCCESS, OUTCOME_CONFLICT}, + actual_outcome=max(rollback_create_status, comp_rb_status), + detail=( + f"main={rollback_create_status}:{rollback_create_detail} " + f"competing={comp_rb_status}:{comp_rb_detail}" + ), + ok=rollback_create_status == OUTCOME_FAILURE + and comp_rb_status in {OUTCOME_SUCCESS, OUTCOME_CONFLICT, OUTCOME_FAILURE}, + ) + ) + + print(f"[cc/{iteration}.11] cc_parallel_updates_isolation") + dep_a, cfg_a, _snap_a, _ = await self._create_update_seed(label=f"cc_iso_a_{iteration}", snapshot_count=1) + dep_b, cfg_b, _snap_b, _ = await self._create_update_seed(label=f"cc_iso_b_{iteration}", snapshot_count=1) + isolation_race = await self._run_parallel_calls( + { + "a": lambda: self._run_update( + dep_a, + DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="isolation-a")), + ), + "b": lambda: self._run_update( + dep_b, + DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="isolation-b")), + ), + } + ) + iso_a_status, iso_a_detail, _ = isolation_race["a"] + iso_b_status, iso_b_detail, _ = isolation_race["b"] + cfg_list_a_status, _cfg_a_detail, cfg_list_a = await self._run_list_configs(dep_a) + cfg_list_b_status, _cfg_b_detail, cfg_list_b = await self._run_list_configs(dep_b) + cfg_ids_a = self._extract_config_ids(cfg_list_a) + cfg_ids_b = self._extract_config_ids(cfg_list_b) + isolation_ok = ( + cfg_list_a_status == OUTCOME_SUCCESS and cfg_list_b_status == OUTCOME_SUCCESS and cfg_ids_a and cfg_ids_b + ) + if cfg_a: + isolation_ok = isolation_ok and str(cfg_a) in cfg_ids_a + if cfg_b: + isolation_ok = isolation_ok and str(cfg_b) in cfg_ids_b + results.append( + self._build_result( + name="cc_parallel_updates_isolation", + expected={OUTCOME_SUCCESS}, + actual_outcome=max(iso_a_status, iso_b_status), + detail=f"a={iso_a_status}:{iso_a_detail} b={iso_b_status}:{iso_b_detail}", + ok=isolation_ok and iso_a_status == OUTCOME_SUCCESS and iso_b_status == OUTCOME_SUCCESS, + ) + ) + return results + + async def _run_parallel_calls( + self, + calls: dict[str, Callable[[], Awaitable[tuple[str, str, Any | None]]]], + ) -> dict[str, tuple[str, str, Any | None]]: + tasks = {name: asyncio.create_task(call()) for name, call in calls.items()} + gathered = await asyncio.gather(*tasks.values(), return_exceptions=True) + results: dict[str, tuple[str, str, Any | None]] = {} + for name, outcome in zip(tasks, gathered, strict=False): + if isinstance(outcome, Exception): + results[name] = (OUTCOME_FAILURE, str(outcome), None) + else: + results[name] = outcome + return results + + def _track_created_result(self, created: dict[str, Any] | None) -> None: + if not created: + return + deployment_id = created.get("deployment_id") + config_id = created.get("config_id") + snapshot_ids = created.get("snapshot_ids") or set() + if deployment_id: + self.created_deployment_ids.add(str(deployment_id)) + if config_id: + self.created_config_ids.add(str(config_id)) + self.created_snapshot_ids.update({str(item) for item in snapshot_ids if item}) + + def _has_unique_snapshot_ids(self, snapshot_ids: set[str]) -> bool: + snapshot_list = [str(item) for item in snapshot_ids] + return len(snapshot_list) == len(set(snapshot_list)) + + def _extract_config_ids(self, config_result: Any) -> set[str]: + configs = getattr(config_result, "configs", []) if config_result else [] + return {str(config.id) for config in configs if config and getattr(config, "id", None)} + + def _stage_hook_mapping(self) -> dict[str, tuple[Any, str]]: + return { + "create_config": (service_module, "process_config"), + "create_snapshots": (service_module, "process_raw_flows_with_app_id"), + "create_agent": (self.service, "_create_agent_deployment"), + "update_create_config": (update_helpers_module, "_create_update_connection_with_conflict_mapping"), + "update_create_tools": (update_helpers_module, "create_and_upload_wxo_flow_tools_with_bindings"), + "update_bindings": (update_helpers_module, "_update_existing_tool_connection_deltas"), + "update_rollback_resources": (update_helpers_module, "rollback_update_resources"), + } + + async def _run_with_stage_hook( + self, + *, + stage: str, + operation: Callable[[], Awaitable[tuple[str, str, Any | None]]], + hook_before: Callable[..., Awaitable[None]] | None = None, + hook_after: Callable[..., Awaitable[None]] | None = None, + ) -> tuple[str, str, Any | None]: + originals: list[tuple[Any, str, Any]] = [] + self._apply_stage_hook( + stage=stage, + originals=originals, + hook_before=hook_before, + hook_after=hook_after, + ) + try: + return await operation() + finally: + for target, attr_name, original in originals: + setattr(target, attr_name, original) + + def _apply_stage_hook( + self, + *, + stage: str, + originals: list[tuple[Any, str, Any]], + hook_before: Callable[..., Awaitable[None]] | None = None, + hook_after: Callable[..., Awaitable[None]] | None = None, + ) -> None: + mapping = self._stage_hook_mapping() + target_and_method = mapping.get(stage) + if target_and_method is None: + return + target, method_name = target_and_method + original = getattr(target, method_name) + originals.append((target, method_name, original)) + before_called = {"value": False} + after_called = {"value": False} + + if isinstance(target, types.ModuleType): + + async def _wrapped_module( + *args, + __orig=original, + __before=hook_before, + __after=hook_after, + __before_called=before_called, + __after_called=after_called, + **kwargs, + ): + if __before is not None and not __before_called["value"]: + __before_called["value"] = True + await __before(*args, **kwargs) + result = await __orig(*args, **kwargs) + if __after is not None and not __after_called["value"]: + __after_called["value"] = True + await __after(result, *args, **kwargs) + return result + + setattr(target, method_name, _wrapped_module) + return + + async def _wrapped_method( + _self, + *args, + __orig=original, + __before=hook_before, + __after=hook_after, + __before_called=before_called, + __after_called=after_called, + **kwargs, + ): + if __before is not None and not __before_called["value"]: + __before_called["value"] = True + await __before(*args, **kwargs) + result = await __orig(*args, **kwargs) + if __after is not None and not __after_called["value"]: + __after_called["value"] = True + await __after(result, *args, **kwargs) + return result + + setattr(target, method_name, MethodType(_wrapped_method, target)) + + async def _safe_delete_snapshot(self, *, clients: Any, snapshot_id: str) -> None: + with suppress(ClientAPIException): + await asyncio.to_thread(clients.tool.delete, snapshot_id) + + async def _safe_delete_config(self, *, clients: Any, config_id: str) -> None: + with suppress(ClientAPIException): + await asyncio.to_thread(clients.connections.delete, config_id) + + async def _delete_resources_before_rollback_hook(self, *args: Any, **kwargs: Any) -> None: # noqa: ARG002 + clients = kwargs.get("clients") + if clients is None: + return + for tool_id in kwargs.get("created_tool_ids") or []: + await self._safe_delete_snapshot(clients=clients, snapshot_id=str(tool_id)) + app_id = kwargs.get("created_app_id") + if app_id: + await self._safe_delete_config(clients=clients, config_id=str(app_id)) + for created_app_id in kwargs.get("created_app_ids") or []: + await self._safe_delete_config(clients=clients, config_id=str(created_app_id)) + + async def _run_update_failpoint_scenarios(self) -> list[ScenarioResult]: + results: list[ScenarioResult] = [] + print("\n[fp-upd] creating seed deployment") + deployment_id, config_id, snapshot_ids, _ = await self._create_update_seed( + label="fp_upd_seed", + snapshot_count=1, + ) + if not config_id: + results.append( + self._build_result( + name="fp_update_seed_missing_config", + expected={OUTCOME_SUCCESS}, + actual_outcome=OUTCOME_FAILURE, + detail="seed deployment config id is missing", + ok=False, + ) + ) + return results + seed_tool_id = next(iter(snapshot_ids), "") + if not seed_tool_id: + results.append( + self._build_result( + name="fp_update_seed_missing_snapshot", + expected={OUTCOME_SUCCESS}, + actual_outcome=OUTCOME_FAILURE, + detail="seed deployment snapshot id is missing", + ok=False, + ) + ) + return results + failpoint_prefix = f"e2e_fp_upd_{uuid4().hex[:6]}_" + failpoint_raw_app_id = self._mk_name("fp_upd_cfg") + + update_payload = DeploymentUpdate( + spec=BaseDeploymentDataUpdate(description="trigger update failpoint"), + provider_data={ + "resource_name_prefix": failpoint_prefix, + "tools": {"existing_ids": [seed_tool_id]}, + "connections": {"raw_payloads": [{"app_id": failpoint_raw_app_id, "environment_variables": {}}]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": seed_tool_id}, + "app_ids": [failpoint_raw_app_id], + } + ], + }, + ) + + print("[fp-upd/1] fp_update_bindings_failure_triggers_rollback") + status_code, detail, _ = await self._run_update( + deployment_id, + update_payload, + inject={ + "update_bindings": { + "fail_first_n": 1, + "error_type": "runtime", + "message": "fp_update_bindings_failure", + } + }, + ) + results.append( + self._build_result( + name="fp_update_bindings_failure_triggers_rollback", + expected={OUTCOME_FAILURE}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_FAILURE, + ) + ) + + print("[fp-upd/2] fp_update_bindings_failure_with_rollback_failure") + status_code, detail, _ = await self._run_update( + deployment_id, + update_payload, + inject={ + "update_bindings": { + "fail_first_n": 1, + "error_type": "runtime", + "message": "fp_update_bindings_failure_again", + }, + "update_rollback_resources": { + "fail_first_n": 1, + "error_type": "runtime", + "message": "fp_update_rollback_failure", + }, + }, + ) + results.append( + self._build_result( + name="fp_update_bindings_failure_with_rollback_failure", + expected={OUTCOME_FAILURE}, + actual_outcome=status_code, + detail=detail, + ok=status_code == OUTCOME_FAILURE, + ) + ) + + return results + + def _apply_injections(self, inject: dict[str, dict[str, Any]], originals: list[tuple[Any, str, Any]]) -> None: + mapping = { + "create_config": (service_module, "process_config"), + "create_snapshots": (service_module, "process_raw_flows_with_app_id"), + "create_agent": (self.service, "_create_agent_deployment"), + "rollback_delete_agent": (retry_module, "delete_agent_if_exists"), + "rollback_delete_tool": (retry_module, "delete_tool_if_exists"), + "rollback_delete_config": (retry_module, "delete_config_if_exists"), + "update_bindings": (update_helpers_module, "_update_existing_tool_connection_deltas"), + "update_rollback_resources": (update_helpers_module, "rollback_update_resources"), + } + + for stage, config in inject.items(): + target_and_method = mapping.get(stage) + if target_and_method is None: + continue + target, method_name = target_and_method + original = getattr(target, method_name) + originals.append((target, method_name, original)) + counter = {"value": 0} + fail_first_n = int(config.get("fail_first_n", 0)) + error_type = str(config.get("error_type", "runtime")).strip().lower() + message = str(config.get("message") or f"injected failure: {stage}") + if isinstance(target, types.ModuleType): + + async def _wrapped_module( + *args, + __orig=original, + __ctr=counter, + __n=fail_first_n, + __type=error_type, + __msg=message, + **kwargs, + ): + __ctr["value"] += 1 + if __ctr["value"] <= __n: + if __type == "domain_conflict": + raise DeploymentConflictError(message=__msg) + if __type == "domain_not_found": + raise DeploymentNotFoundError(message=__msg) + if __type == "domain_invalid_content": + raise InvalidContentError(message=__msg) + if __type == "domain_invalid_operation": + raise InvalidDeploymentOperationError(message=__msg) + if __type == "domain_failure": + raise DeploymentError(message=__msg, error_code="deployment_error") + raise RuntimeError(__msg) + return await __orig(*args, **kwargs) + + setattr(target, method_name, _wrapped_module) + continue + + async def _wrapped_method( + _self, + *args, + __orig=original, + __ctr=counter, + __n=fail_first_n, + __type=error_type, + __msg=message, + **kwargs, + ): + __ctr["value"] += 1 + if __ctr["value"] <= __n: + if __type == "domain_conflict": + raise DeploymentConflictError(message=__msg) + if __type == "domain_not_found": + raise DeploymentNotFoundError(message=__msg) + if __type == "domain_invalid_content": + raise InvalidContentError(message=__msg) + if __type == "domain_invalid_operation": + raise InvalidDeploymentOperationError(message=__msg) + if __type == "domain_failure": + raise DeploymentError(message=__msg, error_code="deployment_error") + raise RuntimeError(__msg) + return await __orig(*args, **kwargs) + + setattr(target, method_name, MethodType(_wrapped_method, target)) + + def _build_create_payload( + self, + *, + snapshots: list[BaseFlowArtifact], + config: DeploymentConfig | None = None, + config_reference_id: str | None = None, + resource_name_prefix: str | None = None, + ) -> DeploymentCreate: + prefix = resource_name_prefix or f"e2e_{uuid4().hex[:8]}_" + provider_spec = {"resource_name_prefix": prefix} + spec = BaseDeploymentData( + name=self._mk_name("dep_agent"), + description="direct adapter scenario", + type=DeploymentType.AGENT, + provider_spec=provider_spec, + ) + config_item = ConfigItem(reference_id=config_reference_id) if config_reference_id else None + if config is not None: + config_item = ConfigItem(raw_payload=config) + return DeploymentCreate(spec=spec, snapshot=SnapshotItems(raw_payloads=snapshots), config=config_item) + + def _build_flow_payload(self, *, label: str, name_override: str | None = None) -> BaseFlowArtifact: + return BaseFlowArtifact( + id=UUID(str(uuid4())), + name=name_override or self._mk_name(label), + description="direct adapter flow payload", + data=self._build_flow_data_payload(), + tags=["e2e", "watsonx-direct-adapter"], + provider_data={"project_id": self.project_id}, + ) + + def _build_config_payload(self, *, label: str) -> DeploymentConfig: + return DeploymentConfig( + name=self._mk_name(label), + description="direct adapter config payload", + environment_variables={}, + ) + + def _build_flow_data_payload(self) -> dict[str, Any]: + chat_input_node_id = f"ChatInput-{uuid4().hex[:8]}" + chat_output_node_id = f"ChatOutput-{uuid4().hex[:8]}" + return { + "nodes": [ + { + "id": chat_input_node_id, + "type": "genericNode", + "position": {"x": 100, "y": 100}, + "data": { + "type": "ChatInput", + "id": chat_input_node_id, + "node": {"template": {"_type": "CustomComponent"}}, + }, + }, + { + "id": chat_output_node_id, + "type": "genericNode", + "position": {"x": 400, "y": 100}, + "data": { + "type": "ChatOutput", + "id": chat_output_node_id, + "node": {"template": {"_type": "CustomComponent"}}, + }, + }, + ], + "edges": [], + "viewport": {"x": 0, "y": 0, "zoom": 1}, + } + + async def _cleanup_resources(self) -> None: + clients = await self.service._get_provider_clients(user_id=self.user_id, db=self.db) # noqa: SLF001 + print("\nCleanup Resources") + print("-" * 90) + print( + f"cleanup targets: deployments={len(self.created_deployment_ids)} " + f"snapshots={len(self.created_snapshot_ids)} " + f"configs={len(self.created_config_ids)}" + ) + + deleted_deployments = 0 + deleted_snapshots = 0 + deleted_configs = 0 + + for deployment_id in sorted(self.created_deployment_ids): + print(f"[cleanup] deleting deployment {deployment_id}...") + with suppress(Exception): + await self.service.delete(user_id=self.user_id, deployment_id=deployment_id, db=self.db) + deleted_deployments += 1 + print(f"[cleanup] deleted deployment {deployment_id}") + + for snapshot_id in sorted(self.created_snapshot_ids): + print(f"[cleanup] deleting snapshot {snapshot_id}...") + try: + await asyncio.to_thread(clients.tool.delete, snapshot_id) + deleted_snapshots += 1 + print(f"[cleanup] deleted snapshot {snapshot_id}") + except ClientAPIException as exc: + if exc.response.status_code != HTTP_STATUS_NOT_FOUND: + print(f"[cleanup-warning] snapshot {snapshot_id}: {exc}") + else: + print(f"[cleanup] snapshot {snapshot_id} already deleted (404)") + + for config_id in sorted(self.created_config_ids): + print(f"[cleanup] deleting config {config_id}...") + try: + await asyncio.to_thread(clients.connections.delete, config_id) + deleted_configs += 1 + print(f"[cleanup] deleted config {config_id}") + except ClientAPIException as exc: + if exc.response.status_code != HTTP_STATUS_NOT_FOUND: + print(f"[cleanup-warning] config {config_id}: {exc}") + else: + print(f"[cleanup] config {config_id} already deleted (404)") + + print( + f"cleanup completed: deployments_deleted={deleted_deployments} " + f"snapshots_deleted={deleted_snapshots} " + f"configs_deleted={deleted_configs}" + ) + print("-" * 90) + + def _mk_name(self, prefix: str) -> str: + raw = f"{prefix}_{self.run_suffix}_{uuid4().hex[:6]}" + normalized = _INVALID_WXO_NAME_CHARS.sub("_", raw) + if not normalized or not normalized[0].isalpha(): + normalized = f"n_{normalized}" + return normalized + + def _scenario_group(self, name: str) -> str: + prefix = name.split("_", 1)[0] + if prefix in {"live", "upd", "cc", "fp"}: + return prefix + return "other" + + def _scenario_label(self, name: str) -> str: + if "_" not in name: + return name + return name.split("_", 1)[1] + + def _format_expected_outcomes(self, outcomes: set[str]) -> str: + ordered = sorted(outcomes, key=lambda item: (item != OUTCOME_SUCCESS, item)) + return ", ".join(ordered) + + def _print_result_row(self, result: ScenarioResult) -> None: + verdict = "PASS" if result.ok else "FAIL" + scenario_label = self._scenario_label(result.name) + expected = self._format_expected_outcomes(result.expected_outcomes) + print(f"{verdict:<5} | {scenario_label:<44} | got: {result.actual_outcome}") + print(f"{'':5} | {'':44} | expected: {expected}") + if not result.ok: + wrapped_detail = textwrap.wrap(result.detail, width=108) or [result.detail] + for index, line in enumerate(wrapped_detail): + prefix = "detail: " if index == 0 else " " + print(f"{'':5} | {'':44} | {prefix}{line}") + + def _print_summary(self, results: list[ScenarioResult]) -> None: + print("\nScenario Summary") + print("-" * 90) + total = len(results) + passed = sum(1 for result in results if result.ok) + failed = total - passed + print(f"Total={total} Passed={passed} Failed={failed}") + print("-" * 90) + + grouped_results: dict[str, list[ScenarioResult]] = {"live": [], "upd": [], "cc": [], "fp": [], "other": []} + for result in results: + grouped_results[self._scenario_group(result.name)].append(result) + + for group in ("live", "upd", "cc", "fp", "other"): + group_items = grouped_results[group] + if not group_items: + continue + print(f"\n[{group}] ({len(group_items)})") + for result in group_items: + self._print_result_row(result) + print("-" * 90) + + +def _get_required_env(name: str) -> str: + value = os.getenv(name, "").strip() + if not value: + msg = f"Environment variable '{name}' is required." + raise RuntimeError(msg) + return value + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run direct Watsonx adapter matrix (live + failpoints).") + parser.add_argument("--project-id", default=os.getenv("WXO_PROJECT_ID", "e2e-project")) + parser.add_argument("--mode", choices=["live", "failpoint", "both"], default=os.getenv("WXO_E2E_MODE", "both")) + parser.add_argument("--keep-resources", action="store_true") + return parser.parse_args() + + +async def _main() -> int: + load_dotenv() + args = _parse_args() + runner = WatsonxAdapterDirectE2E( + provider_backend_url=_get_required_env("WXO_INSTANCE_URL"), + provider_api_key=_get_required_env("WXO_API_KEY"), + project_id=args.project_id, + mode=args.mode, + keep_resources=args.keep_resources, + ) + return await runner.run() + + +if __name__ == "__main__": + raise SystemExit(asyncio.run(_main())) diff --git a/src/backend/base/langflow/api/v1/deployments.py b/src/backend/base/langflow/api/v1/deployments.py index c90c0b14af..606f896231 100644 --- a/src/backend/base/langflow/api/v1/deployments.py +++ b/src/backend/base/langflow/api/v1/deployments.py @@ -59,6 +59,15 @@ DeploymentIdQuery = Annotated[ # - Body ``provider_id`` is included on ``DeploymentCreateRequest`` and ``ExecutionCreateRequest`` # to allow provider routing without an extra DB lookup when the caller already has the context. # - Deployment-scoped routes derive provider context from persisted Langflow relationships. +# +# TODO(deployments-routing): Before replacing 501 stubs with live routing: +# - Resolve provider_key from provider_id/deployment_id and fetch adapter via registry. +# - If adapter is unavailable in current runtime (e.g. optional SDK not installed), +# return a deterministic domain error/HTTP status instead of 500. +# - Add tests for: +# * provider account exists but adapter key not registered +# * adapter import was skipped at startup due to ModuleNotFoundError +# * direct WXO provider routes in unsupported runtimes (py3.10 scenarios) # --------------------------------------------------------------------------- diff --git a/src/backend/base/langflow/services/adapters/__init__.py b/src/backend/base/langflow/services/adapters/__init__.py new file mode 100644 index 0000000000..11f5401501 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/__init__.py @@ -0,0 +1 @@ +"""Adapter namespaces for Langflow service-scoped plugin registries.""" diff --git a/src/backend/base/langflow/services/adapters/deployment/__init__.py b/src/backend/base/langflow/services/adapters/deployment/__init__.py new file mode 100644 index 0000000000..46490c04b1 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/__init__.py @@ -0,0 +1 @@ +"""Langflow deployment adapter implementations.""" diff --git a/src/backend/base/langflow/services/adapters/deployment/context.py b/src/backend/base/langflow/services/adapters/deployment/context.py new file mode 100644 index 0000000000..6f65b0ca5d --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/context.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +from contextlib import contextmanager +from contextvars import ContextVar +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar + +if TYPE_CHECKING: + from contextvars import Token + from uuid import UUID + + +@dataclass(frozen=True, slots=True) +class DeploymentAdapterContext: + provider_id: UUID + + +class DeploymentProviderIDContext: + _current: ClassVar[ContextVar[DeploymentAdapterContext | None]] = ContextVar( + "langflow_current_deployment_context", + default=None, + ) + + @classmethod + def get_current(cls) -> DeploymentAdapterContext | None: + return cls._current.get() + + @classmethod + def set_current(cls, context: DeploymentAdapterContext) -> Token[DeploymentAdapterContext | None]: + return cls._current.set(context) + + @classmethod + def reset_current(cls, token: Token[DeploymentAdapterContext | None]) -> None: + cls._current.reset(token) + + @classmethod + @contextmanager + def scope(cls, context: DeploymentAdapterContext): + token: Token[DeploymentAdapterContext | None] = cls.set_current(context) + try: + yield + finally: + cls.reset_current(token) diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/__init__.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/__init__.py new file mode 100644 index 0000000000..b24151aef6 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/__init__.py @@ -0,0 +1,20 @@ +"""Watsonx Orchestrate deployment adapter.""" + +from lfx.services.adapters.registry import register_adapter +from lfx.services.adapters.schema import AdapterType + +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ( + WATSONX_ORCHESTRATE_DEPLOYMENT_ADAPTER_KEY, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.service import WatsonxOrchestrateDeploymentService +from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOCredentials + +register_adapter( + AdapterType.DEPLOYMENT, + WATSONX_ORCHESTRATE_DEPLOYMENT_ADAPTER_KEY, +)(WatsonxOrchestrateDeploymentService) + +__all__ = [ + "WatsonxOrchestrateDeploymentService", + "WxOCredentials", +] diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/client.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/client.py new file mode 100644 index 0000000000..5334cc23aa --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/client.py @@ -0,0 +1,275 @@ +"""Client creation, authentication, and credential resolution for the Watsonx Orchestrate adapter. + +This module uses request/execution-context memoization for provider clients: + +- `get_provider_clients()` resolves provider context and prebuilt credentials/authenticator. +- The resulting `WxOClient` is memoized in a ContextVar for the active async execution context. +- Subsequent calls with the same `(provider_id, user_id)` in that context reuse the same + `WxOClient` instance and skip repeated DB/decryption work. + +Important behavior notes: +- Memoization is execution-context scoped (not cross-request/global state). +- The context stores a single `(key, client)` entry because deployment routing enforces one + provider context per request path. +- If a different `(provider_id, user_id)` is requested in the same context, resolution fails. +""" + +from __future__ import annotations + +from contextvars import ContextVar +from dataclasses import dataclass +from typing import TYPE_CHECKING, ClassVar + +from ibm_cloud_sdk_core.authenticators import Authenticator, IAMAuthenticator, MCSPAuthenticator +from ibm_watsonx_orchestrate_core.types.connections import KeyValueConnectionCredentials +from lfx.services.adapters.deployment.exceptions import AuthSchemeError, CredentialResolutionError +from lfx.services.adapters.deployment.schema import EnvVarSource, EnvVarValueSpec, IdLike + +from langflow.services.adapters.deployment.context import DeploymentProviderIDContext +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import WxOAuthURL +from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient, WxOCredentials +from langflow.services.auth import utils as auth_utils +from langflow.services.database.models.deployment_provider_account.crud import get_provider_account_by_id +from langflow.services.deps import get_variable_service + +if TYPE_CHECKING: + from contextvars import Token + from uuid import UUID + + from sqlalchemy.ext.asyncio import AsyncSession + + +@dataclass(frozen=True, slots=True) +class WxOProviderClientsContext: + provider_id: str + user_id: str + clients: WxOClient + + +class WxOProviderClientsRequestContext: + _current: ClassVar[ContextVar[WxOProviderClientsContext | None]] = ContextVar( + "langflow_wxo_provider_clients_request_context", + default=None, + ) + + @classmethod + def get_current(cls) -> WxOProviderClientsContext | None: + return cls._current.get() + + @classmethod + def set_current(cls, context: WxOProviderClientsContext) -> Token[WxOProviderClientsContext | None]: + return cls._current.set(context) + + @classmethod + def reset_current(cls, token: Token[WxOProviderClientsContext | None]) -> None: + cls._current.reset(token) + + @classmethod + def clear_current(cls) -> None: + cls._current.set(None) + + +def _provider_client_context_key(*, provider_id: UUID, user_id: UUID | str) -> tuple[str, str]: + return (str(provider_id), str(user_id)) + + +def clear_provider_clients_request_context() -> None: + """Clear execution-context memoized provider clients for the current async context. + + This is mainly useful in tests and explicit context lifecycle control. + """ + WxOProviderClientsRequestContext.clear_current() + + +def get_request_context_provider_clients(*, provider_id: UUID, user_id: UUID | str) -> WxOClient | None: + """Return memoized provider clients for the active execution context, if present. + + Returns `None` when: + - no provider clients have been memoized in this context yet, or + - the memoized entry belongs to a different `(provider_id, user_id)` pair. + """ + request_context = WxOProviderClientsRequestContext.get_current() + if request_context is None: + return None + if (request_context.provider_id, request_context.user_id) == _provider_client_context_key( + provider_id=provider_id, + user_id=user_id, + ): + return request_context.clients + return None + + +def _validate_request_context_provider_key(*, provider_id: UUID, user_id: UUID | str) -> None: + request_context = WxOProviderClientsRequestContext.get_current() + if request_context is None: + return + if (request_context.provider_id, request_context.user_id) != _provider_client_context_key( + provider_id=provider_id, + user_id=user_id, + ): + msg = ( + "A different deployment provider context was requested in the same execution context. " + "This indicates an invalid mixed provider resolution flow." + ) + raise CredentialResolutionError(message=msg) + + +def set_request_context_provider_clients(*, provider_id: UUID, user_id: UUID | str, clients: WxOClient) -> None: + """Memoize provider clients for the active execution context.""" + _validate_request_context_provider_key(provider_id=provider_id, user_id=user_id) + context = WxOProviderClientsContext( + provider_id=str(provider_id), + user_id=str(user_id), + clients=clients, + ) + WxOProviderClientsRequestContext.set_current(context) + + +def get_authenticator(instance_url: str, api_key: str) -> Authenticator: + """Return the appropriate authenticator for the Watsonx Orchestrate API.""" + if ".cloud.ibm.com" in instance_url: + return IAMAuthenticator(apikey=api_key, url=WxOAuthURL.IBM_IAM.value) + elif ".ibm.com" in instance_url: # noqa: RET505 - explicitness + return MCSPAuthenticator(apikey=api_key, url=WxOAuthURL.MCSP.value) + + msg = f"Could not determine authentication scheme for instance URL: {instance_url}" + raise AuthSchemeError(message=msg) + + +async def resolve_wxo_client_credentials( + *, + user_id: UUID | str, + db: AsyncSession, + provider_id: UUID, +) -> WxOCredentials: + """Resolve Watsonx Orchestrate client credentials from deployment provider account. + + The decrypted API key is used only to instantiate the SDK authenticator and is not + retained in adapter credential objects. + """ + try: + provider_account = await get_provider_account_by_id( + db, + provider_id=provider_id, + user_id=user_id, + ) + if provider_account is None: + msg = "Failed to find deployment provider account credentials." + raise CredentialResolutionError(message=msg) + + instance_url = (provider_account.backend_url or "").strip() + api_key = auth_utils.decrypt_api_key((provider_account.api_key or "").strip()) + if not instance_url or not api_key: + msg = "Watsonx Orchestrate backend URL and API key must be configured." + raise CredentialResolutionError(message=msg) + + except CredentialResolutionError: + raise + except Exception as exc: + msg = "An unexpected error occurred while resolving Watsonx Orchestrate client credentials." + raise CredentialResolutionError(message=msg) from exc + + authenticator = get_authenticator(instance_url=instance_url, api_key=api_key) + return WxOCredentials(instance_url=instance_url, authenticator=authenticator) + + +async def get_provider_clients( + *, + user_id: UUID | str, + db: AsyncSession, +) -> WxOClient: + """Resolve and return provider clients for the active deployment provider context. + + Fast-path: return execution-context memoized clients when `(provider_id, user_id)` matches. + Slow-path: resolve credentials from DB, build authenticator, construct `WxOClient`, then memoize. + """ + request_context = DeploymentProviderIDContext.get_current() + if request_context is None: + msg = "Deployment account context is not available for adapter resolution." + raise CredentialResolutionError(message=msg) + provider_id = request_context.provider_id + _validate_request_context_provider_key(provider_id=provider_id, user_id=user_id) + if context_clients := get_request_context_provider_clients(provider_id=provider_id, user_id=user_id): + return context_clients + + credentials: WxOCredentials = await resolve_wxo_client_credentials( + user_id=user_id, + db=db, + provider_id=provider_id, + ) + + clients = WxOClient( + instance_url=credentials.instance_url, + authenticator=credentials.authenticator, + ) + set_request_context_provider_clients(provider_id=provider_id, user_id=user_id, clients=clients) + return clients + + +async def resolve_runtime_credentials( + *, + user_id: IdLike, + environment_variables: dict[str, EnvVarValueSpec], + db: AsyncSession, +) -> KeyValueConnectionCredentials: + """Resolve runtime credentials from environment variables.""" + resolved: dict[str, str] = {} + for credential_key, env_var_value in environment_variables.items(): + resolved[credential_key] = await resolve_env_var_value( + env_var_value, + user_id=user_id, + db=db, + ) + return KeyValueConnectionCredentials(resolved) + + +async def resolve_env_var_value( + env_var_value: EnvVarValueSpec, + *, + user_id: IdLike, + db: AsyncSession, +) -> str: + if env_var_value.source == EnvVarSource.RAW: + return env_var_value.value + return await resolve_variable_value( + env_var_value.value, + user_id=user_id, + db=db, + ) + + +async def resolve_variable_value( + variable_name: str, + *, + user_id: UUID | str, + db: AsyncSession, + optional: bool = False, + default_value: str | None = None, +) -> str: + variable_service = get_variable_service() + if variable_service is None: + msg = "Variable service is not available." + raise CredentialResolutionError(message=msg) + try: + value = await variable_service.get_variable( + user_id=user_id, + name=variable_name, + field="value", + session=db, + ) + if value is not None: + return value + except CredentialResolutionError: + raise + except Exception as exc: + if not optional: + msg = "Failed to resolve a credential variable for the watsonx Orchestrate deployment provider." + raise CredentialResolutionError(message=msg) from exc + if optional: + return default_value or "" + msg = ( + "Failed to find a necessary credential for the " + "watsonx Orchestrate deployment provider. " + "Please ensure all credentials are provided and valid." + ) + raise CredentialResolutionError(message=msg) diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/constants.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/constants.py new file mode 100644 index 0000000000..2ff1e930cb --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/constants.py @@ -0,0 +1,50 @@ +"""Constants, enums, and configuration values for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import re +from enum import Enum + +from lfx.services.adapters.deployment.schema import DeploymentType + +SUPPORTED_ADAPTER_DEPLOYMENT_TYPES: frozenset[DeploymentType] = frozenset({DeploymentType.AGENT}) +CREATE_MAX_RETRIES = 3 +UPDATE_MAX_RETRIES = 3 +ROLLBACK_MAX_RETRIES = 5 +RETRY_INITIAL_DELAY_SECONDS = 0.5 +PROVIDER_SPEC_RESOURCE_NAME_PREFIX_KEY = "resource_name_prefix" +DEFAULT_WXO_AGENT_LLM = "groq/openai/gpt-oss-120b" + +WXO_SANITIZE_RE = re.compile(r"[^a-zA-Z0-9_]") +WXO_TRANSLATE = str.maketrans({" ": "_", "-": "_"}) + +ERROR_PREFIX = "An error occurred while" +ERROR_SUFFIX_IN = "in Watsonx Orchestrate." + + +class WxOAuthURL(str, Enum): + MCSP = "https://iam.platform.saas.ibm.com" + IBM_IAM = "https://iam.cloud.ibm.com" + + +class ErrorPrefix(str, Enum): + CREATE = f"{ERROR_PREFIX} creating a deployment {ERROR_SUFFIX_IN}" + LIST = f"{ERROR_PREFIX} listing deployments {ERROR_SUFFIX_IN}" + GET = f"{ERROR_PREFIX} getting a deployment {ERROR_SUFFIX_IN}" + UPDATE = f"{ERROR_PREFIX} updating a deployment {ERROR_SUFFIX_IN}" + REDEPLOY = f"{ERROR_PREFIX} redeploying a deployment {ERROR_SUFFIX_IN}" + CLONE = f"{ERROR_PREFIX} cloning a deployment {ERROR_SUFFIX_IN}" + DELETE = f"{ERROR_PREFIX} deleting a deployment {ERROR_SUFFIX_IN}" + HEALTH = f"{ERROR_PREFIX} getting a deployment health {ERROR_SUFFIX_IN}" + CREATE_CONFIG = f"{ERROR_PREFIX} creating a deployment config {ERROR_SUFFIX_IN}" + LIST_CONFIGS = f"{ERROR_PREFIX} listing deployment configs {ERROR_SUFFIX_IN}" + GET_CONFIG = f"{ERROR_PREFIX} getting a deployment config {ERROR_SUFFIX_IN}" + UPDATE_CONFIG = f"{ERROR_PREFIX} updating a deployment config {ERROR_SUFFIX_IN}" + DELETE_CONFIG = f"{ERROR_PREFIX} deleting a deployment config {ERROR_SUFFIX_IN}" + CREATE_EXECUTION = f"{ERROR_PREFIX} creating a deployment execution {ERROR_SUFFIX_IN}" + GET_EXECUTION = f"{ERROR_PREFIX} getting a deployment execution {ERROR_SUFFIX_IN}" + + +# NOTE: this key must match the value of the provider_key column +# in the deployment_provider_account table. +WATSONX_ORCHESTRATE_DEPLOYMENT_ADAPTER_KEY = "watsonx-orchestrate" diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/__init__.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/__init__.py new file mode 100644 index 0000000000..464b02e032 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/__init__.py @@ -0,0 +1 @@ +"""Core internal modules for the Watsonx Orchestrate deployment adapter.""" diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/config.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/config.py new file mode 100644 index 0000000000..243004d0bc --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/config.py @@ -0,0 +1,152 @@ +"""Config management functions for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING + +from ibm_watsonx_orchestrate_core.types.connections import ( + ConnectionConfiguration, + ConnectionEnvironment, + ConnectionPreference, + ConnectionSecurityScheme, +) +from lfx.services.adapters.deployment.exceptions import ( + InvalidContentError, + InvalidDeploymentOperationError, +) +from lfx.services.adapters.deployment.schema import ConfigItem, DeploymentConfig, IdLike + +from langflow.services.adapters.deployment.watsonx_orchestrate.client import resolve_runtime_credentials +from langflow.services.adapters.deployment.watsonx_orchestrate.utils import validate_wxo_name + +if TYPE_CHECKING: + from ibm_watsonx_orchestrate_clients.connections.connections_client import ConnectionsClient, GetConnectionResponse + from sqlalchemy.ext.asyncio import AsyncSession + + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + + +async def create_config( + *, + clients: WxOClient, + config: DeploymentConfig, + user_id: IdLike, + db: AsyncSession, +) -> str: + """Create/update a wxO draft key-value connection config plus runtime credentials.""" + app_id = validate_wxo_name(config.name) + + await asyncio.to_thread(clients.connections.create, payload={"app_id": app_id}) + + wxo_config = ConnectionConfiguration( + app_id=app_id, + environment=ConnectionEnvironment.DRAFT, + preference=ConnectionPreference.TEAM, + security_scheme=ConnectionSecurityScheme.KEY_VALUE, + ) + await asyncio.to_thread( + clients.connections.create_config, + app_id=app_id, + payload=wxo_config.model_dump(exclude_unset=True, exclude_none=True), + ) + + runtime_credentials = await resolve_runtime_credentials( + environment_variables=config.environment_variables or {}, + user_id=user_id, + db=db, + ) + + await asyncio.to_thread( + clients.connections.create_credentials, + app_id=app_id, + env=ConnectionEnvironment.DRAFT, + use_app_credentials=False, + payload={"runtime_credentials": runtime_credentials.model_dump()}, + ) + + return app_id + + +async def process_config( + user_id: IdLike, + db: AsyncSession, + deployment_name: str, + config: ConfigItem | None, + *, + clients: WxOClient, +) -> str: + """Create and bind deployment config using deployment name as app_id.""" + validate_config_create_input(config) + + environment_variables = None + description = "" + + if config and config.raw_payload: + environment_variables = config.raw_payload.environment_variables + description = config.raw_payload.description or "" + + config_payload = DeploymentConfig( + name=deployment_name, + description=description, + environment_variables=environment_variables, + ) + app_id: str = await create_config( + clients=clients, + config=config_payload, + user_id=user_id, + db=db, + ) + + return app_id + + +def validate_config_create_input(config: ConfigItem | None) -> None: + if config and config.reference_id is not None: + msg = ( + "Config reference binding is not supported for deployment creation in " + "watsonx Orchestrate. Provide raw config payload or omit config." + ) + raise InvalidDeploymentOperationError(message=msg) + + +def resolve_create_app_id( + *, + prefixed_deployment_name: str, + config: ConfigItem | None, +) -> str: + validate_config_create_input(config) + if config is None or config.raw_payload is None: + return f"{prefixed_deployment_name}_app_id" + + normalized_config_name = validate_wxo_name(config.raw_payload.name) + return f"{prefixed_deployment_name}_{normalized_config_name}_app_id" + + +async def validate_connection(connections_client: ConnectionsClient, *, app_id: str) -> GetConnectionResponse: + connection = await asyncio.to_thread(connections_client.get_draft_by_app_id, app_id=app_id) + if not connection: + msg = f"Connection '{app_id}' not found. Ensure the connection exists with a draft configuration." + raise InvalidContentError(message=msg) + + config = await asyncio.to_thread(connections_client.get_config, app_id=app_id, env=ConnectionEnvironment.DRAFT) + if not config: + msg = f"Connection '{app_id}' is missing draft config. Deployments require draft mode." + raise InvalidContentError(message=msg) + + if config.security_scheme != ConnectionSecurityScheme.KEY_VALUE: + msg = f"Connection '{app_id}' must use key-value credentials for Langflow flows." + raise InvalidContentError(message=msg) + + runtime_credentials = await asyncio.to_thread( + connections_client.get_credentials, + app_id=app_id, + env=ConnectionEnvironment.DRAFT, + use_app_credentials=False, + ) + + if not runtime_credentials: + msg = f"Connection '{app_id}' is missing draft runtime credentials." + raise InvalidContentError(message=msg) + + return connection diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/execution.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/execution.py new file mode 100644 index 0000000000..8f9c1ab83c --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/execution.py @@ -0,0 +1,161 @@ +"""Execution creation, status, and output extraction for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING, Any + +from fastapi import status +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from lfx.services.adapters.deployment.exceptions import DeploymentError, DeploymentNotFoundError, InvalidContentError + +from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + +if TYPE_CHECKING: + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + + +def build_orchestrate_runs_query(provider_input: dict[str, Any] | None) -> str: + if not provider_input: + return "" + + query_segments: list[str] = [] + for key in ("stream", "multiple_content", "stream_timeout"): + if key not in provider_input or provider_input[key] is None: + continue + value = provider_input[key] + normalized_value = str(value).lower() if isinstance(value, bool) else str(value) + query_segments.append(f"{key}={normalized_value}") + + if not query_segments: + return "" + return f"?{'&'.join(query_segments)}" + + +def build_orchestrate_run_payload( + *, + provider_data: dict[str, Any], + deployment_id: str, +) -> dict[str, Any]: + message_payload = provider_data.get("message") + if message_payload is None: + message_payload = resolve_execution_message(provider_data.get("input")) + + payload: dict[str, Any] = { + "message": message_payload, + "agent_id": str(provider_data.get("agent_id") or deployment_id), + } + + extra_fields = [ + "thread_id", + "llm_params", + "guardrails", + "context", + "additional_parameters", + "environment_id", + "version", + "context_variables", + ] + + payload.update({k: v for k in extra_fields if (v := provider_data.get(k)) is not None}) + + return payload + + +async def create_agent_run( + client: WxOClient, + *, + provider_data: dict[str, Any], + deployment_id: str, +) -> dict[str, Any]: + """Create an orchestrate run through the WxOClient wrapper.""" + query_suffix = build_orchestrate_runs_query(provider_data) + try: + run_payload = build_orchestrate_run_payload( + provider_data=provider_data, + deployment_id=deployment_id, + ) + except ValueError as exc: + raise InvalidContentError(message=str(exc)) from exc + try: + response = await asyncio.to_thread( + client.post_run, + query_suffix=query_suffix, + data=run_payload, + ) + except ClientAPIException as exc: + if exc.response.status_code == status.HTTP_404_NOT_FOUND: + msg = f"Agent Deployment '{deployment_id}' was not found in Watsonx Orchestrate." + raise DeploymentNotFoundError(message=msg) from exc + if exc.response.status_code == status.HTTP_422_UNPROCESSABLE_CONTENT: + msg = ( + "Deployment execution request is unprocessable by Watsonx Orchestrate. " + f"{extract_error_detail(exc.response.text)}" + ) + raise InvalidContentError(message=msg) from exc + raise + return create_agent_run_result(response or {}) + + +def resolve_execution_message(execution_input: str | dict[str, Any] | None) -> dict[str, Any]: + if isinstance(execution_input, str): + if not execution_input.strip(): + msg = "Agent execution input message must not be empty." + raise ValueError(msg) + return {"role": "user", "content": execution_input} + + if isinstance(execution_input, dict): + if "role" in execution_input and "content" in execution_input: + return execution_input + + if "message" in execution_input and isinstance(execution_input["message"], dict): + return execution_input["message"] + + content = execution_input.get("content") + if isinstance(content, str) and content.strip(): + return {"role": "user", "content": content} + + msg = ( + "Agent execution requires input content. Provide a non-empty string input " + "or a message payload with 'role' and 'content'." + ) + raise ValueError(msg) + + +def create_agent_run_result(payload: dict[str, Any] | None) -> dict[str, Any]: + if not payload: + msg = "Watsonx Orchestrate returned an empty response for the execution request." + raise DeploymentError(message=msg, error_code="empty_provider_response") + + result: dict[str, Any] = {"status": payload.get("status") or "accepted"} + run_id = str(payload.get("run_id") or payload.get("id") or "").strip() + if not run_id: + msg = "Watsonx Orchestrate accepted the execution but did not return a run_id." + raise DeploymentError(message=msg, error_code="missing_run_id") + result["run_id"] = run_id + return result + + +async def get_agent_run(client: WxOClient, *, run_id: str) -> dict[str, Any]: + payload = await asyncio.to_thread(client.get_run, run_id) + + if not payload: + msg = f"Watsonx Orchestrate returned an empty response when fetching execution '{run_id}'." + raise DeploymentError(message=msg, error_code="empty_provider_response") + + status_value = str(payload.get("status") or "unknown") + result: dict[str, Any] = {"status": status_value} + + passthrough_fields = [ + "agent_id", + "run_id", + "started_at", + "completed_at", + "failed_at", + "cancelled_at", + "last_error", + "result", + ] + result.update({k: v for k in passthrough_fields if (v := payload.get(k)) is not None}) + + return result diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/retry.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/retry.py new file mode 100644 index 0000000000..ff2a4d73a4 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/retry.py @@ -0,0 +1,205 @@ +"""Retry/backoff and rollback logic for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import asyncio +import logging +import random +from collections.abc import Awaitable, Callable +from typing import TYPE_CHECKING, Any, TypeVar + +from fastapi import HTTPException, status +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from lfx.services.adapters.deployment.exceptions import ( + DeploymentConflictError, + InvalidContentError, + InvalidDeploymentOperationError, + InvalidDeploymentTypeError, +) + +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ( + CREATE_MAX_RETRIES, + RETRY_INITIAL_DELAY_SECONDS, + ROLLBACK_MAX_RETRIES, + UPDATE_MAX_RETRIES, +) + +logger = logging.getLogger(__name__) + +if TYPE_CHECKING: + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + +T = TypeVar("T") +Operation = Callable[..., Awaitable[T]] +ShouldRetry = Callable[[Exception], bool] + + +async def retry_with_backoff( + operation: Operation[T], + max_attempts: int, + *args: Any, + should_retry: ShouldRetry | None = None, + **kwargs: Any, +) -> T: + delay_seconds = RETRY_INITIAL_DELAY_SECONDS + for attempt in range(1, max_attempts + 1): + try: + return await operation(*args, **kwargs) + except Exception as exc: + retryable = True if should_retry is None else should_retry(exc) + if not retryable or attempt == max_attempts: + raise + jittered_delay = delay_seconds * (0.5 + random.random()) # noqa: S311 + logger.info( + "Retry attempt %d/%d after %.2fs (%s)", attempt, max_attempts, jittered_delay, type(exc).__name__ + ) + await asyncio.sleep(jittered_delay) + delay_seconds *= 2 + msg = "Retry helper exhausted attempts without result." + raise RuntimeError(msg) + + +async def retry_create(operation: Operation[T], *args: Any, **kwargs: Any) -> T: + return await retry_with_backoff( + operation, + CREATE_MAX_RETRIES, + *args, + should_retry=is_retryable_create_exception, + **kwargs, + ) + + +async def retry_update(operation: Operation[T], *args: Any, **kwargs: Any) -> T: + """Retry write/update operations with the standard provider retry policy.""" + return await retry_with_backoff( + operation, + UPDATE_MAX_RETRIES, + *args, + should_retry=is_retryable_create_exception, + **kwargs, + ) + + +async def retry_rollback(operation: Operation[T], *args: Any, **kwargs: Any) -> T: + return await retry_with_backoff( + operation, + ROLLBACK_MAX_RETRIES, + *args, + should_retry=is_retryable_create_exception, + **kwargs, + ) + + +def is_retryable_create_exception(exc: Exception) -> bool: + non_retryable_status_codes = { + status.HTTP_400_BAD_REQUEST, + status.HTTP_401_UNAUTHORIZED, + status.HTTP_403_FORBIDDEN, + status.HTTP_404_NOT_FOUND, + status.HTTP_409_CONFLICT, + status.HTTP_422_UNPROCESSABLE_CONTENT, + } + if isinstance(exc, ClientAPIException): + return exc.response.status_code not in non_retryable_status_codes + if isinstance(exc, HTTPException): + return exc.status_code not in non_retryable_status_codes + return not isinstance( + exc, + ( + DeploymentConflictError, + InvalidContentError, + InvalidDeploymentOperationError, + InvalidDeploymentTypeError, + ), + ) + + +async def rollback_created_resources( + *, + clients: WxOClient, + agent_id: str | None, + tool_ids: list[str], + app_id: str | None, +) -> None: + logger.info("Rolling back resources: agent_id=%s, tool_ids=%s, app_id=%s", agent_id, tool_ids, app_id) + if agent_id: + try: + await retry_rollback(delete_agent_if_exists, clients, agent_id=agent_id) + except Exception: + logger.exception("Rollback failed for agent_id=%s — resource may be orphaned", agent_id) + if tool_ids: + for tool_id in reversed(tool_ids): + try: + await retry_rollback(delete_tool_if_exists, clients, tool_id=tool_id) + except Exception: + logger.exception("Rollback failed for tool_id=%s — resource may be orphaned", tool_id) + if app_id: + try: + await retry_rollback(delete_config_if_exists, clients, app_id=app_id) + except Exception: + logger.exception("Rollback failed for app_id=%s — resource may be orphaned", app_id) + + +async def rollback_update_resources( + *, + clients: WxOClient, + created_tool_ids: list[str], + created_app_id: str | None, + original_tools: dict[str, dict], +) -> None: + """Best-effort rollback for update operations. + + Restores mutated tools first, then deletes newly created tools, then deletes + newly created config. Unlike ``rollback_created_resources`` this never + deletes the deployment/agent itself. + """ + logger.info( + "Rolling back update resources: created_tool_ids=%s, created_app_id=%s, mutated_tools=%s", + created_tool_ids, + created_app_id, + list(original_tools.keys()), + ) + for tool_id, original_tool in reversed(list(original_tools.items())): + try: + await retry_rollback(asyncio.to_thread, clients.tool.update, tool_id, original_tool) + except Exception: + logger.exception( + "Rollback failed: could not restore tool payload for tool_id=%s — resource may be orphaned", + tool_id, + ) + + for tool_id in reversed(created_tool_ids): + try: + await retry_rollback(delete_tool_if_exists, clients, tool_id=tool_id) + except Exception: + logger.exception("Rollback failed for created tool_id=%s — resource may be orphaned", tool_id) + + if created_app_id: + try: + await retry_rollback(delete_config_if_exists, clients, app_id=created_app_id) + except Exception: + logger.exception("Rollback failed for created app_id=%s — resource may be orphaned", created_app_id) + + +async def delete_agent_if_exists(clients: WxOClient, *, agent_id: str) -> None: + try: + await asyncio.to_thread(clients.agent.delete, agent_id) + except ClientAPIException as exc: + if exc.response.status_code != status.HTTP_404_NOT_FOUND: + raise + + +async def delete_tool_if_exists(clients: WxOClient, *, tool_id: str) -> None: + try: + await asyncio.to_thread(clients.tool.delete, tool_id) + except ClientAPIException as exc: + if exc.response.status_code != status.HTTP_404_NOT_FOUND: + raise + + +async def delete_config_if_exists(clients: WxOClient, *, app_id: str) -> None: + try: + await asyncio.to_thread(clients.connections.delete, app_id) + except ClientAPIException as exc: + if exc.response.status_code != status.HTTP_404_NOT_FOUND: + raise diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/status.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/status.py new file mode 100644 index 0000000000..98ffc06536 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/status.py @@ -0,0 +1,72 @@ +"""Health/status helpers and metadata mapping for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +from typing import Any + +from ibm_watsonx_orchestrate_core.types.connections import ConnectionEnvironment +from lfx.services.adapters.deployment.schema import DeploymentGetResult, DeploymentType, ItemResult + + +def get_deployment_metadata( + data: dict[str, Any], + deployment_type: DeploymentType, + provider_data: dict[str, Any] | None = None, +) -> ItemResult: + result: dict[str, Any] = { + "id": data.get("id"), + "type": deployment_type.value, + "name": data.get("name"), + "created_at": data.get("created_on"), + "updated_at": data.get("updated_at"), + } + if provider_data: + result["provider_data"] = provider_data + + return ItemResult(**result) + + +def get_deployment_detail_metadata( + data: dict[str, Any], + deployment_type: DeploymentType, + provider_data: dict[str, Any] | None = None, + provider_raw: bool = False, # noqa: FBT001,FBT002 +) -> DeploymentGetResult: + result: dict[str, Any] = { + "id": data.get("id"), + "type": deployment_type.value, + "name": data.get("name"), + "description": data.get("description"), + } + if provider_data: + result["provider_data"] = provider_data + if provider_raw: + result["provider_data"] = data if not provider_data else {**provider_data, "provider_raw": data} + + return DeploymentGetResult(**result) + + +def derive_agent_environment(agent: dict[str, Any]) -> str: + environments = agent.get("environments", []) + if not isinstance(environments, list) or not environments: + return "unknown" + + has_draft = False + has_live = False + for env in environments: + if not isinstance(env, dict): + continue + env_name = str(env.get("name", "")).strip().lower() + if env_name == ConnectionEnvironment.DRAFT.value: + has_draft = True + continue + if env_name: + has_live = True + + if has_draft and has_live: + return "both" + if has_live: + return "live" + if has_draft: + return "draft" + return "unknown" diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/tools.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/tools.py new file mode 100644 index 0000000000..21698457e6 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/core/tools.py @@ -0,0 +1,447 @@ +"""Snapshot/flow tool creation, artifact building, and upload for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import asyncio +import copy +import importlib.metadata as md +import io +import json +import logging +import zipfile +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +from cachetools import func +from ibm_watsonx_orchestrate_core.types.tools.langflow_tool import LangflowTool +from ibm_watsonx_orchestrate_core.types.tools.langflow_tool import create_langflow_tool as _create_langflow_tool +from lfx.services.adapters.deployment.exceptions import InvalidContentError, InvalidDeploymentOperationError +from lfx.utils.flow_requirements import generate_requirements_from_flow + +from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_create +from langflow.services.adapters.deployment.watsonx_orchestrate.utils import ( + dedupe_list, + normalize_wxo_name, + require_tool_id, +) +from langflow.utils.version import get_version_info + +logger = logging.getLogger(__name__) + +if TYPE_CHECKING: + from lfx.services.adapters.deployment.schema import BaseFlowArtifact, SnapshotItems + + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + +# TODO: ensure all fields from here are used +# https://developer.watson-orchestrate.ibm.com/apis/tools/patch-a-tool +# as it is a PUT endpoint (don't want to lose any fields) +_WRITABLE_TOOL_FIELDS = ( + "description", + "permission", + "name", + "display_name", + "input_schema", + "output_schema", + "binding", + "tags", + "is_async", + "restrictions", + "bundled_agent_id", +) + + +@dataclass(slots=True) +class FlowToolBindingSpec: + flow_payload: BaseFlowArtifact + connections: dict[str, str] + + +class ToolUploadBatchError(RuntimeError): + """Raised when a concurrent tool-upload batch partially succeeds.""" + + def __init__(self, *, created_tool_ids: list[str], errors: list[Exception]) -> None: + self.created_tool_ids = created_tool_ids + self.errors = errors + super().__init__("One or more tool uploads failed.") + + +def to_writable_tool_payload(tool: dict[str, Any]) -> dict[str, Any]: + """Build tool payload accepted by wxO tool update endpoint.""" + return {field: copy.deepcopy(tool[field]) for field in _WRITABLE_TOOL_FIELDS if field in tool} + + +def _ensure_dict(parent: dict[str, Any], key: str) -> dict[str, Any]: + """Return ``parent[key]`` as a dict, replacing non-dict values with ``{}``.""" + value = parent.setdefault(key, {}) + if not isinstance(value, dict): + logger.warning( + "Expected dict at key '%s' but found %s; replacing with empty dict", + key, + type(value).__name__, + ) + value = {} + parent[key] = value + return value + + +def ensure_langflow_connections_binding(tool_payload: dict[str, Any]) -> dict[str, str]: + """Ensure ``binding.langflow.connections`` exists in *tool_payload* and return the mutable dict. + + Non-dict values at any nesting level are silently replaced with ``{}``. + We intentionally do *not* raise on a malformed shape because + callers of this function are *writing* connection bindings into + these payloads (and the pre-mutation snapshot is captured for rollback), + replacing an unexpected value is traded with explcitiness + to prevent a stubbornly failing update. + """ + binding = _ensure_dict(tool_payload, "binding") + langflow = _ensure_dict(binding, "langflow") + return _ensure_dict(langflow, "connections") + + +async def update_existing_tool_connection_bindings( + *, + clients: WxOClient, + existing_target_tool_ids: list[str], + resolved_connections: dict[str, str], + original_tools: dict[str, dict[str, Any]], +) -> None: + """Apply resolved connection bindings to existing tools. + + Captures original writable payloads for rollback before any update call. + Raises ``InvalidContentError`` when any expected tool id is missing. + """ + if not existing_target_tool_ids: + return + + tools = await asyncio.to_thread(clients.tool.get_drafts_by_ids, existing_target_tool_ids) + tool_by_id = {str(tool.get("id")): tool for tool in tools if isinstance(tool, dict) and tool.get("id")} + missing_tool_ids = [tool_id for tool_id in existing_target_tool_ids if tool_id not in tool_by_id] + if missing_tool_ids: + missing_ids = ", ".join(missing_tool_ids) + msg = f"Snapshot tool(s) not found: {missing_ids}" + raise InvalidContentError(message=msg) + + tool_updates: list[tuple[str, dict[str, Any]]] = [] + for tool_id in existing_target_tool_ids: + original_tool = to_writable_tool_payload(tool_by_id[tool_id]) + original_tools[tool_id] = original_tool + writable_tool = copy.deepcopy(original_tool) + connections = ensure_langflow_connections_binding(writable_tool) + connections.update(resolved_connections) + tool_updates.append((tool_id, writable_tool)) + + await asyncio.gather( + *( + retry_create(asyncio.to_thread, clients.tool.update, tool_id, writable_tool) + for tool_id, writable_tool in tool_updates + ) + ) + + +def extract_langflow_artifact_from_zip(artifact_zip_bytes: bytes, *, snapshot_id: str) -> dict[str, Any]: + """Read and parse the Langflow flow JSON from a wxO snapshot artifact zip.""" + try: + with zipfile.ZipFile(io.BytesIO(artifact_zip_bytes), "r") as zip_artifact: + json_members = [name for name in zip_artifact.namelist() if name.lower().endswith(".json")] + if not json_members: + msg = f"Snapshot '{snapshot_id}' artifact does not include a flow JSON file." + raise InvalidContentError(message=msg) + + flow_json_member = json_members[0] + flow_json_raw = zip_artifact.read(flow_json_member) + except InvalidContentError: + raise + except zipfile.BadZipFile as exc: + msg = f"Snapshot '{snapshot_id}' artifact is not a valid zip archive." + raise InvalidContentError(message=msg) from exc + + try: + return json.loads(flow_json_raw.decode("utf-8")) + except UnicodeDecodeError as exc: + msg = f"Snapshot '{snapshot_id}' flow artifact is not valid UTF-8 JSON." + raise InvalidContentError(message=msg) from exc + except json.JSONDecodeError as exc: + msg = f"Snapshot '{snapshot_id}' flow artifact contains invalid JSON." + raise InvalidContentError(message=msg) from exc + + +def build_langflow_artifact_bytes( + *, + tool: LangflowTool, + flow_definition: dict[str, Any], + flow_filename: str | None = None, +) -> bytes: + filename = flow_filename or f"{tool.__tool_spec__.name}.json" + lfx_requirement = _resolve_lfx_requirement() + requirements = generate_requirements_from_flow( + flow_definition, + include_lfx=False, + pin_versions=True, + ) + requirements = [lfx_requirement, *requirements] + requirements = dedupe_list(requirements) + requirements_content = "\n".join(requirements) + "\n" + flow_content = json.dumps(flow_definition, indent=2) + + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as zip_tool_artifacts: + zip_tool_artifacts.writestr(filename, flow_content) + zip_tool_artifacts.writestr("requirements.txt", requirements_content) + zip_tool_artifacts.writestr("bundle-format", "2.0.0\n") + return buffer.getvalue() + + +def upload_tool_artifact_bytes( + clients: WxOClient, + *, + tool_id: str, + artifact_bytes: bytes, +) -> dict[str, Any]: + file_obj = io.BytesIO(artifact_bytes) + return clients.upload_tool_artifact( + tool_id, + files={"file": (f"{tool_id}.zip", file_obj, "application/zip", {"Expires": "0"})}, + ) + + +def create_wxo_flow_tool( + *, + flow_payload: BaseFlowArtifact, + connections: dict[str, str], + tool_name_prefix: str, +) -> tuple[dict[str, Any], bytes]: + """Create a Watsonx Orchestrate flow tool specification. + + Given a flow payload and connections dictionary, + create a Watsonx Orchestrate flow tool specification + and the supporting artifacts of the requirements.txt + and the flow json file. + + Args: + flow_payload: The flow payload to create the tool specification for. + connections: The connections dictionary to create the tool specification for. + tool_name_prefix: Deterministic prefix for the resulting tool name. + + Returns: + Tuple[dict[str, Any], bytes]: a tuple containing: + - tool_payload: The Watsonx Orchestrate flow tool specification. + - artifacts: The supporting artifacts (the requirements.txt + and the flow json file) for the tool. + """ + flow_definition = flow_payload.model_dump() + + flow_provider_data = flow_definition.pop("provider_data", None) + + if not isinstance(flow_provider_data, dict): + msg = "Flow payload must include provider_data with a non-empty project_id for Watsonx deployment." + raise InvalidContentError(message=msg) + project_id = str(flow_provider_data.get("project_id") or "").strip() + if not project_id: + msg = "Flow payload must include provider_data with a non-empty project_id for Watsonx deployment." + raise InvalidContentError(message=msg) + + flow_definition.update( + { + "name": normalize_wxo_name(flow_definition.get("name") or ""), + "id": str(flow_definition.get("id")), + } + ) + + # Fallback for flows that don't include last_tested_version in payload + if not flow_definition.get("last_tested_version"): + detected_version = (get_version_info() or {}).get("version") + if not detected_version: + msg = "Unable to determine running Langflow version for snapshot creation." + raise InvalidContentError(message=msg) + flow_definition["last_tested_version"] = detected_version + + tool: LangflowTool = create_langflow_tool( + tool_definition=flow_definition, + connections=connections, + show_details=False, + ) + + tool_payload = tool.__tool_spec__.model_dump( + mode="json", + exclude_unset=True, + exclude_none=True, + by_alias=True, + ) + + current_name = str(tool_payload.get("name") or "").strip() + + if current_name: + normalized_current_name = normalize_wxo_name(current_name) + tool_payload["name"] = f"{tool_name_prefix}{normalized_current_name}" + + (tool_payload.setdefault("binding", {}).setdefault("langflow", {})["project_id"]) = project_id + + artifacts: bytes = build_langflow_artifact_bytes( + tool=tool, + flow_definition=flow_definition, + ) + + return tool_payload, artifacts + + +def create_langflow_tool( + *, + tool_definition: dict[str, Any], + connections: dict[str, str], + show_details: bool, +) -> LangflowTool: + """Module-level wrapper to keep tool creation monkeypatchable in tests.""" + return _create_langflow_tool( + tool_definition=tool_definition, + connections=connections, + show_details=show_details, + ) + + +async def create_and_upload_wxo_flow_tools( + *, + clients: WxOClient, + flow_payloads: list[BaseFlowArtifact], + connections: dict[str, str], + tool_name_prefix: str, +) -> list[str]: + tool_bindings = [ + FlowToolBindingSpec( + flow_payload=flow_payload, + connections=connections, + ) + for flow_payload in flow_payloads + ] + return await create_and_upload_wxo_flow_tools_with_bindings( + clients=clients, + tool_bindings=tool_bindings, + tool_name_prefix=tool_name_prefix, + ) + + +async def create_and_upload_wxo_flow_tools_with_bindings( + *, + clients: WxOClient, + tool_bindings: list[FlowToolBindingSpec], + tool_name_prefix: str, +) -> list[str]: + specs = [ + create_wxo_flow_tool( + flow_payload=tool_binding.flow_payload, + connections=tool_binding.connections, + tool_name_prefix=tool_name_prefix, + ) + for tool_binding in tool_bindings + ] + created_tool_ids_journal: list[str] = [] + results = await asyncio.gather( + *( + upload_wxo_flow_tool( + clients=clients, + tool_payload=tool_payload, + artifact_bytes=artifact_bytes, + created_tool_ids_journal=created_tool_ids_journal, + ) + for tool_payload, artifact_bytes in specs + ), + return_exceptions=True, + ) + errors: list[Exception] = [] + created_tool_ids: list[str] = [] + for result in results: + if isinstance(result, BaseException): + if isinstance(result, Exception): + errors.append(result) + else: + errors.append(RuntimeError(f"Tool upload failed with non-standard exception: {type(result).__name__}")) + continue + created_tool_ids.append(result) + if errors: + raise ToolUploadBatchError(created_tool_ids=dedupe_list(created_tool_ids_journal), errors=errors) + return created_tool_ids + + +async def upload_wxo_flow_tool( + *, + clients: WxOClient, + tool_payload: dict[str, Any], + artifact_bytes: bytes, + created_tool_ids_journal: list[str] | None = None, +) -> str: + tool_response = await retry_create(asyncio.to_thread, clients.tool.create, tool_payload) + tool_id = require_tool_id(tool_response) + if created_tool_ids_journal is not None: + created_tool_ids_journal.append(tool_id) + + await retry_create( + asyncio.to_thread, + upload_tool_artifact_bytes, + clients, + tool_id=tool_id, + artifact_bytes=artifact_bytes, + ) + return tool_id + + +def build_snapshot_tool_names( + *, + snapshots: SnapshotItems | None, + tool_name_prefix: str, +) -> list[str]: + if snapshots is None: + return [] + + tool_names: list[str] = [] + for snapshot in snapshots.raw_payloads: + normalized_tool_name = normalize_wxo_name(str(snapshot.name)) + if not normalized_tool_name: + msg = "Snapshot name must include at least one alphanumeric character." + raise InvalidContentError(message=msg) + tool_names.append(f"{tool_name_prefix}{normalized_tool_name}") + return tool_names + + +async def process_raw_flows_with_app_id( + clients: WxOClient, + app_id: str, + flows: list[BaseFlowArtifact], + tool_name_prefix: str, +) -> list[str]: + """Create langflow tools in wxO and connect them to the given app_id.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import validate_connection + + if not tool_name_prefix.strip(): + msg = "Snapshot creation requires a non-empty tool_name_prefix." + raise InvalidDeploymentOperationError(message=msg) + + connection = await validate_connection(clients.connections, app_id=app_id) + + return await create_and_upload_wxo_flow_tools( + clients=clients, + flow_payloads=flows, + connections={app_id: connection.connection_id}, + tool_name_prefix=tool_name_prefix, + ) + + +# TODO(WXO): find a way to make this fallback not hard-coded. +_LFX_MINIMUM_REQUIREMENT = "lfx>=0.3.0" + + +@func.ttl_cache(maxsize=1, ttl=60) +def _pin_requirement_name(package_name: str) -> str: + return f"{package_name}=={md.version(package_name)}" + + +def _resolve_lfx_requirement() -> str: + """Pin lfx to the installed version, falling back to a minimum spec.""" + try: + return _pin_requirement_name("lfx") + except (md.PackageNotFoundError, ValueError): + logger.warning( + "Could not determine installed lfx version; falling back to minimum requirement '%s'", + _LFX_MINIMUM_REQUIREMENT, + ) + return _LFX_MINIMUM_REQUIREMENT diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/payloads.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/payloads.py new file mode 100644 index 0000000000..978da0a8ba --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/payloads.py @@ -0,0 +1,248 @@ +"""Watsonx Orchestrate custom deployment update payload contracts.""" + +from __future__ import annotations + +from collections import Counter +from typing import Annotated, Literal + +from lfx.services.adapters.deployment.schema import BaseFlowArtifact, EnvVarKey, EnvVarValueSpec, NormalizedId +from lfx.services.adapters.payload import AdapterPayload +from pydantic import BaseModel, ConfigDict, Field, StringConstraints, field_validator, model_validator + +RawToolName = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1)] + + +class WatsonxConnectionRawPayload(BaseModel): + """Connection payload for creating a new watsonx connection/config.""" + + app_id: NormalizedId = Field( + description=( + "App id used for operation references. resource_name_prefix is applied only when creating resources." + ) + ) + environment_variables: dict[EnvVarKey, EnvVarValueSpec] | None = Field(None, description="Environment variables.") + provider_config: AdapterPayload | None = Field(None, description="Provider-specific connection configuration.") + + +class WatsonxUpdateTools(BaseModel): + """Tool pool available to update operations.""" + + model_config = ConfigDict(extra="forbid") + + existing_ids: list[NormalizedId] | None = Field( + default=None, + description="Known existing provider tool ids available for operation references.", + ) + raw_payloads: list[BaseFlowArtifact] | None = Field( + default=None, + description="Raw tool payloads keyed by BaseFlowArtifact.name.", + ) + + @field_validator("existing_ids") + @classmethod + def dedupe_existing_ids(cls, value: list[str] | None) -> list[str] | None: + if value is None: + return None + return list(dict.fromkeys(value)) + + @model_validator(mode="after") + def validate_unique_raw_tool_names(self) -> WatsonxUpdateTools: + raw_payloads = self.raw_payloads or [] + name_counts = Counter(payload.name for payload in raw_payloads) + duplicates = sorted(name for name, count in name_counts.items() if count > 1) + if duplicates: + msg = f"tools.raw_payloads contains duplicate names: {duplicates}" + raise ValueError(msg) + return self + + +class WatsonxUpdateConnections(BaseModel): + """Connection pool available to update operations.""" + + model_config = ConfigDict(extra="forbid") + + existing_app_ids: list[NormalizedId] | None = Field( + default=None, + description="Known existing app ids available for operation references.", + ) + raw_payloads: list[WatsonxConnectionRawPayload] | None = Field( + default=None, + description=( + "Raw connection payloads keyed by app_id. resource_name_prefix is applied only when resources are created." + ), + ) + + @field_validator("existing_app_ids") + @classmethod + def dedupe_existing_app_ids(cls, value: list[str] | None) -> list[str] | None: + if value is None: + return None + return list(dict.fromkeys(value)) + + @model_validator(mode="after") + def validate_unique_raw_app_ids(self) -> WatsonxUpdateConnections: + raw_payloads = self.raw_payloads or [] + app_id_counts = Counter(payload.app_id for payload in raw_payloads) + duplicates = sorted(app_id for app_id, count in app_id_counts.items() if count > 1) + if duplicates: + msg = f"connections.raw_payloads contains duplicate app_id values: {duplicates}" + raise ValueError(msg) + return self + + +class WatsonxToolReference(BaseModel): + """Tool selector for bind operations.""" + + model_config = ConfigDict(extra="forbid") + + reference_id: NormalizedId | None = Field( + default=None, + description="Existing provider tool id.", + ) + name_of_raw: RawToolName | None = Field( + default=None, + description="Name of a tool entry declared in tools.raw_payloads.", + ) + + @model_validator(mode="after") + def validate_exactly_one_selector(self) -> WatsonxToolReference: + has_reference_id = self.reference_id is not None + has_name_of_raw = self.name_of_raw is not None + if has_reference_id == has_name_of_raw: + msg = "Exactly one of 'tool.reference_id' or 'tool.name_of_raw' must be provided." + raise ValueError(msg) + return self + + +class WatsonxBindOperation(BaseModel): + """Bind a selected tool to a selected app id.""" + + model_config = ConfigDict(extra="forbid") + + op: Literal["bind"] + tool: WatsonxToolReference + app_ids: list[NormalizedId] = Field( + min_length=1, + description=( + "Operation app ids to bind. Must match declared connection pools " + "(connections.existing_app_ids or connections.raw_payloads[*].app_id)." + ), + ) + + @field_validator("app_ids") + @classmethod + def dedupe_app_ids(cls, value: list[str]) -> list[str]: + return list(dict.fromkeys(value)) + + +class WatsonxUnbindOperation(BaseModel): + """Unbind app connection from a tool.""" + + model_config = ConfigDict(extra="forbid") + + op: Literal["unbind"] + tool_id: NormalizedId = Field(description="Existing provider tool id.") + app_ids: list[NormalizedId] = Field( + min_length=1, + description=("Operation app ids to unbind. Must reference connections.existing_app_ids only."), + ) + + @field_validator("app_ids") + @classmethod + def dedupe_app_ids(cls, value: list[str]) -> list[str]: + return list(dict.fromkeys(value)) + + +class WatsonxRemoveToolOperation(BaseModel): + """Detach an existing tool from the deployment.""" + + model_config = ConfigDict(extra="forbid") + + op: Literal["remove_tool"] + tool_id: NormalizedId = Field(description="Existing provider tool id to remove from deployment.") + + +WatsonxUpdateOperation = Annotated[ + WatsonxBindOperation | WatsonxUnbindOperation | WatsonxRemoveToolOperation, + Field(discriminator="op"), +] + + +class WatsonxDeploymentUpdatePayload(BaseModel): + """Watsonx provider_data contract for deployment update patch operations. + + Notes: + - operations[*].app_ids are operation-side ids. + - resource_name_prefix is applied only when creating resources + (for raw connections and raw tools). + """ + + model_config = ConfigDict(extra="forbid") + + resource_name_prefix: str | None = Field( + default=None, + description=( + "Prefix applied only when creating resources: " + "derived app ids from connections.raw_payloads[*].app_id and created tool names." + ), + ) + tools: WatsonxUpdateTools = Field(default_factory=WatsonxUpdateTools) + connections: WatsonxUpdateConnections = Field(default_factory=WatsonxUpdateConnections) + operations: list[WatsonxUpdateOperation] = Field(min_length=1) + + @model_validator(mode="after") + def validate_operation_references(self) -> WatsonxDeploymentUpdatePayload: + raw_tool_names = {payload.name for payload in (self.tools.raw_payloads or [])} + existing_tool_ids = set(self.tools.existing_ids or []) + + existing_app_ids = set(self.connections.existing_app_ids or []) + raw_app_ids = {payload.app_id for payload in (self.connections.raw_payloads or [])} + collisions = sorted(existing_app_ids.intersection(raw_app_ids)) + if collisions: + msg = f"connections.existing_app_ids collides with raw app ids from connections.raw_payloads: {collisions}" + raise ValueError(msg) + valid_app_ids = existing_app_ids.union(raw_app_ids) + + referenced_app_ids: set[str] = set() + + for operation in self.operations: + if isinstance(operation, WatsonxBindOperation): + if operation.tool.name_of_raw is not None and operation.tool.name_of_raw not in raw_tool_names: + msg = f"bind.tool.name_of_raw not found in tools.raw_payloads: [{operation.tool.name_of_raw!r}]" + raise ValueError(msg) + if operation.tool.reference_id is not None and operation.tool.reference_id not in existing_tool_ids: + msg = f"bind.tool.reference_id not found in tools.existing_ids: [{operation.tool.reference_id!r}]" + raise ValueError(msg) + for app_id in operation.app_ids: + referenced_app_ids.add(app_id) + if app_id not in valid_app_ids: + msg = ( + "operation app_ids must be declared in " + "connections.existing_app_ids or connections.raw_payloads[*].app_id: " + f"[{app_id!r}]" + ) + raise ValueError(msg) + if isinstance(operation, WatsonxUnbindOperation): + for app_id in operation.app_ids: + referenced_app_ids.add(app_id) + if app_id not in valid_app_ids: + msg = ( + "operation app_ids must be declared in " + "connections.existing_app_ids or connections.raw_payloads[*].app_id: " + f"[{app_id!r}]" + ) + raise ValueError(msg) + if app_id in raw_app_ids: + msg = f"unbind.operation app_ids must reference connections.existing_app_ids only: [{app_id!r}]" + raise ValueError(msg) + + unused_existing_app_ids = sorted(existing_app_ids.difference(referenced_app_ids)) + if unused_existing_app_ids: + msg = f"connections.existing_app_ids contains ids not referenced by operations: {unused_existing_app_ids}" + raise ValueError(msg) + unused_raw_app_ids = sorted(raw_app_ids.difference(referenced_app_ids)) + if unused_raw_app_ids: + msg = f"connections.raw_payloads contains app_id values not referenced by operations: {unused_raw_app_ids}" + raise ValueError(msg) + + return self diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/service.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/service.py new file mode 100644 index 0000000000..965ae811d7 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/service.py @@ -0,0 +1,799 @@ +"""Slim WatsonxOrchestrateDeploymentService that delegates to submodules.""" + +from __future__ import annotations + +import asyncio +import logging +from typing import TYPE_CHECKING + +from fastapi import HTTPException, status +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from lfx.services.adapters.deployment.base import BaseDeploymentService +from lfx.services.adapters.deployment.exceptions import ( + AuthenticationError, + AuthorizationError, + DeploymentConflictError, + DeploymentError, + DeploymentNotFoundError, + DeploymentSupportError, + InvalidContentError, + InvalidDeploymentOperationError, + InvalidDeploymentTypeError, + OperationNotSupportedError, +) +from lfx.services.adapters.deployment.payloads import DeploymentPayloadSchemas +from lfx.services.adapters.deployment.schema import ( + BaseDeploymentData, + ConfigListItem, + ConfigListParams, + ConfigListResult, + DeploymentCreate, + DeploymentCreateResult, + DeploymentDeleteResult, + DeploymentDuplicateResult, + DeploymentGetResult, + DeploymentListParams, + DeploymentListResult, + DeploymentListTypesResult, + DeploymentStatusResult, + DeploymentType, + DeploymentUpdate, + DeploymentUpdateResult, + ExecutionCreate, + ExecutionCreateResult, + ExecutionStatusResult, + IdLike, + RedeployResult, + SnapshotItem, + SnapshotListParams, + SnapshotListResult, + _normalize_and_validate_id, +) +from lfx.services.adapters.payload import AdapterPayloadValidationError, PayloadSlot + +from langflow.services.adapters.deployment.watsonx_orchestrate.client import get_provider_clients +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ( + PROVIDER_SPEC_RESOURCE_NAME_PREFIX_KEY, + SUPPORTED_ADAPTER_DEPLOYMENT_TYPES, + ErrorPrefix, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import ( + process_config, + resolve_create_app_id, + validate_config_create_input, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import ( + create_agent_run, + get_agent_run, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import ( + retry_create, + rollback_created_resources, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.status import ( + derive_agent_environment, + get_deployment_detail_metadata, + get_deployment_metadata, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + build_snapshot_tool_names, + process_raw_flows_with_app_id, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.payloads import ( + WatsonxDeploymentUpdatePayload, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.update_helpers import ( + apply_provider_update_plan_with_rollback, + build_provider_update_plan, + build_update_payload_from_spec, + validate_provider_update_request_sections, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.utils import ( + _require_single_deployment_id, + build_agent_payload, + dedupe_list, + extract_agent_tool_ids, + extract_error_detail, + raise_as_deployment_error, + resolve_resource_name_prefix, + validate_wxo_name, +) +from langflow.services.deps import get_settings_service + +logger = logging.getLogger(__name__) + +if TYPE_CHECKING: + from collections.abc import Sequence + from typing import Any + + from ibm_watsonx_orchestrate_clients.agents.agent_client import AgentUpsertResponse + from lfx.services.settings.service import SettingsService + from sqlalchemy.ext.asyncio import AsyncSession + + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + + +class WatsonxOrchestrateDeploymentService(BaseDeploymentService): + """Deployment adapter for Watsonx Orchestrate.""" + + name = "deployment_service" + payload_schemas = DeploymentPayloadSchemas( + deployment_update=PayloadSlot(WatsonxDeploymentUpdatePayload), + ) + + def __init__(self, settings_service: SettingsService | None = None): + super().__init__() + if settings_service is None: + settings_service = get_settings_service() + if settings_service is None: + msg = "Settings service is not available." + raise RuntimeError(msg) + self.settings_service = settings_service + self.set_ready() + + async def _get_provider_clients(self, *, user_id: IdLike, db: AsyncSession) -> WxOClient: + """Resolve provider clients through a service-level seam. + + A dedicated method keeps call sites patchable in unit/e2e tests and + centralizes provider-client resolution behavior for this service. + """ + return await get_provider_clients(user_id=user_id, db=db) + + async def create( + self, + *, + user_id: IdLike, + payload: DeploymentCreate, + db: AsyncSession, + ) -> DeploymentCreateResult: + """Create a deployment in Watsonx Orchestrate.""" + # The wxO API does not have an endpoint to create + # a connection, tool, and agent atomically. + # We have to make a separate api call for each resource. + # -- + # If one of these resources is created successfully + # but the next one fails, then we end up with orphaned resources. + # - We thus use a best-effort creation and rollback strategy: + # Attempt to create each resource with retries. + # If the creation of one resource fails, + # then attempt to delete all previously created + # resources with retries. + # -- + # The caller must supply a resource name prefix via + # provider_spec["resource_name_prefix"]. + # Every created resource (connection, tool, agent) is + # prefixed with this value, which prevents name collisions + # and supports idempotent retries (re-use the same prefix + # across attempts). We recommend using a random prefix. + # -- + logger.info("Creating wxO deployment for user_id=%s", user_id) + agent_create_response: AgentUpsertResponse | None = None + created_tool_ids: list[str] = [] + created_app_id: str | None = None + clients: WxOClient | None = None + derived_spec: BaseDeploymentData | None = None + try: + deployment_spec: BaseDeploymentData = payload.spec + normalized_deployment_name = validate_wxo_name(deployment_spec.name) + validate_config_create_input(payload.config) + + if deployment_spec.type != DeploymentType.AGENT: + msg = ( + f"{ErrorPrefix.CREATE.value}" + f"Deployment type '{deployment_spec.type.value}' " + "is not supported by watsonx Orchestrate." + ) + raise DeploymentSupportError(message=msg) + + caller_prefix = (deployment_spec.provider_spec or {}).get(PROVIDER_SPEC_RESOURCE_NAME_PREFIX_KEY) + if not caller_prefix: + msg = ( + f"{ErrorPrefix.CREATE.value} provider_spec must include '{PROVIDER_SPEC_RESOURCE_NAME_PREFIX_KEY}'." + ) + raise InvalidContentError(message=msg) + + clients = await self._get_provider_clients(user_id=user_id, db=db) + + resource_prefix = resolve_resource_name_prefix( + caller_prefix=caller_prefix, + ) + prefixed_deployment_name = f"{resource_prefix}{normalized_deployment_name}" + + tool_names = build_snapshot_tool_names( + snapshots=payload.snapshot, + tool_name_prefix=resource_prefix, + ) + if len(tool_names) != len(set(tool_names)): + msg = ( + f"{ErrorPrefix.CREATE.value} " + "Duplicate snapshot names detected in the request. " + "Each snapshot must have a unique name." + ) + raise DeploymentConflictError(message=msg) + + prefixed_app_id = resolve_create_app_id( + prefixed_deployment_name=prefixed_deployment_name, + config=payload.config, + ) + + try: + created_app_id = await retry_create( + process_config, + clients=clients, + user_id=user_id, + db=db, + deployment_name=prefixed_app_id, + config=payload.config, + ) + + if payload.snapshot and (flow_payloads := payload.snapshot.raw_payloads): + created_tool_ids = await retry_create( + process_raw_flows_with_app_id, + clients=clients, + app_id=prefixed_app_id, + flows=flow_payloads, + tool_name_prefix=resource_prefix, + ) + + derived_spec = deployment_spec.model_copy(deep=True) + + if derived_spec.provider_spec is None: + derived_spec.provider_spec = {} + + derived_spec.provider_spec.update( + { + "name": prefixed_deployment_name, + "display_name": derived_spec.name, + } + ) + + agent_create_response = await retry_create( + self._create_agent_deployment, + clients=clients, + data=derived_spec, + tool_ids=created_tool_ids, + ) + except Exception: + logger.warning( + "wxO create failed; rolling back agent_id=%s, tool_ids=%s, app_id=%s", + agent_create_response.id if agent_create_response else None, + created_tool_ids, + created_app_id, + ) + await rollback_created_resources( + clients=clients, + agent_id=agent_create_response.id if agent_create_response else None, + tool_ids=created_tool_ids, + app_id=created_app_id, + ) + raise + + except (ClientAPIException, HTTPException) as exc: + if isinstance(exc, ClientAPIException): + status_code = exc.response.status_code + error_detail = extract_error_detail(exc.response.text) + else: + status_code = exc.status_code + error_detail = extract_error_detail(str(exc.detail)) + is_conflict = status_code == status.HTTP_409_CONFLICT or "already exists" in error_detail.lower() + if is_conflict: + msg = ( + f"{ErrorPrefix.CREATE.value} " + "One or more resources already exist. " + "Please ensure the names and/or ids of the " + "following resources to be unique: " + "(1) The deployment specification, " + "(2) The deployment configuration, " + "(3) The deployment snapshot. " + f"error details: {error_detail}" + ) + raise DeploymentConflictError(message=msg) from exc + if status_code == status.HTTP_422_UNPROCESSABLE_CONTENT: + msg = ( + f"{ErrorPrefix.CREATE.value} " + "The deployment request entity is unprocessable. " + "Please ensure the request entity is valid and complete. " + f"error details: {error_detail}" + ) + raise InvalidContentError(message=msg) from exc + msg = f"{ErrorPrefix.CREATE.value} error details: {error_detail}" + raise DeploymentError(message=msg, error_code="deployment_error") from exc + except ( + AuthenticationError, + DeploymentConflictError, + InvalidContentError, + InvalidDeploymentOperationError, + InvalidDeploymentTypeError, + DeploymentSupportError, + ): + raise + except Exception as exc: + logger.exception("Unexpected error during wxO deployment creation") + msg = f"{ErrorPrefix.CREATE.value} Please check server logs for details." + raise DeploymentError(message=msg, error_code="deployment_error") from exc + + if agent_create_response is None or derived_spec is None: + msg = f"{ErrorPrefix.CREATE.value} Deployment response was empty." + raise DeploymentError(message=msg, error_code="deployment_error") + + derived_spec.name = deployment_spec.name # restore the original name + + return DeploymentCreateResult( + id=agent_create_response.id, + config_id=created_app_id, + snapshot_ids=created_tool_ids, + **derived_spec.model_dump(exclude_unset=True), + ) + + async def list_types( + self, + *, + user_id: IdLike, # noqa: ARG002 + db: AsyncSession, # noqa: ARG002 + ) -> DeploymentListTypesResult: + """List deployment types supported by the provider.""" + return DeploymentListTypesResult(deployment_types=list(SUPPORTED_ADAPTER_DEPLOYMENT_TYPES)) + + async def list( + self, + *, + user_id: IdLike, + db: AsyncSession, + params: DeploymentListParams | None = None, + ) -> DeploymentListResult: + """List deployments from Watsonx Orchestrate.""" + client_manager = await self._get_provider_clients(user_id=user_id, db=db) + deployments: list = [] + try: + deployment_types: set[DeploymentType] = set() + invalid_deployment_types: set[DeploymentType] = set() + + if params and params.deployment_types: + deployment_types = set(params.deployment_types) + invalid_deployment_types = deployment_types.difference(SUPPORTED_ADAPTER_DEPLOYMENT_TYPES) + + if invalid_deployment_types: + invalid_values = ", ".join([dtype.value for dtype in invalid_deployment_types]) + msg = ( + f"{ErrorPrefix.LIST.value} watsonx Orchestrate has no such deployment type(s): '{invalid_values}'." + ) + raise InvalidDeploymentTypeError(message=msg) + + query_params: dict[str, Any] = {} + + if params and params.provider_params: + query_params = params.provider_params + + if params and params.deployment_ids and "ids" not in query_params: + query_params["ids"] = [str(_id) for _id in params.deployment_ids] + + # if different deployment types + # are distinct resources in wxO + # then we should probably raise an error if + # the ids query parameter is not empty or null + # this is not a problem today, but might be in the future + + raw_agents = await asyncio.to_thread( + client_manager.get_agents_raw, + params=query_params or None, + ) + deployments = [ + get_deployment_metadata( + data=agent, + deployment_type=DeploymentType.AGENT, + provider_data={ + "snapshot_ids": extract_agent_tool_ids(agent), + "environment": derive_agent_environment(agent), + }, + ) + for agent in raw_agents + ] + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST, + log_msg="Unexpected error while listing wxO deployments", + pass_through=(AuthenticationError, AuthorizationError, InvalidDeploymentTypeError), + ) + + return DeploymentListResult( + deployments=deployments, + ) + + async def get( + self, + *, + user_id: IdLike, + deployment_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, + ) -> DeploymentGetResult: + """Get a deployment (agent) from Watsonx Orchestrate.""" + client_manager = await self._get_provider_clients(user_id=user_id, db=db) + try: + agent = await asyncio.to_thread(client_manager.agent.get_draft_by_id, deployment_id) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.GET, + log_msg="Unexpected error fetching wxO deployment", + pass_through=(AuthenticationError, AuthorizationError, DeploymentNotFoundError), + ) + if not agent: + msg = f"Deployment '{deployment_id}' not found." + raise DeploymentNotFoundError(msg) + return get_deployment_detail_metadata( + data=agent, + deployment_type=DeploymentType.AGENT, + ) + + async def update( + self, + *, + user_id: IdLike, + deployment_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + payload: DeploymentUpdate, + db: AsyncSession, + ) -> DeploymentUpdateResult: + """Update deployment metadata and provider-driven tool/config operations.""" + try: + clients = await self._get_provider_clients(user_id=user_id, db=db) + agent_id = _normalize_and_validate_id(str(deployment_id), field_name="deployment_id") + + agent = await asyncio.to_thread(clients.agent.get_draft_by_id, agent_id) + + if not agent: + msg = f"Deployment '{agent_id}' not found." + raise DeploymentNotFoundError(msg) + + # base agent payload to build for final update call + update_payload: dict[str, Any] = build_update_payload_from_spec(payload.spec) + + validate_provider_update_request_sections(payload) + if payload.provider_data is None: + if not update_payload: + msg = "provider_data is required when update operations do not include spec changes." + raise InvalidContentError(message=msg) + await retry_create( + asyncio.to_thread, + clients.agent.update, + agent_id, + update_payload, + ) + return DeploymentUpdateResult(id=deployment_id, snapshot_ids=[]) + + try: + provider_update = self.payload_schemas.deployment_update.parse(payload.provider_data) + except AdapterPayloadValidationError as exc: + first_error = exc.error.errors()[0] if exc.error.errors() else {} + msg = str(first_error.get("msg") or exc) + raise InvalidContentError(message=msg) from None + + provider_plan = build_provider_update_plan( + agent=agent, + provider_update=provider_update, + ) + + added_snapshot_ids: list[str] = await apply_provider_update_plan_with_rollback( + clients=clients, + user_id=user_id, + db=db, + agent_id=agent_id, + agent=agent, + update_payload=update_payload, + plan=provider_plan, + ) + + return DeploymentUpdateResult( + id=deployment_id, + snapshot_ids=added_snapshot_ids, + ) + + except (ClientAPIException, HTTPException) as exc: + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.UPDATE, + log_msg="Unexpected provider error during wxO deployment update", + ) + except ( + AuthenticationError, + AuthorizationError, + DeploymentNotFoundError, + InvalidContentError, + InvalidDeploymentOperationError, + DeploymentConflictError, + ): + raise + except Exception as exc: + logger.exception("Unexpected error during wxO deployment update") + msg = f"{ErrorPrefix.UPDATE.value} Please check server logs for details." + raise DeploymentError(message=msg, error_code="deployment_error") from exc + + async def redeploy( + self, + *, + user_id: IdLike, # noqa: ARG002 + deployment_id: IdLike, # noqa: ARG002 + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, # noqa: ARG002 + ) -> RedeployResult: + """Trigger a deployment redeployment for the agent in draft environment.""" + msg = "Redeployment is not supported by the watsonx Orchestrate adapter." + raise OperationNotSupportedError(message=msg) + + async def duplicate( + self, + *, + user_id: IdLike, # noqa: ARG002 + deployment_id: IdLike, # noqa: ARG002 + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, # noqa: ARG002 + ) -> DeploymentDuplicateResult: + """Duplicate an existing deployment.""" + msg = "Deployment duplication is not supported by the watsonx Orchestrate adapter." + raise OperationNotSupportedError(message=msg) + + async def delete( + self, + *, + user_id: IdLike, + deployment_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, + ) -> DeploymentDeleteResult: + """Delete only the deployment agent (keep tools/configs reusable).""" + logger.info("Deleting wxO deployment deployment_id=%s", deployment_id) + agent_id = _normalize_and_validate_id(str(deployment_id), field_name="deployment_id") + clients = await self._get_provider_clients(user_id=user_id, db=db) + try: + await asyncio.to_thread(clients.agent.delete, agent_id) + except ClientAPIException as e: + status_code = e.response.status_code + if status_code == status.HTTP_404_NOT_FOUND: + msg = f"{ErrorPrefix.DELETE.value} deployment id '{agent_id}' not found." + raise DeploymentNotFoundError(msg) from e + msg = f"{ErrorPrefix.DELETE.value} error details: {extract_error_detail(e.response.text)}" + raise DeploymentError(msg, error_code="deployment_error") from e + except (AuthenticationError, AuthorizationError, DeploymentNotFoundError): + raise + except Exception as exc: + logger.exception("Unexpected error while deleting wxO deployment %s", agent_id) + msg = f"{ErrorPrefix.DELETE.value} Please check server logs for details." + raise DeploymentError(msg, error_code="deployment_error") from exc + + return DeploymentDeleteResult(id=agent_id) + + # TODO: get status normally if its a live agent + # if its draft, use the current 'exists' or raise not found logic + async def get_status( + self, + *, + user_id: IdLike, + deployment_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, + ) -> DeploymentStatusResult: + """Get deployment status from wxO agent metadata. + + Note: wxO does not expose a dedicated health endpoint for draft Agents. Status is + inferred from agent existence and environment metadata -- "connected" + means the agent draft was found, not that it is healthy at runtime. + """ + agent_id = _normalize_and_validate_id(str(deployment_id), field_name="deployment_id") + + clients = await self._get_provider_clients(user_id=user_id, db=db) + + try: + agent = await asyncio.to_thread(clients.agent.get_draft_by_id, agent_id) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.HEALTH, + log_msg="Unexpected error fetching wxO deployment status", + ) + + if not agent or isinstance(agent, str): # the adk returns a string if not found + raise DeploymentNotFoundError(deployment_id=agent_id) + + return DeploymentStatusResult( + id=agent_id, + provider_data={ + "status": "connected", + "environment": derive_agent_environment(agent), + }, + ) + + async def create_execution( + self, + *, + user_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + payload: ExecutionCreate, + db: AsyncSession, + ) -> ExecutionCreateResult: + """Create a provider-agnostic deployment execution.""" + agent_id = _normalize_and_validate_id(str(payload.deployment_id), field_name="deployment_id") + + clients = await self._get_provider_clients(user_id=user_id, db=db) + + provider_data: dict = payload.provider_data or {} + + try: + agent_run_result = await create_agent_run( + clients, + provider_data=provider_data, + deployment_id=agent_id, + ) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.CREATE_EXECUTION, + log_msg="Unexpected error creating wxO deployment execution", + pass_through=(AuthenticationError, AuthorizationError, DeploymentNotFoundError, InvalidContentError), + ) + + return ExecutionCreateResult( + execution_id=agent_run_result.get("run_id"), + deployment_id=agent_id, + provider_result=agent_run_result, + ) + + async def get_execution( + self, + *, + user_id: IdLike, + execution_id: IdLike, + deployment_type: DeploymentType | None = None, # noqa: ARG002 + db: AsyncSession, + ) -> ExecutionStatusResult: + """Get provider-agnostic deployment execution state/output.""" + run_id = _normalize_and_validate_id(str(execution_id), field_name="execution_id") + + clients = await self._get_provider_clients(user_id=user_id, db=db) + + try: + agent_run_result = await get_agent_run( + clients, + run_id=run_id, + ) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.GET_EXECUTION, + log_msg="Unexpected error fetching wxO deployment execution", + pass_through=(AuthenticationError, AuthorizationError, DeploymentNotFoundError, InvalidContentError), + ) + + return ExecutionStatusResult( + execution_id=run_id, + deployment_id=agent_run_result.get("agent_id"), + provider_result=agent_run_result, + ) + + async def list_configs( + self, + *, + user_id: IdLike, + params: ConfigListParams | None = None, + db: AsyncSession, + ) -> ConfigListResult: + """List configs visible to this adapter.""" + agent_id = _require_single_deployment_id(params, resource_label="config") + clients = await self._get_provider_clients(user_id=user_id, db=db) + + try: + agent = await asyncio.to_thread(clients.agent.get_draft_by_id, agent_id) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST_CONFIGS, + log_msg="Unexpected error while listing wxO deployment configs", + ) + + if not agent: + msg = f"Deployment '{agent_id}' not found." + raise DeploymentNotFoundError(msg) + + tool_ids = agent.get("tools", []) if isinstance(agent, dict) else [] + tool_ids = dedupe_list(tool_ids) + + if not tool_ids: + return ConfigListResult( + configs=[], + provider_result={"deployment_id": agent_id, "tool_ids": []}, + ) + + tools: list[dict] + + try: + tools = await asyncio.to_thread(clients.tool.get_drafts_by_ids, tool_ids) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST_CONFIGS, + log_msg="Unexpected error while listing wxO tools for config extraction", + ) + + app_ids: set[str] = set() + for tool in tools or []: + if not isinstance(tool, dict): + continue + connections: dict = tool.get("binding", {}).get("langflow", {}).get("connections", {}) + if not connections: + continue + app_ids.update(connections.keys()) + + return ConfigListResult( + configs=[ConfigListItem(id=app_id, name=app_id) for app_id in app_ids], + provider_result={"deployment_id": agent_id}, + ) + + async def list_snapshots( + self, + *, + user_id: IdLike, + params: SnapshotListParams | None = None, + db: AsyncSession, + ) -> SnapshotListResult: + """List snapshots visible to this adapter.""" + agent_id = _require_single_deployment_id(params, resource_label="snapshot") + clients = await self._get_provider_clients(user_id=user_id, db=db) + + try: + agent = await asyncio.to_thread(clients.agent.get_draft_by_id, agent_id) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST, + log_msg="Unexpected error while listing wxO deployment snapshots", + ) + + if not agent or not isinstance(agent, dict): + msg = f"Deployment '{agent_id}' not found." + raise DeploymentNotFoundError(msg) + + tools: list[dict] = [] + requested_tool_ids = dedupe_list(agent.get("tools", [])) + if requested_tool_ids: + try: + tools = await asyncio.to_thread(clients.tool.get_drafts_by_ids, requested_tool_ids) + except Exception as exc: # noqa: BLE001 + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST, + log_msg="Unexpected error while listing wxO tools for snapshot extraction", + ) + + snapshots = [ + SnapshotItem( + id=tool["id"], + name=tool.get("name") or tool["id"], + ) + for tool in (tools or []) + if tool.get("id") + ] + if not snapshots and requested_tool_ids: + snapshots = [SnapshotItem(id=tool_id, name=tool_id) for tool_id in requested_tool_ids] + + return SnapshotListResult( + snapshots=snapshots, + provider_result={"deployment_id": agent_id}, + ) + + async def _create_agent_deployment( + self, + *, + clients: WxOClient, + tool_ids: Sequence[str], + data: BaseDeploymentData, + ) -> AgentUpsertResponse: + """Create an agent deployment.""" + payload = build_agent_payload( + data=data, + tool_ids=tool_ids, + ) + return await asyncio.to_thread(clients.agent.create, payload) + + async def teardown(self) -> None: + """Teardown provider-specific resources.""" diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/types.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/types.py new file mode 100644 index 0000000000..eaed3497de --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/types.py @@ -0,0 +1,81 @@ +"""Dataclasses for the Watsonx Orchestrate adapter. + +`WxOClient` eagerly creates SDK clients (`AgentClient`, `ToolClient`, +`ConnectionsClient`, and `BaseWXOClient`) at construction time to +guarantee thread safety when accessed from ``asyncio.to_thread`` workers. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +from ibm_watsonx_orchestrate_clients.agents.agent_client import AgentClient +from ibm_watsonx_orchestrate_clients.common.base_client import BaseWXOClient +from ibm_watsonx_orchestrate_clients.connections.connections_client import ConnectionsClient +from ibm_watsonx_orchestrate_clients.tools.tool_client import ToolClient + +if TYPE_CHECKING: + from ibm_cloud_sdk_core.authenticators import Authenticator + + +@dataclass(frozen=True, slots=True) +class WxOClient: + """Provider client facade with eager SDK client initialization. + + All sub-clients are constructed in ``__post_init__`` from ``instance_url`` + and ``authenticator`` so that they are guaranteed to share the same + URL and authentication context. The dataclass is frozen to prevent + post-construction mutation of credentials. + """ + + instance_url: str + authenticator: Authenticator + base: BaseWXOClient = field(init=False, repr=False) + tool: ToolClient = field(init=False, repr=False) + connections: ConnectionsClient = field(init=False, repr=False) + agent: AgentClient = field(init=False, repr=False) + + def __post_init__(self) -> None: + url = self.instance_url.rstrip("/") + if not url: + msg = "instance_url must be a non-empty string." + raise ValueError(msg) + # Use object.__setattr__ because the dataclass is frozen. + object.__setattr__(self, "instance_url", url) + object.__setattr__(self, "base", BaseWXOClient(base_url=url, authenticator=self.authenticator)) + object.__setattr__(self, "tool", ToolClient(base_url=url, authenticator=self.authenticator)) + object.__setattr__(self, "connections", ConnectionsClient(base_url=url, authenticator=self.authenticator)) + object.__setattr__(self, "agent", AgentClient(base_url=url, authenticator=self.authenticator)) + + # -- SDK private-method wrappers ------------------------------------------ + # Centralise access to SDK-internal _get/_post so breakage from SDK + # upgrades is confined to this single file. + + def get_agents_raw(self, params: dict[str, Any] | None = None) -> Any: + return self.base._get("/agents", params=params) # noqa: SLF001 + + def post_run(self, *, query_suffix: str = "", data: dict[str, Any]) -> Any: + return self.base._post(f"/runs{query_suffix}", data=data) # noqa: SLF001 + + def get_run(self, run_id: str) -> Any: + return self.base._get(f"/runs/{run_id}") # noqa: SLF001 + + def upload_tool_artifact(self, tool_id: str, *, files: dict[str, Any]) -> Any: + return self.base._post(f"/tools/{tool_id}/upload", files=files) # noqa: SLF001 + + +@dataclass(frozen=True, slots=True) +class WxOCredentials: + instance_url: str + authenticator: Authenticator = field(repr=False) + + def __post_init__(self) -> None: + if not self.instance_url or not self.instance_url.strip(): + msg = "instance_url must be a non-empty string." + raise ValueError(msg) + + def __repr__(self) -> str: + return ( + f"WxOCredentials(instance_url={self.instance_url!r}, authenticator={self.authenticator.__class__.__name__})" + ) diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/update_helpers.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/update_helpers.py new file mode 100644 index 0000000000..5e23ac83cf --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/update_helpers.py @@ -0,0 +1,495 @@ +"""Helpers used to flatten wxO deployment update control flow.""" + +from __future__ import annotations + +import asyncio +import copy +import logging +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +from fastapi import HTTPException, status +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from lfx.services.adapters.deployment.exceptions import ( + DeploymentConflictError, + InvalidContentError, + InvalidDeploymentOperationError, +) + +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix +from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import create_config, validate_connection +from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import ( + delete_config_if_exists, + retry_create, + retry_rollback, + retry_update, + rollback_update_resources, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + FlowToolBindingSpec, + ToolUploadBatchError, + create_and_upload_wxo_flow_tools_with_bindings, + ensure_langflow_connections_binding, + to_writable_tool_payload, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.payloads import ( + WatsonxBindOperation, + WatsonxConnectionRawPayload, + WatsonxDeploymentUpdatePayload, + WatsonxRemoveToolOperation, + WatsonxUnbindOperation, +) +from langflow.services.adapters.deployment.watsonx_orchestrate.utils import ( + dedupe_list, + extract_agent_tool_ids, + extract_error_detail, + validate_wxo_name, +) + +if TYPE_CHECKING: + from collections.abc import Iterator + + from lfx.services.adapters.deployment.schema import ( + BaseDeploymentDataUpdate, + BaseFlowArtifact, + DeploymentUpdate, + IdLike, + ) + from sqlalchemy.ext.asyncio import AsyncSession + + from langflow.services.adapters.deployment.watsonx_orchestrate.types import WxOClient + +logger = logging.getLogger(__name__) + + +class OrderedUniqueStrs: + """Ordered, de-duplicating string collection used for deterministic update plans. + + We intentionally avoid a plain list to avoid repeated O(n) membership checks. + Using a dictionary keeps membership/update checks O(1) while preserving insertion order. + + Deterministic order is primarily for rollback correctness + Stable ordering ensures we undo the exact resources we created + in the correct (reverse) sequence. + """ + + def __init__(self, items: dict[str, None] | None = None) -> None: + self._items: dict[str, None] = items or {} + + @classmethod + def from_values(cls, values: list[str]) -> OrderedUniqueStrs: + ordered = cls() + ordered.extend(values) + return ordered + + def __iter__(self) -> Iterator[str]: + return iter(self._items) + + def to_list(self) -> list[str]: + # Snapshot keys only at list-boundary call sites. + return list(self._items) + + def add(self, value: str) -> None: + self._items.setdefault(value, None) + + def extend(self, values: list[str]) -> None: + for value in values: + self.add(value) + + def discard(self, value: str) -> None: + self._items.pop(value, None) + + +class ToolConnectionOps: + def __init__( + self, + *, + bind: OrderedUniqueStrs | None = None, + unbind: OrderedUniqueStrs | None = None, + ) -> None: + self.bind = bind or OrderedUniqueStrs() + self.unbind = unbind or OrderedUniqueStrs() + + +def _get_or_create_tool_connection_ops( + deltas: dict[str, ToolConnectionOps], + *, + tool_id: str, +) -> ToolConnectionOps: + return deltas.setdefault(tool_id, ToolConnectionOps()) + + +@dataclass(slots=True) +class RawConnectionCreatePlan: + operation_app_id: str + provider_app_id: str + payload: WatsonxConnectionRawPayload + + +@dataclass(slots=True) +class RawToolCreatePlan: + raw_name: str + payload: BaseFlowArtifact + app_ids: list[str] + + +@dataclass(slots=True) +class ProviderUpdatePlan: + resource_prefix: str + existing_app_ids: list[str] + raw_connections_to_create: list[RawConnectionCreatePlan] + existing_tool_deltas: dict[str, ToolConnectionOps] + raw_tools_to_create: list[RawToolCreatePlan] + final_existing_tool_ids: list[str] + bind_existing_tool_ids: list[str] + + +def validate_provider_update_request_sections(payload: DeploymentUpdate) -> None: + """Reject legacy top-level update sections in watsonx clean-break mode.""" + if payload.snapshot is not None or payload.config is not None: + msg = ( + "Top-level 'snapshot' and 'config' update sections are no longer supported for " + "watsonx Orchestrate deployment updates. Use provider_data.operations instead." + ) + raise InvalidDeploymentOperationError(message=msg) + + +def build_provider_update_plan( + *, + agent: dict[str, Any], + provider_update: WatsonxDeploymentUpdatePayload, +) -> ProviderUpdatePlan: + """Build a deterministic CPU-only plan for provider_data update operations.""" + resource_prefix = (provider_update.resource_name_prefix or "").strip() + agent_tool_ids = extract_agent_tool_ids(agent) + final_existing_tool_ids = OrderedUniqueStrs.from_values(agent_tool_ids) + + # Per existing tool_id, track app_ids to bind/unbind during this update. + existing_tool_deltas: dict[str, ToolConnectionOps] = {} + # Existing tool_ids explicitly referenced by bind operations (for added snapshot reporting). + bind_existing_tool_ids = OrderedUniqueStrs() + # Per raw tool name, collect app_ids that should be bound when the raw tool is created. + raw_tool_app_ids: dict[str, OrderedUniqueStrs] = {} + + for operation in provider_update.operations: + if isinstance(operation, WatsonxBindOperation): + if operation.tool.reference_id is not None: + tool_id = operation.tool.reference_id + bind_existing_tool_ids.add(tool_id) + final_existing_tool_ids.add(tool_id) + delta = _get_or_create_tool_connection_ops(existing_tool_deltas, tool_id=tool_id) + delta.bind.extend(operation.app_ids) + continue + + raw_name = str(operation.tool.name_of_raw) + raw_apps = raw_tool_app_ids.setdefault(raw_name, OrderedUniqueStrs()) + raw_apps.extend(operation.app_ids) + continue + + if isinstance(operation, WatsonxUnbindOperation): + tool_id = operation.tool_id + delta = _get_or_create_tool_connection_ops(existing_tool_deltas, tool_id=tool_id) + delta.unbind.extend(operation.app_ids) + continue + + if isinstance(operation, WatsonxRemoveToolOperation): + final_existing_tool_ids.discard(operation.tool_id) + continue + + raw_connections_to_create = [ + RawConnectionCreatePlan( + operation_app_id=raw_payload.app_id, + provider_app_id=f"{resource_prefix}{raw_payload.app_id}", + payload=raw_payload, + ) + for raw_payload in (provider_update.connections.raw_payloads or []) + ] + + raw_tool_pool = {raw_payload.name: raw_payload for raw_payload in (provider_update.tools.raw_payloads or [])} + raw_tools_to_create = [ + RawToolCreatePlan(raw_name=raw_name, payload=raw_tool_pool[raw_name], app_ids=app_ids.to_list()) + for raw_name, app_ids in raw_tool_app_ids.items() + ] + + return ProviderUpdatePlan( + resource_prefix=resource_prefix, + existing_app_ids=list(provider_update.connections.existing_app_ids or []), + raw_connections_to_create=raw_connections_to_create, + existing_tool_deltas=existing_tool_deltas, + raw_tools_to_create=raw_tools_to_create, + final_existing_tool_ids=final_existing_tool_ids.to_list(), + bind_existing_tool_ids=bind_existing_tool_ids.to_list(), + ) + + +async def _create_update_connection_with_conflict_mapping( + *, + clients: WxOClient, + app_id: str, + payload: WatsonxConnectionRawPayload, + user_id: IdLike, + db: AsyncSession, +) -> str: + from lfx.services.adapters.deployment.schema import DeploymentConfig + + config_payload = DeploymentConfig( + name=app_id, + description=None, + environment_variables=payload.environment_variables, + provider_config=payload.provider_config, + ) + try: + return await retry_create( + create_config, + clients=clients, + config=config_payload, + user_id=user_id, + db=db, + ) + except (ClientAPIException, HTTPException) as exc: + if isinstance(exc, ClientAPIException): + status_code = exc.response.status_code + error_detail = str(extract_error_detail(exc.response.text)) + else: + status_code = exc.status_code + error_detail = str(extract_error_detail(str(exc.detail))) + is_conflict = status_code == status.HTTP_409_CONFLICT or "already exists" in error_detail.lower() + if is_conflict: + msg = f"{ErrorPrefix.UPDATE.value} error details: {error_detail}" + raise DeploymentConflictError(message=msg) from exc + raise + + +async def _update_existing_tool_connection_deltas( + *, + clients: WxOClient, + existing_tool_deltas: dict[str, ToolConnectionOps], + resolved_connections: dict[str, str], + original_tools: dict[str, dict[str, Any]], +) -> None: + if not existing_tool_deltas: + return + + tool_ids = list(existing_tool_deltas.keys()) + tools = await asyncio.to_thread(clients.tool.get_drafts_by_ids, tool_ids) + tool_by_id = {str(tool.get("id")): tool for tool in tools if isinstance(tool, dict) and tool.get("id")} + missing_tool_ids = [tool_id for tool_id in tool_ids if tool_id not in tool_by_id] + if missing_tool_ids: + missing_ids = ", ".join(missing_tool_ids) + msg = f"Snapshot tool(s) not found: {missing_ids}" + raise InvalidContentError(message=msg) + + tool_updates: list[tuple[str, dict[str, Any]]] = [] + for tool_id in tool_ids: + delta = existing_tool_deltas[tool_id] + original_tool = to_writable_tool_payload(tool_by_id[tool_id]) + original_tools[tool_id] = original_tool + writable_tool = copy.deepcopy(original_tool) + connections = ensure_langflow_connections_binding(writable_tool) + + for app_id in delta.unbind: + connections.pop(app_id, None) + for app_id in delta.bind: + connection_id = resolved_connections.get(app_id) + if not connection_id: + msg = f"No resolved connection id available for app_id '{app_id}'." + raise InvalidContentError(message=msg) + connections[app_id] = connection_id + tool_updates.append((tool_id, writable_tool)) + + await asyncio.gather( + *( + retry_update(asyncio.to_thread, clients.tool.update, tool_id, writable_tool) + for tool_id, writable_tool in tool_updates + ) + ) + + +async def _rollback_created_app_ids( + *, + clients: WxOClient, + created_app_ids: list[str], +) -> None: + for app_id in reversed(created_app_ids): + try: + await retry_rollback(delete_config_if_exists, clients, app_id=app_id) + except Exception: + logger.exception("Rollback failed for created app_id=%s — resource may be orphaned", app_id) + + +def _build_agent_rollback_payload(*, agent: dict[str, Any], final_update_payload: dict[str, Any]) -> dict[str, Any]: + rollback_payload: dict[str, Any] = {} + if "tools" in final_update_payload: + rollback_payload["tools"] = extract_agent_tool_ids(agent) + for update_field in ("name", "display_name", "description"): + if update_field in final_update_payload and update_field in agent: + rollback_payload[update_field] = agent[update_field] + return rollback_payload + + +async def _rollback_agent_update( + *, + clients: WxOClient, + agent_id: str, + rollback_agent_payload: dict[str, Any], +) -> None: + if not rollback_agent_payload: + return + try: + await retry_rollback(asyncio.to_thread, clients.agent.update, agent_id, rollback_agent_payload) + except Exception: + logger.exception("Rollback failed for agent_id=%s — resource may be orphaned", agent_id) + + +async def apply_provider_update_plan_with_rollback( + *, + clients: WxOClient, + user_id: IdLike, + db: AsyncSession, + agent_id: str, + agent: dict[str, Any], + update_payload: dict[str, Any], + plan: ProviderUpdatePlan, +) -> list[str]: + """Apply provider_data update operations with rollback protection.""" + # Rollback journals: + # - created_tool_ids: provider tool ids created during this update. + # - created_app_ids: provider app ids created during this update. + # - original_tools: writable pre-update payloads for mutated existing tools. + created_tool_ids: list[str] = [] + created_app_ids: list[str] = [] + original_tools: dict[str, dict[str, Any]] = {} + + # Working state: + # - resolved_connections: app_id -> connection_id map used for bind/update calls. + # - added_snapshot_ids: snapshot/tool ids to return in update result. + # - final_update_payload: outbound agent patch payload (spec + tools). + # - rollback_agent_payload: best-effort restore payload for agent update rollback. + resolved_connections: dict[str, str] = {} + added_snapshot_ids: list[str] = [] + final_update_payload = dict(update_payload) + rollback_agent_payload: dict[str, Any] = {} + + try: + if plan.existing_app_ids: + existing_connections = await asyncio.gather( + *( + retry_create(validate_connection, clients.connections, app_id=app_id) + for app_id in plan.existing_app_ids + ) + ) + for app_id, connection in zip(plan.existing_app_ids, existing_connections, strict=True): + resolved_connections[app_id] = connection.connection_id + + if plan.raw_connections_to_create: + created_connections = await asyncio.gather( + *( + _create_update_connection_with_conflict_mapping( + clients=clients, + app_id=create_plan.provider_app_id, + payload=create_plan.payload, + user_id=user_id, + db=db, + ) + for create_plan in plan.raw_connections_to_create + ) + ) + created_app_ids.extend(created_connections) + validated_created_connections = await asyncio.gather( + *( + retry_create( + validate_connection, + clients.connections, + app_id=create_plan.provider_app_id, + ) + for create_plan in plan.raw_connections_to_create + ) + ) + for create_plan, connection in zip( + plan.raw_connections_to_create, validated_created_connections, strict=True + ): + resolved_connections[create_plan.operation_app_id] = connection.connection_id + + if plan.raw_tools_to_create: + tool_bindings = [ + FlowToolBindingSpec( + flow_payload=raw_plan.payload, + connections={app_id: resolved_connections[app_id] for app_id in raw_plan.app_ids}, + ) + for raw_plan in plan.raw_tools_to_create + ] + try: + raw_create_results = await create_and_upload_wxo_flow_tools_with_bindings( + clients=clients, + tool_bindings=tool_bindings, + tool_name_prefix=plan.resource_prefix, + ) + except ToolUploadBatchError as exc: + created_tool_ids.extend(exc.created_tool_ids) + added_snapshot_ids.extend(exc.created_tool_ids) + for i, err in enumerate(exc.errors): + logger.exception("Tool upload batch error [%d/%d]: %s", i + 1, len(exc.errors), err) + raise exc.errors[0] from exc + for raw_plan, created_tool_id in zip(plan.raw_tools_to_create, raw_create_results, strict=True): + tool_id = str(created_tool_id).strip() + if not tool_id: + msg = f"Failed to create tool for raw payload '{raw_plan.raw_name}'." + raise InvalidContentError(message=msg) + created_tool_ids.append(tool_id) + added_snapshot_ids.append(tool_id) + + if plan.existing_tool_deltas: + await _update_existing_tool_connection_deltas( + clients=clients, + existing_tool_deltas=plan.existing_tool_deltas, + resolved_connections=resolved_connections, + original_tools=original_tools, + ) + + added_snapshot_ids.extend(plan.bind_existing_tool_ids) + final_tools = dedupe_list([*plan.final_existing_tool_ids, *created_tool_ids]) + final_update_payload["tools"] = final_tools + rollback_agent_payload = _build_agent_rollback_payload( + agent=agent, + final_update_payload=final_update_payload, + ) + if final_update_payload: + await retry_update(asyncio.to_thread, clients.agent.update, agent_id, final_update_payload) + except Exception: + await _rollback_agent_update( + clients=clients, + agent_id=agent_id, + rollback_agent_payload=rollback_agent_payload, + ) + await rollback_update_resources( + clients=clients, + created_tool_ids=created_tool_ids, + created_app_id=None, + original_tools=original_tools, + ) + await _rollback_created_app_ids( + clients=clients, + created_app_ids=created_app_ids, + ) + raise + + return dedupe_list(added_snapshot_ids) + + +def build_update_payload_from_spec(spec: BaseDeploymentDataUpdate | None) -> dict[str, Any]: + """Build agent update payload from deployment spec updates.""" + update_payload: dict[str, Any] = {} + if not spec: + return update_payload + + spec_updates = spec.model_dump(exclude_unset=True) + if "name" in spec_updates: + update_payload.update( + { + "name": validate_wxo_name(spec_updates["name"]), + "display_name": spec_updates["name"], + } + ) + if "description" in spec_updates: + update_payload["description"] = spec_updates["description"] + return update_payload diff --git a/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/utils.py b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/utils.py new file mode 100644 index 0000000000..db7eb1a3c0 --- /dev/null +++ b/src/backend/base/langflow/services/adapters/deployment/watsonx_orchestrate/utils.py @@ -0,0 +1,207 @@ +"""Name validation, error helpers, and misc utilities for the Watsonx Orchestrate adapter.""" + +from __future__ import annotations + +import json +import logging +from typing import TYPE_CHECKING, Any, NoReturn + +from fastapi import HTTPException +from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException +from lfx.services.adapters.deployment.exceptions import ( + DeploymentError, + DeploymentServiceError, + InvalidContentError, + OperationNotSupportedError, + raise_for_status_and_detail, +) +from lfx.services.adapters.deployment.schema import _normalize_and_validate_id + +from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ( + DEFAULT_WXO_AGENT_LLM, + WXO_SANITIZE_RE, + WXO_TRANSLATE, + ErrorPrefix, +) + +if TYPE_CHECKING: + from collections.abc import Sequence + + from lfx.services.adapters.deployment.schema import ( + BaseDeploymentData, + ConfigListParams, + SnapshotListParams, + ) + +logger = logging.getLogger(__name__) + + +def normalize_wxo_name(s: str) -> str: + return WXO_SANITIZE_RE.sub("", s.translate(WXO_TRANSLATE)) + + +def validate_wxo_name(name: str) -> str: + """Normalize and validate a wxO resource name.""" + normalized_name = normalize_wxo_name(str(name)) + if not normalized_name: + msg = "Deployment name must include at least one alphanumeric character." + raise InvalidContentError(message=msg) + if not normalized_name[0].isalpha(): + msg = "Deployment name must start with a letter." + raise InvalidContentError(message=msg) + return normalized_name + + +def resolve_resource_name_prefix( + *, + caller_prefix: str, +) -> str: + """Validate and return the caller-supplied resource name prefix for WxO resource creation.""" + if not isinstance(caller_prefix, str) or not caller_prefix.strip(): + msg = "resource_name_prefix must be a non-empty string." + raise InvalidContentError(message=msg) + validated = normalize_wxo_name(caller_prefix) + if not validated: + msg = "resource_name_prefix must contain at least one alphanumeric character." + raise InvalidContentError(message=msg) + if not validated[0].isalpha(): + msg = "resource_name_prefix must start with a letter." + raise InvalidContentError(message=msg) + return validated + + +def require_tool_id(tool_response: dict[str, Any]) -> str: + tool_id = tool_response.get("id") + if not tool_id: + msg = "wxO did not return a tool id for snapshot creation." + raise InvalidContentError(message=msg) + return tool_id + + +def dedupe_list(items: list[str]) -> list[str]: + return list(dict.fromkeys(items)) + + +def normalize_and_dedupe_ids(values: list[Any] | None, *, field_name: str) -> list[str]: + """Normalize id values to non-empty strings and dedupe while preserving order.""" + if not values: + return [] + return dedupe_list([_normalize_and_validate_id(str(value), field_name=field_name) for value in values]) + + +def _require_single_deployment_id( + params: ConfigListParams | SnapshotListParams | None, + *, + resource_label: str, +) -> str: + deployment_ids = params.deployment_ids if params else None + if not deployment_ids: + msg = ( + f"watsonx Orchestrate {resource_label} listing requires exactly one " + "deployment_id. Global listing is not supported by this adapter." + ) + raise OperationNotSupportedError(message=msg) + if len(deployment_ids) != 1: + msg = ( + f"watsonx Orchestrate {resource_label} listing currently supports " + "exactly one deployment_id and only deployment-scoped listing." + ) + raise InvalidContentError(message=msg) + return _normalize_and_validate_id(str(deployment_ids[0]), field_name="deployment_id") + + +def extract_error_detail(response_text: str) -> str: + """Extract a human-readable error detail from a ClientAPIException response. + + The response body may contain a ``detail`` value that is a string, a dict + with a ``msg`` key, or a list of such dicts. This helper normalises all + three shapes into a single value suitable for inclusion in an error message. + """ + fallback = response_text or "" + try: + payload = json.loads(response_text) + except (TypeError, ValueError, json.JSONDecodeError): + return fallback + if not isinstance(payload, dict): + return fallback + + detail = payload.get("detail") + if detail in (None, "", [], {}): + for field in ("message", "details", "error"): + detail = payload.get(field) + if detail not in (None, "", [], {}): + break + else: + return fallback + + if isinstance(detail, list): + detail = detail[0] if detail else None + if isinstance(detail, dict): + detail = detail.get("msg") or detail + + return str(detail) if detail not in (None, "", [], {}) else fallback + + +def _resolve_exc_detail(exc: ClientAPIException | HTTPException) -> str: + if isinstance(exc, ClientAPIException): + raw_text = getattr(exc.response, "text", "") + return extract_error_detail(raw_text) + return str(extract_error_detail(str(exc.detail))) + + +def _resolve_exc_status_code(exc: ClientAPIException | HTTPException) -> int | None: + if isinstance(exc, ClientAPIException): + return int(getattr(exc.response, "status_code", 0) or 0) or None + return int(exc.status_code) + + +def raise_as_deployment_error( + exc: Exception, + *, + error_prefix: ErrorPrefix, + log_msg: str, + pass_through: tuple[type[DeploymentServiceError], ...] = (), +) -> NoReturn: + if isinstance(exc, pass_through): + raise exc + if isinstance(exc, DeploymentServiceError): + logger.exception(log_msg) + msg = f"{error_prefix.value} Please check server logs for details." + raise DeploymentError(message=msg, error_code="deployment_error") from exc + if isinstance(exc, (ClientAPIException, HTTPException)): + status_code = _resolve_exc_status_code(exc) + detail = _resolve_exc_detail(exc) + raise_for_status_and_detail( + status_code=status_code, + detail=detail, + message_prefix=error_prefix.value, + ) + logger.exception(log_msg) + msg = f"{error_prefix.value} Please check server logs for details." + raise DeploymentError(message=msg, error_code="deployment_error") from exc + + +def build_agent_payload( + *, + data: BaseDeploymentData, + tool_ids: Sequence[str], +) -> dict[str, Any]: + if data.provider_spec is None: + msg = "Deployment data must include provider_spec with a non-empty name and display_name." + raise InvalidContentError(message=msg) + return { + "name": data.provider_spec["name"], + "display_name": data.provider_spec["display_name"], + "description": str(data.description or "").strip() or f"Langflow deployment {data.name}", + "tools": list(tool_ids), + "style": "default", + # TODO: make configurable; the llm field is required by the wxO api + # but retrieving available llms requires an extra api request. + "llm": DEFAULT_WXO_AGENT_LLM, + } + + +def extract_agent_tool_ids(agent: dict[str, Any]) -> list[str]: + # Shape source: + # - SDK/API agent payload uses "tools" as list[str] in this adapter flow. + return [str(tool_id) for tool_id in agent.get("tools", []) if tool_id] diff --git a/src/backend/base/langflow/services/utils.py b/src/backend/base/langflow/services/utils.py index f16895a463..1b80e61644 100644 --- a/src/backend/base/langflow/services/utils.py +++ b/src/backend/base/langflow/services/utils.py @@ -267,6 +267,27 @@ def register_all_service_factories() -> None: service_manager.set_factory_registered() +def register_builtin_adapters() -> None: + """Import built-in adapter modules so ``@register_adapter`` decorators fire. + + Mirrors ``register_all_service_factories()`` for the adapter registry system. + Each import triggers the ``@register_adapter`` decorator at module scope, + registering the adapter class on the AdapterRegistry singleton. + + TODO: Watsonx risks are documented here because registration is runtime-optional: + missing ``ibm_*`` modules should skip adapter registration, but broad + ``ModuleNotFoundError`` handling can also hide internal import regressions. + Future deployment API routing must treat "provider exists but adapter is not + registered in this runtime" as an explicit, deterministic error path. + Keep direct adapter imports limited to guarded paths and maintain CI + coverage that confirms Watsonx tests run (not skip) in eligible environments. + """ + try: + import langflow.services.adapters.deployment.watsonx_orchestrate # noqa: F401 + except ModuleNotFoundError as exc: + logger.info("Skipping Watsonx Orchestrate adapter registration: %s", exc) + + async def initialize_services(*, fix_migration: bool = False) -> None: """Initialize all the services needed.""" from langflow.helpers.windows_postgres_helper import configure_windows_postgres_event_loop @@ -275,6 +296,7 @@ async def initialize_services(*, fix_migration: bool = False) -> None: # Register all service factories first register_all_service_factories() + register_builtin_adapters() cache_service = get_service(ServiceType.CACHE_SERVICE, default=CacheServiceFactory()) # Test external cache connection diff --git a/src/backend/base/pyproject.toml b/src/backend/base/pyproject.toml index c8ea735777..e7b07c5379 100644 --- a/src/backend/base/pyproject.toml +++ b/src/backend/base/pyproject.toml @@ -163,7 +163,16 @@ ignore_missing_imports = true requires = ["hatchling"] build-backend = "hatchling.build" +# TODO(WXO): Remove testpypi index and uv.sources once the ibm-watsonx-orchestrate +# packages publish their official release to PyPI (expected late March 2026). +[[tool.uv.index]] +name = "testpypi" +url = "https://test.pypi.org/simple" +explicit = true +[tool.uv.sources] +ibm-watsonx-orchestrate-core = [{ index = "testpypi" }] +ibm-watsonx-orchestrate-clients = [{ index = "testpypi" }] [project.urls] Repository = "https://github.com/langflow-ai/langflow" @@ -353,6 +362,16 @@ astrapy = ["astrapy>=2.1.0,<3.0.0"] # Windows-specific gassist = ["gassist>=0.0.1; sys_platform == 'win32'"] +# SDKs for IBM watsonx Orchestrate deployment adapter. +# TODO(WXO): Pin to stable versions once official PyPI releases land (late March 2026). +ibm-watsonx = [ + "ibm-cloud-sdk-core~=3.24.4", + "ibm-watsonx-orchestrate-clients==2.3.2.dev5117; python_version >= '3.11'", + "ibm-watsonx-orchestrate-core==2.3.2.dev5117; python_version >= '3.11'", + "rich", + "PyYAML" +] + # Complete installation complete = [ "langflow-base[couchbase]", @@ -466,6 +485,8 @@ complete = [ "langflow-base[astrapy]", # Windows-specific "langflow-base[gassist]", + # wxO deployment adapter + "langflow-base[ibm-watsonx]", ] all = [ diff --git a/src/backend/tests/unit/services/deployment/__init__.py b/src/backend/tests/unit/services/deployment/__init__.py new file mode 100644 index 0000000000..5f507ad274 --- /dev/null +++ b/src/backend/tests/unit/services/deployment/__init__.py @@ -0,0 +1 @@ +# This file marks the deployment tests directory as a package for linting. diff --git a/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate.py b/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate.py new file mode 100644 index 0000000000..315547a7f2 --- /dev/null +++ b/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate.py @@ -0,0 +1,3407 @@ +from __future__ import annotations + +import importlib +import io +import zipfile +from types import SimpleNamespace +from uuid import UUID + +import pytest +from fastapi import HTTPException, status +from lfx.services.adapters.deployment.exceptions import ( + AuthorizationError, + CredentialResolutionError, + DeploymentConflictError, + DeploymentError, + DeploymentNotFoundError, + DeploymentSupportError, + InvalidContentError, + InvalidDeploymentOperationError, + OperationNotSupportedError, +) +from lfx.services.adapters.deployment.schema import ( + BaseDeploymentData, + BaseDeploymentDataUpdate, + BaseFlowArtifact, + ConfigItem, + ConfigListParams, + DeploymentConfig, + DeploymentCreate, + DeploymentListParams, + DeploymentType, + DeploymentUpdate, + EnvVarValueSpec, + ExecutionCreate, + SnapshotItems, + SnapshotListParams, +) + +try: + import langflow.services.adapters.deployment.watsonx_orchestrate # noqa: F401 +except ModuleNotFoundError: + pytest.skip( + "Skipping Watsonx deployment tests: optional IBM SDK dependencies not available.", + allow_module_level=True, + ) + +tools_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.core.tools") +service_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.service") +update_helpers_module = importlib.import_module( + "langflow.services.adapters.deployment.watsonx_orchestrate.update_helpers" +) +payloads_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.payloads") +client_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.client") +types_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.types") +deployment_context_module = importlib.import_module("langflow.services.adapters.deployment.context") +WatsonxOrchestrateDeploymentService = importlib.import_module( + "langflow.services.adapters.deployment.watsonx_orchestrate" +).WatsonxOrchestrateDeploymentService +WxOCredentials = importlib.import_module( + "langflow.services.adapters.deployment.watsonx_orchestrate.types" +).WxOCredentials + + +class DummySettingsService: + def __init__(self): + self.settings = SimpleNamespace() + + +class FakeAgentClient: + def __init__( + self, + deployment: dict, + listed_agents: list[dict] | None = None, + get_payloads: dict[str, dict | list[dict]] | None = None, + ): + self._deployment = deployment + self._listed_agents = listed_agents or [] + self._get_payloads = get_payloads or {} + self.update_calls: list[tuple[str, dict]] = [] + self.post_calls: list[tuple[str, dict]] = [] + self.create_calls: list[dict] = [] + self.delete_calls: list[str] = [] + + def get_draft_by_id(self, deployment_id: str): # noqa: ARG002 + return self._deployment + + def get_drafts_by_ids(self, deployment_ids: list[str]): + return [agent for agent in self._listed_agents if str(agent.get("id") or "").strip() in set(deployment_ids)] + + def get_drafts_by_names(self, agent_names: list[str]): + return [agent for agent in self._listed_agents if str(agent.get("name") or "").strip() in set(agent_names)] + + def get_draft_by_name(self, agent_name: str): + return [agent for agent in self._listed_agents if agent.get("name") == agent_name] + + def get(self): + return self._listed_agents + + def update(self, deployment_id: str, payload: dict): + self.update_calls.append((deployment_id, payload)) + + def create(self, payload: dict): + self.create_calls.append(payload) + return SimpleNamespace(id="dep-created") + + def delete(self, deployment_id: str): + self.delete_calls.append(deployment_id) + + def _post(self, path: str, data: dict): + self.post_calls.append((path, data)) + return { + "thread_id": "thread-1", + "run_id": "run-1", + "task_id": "task-1", + "message_id": "message-1", + } + + def _get(self, path: str, params: dict | None = None): + if path == "/agents": + ids = params.get("ids", []) if isinstance(params, dict) else [] + names = params.get("names", []) if isinstance(params, dict) else [] + id_set = {str(item) for item in ids} + name_set = {str(item) for item in names} + if id_set or name_set: + return [ + agent + for agent in self._listed_agents + if str(agent.get("id") or "").strip() in id_set or str(agent.get("name") or "").strip() in name_set + ] + return self._listed_agents + return self._get_payloads.get(path, {}) + + +class FakeToolClient: + def __init__(self, tools: list[dict], existing_names: set[str] | None = None): + self._tools_by_id = {str(tool.get("id")): dict(tool) for tool in tools if tool.get("id")} + self._existing_names = existing_names or set() + self.delete_calls: list[str] = [] + self.update_calls: list[tuple[str, dict]] = [] + self.create_calls: list[dict] = [] + + def get_drafts_by_ids(self, tool_ids: list[str]): + return [dict(self._tools_by_id[tool_id]) for tool_id in tool_ids if tool_id in self._tools_by_id] + + def get_draft_by_name(self, tool_name: str): + if tool_name in self._existing_names: + return [{"name": tool_name}] + return [] + + def update(self, tool_id: str, payload: dict): + self.update_calls.append((tool_id, payload)) + current = self._tools_by_id.get(tool_id, {"id": tool_id}) + merged = dict(current) + merged.update(payload) + self._tools_by_id[tool_id] = merged + + def create(self, payload: dict): + self.create_calls.append(payload) + tool_id = f"created-tool-{len(self.create_calls)}" + self._tools_by_id[tool_id] = {"id": tool_id, **payload} + return {"id": tool_id} + + def delete(self, tool_id: str): + self.delete_calls.append(tool_id) + + +class FakeConnectionsClient: + def __init__(self, existing_app_id: str | None = None): + self._connections_by_app_id: dict[str, str] = {} + if existing_app_id: + self._connections_by_app_id[existing_app_id] = "conn-1" + self.delete_calls: list[str] = [] + self.delete_credentials_calls: list[tuple[str, object, bool]] = [] + self._list_entries: list[dict] = [] + self.create_calls: list[dict] = [] + self.create_config_calls: list[tuple[str, dict]] = [] + self.create_credentials_calls: list[tuple[str, object, bool, dict]] = [] + + def get_draft_by_app_id(self, app_id: str): + if app_id in self._connections_by_app_id: + return SimpleNamespace(connection_id=self._connections_by_app_id[app_id]) + return None + + def get_config(self, app_id: str, env): # noqa: ARG002 + return SimpleNamespace(security_scheme="key_value") + + def get_credentials(self, app_id: str, env, *, use_app_credentials: bool): # noqa: ARG002 + return {"runtime_credentials": {"TOKEN": "value"}} + + def delete_credentials(self, app_id: str, env, *, use_app_credentials: bool): + self.delete_credentials_calls.append((app_id, env, use_app_credentials)) + + def delete(self, app_id: str): + self.delete_calls.append(app_id) + self._connections_by_app_id.pop(app_id, None) + + def create(self, payload: dict): + self.create_calls.append(payload) + app_id = str(payload.get("app_id")) + self._connections_by_app_id[app_id] = f"conn-{app_id}" + + def create_config(self, app_id: str, payload: dict): + self.create_config_calls.append((app_id, payload)) + + def create_credentials(self, app_id: str, env, *, use_app_credentials: bool, payload: dict): + self.create_credentials_calls.append((app_id, env, use_app_credentials, payload)) + + def list(self): + return self._list_entries + + +class FakeBaseClient: + def __init__( + self, + *, + post_response: dict | None = None, + get_payloads: dict[str, dict] | None = None, + ): + self.post_response = post_response or { + "thread_id": "thread-1", + "run_id": "run-1", + "task_id": "task-1", + "message_id": "message-1", + } + self._get_payloads = get_payloads or {} + self.post_calls: list[tuple[str, dict]] = [] + self.get_calls: list[str] = [] + + def _post(self, path: str, data: dict): + self.post_calls.append((path, data)) + return self.post_response + + def _get(self, path: str, params: dict | None = None): # noqa: ARG002 + self.get_calls.append(path) + return self._get_payloads.get(path, {}) + + +def _with_wxo_wrappers(ns): + """Attach WxOClient SDK wrapper methods to a SimpleNamespace test double.""" + if hasattr(ns, "_base") and ns._base is not None: + ns.get_agents_raw = lambda params=None: ns._base._get("/agents", params=params) + ns.post_run = lambda *, query_suffix="", data: ns._base._post(f"/runs{query_suffix}", data) + ns.get_run = lambda run_id: ns._base._get(f"/runs/{run_id}") + return ns + + +@pytest.mark.anyio +async def test_process_config_uses_raw_payload_but_overrides_name(monkeypatch): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import process_config + + captured = {} + + async def mock_create_config(*, clients, config, user_id, db): # noqa: ARG001 + captured["name"] = config.name + captured["env_vars"] = config.environment_variables + return config.name + + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.core.config.create_config", + mock_create_config, + ) + + app_id = await process_config( + clients=SimpleNamespace(), + user_id="user-1", + db=object(), + deployment_name="my_deployment", + config=ConfigItem( + raw_payload=DeploymentConfig( + name="caller_supplied_name", + description="from payload", + environment_variables=None, + ) + ), + ) + + assert app_id == "my_deployment" + assert captured["name"] == "my_deployment" + assert captured["env_vars"] is None + + +@pytest.mark.anyio +async def test_process_config_rejects_reference_id(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import process_config + + with pytest.raises(InvalidDeploymentOperationError, match="Config reference binding is not supported"): + await process_config( + clients=SimpleNamespace(), + user_id="user-1", + db=object(), + deployment_name="my_deployment", + config=ConfigItem(reference_id="existing-config"), + ) + + +@pytest.mark.anyio +async def test_create_rejects_config_reference_before_name_precheck(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(InvalidDeploymentOperationError, match="Config reference binding is not supported"): + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate( + spec=BaseDeploymentData( + name="my deployment", + description="desc", + type=DeploymentType.AGENT, + ), + config=ConfigItem(reference_id="existing-config"), + ), + ) + + +@pytest.mark.anyio +async def test_create_rejects_missing_resource_name_prefix(): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + with pytest.raises(InvalidContentError, match="resource_name_prefix"): + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate( + spec=BaseDeploymentData( + name="my deployment", + description="desc", + type=DeploymentType.AGENT, + ), + ), + ) + + +@pytest.mark.anyio +async def test_resolve_runtime_credentials_supports_variable_and_raw_sources(monkeypatch): + async def mock_resolve_variable_value(variable_name: str, *, user_id, db, **kwargs): # noqa: ARG001 + return f"resolved::{variable_name}" + + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.client.resolve_variable_value", + mock_resolve_variable_value, + ) + + from langflow.services.adapters.deployment.watsonx_orchestrate.client import resolve_runtime_credentials + + runtime_credentials = await resolve_runtime_credentials( + user_id="user-1", + db=object(), + environment_variables={ + "FROM_RAW": EnvVarValueSpec(source="raw", value="raw-token"), + "FROM_VAR": EnvVarValueSpec(source="variable", value="OPENAI_API_KEY"), + }, + ) + + assert runtime_credentials.model_dump() == { + "FROM_RAW": "raw-token", + "FROM_VAR": "resolved::OPENAI_API_KEY", + } + + +@pytest.mark.anyio +async def test_update_rejects_legacy_top_level_snapshot_or_config(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}), + tool=FakeToolClient([{"id": "tool-1", "binding": {"langflow": {"connections": {}}}}]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + with pytest.raises(InvalidDeploymentOperationError, match="no longer supported"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate(snapshot={"add_ids": ["tool-1"]}), + db=object(), + ) + + +@pytest.mark.anyio +async def test_update_provider_data_binds_existing_tool_and_updates_agent_tools(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}) + fake_tool = FakeToolClient( + [ + {"id": "tool-1", "name": "tool-1", "binding": {"langflow": {"connections": {}}}}, + {"id": "tool-3", "name": "tool-3", "binding": {"langflow": {"connections": {}}}}, + ] + ) + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=fake_tool, + connections=FakeConnectionsClient(existing_app_id="cfg-new"), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + assert app_id == "cfg-new" + return SimpleNamespace(connection_id="conn-new") + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "validate_connection", mock_validate_connection) + + result = await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": ["tool-3"]}, + "connections": {"existing_app_ids": ["cfg-new"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-3"}, + "app_ids": ["cfg-new"], + } + ], + } + ), + db=object(), + ) + + assert result.snapshot_ids == ["tool-3"] + assert [tool_id for tool_id, _payload in fake_tool.update_calls] == ["tool-3"] + _, updated_tool_payload = fake_tool.update_calls[0] + assert updated_tool_payload["binding"]["langflow"]["connections"]["cfg-new"] == "conn-new" + _, agent_payload = fake_agent.update_calls[0] + assert agent_payload["tools"] == ["tool-1", "tool-3"] + + +@pytest.mark.anyio +async def test_update_provider_data_creates_raw_connection_and_raw_tool(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}) + fake_tool = FakeToolClient([{"id": "tool-1", "name": "tool-1", "binding": {"langflow": {"connections": {}}}}]) + fake_connections = FakeConnectionsClient() + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=fake_tool, + connections=fake_connections, + ) + captured: dict[str, object] = {} + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_create_config(*, clients, config, user_id, db): # noqa: ARG001 + captured["created_app_id"] = config.name + fake_connections._connections_by_app_id[config.name] = f"conn-{config.name}" + return config.name + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + return SimpleNamespace(connection_id=f"conn-{app_id}") + + async def mock_create_and_upload(*, clients, tool_bindings, tool_name_prefix): + _ = clients + first_binding = tool_bindings[0] + captured["connections"] = first_binding.connections + captured["tool_name_prefix"] = tool_name_prefix + return ["new-tool-1"] + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "create_config", mock_create_config) + monkeypatch.setattr(update_helpers_module, "validate_connection", mock_validate_connection) + monkeypatch.setattr( + update_helpers_module, + "create_and_upload_wxo_flow_tools_with_bindings", + mock_create_and_upload, + ) + + result = await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + provider_data={ + "resource_name_prefix": "lf_", + "tools": { + "raw_payloads": [ + { + "id": str(UUID("00000000-0000-0000-0000-000000000011")), + "name": "snapshot-new-1", + "description": "desc", + "data": {"nodes": [], "edges": []}, + "tags": [], + "provider_data": {"project_id": "project-1"}, + } + ] + }, + "connections": { + "raw_payloads": [ + { + "app_id": "cfg", + "environment_variables": {"API_KEY": {"source": "raw", "value": "secret"}}, + } + ] + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "snapshot-new-1"}, + "app_ids": ["cfg"], + } + ], + } + ), + db=object(), + ) + + assert captured["created_app_id"] == "lf_cfg" + assert captured["connections"] == {"cfg": "conn-lf_cfg"} + assert captured["tool_name_prefix"] == "lf_" + assert result.snapshot_ids == ["new-tool-1"] + _, agent_payload = fake_agent.update_calls[0] + assert agent_payload["tools"] == ["tool-1", "new-tool-1"] + + +@pytest.mark.anyio +async def test_update_provider_data_mixed_operations_preserve_encounter_order(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": ["tool-1", "tool-2"]}) + fake_tool = FakeToolClient( + [ + { + "id": "tool-1", + "name": "tool-1", + "binding": {"langflow": {"connections": {"cfg-1": "conn-old-1", "cfg-2": "conn-old-2"}}}, + }, + {"id": "tool-2", "name": "tool-2", "binding": {"langflow": {"connections": {}}}}, + {"id": "tool-3", "name": "tool-3", "binding": {"langflow": {"connections": {}}}}, + ] + ) + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=fake_tool, + connections=FakeConnectionsClient(existing_app_id="cfg-1"), + ) + validate_calls: list[str] = [] + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + validate_calls.append(app_id) + return SimpleNamespace(connection_id=f"conn-{app_id}") + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "validate_connection", mock_validate_connection) + + result = await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + provider_data={ + "tools": {"existing_ids": ["tool-1", "tool-2", "tool-3"]}, + "connections": {"existing_app_ids": ["cfg-1", "cfg-2"]}, + "operations": [ + {"op": "bind", "tool": {"reference_id": "tool-3"}, "app_ids": ["cfg-2", "cfg-1"]}, + {"op": "unbind", "tool_id": "tool-1", "app_ids": ["cfg-1", "cfg-2"]}, + {"op": "remove_tool", "tool_id": "tool-2"}, + ], + } + ), + db=object(), + ) + + assert validate_calls == ["cfg-1", "cfg-2"] + assert result.snapshot_ids == ["tool-3"] + + # Existing tool updates should follow first encounter order: bind(tool-3) then unbind(tool-1). + assert [tool_id for tool_id, _payload in fake_tool.update_calls] == ["tool-3", "tool-1"] + + _, tool3_payload = fake_tool.update_calls[0] + assert list(tool3_payload["binding"]["langflow"]["connections"]) == ["cfg-2", "cfg-1"] + assert tool3_payload["binding"]["langflow"]["connections"] == {"cfg-2": "conn-cfg-2", "cfg-1": "conn-cfg-1"} + + _, tool1_payload = fake_tool.update_calls[1] + assert tool1_payload["binding"]["langflow"]["connections"] == {} + + _, agent_payload = fake_agent.update_calls[0] + assert agent_payload["tools"] == ["tool-1", "tool-3"] + + +def test_ordered_unique_strs_preserves_encounter_order_and_safe_discard(): + ordered = update_helpers_module.OrderedUniqueStrs() + ordered.extend(["b", "a", "b", "c"]) + ordered.add("a") + ordered.add("d") + ordered.discard("c") + ordered.discard("missing") + + assert ordered.to_list() == ["b", "a", "d"] + + +def test_build_provider_update_plan_preserves_operation_encounter_order(): + provider_update = payloads_module.WatsonxDeploymentUpdatePayload.model_validate( + { + "resource_name_prefix": "lf_", + "tools": { + "existing_ids": ["tool-a", "tool-b", "tool-c"], + "raw_payloads": [ + { + "id": str(UUID("00000000-0000-0000-0000-000000000041")), + "name": "snapshot-raw-1", + "description": "desc", + "data": {"nodes": [], "edges": []}, + "tags": [], + "provider_data": {"project_id": "project-1"}, + } + ], + }, + "connections": { + "existing_app_ids": ["cfg-1", "cfg-2", "cfg-3"], + "raw_payloads": [ + {"app_id": "cfg-raw-1", "environment_variables": {"API_KEY": {"source": "raw", "value": "x"}}}, + {"app_id": "cfg-raw-2", "environment_variables": {"API_KEY": {"source": "raw", "value": "y"}}}, + ], + }, + "operations": [ + {"op": "bind", "tool": {"reference_id": "tool-c"}, "app_ids": ["cfg-2", "cfg-1", "cfg-2"]}, + {"op": "bind", "tool": {"reference_id": "tool-a"}, "app_ids": ["cfg-1"]}, + {"op": "unbind", "tool_id": "tool-c", "app_ids": ["cfg-3", "cfg-1", "cfg-3"]}, + {"op": "remove_tool", "tool_id": "tool-b"}, + {"op": "bind", "tool": {"name_of_raw": "snapshot-raw-1"}, "app_ids": ["cfg-raw-2", "cfg-raw-1"]}, + ], + } + ) + plan = update_helpers_module.build_provider_update_plan( + agent={"id": "dep-1", "tools": ["tool-a", "tool-b"]}, + provider_update=provider_update, + ) + + assert plan.bind_existing_tool_ids == ["tool-c", "tool-a"] + assert plan.final_existing_tool_ids == ["tool-a", "tool-c"] + assert plan.existing_app_ids == ["cfg-1", "cfg-2", "cfg-3"] + assert [item.operation_app_id for item in plan.raw_connections_to_create] == ["cfg-raw-1", "cfg-raw-2"] + assert [item.provider_app_id for item in plan.raw_connections_to_create] == ["lf_cfg-raw-1", "lf_cfg-raw-2"] + assert len(plan.raw_tools_to_create) == 1 + assert plan.raw_tools_to_create[0].app_ids == ["cfg-raw-2", "cfg-raw-1"] + + delta = plan.existing_tool_deltas["tool-c"] + assert delta.bind.to_list() == ["cfg-2", "cfg-1"] + assert delta.unbind.to_list() == ["cfg-3", "cfg-1"] + + +@pytest.mark.anyio +async def test_update_existing_tool_connection_deltas_uses_bind_order_in_errors(): + fake_tool = FakeToolClient([{"id": "tool-c", "name": "tool-c", "binding": {"langflow": {"connections": {}}}}]) + clients = SimpleNamespace(tool=fake_tool) + delta = update_helpers_module.ToolConnectionOps() + delta.bind.extend(["cfg-missing-first", "cfg-present"]) + + with pytest.raises(InvalidContentError, match="cfg-missing-first"): + await update_helpers_module._update_existing_tool_connection_deltas( + clients=clients, + existing_tool_deltas={"tool-c": delta}, + resolved_connections={"cfg-present": "conn-present"}, + original_tools={}, + ) + + +@pytest.mark.anyio +async def test_list_deployments_filters_with_provider_draft_filters(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient( + {"id": "dep-1", "tools": []}, + listed_agents=[ + {"id": "dep-1", "name": "deployment-1", "tools": []}, + {"id": "dep-2", "name": "deployment-2", "tools": []}, + {"id": "dep-3", "name": "deployment-3", "tools": []}, + ], + ) + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_agent, + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.list( + user_id="user-1", + db=object(), + params=DeploymentListParams( + deployment_types=[DeploymentType.AGENT], + provider_params={"ids": ["dep-2"], "names": ["deployment-3"]}, + ), + ) + + assert sorted(item.id for item in result.deployments) == ["dep-2", "dep-3"] + + +@pytest.mark.anyio +async def test_update_provider_data_maps_raw_connection_conflict_to_deployment_conflict(monkeypatch): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}), + tool=FakeToolClient([{"id": "tool-1", "binding": {"langflow": {"connections": {}}}}]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_create_config(*, clients, config, user_id, db): # noqa: ARG001 + response = SimpleNamespace(status_code=409, text='{"detail":"already exists"}') + raise ClientAPIException(response=response) + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "create_config", mock_create_config) + + with pytest.raises(DeploymentConflictError, match="error details"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + provider_data={ + "resource_name_prefix": "lf_", + "tools": { + "raw_payloads": [ + { + "id": str(UUID("00000000-0000-0000-0000-000000000012")), + "name": "snapshot-new-1", + "description": "desc", + "data": {"nodes": [], "edges": []}, + "tags": [], + "provider_data": {"project_id": "project-1"}, + } + ] + }, + "connections": { + "raw_payloads": [ + { + "app_id": "cfg", + "environment_variables": {"API_KEY": {"source": "raw", "value": "secret"}}, + } + ] + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "snapshot-new-1"}, + "app_ids": ["cfg"], + } + ], + } + ), + db=object(), + ) + + +@pytest.mark.anyio +@pytest.mark.parametrize( + ("provider_data", "error_message"), + [ + ( + { + "tools": {"existing_ids": ["tool-1"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-1"}, + "app_ids": ["undeclared_app_for_bind"], + } + ], + }, + "operation app_ids must be declared in connections\\.existing_app_ids or " + "connections\\.raw_payloads\\[\\*\\]\\.app_id", + ), + ( + { + "tools": {"existing_ids": ["tool-1"]}, + "connections": {"existing_app_ids": ["cfg-1"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-missing"}, + "app_ids": ["cfg-1"], + } + ], + }, + "bind.tool.reference_id not found in tools.existing_ids", + ), + ], +) +async def test_update_provider_data_validation_errors_raise_invalid_content( + monkeypatch, provider_data: dict, error_message: str +): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}), + tool=FakeToolClient([{"id": "tool-1", "binding": {"langflow": {"connections": {}}}}]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(InvalidContentError, match=error_message): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate(provider_data=provider_data), + db=object(), + ) + + +@pytest.mark.anyio +async def test_update_provider_data_rolls_back_mutated_tools_with_writable_payload(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + class FailingAgentClient(FakeAgentClient): + def update(self, deployment_id: str, payload: dict): + self.update_calls.append((deployment_id, payload)) + msg = "agent update failed" + raise RuntimeError(msg) + + fake_tool = FakeToolClient( + [ + { + "id": "tool-1", + "name": "tool-1", + "display_name": "Tool 1", + "description": "desc", + "binding": {"langflow": {"connections": {"old": "conn-old"}}}, + "created_at": "read-only-field", + } + ] + ) + fake_clients = SimpleNamespace( + agent=FailingAgentClient({"id": "dep-1", "tools": ["tool-1"]}), + tool=fake_tool, + connections=FakeConnectionsClient(existing_app_id="cfg-1"), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + return SimpleNamespace(connection_id="conn-new") + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "validate_connection", mock_validate_connection) + + with pytest.raises(DeploymentError, match="Please check server logs for details"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + spec=BaseDeploymentDataUpdate(description="trigger update"), + provider_data={ + "tools": {"existing_ids": ["tool-1"]}, + "connections": {"existing_app_ids": ["cfg-1"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-1"}, + "app_ids": ["cfg-1"], + } + ], + }, + ), + db=object(), + ) + + assert len(fake_tool.update_calls) == 2 + first_payload = fake_tool.update_calls[0][1] + rollback_payload = fake_tool.update_calls[1][1] + assert "id" not in first_payload + assert "created_at" not in first_payload + assert first_payload["binding"]["langflow"]["connections"]["cfg-1"] == "conn-new" + assert rollback_payload["binding"]["langflow"]["connections"] == {"old": "conn-old"} + + +@pytest.mark.anyio +async def test_update_provider_data_rolls_back_partially_created_raw_tools(monkeypatch): + core_tools_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.core.tools") + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_connections = FakeConnectionsClient() + fake_tool = FakeToolClient([]) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=fake_tool, + connections=fake_connections, + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + async def mock_create_config(*, clients, config, user_id, db): # noqa: ARG001 + fake_connections._connections_by_app_id[config.name] = f"conn-{config.name}" + return config.name + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + return SimpleNamespace(connection_id=f"conn-{app_id}") + + async def mock_create_and_upload_with_bindings(*, clients, tool_bindings, tool_name_prefix): + _ = clients, tool_bindings, tool_name_prefix + raise core_tools_module.ToolUploadBatchError( + created_tool_ids=["created-tool-1"], + errors=[RuntimeError("upload failed")], + ) + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr(update_helpers_module, "create_config", mock_create_config) + monkeypatch.setattr(update_helpers_module, "validate_connection", mock_validate_connection) + monkeypatch.setattr( + update_helpers_module, + "create_and_upload_wxo_flow_tools_with_bindings", + mock_create_and_upload_with_bindings, + ) + + with pytest.raises(DeploymentError, match="Please check server logs for details"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate( + provider_data={ + "resource_name_prefix": "lf_", + "tools": { + "raw_payloads": [ + { + "id": str(UUID("00000000-0000-0000-0000-000000000021")), + "name": "snapshot-new-1", + "description": "desc", + "data": {"nodes": [], "edges": []}, + "tags": [], + "provider_data": {"project_id": "project-1"}, + } + ] + }, + "connections": { + "raw_payloads": [ + { + "app_id": "cfg", + "environment_variables": {"API_KEY": {"source": "raw", "value": "secret"}}, + } + ] + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "snapshot-new-1"}, + "app_ids": ["cfg"], + } + ], + } + ), + db=object(), + ) + + assert fake_tool.delete_calls == ["created-tool-1"] + assert fake_connections.delete_calls == ["lf_cfg"] + + +@pytest.mark.anyio +async def test_process_raw_flows_with_app_id_awaits_connection_validation(monkeypatch): + from langflow.services.adapters.deployment.watsonx_orchestrate.core import tools as tools_core_module + + fake_clients = SimpleNamespace( + tool=SimpleNamespace(), + connections=SimpleNamespace(), + ) + captured: dict[str, object] = {} + + async def mock_validate_connection(connections_client, *, app_id): # noqa: ARG001 + return SimpleNamespace(connection_id="conn-123") + + async def mock_create_and_upload_wxo_flow_tools( + *, + clients, + flow_payloads, + connections, + tool_name_prefix, + ): + captured["clients"] = clients + captured["flow_payloads"] = flow_payloads + captured["connections"] = connections + captured["tool_name_prefix"] = tool_name_prefix + return ["tool-1"] + + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.core.config.validate_connection", + mock_validate_connection, + ) + monkeypatch.setattr( + tools_core_module, + "create_and_upload_wxo_flow_tools", + mock_create_and_upload_wxo_flow_tools, + ) + + result = await tools_core_module.process_raw_flows_with_app_id( + clients=fake_clients, + app_id="app-1", + flows=[], + tool_name_prefix="lf_test_", + ) + + assert result == ["tool-1"] + assert captured["connections"] == {"app-1": "conn-123"} + assert captured["tool_name_prefix"] == "lf_test_" + + +def test_create_wxo_flow_tool_keeps_load_from_db_global_values_unprefixed(monkeypatch): + captured_tool_definition = {} + flow_payload = BaseFlowArtifact( + id="00000000-0000-0000-0000-000000000001", + name="flow", + description="desc", + data={ + "nodes": [ + { + "data": { + "node": { + "template": { + "api_key": { + "load_from_db": True, + "value": "OPENAI_API_KEY", + }, + "plain_value": { + "load_from_db": False, + "value": "DO_NOT_TOUCH", + }, + } + } + } + } + ], + "edges": [], + }, + tags=[], + provider_data={"project_id": "project-123"}, + ) + + fake_tool = SimpleNamespace( + __tool_spec__=SimpleNamespace( + model_dump=lambda **kwargs: {"name": "flow"}, # noqa: ARG005 + ) + ) + + def mock_create_langflow_tool(*, tool_definition, connections, show_details): # noqa: ARG001 + captured_tool_definition.update(tool_definition) + return fake_tool + + monkeypatch.setattr(tools_module, "create_langflow_tool", mock_create_langflow_tool) + monkeypatch.setattr( + tools_module, + "build_langflow_artifact_bytes", + lambda **kwargs: b"artifact", # noqa: ARG005 + ) + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import create_wxo_flow_tool + + create_wxo_flow_tool( + flow_payload=flow_payload, + connections={}, + tool_name_prefix="lf_test_", + ) + + template = captured_tool_definition["data"]["nodes"][0]["data"]["node"]["template"] + assert template["api_key"]["value"] == "OPENAI_API_KEY" + assert template["plain_value"]["value"] == "DO_NOT_TOUCH" + + +def test_create_wxo_flow_tool_requires_provider_data_project_id(): + flow_payload = BaseFlowArtifact( + id="00000000-0000-0000-0000-000000000001", + name="flow", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + ) + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import create_wxo_flow_tool + + with pytest.raises( + InvalidContentError, + match="Flow payload must include provider_data with a non-empty project_id", + ): + create_wxo_flow_tool( + flow_payload=flow_payload, + connections={}, + tool_name_prefix="lf_test_", + ) + + +def test_create_wxo_flow_tool_prefixes_name_for_raw_payload(monkeypatch): + flow_payload = BaseFlowArtifact( + id="00000000-0000-0000-0000-000000000001", + name="basicllmwxo", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-123"}, + ) + + fake_tool = SimpleNamespace( + __tool_spec__=SimpleNamespace( + model_dump=lambda **kwargs: {"name": "basicllmwxo"}, # noqa: ARG005 + ) + ) + monkeypatch.setattr( + tools_module, + "create_langflow_tool", + lambda **kwargs: fake_tool, # noqa: ARG005 + ) + monkeypatch.setattr( + tools_module, + "build_langflow_artifact_bytes", + lambda **kwargs: b"artifact", # noqa: ARG005 + ) + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import create_wxo_flow_tool + + tool_payload, artifact_bytes = create_wxo_flow_tool( + flow_payload=flow_payload, + connections={}, + tool_name_prefix="lf_abcdef_", + ) + + assert tool_payload["name"] == "lf_abcdef_basicllmwxo" + assert tool_payload["binding"]["langflow"]["project_id"] == "project-123" + assert artifact_bytes == b"artifact" + + +@pytest.mark.anyio +async def test_create_wires_snapshot_ids_to_agent_and_prefixed_names(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + captured: dict[str, object] = {} + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + async def mock_process_config(user_id, db, deployment_name, config, *, clients): # noqa: ARG001 + captured["config_deployment_name"] = deployment_name + return deployment_name + + monkeypatch.setattr( + service_module, + "process_config", + mock_process_config, + ) + + async def mock_process_raw_flows_with_app_id( + clients, # noqa: ARG001 + app_id, + flows, + tool_name_prefix, + ): + captured["snapshot_app_id"] = app_id + captured["snapshot_flows"] = flows + captured["tool_name_prefix"] = tool_name_prefix + return ["tool-1", "tool-2"] + + monkeypatch.setattr( + service_module, + "process_raw_flows_with_app_id", + mock_process_raw_flows_with_app_id, + ) + + deployment_payload = DeploymentCreate( + spec=BaseDeploymentData( + name="my deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_abcdef_"}, + ), + config=ConfigItem( + raw_payload=DeploymentConfig( + name="ignored", + description="from payload", + environment_variables=None, + ) + ), + snapshot=SnapshotItems( + raw_payloads=[ + BaseFlowArtifact( + id=UUID("00000000-0000-0000-0000-000000000001"), + name="snapshot-one", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-123"}, + ) + ] + ), + ) + + result = await service.create( + user_id="user-1", + payload=deployment_payload, + db=object(), + ) + + assert result.id == "dep-created" + assert result.config_id == "lf_abcdef_my_deployment_ignored_app_id" + assert result.snapshot_ids == ["tool-1", "tool-2"] + assert captured["config_deployment_name"] == "lf_abcdef_my_deployment_ignored_app_id" + assert captured["snapshot_app_id"] == "lf_abcdef_my_deployment_ignored_app_id" + assert len(captured["snapshot_flows"]) == 1 + assert captured["tool_name_prefix"] == "lf_abcdef_" + + assert fake_clients.agent.create_calls + assert fake_clients.agent.create_calls[0]["tools"] == ["tool-1", "tool-2"] + assert fake_clients.agent.create_calls[0]["name"] == "lf_abcdef_my_deployment" + + +@pytest.mark.anyio +async def test_create_uses_caller_provided_resource_name_prefix(monkeypatch): + """When provider_spec includes resource_name_prefix, the service uses it.""" + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + captured: dict[str, object] = {} + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + async def mock_process_config(user_id, db, deployment_name, config, *, clients): # noqa: ARG001 + captured["config_deployment_name"] = deployment_name + return deployment_name + + monkeypatch.setattr( + service_module, + "process_config", + mock_process_config, + ) + + async def mock_process_raw_flows_with_app_id( + clients, # noqa: ARG001 + app_id, + flows, + tool_name_prefix, + ): + captured["snapshot_app_id"] = app_id + captured["snapshot_flows"] = flows + captured["tool_name_prefix"] = tool_name_prefix + return ["tool-1", "tool-2"] + + monkeypatch.setattr( + service_module, + "process_raw_flows_with_app_id", + mock_process_raw_flows_with_app_id, + ) + + deployment_payload = DeploymentCreate( + spec=BaseDeploymentData( + name="my deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "idempotent_abc_"}, + ), + config=ConfigItem( + raw_payload=DeploymentConfig( + name="ignored", + description="from payload", + environment_variables=None, + ) + ), + snapshot=SnapshotItems( + raw_payloads=[ + BaseFlowArtifact( + id=UUID("00000000-0000-0000-0000-000000000001"), + name="snapshot-one", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-123"}, + ) + ] + ), + ) + + result = await service.create( + user_id="user-1", + payload=deployment_payload, + db=object(), + ) + + assert result.id == "dep-created" + assert captured["config_deployment_name"] == "idempotent_abc_my_deployment_ignored_app_id" + assert captured["snapshot_app_id"] == "idempotent_abc_my_deployment_ignored_app_id" + assert captured["tool_name_prefix"] == "idempotent_abc_" + + assert fake_clients.agent.create_calls + assert fake_clients.agent.create_calls[0]["name"] == "idempotent_abc_my_deployment" + + +@pytest.mark.anyio +async def test_create_rolls_back_and_preserves_original_error_when_cleanup_fails(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_connections = FakeConnectionsClient() + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=fake_connections, + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + async def mock_process_config(user_id, db, deployment_name, config, *, clients): # noqa: ARG001 + return deployment_name + + monkeypatch.setattr( + service_module, + "process_config", + mock_process_config, + ) + + async def mock_process_raw_flows_with_app_id( + clients, # noqa: ARG001 + app_id, # noqa: ARG001 + flows, # noqa: ARG001 + tool_name_prefix, # noqa: ARG001 + ): + msg = "boom" + raise RuntimeError(msg) + + def failing_delete(app_id: str): + _ = app_id + msg = "cleanup failed" + raise RuntimeError(msg) + + monkeypatch.setattr( + service_module, + "process_raw_flows_with_app_id", + mock_process_raw_flows_with_app_id, + ) + monkeypatch.setattr(fake_connections, "delete", failing_delete) + + deployment_payload = DeploymentCreate( + spec=BaseDeploymentData( + name="my deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_abcdef_"}, + ), + config=ConfigItem( + raw_payload=DeploymentConfig( + name="ignored", + description="from payload", + environment_variables=None, + ) + ), + snapshot=SnapshotItems( + raw_payloads=[ + BaseFlowArtifact( + id=UUID("00000000-0000-0000-0000-000000000001"), + name="snapshot-one", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-123"}, + ) + ] + ), + ) + + with pytest.raises(DeploymentError, match="Please check server logs for details"): + await service.create( + user_id="user-1", + payload=deployment_payload, + db=object(), + ) + + +@pytest.mark.anyio +async def test_create_execution_posts_runs_payload(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_base = FakeBaseClient() + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_base, + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.create_execution( + user_id="user-1", + db=object(), + payload=ExecutionCreate( + deployment_id="dep-1", + provider_data={"input": "hello from test", "thread_id": "thread-123", "stream": False}, + ), + ) + + assert result.deployment_id == "dep-1" + assert result.execution_id == "run-1" + assert result.provider_result == {"status": "accepted", "run_id": "run-1"} + assert fake_base.post_calls + path, payload = fake_base.post_calls[0] + assert path == "/runs?stream=false" + assert payload["agent_id"] == "dep-1" + assert payload["thread_id"] == "thread-123" + assert payload["message"] == {"role": "user", "content": "hello from test"} + + +@pytest.mark.anyio +async def test_get_execution_returns_completed_output(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_base = FakeBaseClient( + get_payloads={ + "/runs/run-1": { + "run_id": "run-1", + "status": "completed", + "agent_id": "dep-1", + "completed_at": "2026-03-08T18:23:25.277362Z", + "result": {"data": "Final assistant response"}, + } + } + ) + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_base, + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.get_execution( + user_id="user-1", + db=object(), + execution_id="run-1", + ) + + assert result.execution_id == "run-1" + assert result.provider_result["status"] == "completed" + assert result.provider_result["agent_id"] == "dep-1" + assert result.provider_result["run_id"] == "run-1" + assert result.provider_result["completed_at"] == "2026-03-08T18:23:25.277362Z" + + +@pytest.mark.anyio +async def test_get_execution_fetches_result_payload(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_base = FakeBaseClient( + get_payloads={ + "/runs/run-1": { + "run_id": "run-1", + "status": "completed", + "agent_id": "dep-1", + "result": {"output": "some result"}, + } + } + ) + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_base, + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.get_execution( + user_id="user-1", + db=object(), + execution_id="run-1", + ) + + assert result.provider_result["status"] == "completed" + assert result.provider_result["agent_id"] == "dep-1" + assert result.provider_result["result"] == {"output": "some result"} + + +@pytest.mark.anyio +async def test_get_execution_requires_execution_id(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_base = FakeBaseClient() + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_base, + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(ValueError, match="execution_id"): + await service.get_execution( + user_id="user-1", + db=object(), + execution_id="", + ) + + +@pytest.mark.anyio +async def test_get_execution_handles_client_api_exception(monkeypatch): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_base = FakeBaseClient() + + def failing_get(path: str, params=None): # noqa: ARG001 + resp = SimpleNamespace(status_code=500, text='{"detail": "internal error"}') + raise ClientAPIException(response=resp) + + fake_base._get = failing_get + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_base, + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentError, match="getting a deployment execution"): + await service.get_execution( + user_id="user-1", + db=object(), + execution_id="run-1", + ) + + +@pytest.mark.anyio +async def test_list_configs_single_deployment_scope(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}) + fake_tool = FakeToolClient( + [ + { + "id": "tool-1", + "name": "tool-one", + "binding": { + "langflow": { + "connections": { + "cfg-1": "conn-1", + } + } + }, + "created_at": "2026-03-08T18:23:25.277362Z", + } + ] + ) + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=fake_tool, + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.list_configs( + user_id="user-1", + db=object(), + params=ConfigListParams(deployment_ids=["dep-1"]), + ) + + assert len(result.configs) == 1 + assert result.configs[0].id == "cfg-1" + assert result.configs[0].name == "cfg-1" + + +@pytest.mark.anyio +async def test_list_snapshots_single_deployment_scope(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": ["tool-1", "tool-2"]}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.list_snapshots( + user_id="user-1", + db=object(), + params=SnapshotListParams(deployment_ids=["dep-1"]), + ) + + assert [snapshot.id for snapshot in result.snapshots] == ["tool-1", "tool-2"] + + +@pytest.mark.anyio +async def test_list_configs_without_deployment_id_raises(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(OperationNotSupportedError, match="requires exactly one deployment_id"): + await service.list_configs(user_id="user-1", db=object(), params=None) + + +@pytest.mark.anyio +async def test_list_snapshots_without_deployment_id_raises(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(OperationNotSupportedError, match="requires exactly one deployment_id"): + await service.list_snapshots(user_id="user-1", db=object(), params=None) + + +# --------------------------------------------------------------------------- +# Retry / backoff tests +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_retry_with_backoff_succeeds_on_first_try(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_with_backoff + + call_count = 0 + + async def op(): + nonlocal call_count + call_count += 1 + return "ok" + + result = await retry_with_backoff(op, max_attempts=3) + assert result == "ok" + assert call_count == 1 + + +@pytest.mark.anyio +async def test_retry_with_backoff_forwards_args_and_kwargs(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_with_backoff + + received: list[tuple[str, str]] = [] + + async def op(prefix: str, *, suffix: str) -> str: + received.append((prefix, suffix)) + return f"{prefix}-{suffix}" + + result = await retry_with_backoff(op, 3, "left", suffix="right") + assert result == "left-right" + assert received == [("left", "right")] + + +@pytest.mark.anyio +async def test_retry_create_with_to_thread_forwards_kwargs(): + import asyncio + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_create + + def sync_add(a: int, *, b: int) -> int: + return a + b + + result = await retry_create(asyncio.to_thread, sync_add, 2, b=5) + assert result == 7 + + +@pytest.mark.anyio +async def test_retry_with_backoff_retries_then_succeeds(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_with_backoff + + call_count = 0 + + async def op(): + nonlocal call_count + call_count += 1 + if call_count < 3: + msg = "transient" + raise RuntimeError(msg) + return "ok" + + result = await retry_with_backoff(op, max_attempts=3) + assert result == "ok" + assert call_count == 3 + + +@pytest.mark.anyio +async def test_retry_with_backoff_gives_up_after_max_attempts(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_with_backoff + + call_count = 0 + + async def op(): + nonlocal call_count + call_count += 1 + msg = "always fails" + raise RuntimeError(msg) + + with pytest.raises(RuntimeError, match="always fails"): + await retry_with_backoff(op, max_attempts=3) + assert call_count == 3 + + +@pytest.mark.anyio +async def test_retry_with_backoff_respects_should_retry_predicate(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import retry_with_backoff + + call_count = 0 + + async def op(): + nonlocal call_count + call_count += 1 + msg = "non-retryable" + raise ValueError(msg) + + with pytest.raises(ValueError, match="non-retryable"): + await retry_with_backoff( + op, + max_attempts=5, + should_retry=lambda exc: not isinstance(exc, ValueError), + ) + assert call_count == 1 + + +def test_is_retryable_create_exception_non_retryable_status_codes(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import is_retryable_create_exception + + non_retryable = {400, 401, 403, 404, 409, 422} + for code in non_retryable: + exc = HTTPException(status_code=code) + assert is_retryable_create_exception(exc) is False, f"status {code} should not be retryable" + + +def test_is_retryable_create_exception_retryable_status_codes(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import is_retryable_create_exception + + for code in (500, 502, 503, 429): + exc = HTTPException(status_code=code) + assert is_retryable_create_exception(exc) is True, f"status {code} should be retryable" + + +def test_is_retryable_create_exception_domain_exceptions_not_retryable(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import is_retryable_create_exception + + assert is_retryable_create_exception(DeploymentConflictError()) is False + assert is_retryable_create_exception(InvalidContentError()) is False + assert is_retryable_create_exception(InvalidDeploymentOperationError()) is False + + +def test_is_retryable_create_exception_generic_exception_is_retryable(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import is_retryable_create_exception + + assert is_retryable_create_exception(RuntimeError("boom")) is True + + +@pytest.mark.anyio +async def test_rollback_created_resources_deletes_all(monkeypatch): + from langflow.services.adapters.deployment.watsonx_orchestrate.core import retry as retry_module + + deleted = {"agents": [], "tools": [], "configs": []} + + async def fake_delete_agent(clients, *, agent_id): # noqa: ARG001 + deleted["agents"].append(agent_id) + + async def fake_delete_tool(clients, *, tool_id): # noqa: ARG001 + deleted["tools"].append(tool_id) + + async def fake_delete_config(clients, *, app_id): # noqa: ARG001 + deleted["configs"].append(app_id) + + monkeypatch.setattr(retry_module, "delete_agent_if_exists", fake_delete_agent) + monkeypatch.setattr(retry_module, "delete_tool_if_exists", fake_delete_tool) + monkeypatch.setattr(retry_module, "delete_config_if_exists", fake_delete_config) + + fake_clients = SimpleNamespace() + await retry_module.rollback_created_resources( + clients=fake_clients, + agent_id="agent-1", + tool_ids=["tool-1", "tool-2"], + app_id="app-1", + ) + + assert deleted["agents"] == ["agent-1"] + assert deleted["tools"] == ["tool-2", "tool-1"] + assert deleted["configs"] == ["app-1"] + + +@pytest.mark.anyio +async def test_rollback_continues_after_individual_failures(monkeypatch): + from langflow.services.adapters.deployment.watsonx_orchestrate.core import retry as retry_module + + deleted = {"configs": []} + + async def fail_delete_agent(clients, *, agent_id): # noqa: ARG001 + msg = "agent delete failed" + raise RuntimeError(msg) + + async def fail_delete_tool(clients, *, tool_id): # noqa: ARG001 + msg = "tool delete failed" + raise RuntimeError(msg) + + async def fake_delete_config(clients, *, app_id): # noqa: ARG001 + deleted["configs"].append(app_id) + + monkeypatch.setattr(retry_module, "delete_agent_if_exists", fail_delete_agent) + monkeypatch.setattr(retry_module, "delete_tool_if_exists", fail_delete_tool) + monkeypatch.setattr(retry_module, "delete_config_if_exists", fake_delete_config) + monkeypatch.setattr(retry_module, "ROLLBACK_MAX_RETRIES", 1) + + fake_clients = SimpleNamespace() + await retry_module.rollback_created_resources( + clients=fake_clients, + agent_id="agent-1", + tool_ids=["tool-1"], + app_id="app-1", + ) + + assert deleted["configs"] == ["app-1"] + + +@pytest.mark.anyio +async def test_rollback_update_resources_restores_then_deletes(monkeypatch): + from langflow.services.adapters.deployment.watsonx_orchestrate.core import retry as retry_module + + restored: list[tuple[str, dict]] = [] + deleted = {"tools": [], "configs": []} + + fake_clients = SimpleNamespace( + tool=SimpleNamespace(update=lambda tool_id, payload: restored.append((tool_id, payload))), + ) + + async def fake_delete_tool(clients, *, tool_id): # noqa: ARG001 + deleted["tools"].append(tool_id) + + async def fake_delete_config(clients, *, app_id): # noqa: ARG001 + deleted["configs"].append(app_id) + + monkeypatch.setattr(retry_module, "delete_tool_if_exists", fake_delete_tool) + monkeypatch.setattr(retry_module, "delete_config_if_exists", fake_delete_config) + + await retry_module.rollback_update_resources( + clients=fake_clients, + created_tool_ids=["tool-new-1", "tool-new-2"], + created_app_id="cfg-new", + original_tools={ + "tool-old-1": {"name": "t1"}, + "tool-old-2": {"name": "t2"}, + }, + ) + + assert [tool_id for tool_id, _payload in restored] == ["tool-old-2", "tool-old-1"] + assert deleted["tools"] == ["tool-new-2", "tool-new-1"] + assert deleted["configs"] == ["cfg-new"] + + +# --------------------------------------------------------------------------- +# Service method tests: get, delete, get_status, update happy path +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_get_deployment_returns_agent(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient( + {"id": "dep-1", "name": "my_agent", "description": "desc"}, + ), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.get(user_id="user-1", deployment_id="dep-1", db=object()) + assert result.id == "dep-1" + assert result.name == "my_agent" + + +@pytest.mark.anyio +async def test_get_deployment_not_found_raises(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient(None), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentNotFoundError, match="not found"): + await service.get(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_get_deployment_handles_client_api_exception(monkeypatch): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + class FailingAgentClient(FakeAgentClient): + def get_draft_by_id(self, deployment_id: str): # noqa: ARG002 + resp = SimpleNamespace(status_code=500, text='{"detail": "internal error"}') + raise ClientAPIException(response=resp) + + fake_clients = SimpleNamespace( + agent=FailingAgentClient(None), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentError, match="getting a deployment"): + await service.get(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_delete_deployment_calls_agent_delete(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.delete(user_id="user-1", deployment_id="dep-1", db=object()) + assert result.id == "dep-1" + assert fake_agent.delete_calls == ["dep-1"] + + +@pytest.mark.anyio +async def test_delete_deployment_not_found_raises(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + class FailingAgentClient(FakeAgentClient): + def delete(self, deployment_id: str): # noqa: ARG002 + resp = SimpleNamespace(status_code=status.HTTP_404_NOT_FOUND, text="not found") + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + raise ClientAPIException(response=resp) + + fake_clients = SimpleNamespace( + agent=FailingAgentClient({"id": "dep-1", "tools": []}), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentNotFoundError, match="not found"): + await service.delete(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_get_status_connected(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient( + {"id": "dep-1", "environments": [{"name": "draft", "id": "env-1"}]}, + ), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.get_status(user_id="user-1", deployment_id="dep-1", db=object()) + assert result.id == "dep-1" + assert result.provider_data["status"] == "connected" + + +@pytest.mark.anyio +async def test_get_status_not_found(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient(None), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentNotFoundError, match="dep-1"): + await service.get_status(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_update_deployment_name_and_description(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}) + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=FakeToolClient([{"id": "tool-1"}]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + update_data = DeploymentUpdate( + spec=BaseDeploymentDataUpdate(name="new name", description="new desc"), + ) + + result = await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=update_data, + db=object(), + ) + assert result.id == "dep-1" + assert len(fake_agent.update_calls) == 1 + agent_id, payload = fake_agent.update_calls[0] + assert agent_id == "dep-1" + assert payload["name"] == "new_name" + assert payload["display_name"] == "new name" + assert payload["description"] == "new desc" + + +@pytest.mark.anyio +async def test_update_deployment_not_found_raises(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient(None), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + update_data = DeploymentUpdate(spec=BaseDeploymentDataUpdate(name="x")) + + with pytest.raises(DeploymentNotFoundError, match="not found"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=update_data, + db=object(), + ) + + +@pytest.mark.anyio +async def test_list_deployments_without_params(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient( + {"id": "dep-1", "tools": []}, + listed_agents=[ + {"id": "dep-1", "name": "agent-1", "tools": []}, + ], + ) + fake_clients = _with_wxo_wrappers( + SimpleNamespace( + _base=fake_agent, + agent=fake_agent, + ) + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.list(user_id="user-1", db=object(), params=None) + assert len(result.deployments) == 1 + assert result.deployments[0].id == "dep-1" + + +@pytest.mark.anyio +async def test_list_types_returns_supported_types(): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + result = await service.list_types(user_id="user-1", db=object()) + assert DeploymentType.AGENT in result.deployment_types + assert len(result.deployment_types) == 1 + + +@pytest.mark.anyio +async def test_get_status_handles_client_api_exception(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + class FailingAgentClient(FakeAgentClient): + def get_draft_by_id(self, deployment_id: str): # noqa: ARG002 + resp = SimpleNamespace(status_code=500, text='{"detail": "internal error"}') + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + raise ClientAPIException(response=resp) + + fake_clients = SimpleNamespace( + agent=FailingAgentClient(None), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentError, match="getting a deployment health"): + await service.get_status(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_update_spec_only_description_sends_update(monkeypatch): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_clients = SimpleNamespace( + agent=fake_agent, + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + result = await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=DeploymentUpdate(spec=BaseDeploymentDataUpdate(description="only desc")), + db=object(), + ) + assert result.id == "dep-1" + assert len(fake_agent.update_calls) == 1 + _, payload = fake_agent.update_calls[0] + assert payload == {"description": "only desc"} + assert "name" not in payload + + +# --------------------------------------------------------------------------- +# Client authentication tests +# --------------------------------------------------------------------------- + + +def test_get_authenticator_ibm_cloud(): + from langflow.services.adapters.deployment.watsonx_orchestrate.client import get_authenticator + + auth = get_authenticator("https://api.region-foobar.cloud.ibm.com", "test-key") + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + assert isinstance(auth, IAMAuthenticator) + + +def test_get_authenticator_mcsp(): + from langflow.services.adapters.deployment.watsonx_orchestrate.client import get_authenticator + + auth = get_authenticator("https://api.wxo.ibm.com", "test-key") + from ibm_cloud_sdk_core.authenticators import MCSPAuthenticator + + assert isinstance(auth, MCSPAuthenticator) + + +def test_get_authenticator_unknown_url(): + from langflow.services.adapters.deployment.watsonx_orchestrate.client import get_authenticator + from lfx.services.adapters.deployment.exceptions import AuthSchemeError + + with pytest.raises(AuthSchemeError, match="Could not determine"): + get_authenticator("https://example.com", "test-key") + + +@pytest.mark.anyio +async def test_get_provider_clients_uses_request_scoped_context_memoization(monkeypatch): + resolve_calls = 0 + + async def mock_resolve_wxo_client_credentials(*, user_id, db, provider_id): # noqa: ARG001 + nonlocal resolve_calls + resolve_calls += 1 + return WxOCredentials( + instance_url="https://api.region-foobar.cloud.ibm.com/", + authenticator=object(), + ) + + monkeypatch.setattr(client_module, "resolve_wxo_client_credentials", mock_resolve_wxo_client_credentials) + client_module.clear_provider_clients_request_context() + + provider_context = deployment_context_module.DeploymentAdapterContext(provider_id=UUID(int=1)) + with deployment_context_module.DeploymentProviderIDContext.scope(provider_context): + first = await client_module.get_provider_clients(user_id="user-1", db=object()) + second = await client_module.get_provider_clients(user_id="user-1", db=object()) + + assert first is second + assert resolve_calls == 1 + + +@pytest.mark.anyio +async def test_get_provider_clients_rejects_mixed_provider_contexts(monkeypatch): + resolve_calls = 0 + + async def mock_resolve_wxo_client_credentials(*, user_id, db, provider_id): # noqa: ARG001 + nonlocal resolve_calls + resolve_calls += 1 + return WxOCredentials( + instance_url="https://api.region-foobar.cloud.ibm.com/", + authenticator=object(), + ) + + monkeypatch.setattr(client_module, "resolve_wxo_client_credentials", mock_resolve_wxo_client_credentials) + client_module.clear_provider_clients_request_context() + + with deployment_context_module.DeploymentProviderIDContext.scope( + deployment_context_module.DeploymentAdapterContext(provider_id=UUID(int=1)) + ): + await client_module.get_provider_clients(user_id="user-1", db=object()) + with ( + deployment_context_module.DeploymentProviderIDContext.scope( + deployment_context_module.DeploymentAdapterContext(provider_id=UUID(int=2)) + ), + pytest.raises(CredentialResolutionError, match="different deployment provider context"), + ): + await client_module.get_provider_clients(user_id="user-1", db=object()) + + assert resolve_calls == 1 + + +def test_wxo_client_initializes_subclients_eagerly(monkeypatch): + init_counts = {"tool": 0, "connections": 0, "agent": 0, "base": 0} + + class FakeToolClient: + def __init__(self, base_url: str, authenticator): # noqa: ARG002 + init_counts["tool"] += 1 + self.base_url = base_url + + class FakeConnectionsClient: + def __init__(self, base_url: str, authenticator): # noqa: ARG002 + init_counts["connections"] += 1 + self.base_url = base_url + + class FakeAgentClient: + def __init__(self, base_url: str, authenticator): # noqa: ARG002 + init_counts["agent"] += 1 + self.base_url = base_url + + class FakeBaseWXOClient: + def __init__(self, base_url: str, authenticator): # noqa: ARG002 + init_counts["base"] += 1 + self.base_url = base_url + + monkeypatch.setattr(types_module, "ToolClient", FakeToolClient) + monkeypatch.setattr(types_module, "ConnectionsClient", FakeConnectionsClient) + monkeypatch.setattr(types_module, "AgentClient", FakeAgentClient) + monkeypatch.setattr(types_module, "BaseWXOClient", FakeBaseWXOClient) + + wxo_client = types_module.WxOClient( + instance_url="https://api.region-foobar.cloud.ibm.com", + authenticator=object(), + ) + # All sub-clients should be created eagerly at construction + assert init_counts == {"tool": 1, "connections": 1, "agent": 1, "base": 1} + + # Subsequent access does not re-create + _ = wxo_client.agent + assert init_counts["agent"] == 1 + + +# --------------------------------------------------------------------------- +# Utility function tests +# --------------------------------------------------------------------------- + + +def test_normalize_wxo_name(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import normalize_wxo_name + + assert normalize_wxo_name("Hello World!") == "Hello_World" + assert normalize_wxo_name("test-name-123") == "test_name_123" + assert normalize_wxo_name(" spaces ") == "__spaces__" + assert normalize_wxo_name("") == "" + + +def test_validate_wxo_name_valid(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import validate_wxo_name + + assert validate_wxo_name("my_deployment") == "my_deployment" + assert validate_wxo_name("My Deployment!") == "My_Deployment" + + +def test_validate_wxo_name_empty(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import validate_wxo_name + + with pytest.raises(InvalidContentError, match="alphanumeric"): + validate_wxo_name("!!!") + + +def test_validate_wxo_name_starts_with_digit(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import validate_wxo_name + + with pytest.raises(InvalidContentError, match="start with a letter"): + validate_wxo_name("123abc") + + +@pytest.mark.anyio +async def test_create_and_upload_wxo_flow_tools_with_bindings_journals_created_ids_on_upload_failure(monkeypatch): + created_calls: list[dict] = [] + + def mock_create_tool(payload: dict): + created_calls.append(payload) + return {"id": f"tool-{len(created_calls)}"} + + def mock_upload_tool_artifact(tool_id: str, files: dict): # noqa: ARG001 + if tool_id == "tool-1": + raise InvalidContentError(message="artifact upload failed") + return {"status": "ok"} + + fake_clients = SimpleNamespace( + tool=SimpleNamespace(create=mock_create_tool), + upload_tool_artifact=mock_upload_tool_artifact, + ) + monkeypatch.setattr( + tools_module, + "create_wxo_flow_tool", + lambda flow_payload, connections, tool_name_prefix: ( # noqa: ARG005 + {"name": flow_payload.name, "description": flow_payload.description}, + b"artifact", + ), + ) + + bindings = [ + tools_module.FlowToolBindingSpec( + flow_payload=BaseFlowArtifact( + id=UUID("00000000-0000-0000-0000-000000000031"), + name="snapshot-one", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-1"}, + ), + connections={"cfg-1": "conn-1"}, + ), + tools_module.FlowToolBindingSpec( + flow_payload=BaseFlowArtifact( + id=UUID("00000000-0000-0000-0000-000000000032"), + name="snapshot-two", + description="desc", + data={"nodes": [], "edges": []}, + tags=[], + provider_data={"project_id": "project-1"}, + ), + connections={"cfg-1": "conn-1"}, + ), + ] + + with pytest.raises(tools_module.ToolUploadBatchError) as exc: + await tools_module.create_and_upload_wxo_flow_tools_with_bindings( + clients=fake_clients, + tool_bindings=bindings, + tool_name_prefix="lf_", + ) + + assert set(exc.value.created_tool_ids) == {"tool-1", "tool-2"} + assert len(created_calls) == 2 + + +def test_resolve_resource_name_prefix_uses_caller_provided_prefix(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import resolve_resource_name_prefix + + assert resolve_resource_name_prefix(caller_prefix="custom_abc_") == "custom_abc_" + + +def test_resolve_resource_name_prefix_rejects_empty_string(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import resolve_resource_name_prefix + + with pytest.raises(InvalidContentError, match="non-empty string"): + resolve_resource_name_prefix(caller_prefix="") + + +def test_resolve_resource_name_prefix_rejects_non_alpha_start(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import resolve_resource_name_prefix + + with pytest.raises(InvalidContentError, match="start with a letter"): + resolve_resource_name_prefix(caller_prefix="123_prefix_") + + +def test_resolve_resource_name_prefix_rejects_only_special_chars(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import resolve_resource_name_prefix + + with pytest.raises(InvalidContentError, match="alphanumeric"): + resolve_resource_name_prefix(caller_prefix="!!!") + + +def test_resolve_resource_name_prefix_normalizes_caller_prefix(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import resolve_resource_name_prefix + + assert resolve_resource_name_prefix(caller_prefix="my-prefix!") == "my_prefix" + + +def test_extract_error_detail_json_string(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + assert extract_error_detail('{"detail": "something went wrong"}') == "something went wrong" + + +def test_extract_error_detail_json_list(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + assert extract_error_detail('{"detail": [{"msg": "field required"}]}') == "field required" + + +def test_extract_error_detail_json_dict(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + result = extract_error_detail('{"detail": {"msg": "invalid"}}') + assert result == "invalid" + + +def test_extract_error_detail_non_json(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + assert extract_error_detail("plain text error") == "plain text error" + + +def test_extract_error_detail_with_null_detail_falls_back_to_body(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + assert extract_error_detail('{"detail": null}') == '{"detail": null}' + + +def test_extract_error_detail_uses_message_field_when_detail_missing(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_error_detail + + payload = '{"statusCode":409,"message":"The connection ID already exists.","details":"duplicate"}' + assert extract_error_detail(payload) == "The connection ID already exists." + + +def test_dedupe_list(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import dedupe_list + + assert dedupe_list(["a", "b", "a", "c", "b"]) == ["a", "b", "c"] + assert dedupe_list([]) == [] + + +def test_raise_as_deployment_error_wraps_service_error_by_default(): + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + original = InvalidContentError(message="invalid payload") + + with pytest.raises(DeploymentError, match="Please check server logs for details"): + raise_as_deployment_error( + original, + error_prefix=ErrorPrefix.LIST, + log_msg="unexpected list failure", + ) + + +def test_raise_as_deployment_error_reraises_allowed_service_error(): + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + original = InvalidContentError(message="invalid payload") + + with pytest.raises(InvalidContentError, match="invalid payload"): + raise_as_deployment_error( + original, + error_prefix=ErrorPrefix.LIST, + log_msg="unexpected list failure", + pass_through=(InvalidContentError,), + ) + + +def test_raise_as_deployment_error_client_api_falls_back_to_raw_body(): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + resp = SimpleNamespace(status_code=500, text='{"error":"boom"}') + exc = ClientAPIException(response=resp) + + with pytest.raises(DeploymentError, match="error details: boom"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.LIST_CONFIGS, + log_msg="unexpected config list failure", + ) + + +def test_raise_as_deployment_error_http_exception_uses_detail(): + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + exc = HTTPException(status_code=400, detail="bad request") + + with pytest.raises(DeploymentError, match="error details: bad request"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.HEALTH, + log_msg="unexpected health check failure", + ) + + +def test_raise_as_deployment_error_maps_not_found(): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + resp = SimpleNamespace(status_code=500, text='{"detail":"Agent \'abc\' not found"}') + exc = ClientAPIException(response=resp) + + with pytest.raises(DeploymentNotFoundError, match="not found"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.UPDATE, + log_msg="unexpected update failure", + ) + + +def test_raise_as_deployment_error_maps_conflict(): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + resp = SimpleNamespace(status_code=500, text='{"detail":"resource already exists"}') + exc = ClientAPIException(response=resp) + + with pytest.raises(DeploymentConflictError, match="already exists"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.UPDATE, + log_msg="unexpected update conflict", + ) + + +def test_raise_as_deployment_error_maps_unprocessable_content(): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + resp = SimpleNamespace(status_code=422, text='{"detail":"unprocessable"}') + exc = ClientAPIException(response=resp) + + with pytest.raises(InvalidContentError, match="unprocessable"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.UPDATE, + log_msg="unexpected update validation failure", + ) + + +def test_raise_as_deployment_error_maps_forbidden_to_authorization_error(): + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + from langflow.services.adapters.deployment.watsonx_orchestrate.constants import ErrorPrefix + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import raise_as_deployment_error + + resp = SimpleNamespace(status_code=403, text='{"detail":"forbidden"}') + exc = ClientAPIException(response=resp) + + with pytest.raises(AuthorizationError, match="forbidden"): + raise_as_deployment_error( + exc, + error_prefix=ErrorPrefix.UPDATE, + log_msg="unexpected update authorization failure", + ) + + +def test_build_agent_payload_requires_provider_spec(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import build_agent_payload + + data = SimpleNamespace(provider_spec=None, description="desc", name="test") + with pytest.raises(InvalidContentError, match="provider_spec"): + build_agent_payload(data=data, tool_ids=[]) + + +def test_build_agent_payload_structure(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import build_agent_payload + + data = SimpleNamespace( + provider_spec={"name": "agent_name", "display_name": "Agent Name"}, + description="test description", + name="test", + ) + payload = build_agent_payload(data=data, tool_ids=["tool-1", "tool-2"]) + assert payload["name"] == "agent_name" + assert payload["display_name"] == "Agent Name" + assert payload["description"] == "test description" + assert payload["tools"] == ["tool-1", "tool-2"] + assert "llm" in payload + + +def test_extract_agent_tool_ids(): + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import extract_agent_tool_ids + + assert extract_agent_tool_ids({"tools": ["t1", "t2", None, ""]}) == ["t1", "t2"] + assert extract_agent_tool_ids({}) == [] + + +# --------------------------------------------------------------------------- +# Execution helper tests +# --------------------------------------------------------------------------- + + +def test_resolve_execution_message_string(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import resolve_execution_message + + result = resolve_execution_message("hello") + assert result == {"role": "user", "content": "hello"} + + +def test_resolve_execution_message_dict_with_role_content(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import resolve_execution_message + + msg = {"role": "assistant", "content": "hi"} + assert resolve_execution_message(msg) == msg + + +def test_resolve_execution_message_dict_with_nested_message(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import resolve_execution_message + + msg = {"message": {"role": "user", "content": "nested"}} + assert resolve_execution_message(msg) == {"role": "user", "content": "nested"} + + +def test_resolve_execution_message_empty_string_raises(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import resolve_execution_message + + with pytest.raises(ValueError, match="must not be empty"): + resolve_execution_message(" ") + + +def test_resolve_execution_message_none_raises(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import resolve_execution_message + + with pytest.raises(ValueError, match="requires input content"): + resolve_execution_message(None) + + +def test_build_orchestrate_runs_query(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import build_orchestrate_runs_query + + assert build_orchestrate_runs_query(None) == "" + assert build_orchestrate_runs_query({}) == "" + assert "stream=true" in build_orchestrate_runs_query({"stream": True}) + assert "stream_timeout=30" in build_orchestrate_runs_query({"stream_timeout": 30}) + + +def test_create_agent_run_result_empty_raises(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import create_agent_run_result + + with pytest.raises(DeploymentError, match="empty response"): + create_agent_run_result(None) + with pytest.raises(DeploymentError, match="empty response"): + create_agent_run_result({}) + + +def test_create_agent_run_result_with_run_id(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import create_agent_run_result + + result = create_agent_run_result({"status": "running", "run_id": "r-1"}) + assert result == {"status": "running", "run_id": "r-1"} + + +# --------------------------------------------------------------------------- +# Status helper tests +# --------------------------------------------------------------------------- + + +def test_derive_agent_environment_draft(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.status import derive_agent_environment + + assert derive_agent_environment({"environments": [{"name": "draft"}]}) == "draft" + + +def test_derive_agent_environment_live(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.status import derive_agent_environment + + assert derive_agent_environment({"environments": [{"name": "production"}]}) == "live" + + +def test_derive_agent_environment_both(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.status import derive_agent_environment + + assert derive_agent_environment({"environments": [{"name": "draft"}, {"name": "prod"}]}) == "both" + + +def test_derive_agent_environment_empty(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.status import derive_agent_environment + + assert derive_agent_environment({}) == "unknown" + assert derive_agent_environment({"environments": []}) == "unknown" + + +# --------------------------------------------------------------------------- +# Artifact build tests +# --------------------------------------------------------------------------- + + +def test_build_langflow_artifact_bytes_structure(): + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + build_langflow_artifact_bytes, + ) + + flow_definition = {"nodes": [{"id": "n1"}], "edges": []} + tool = SimpleNamespace( + __tool_spec__=SimpleNamespace(name="test_tool"), + requirements=["lfx>=0.3.0"], + ) + + artifact_bytes = build_langflow_artifact_bytes( + tool=tool, + flow_definition=flow_definition, + ) + + assert isinstance(artifact_bytes, bytes) + + with zipfile.ZipFile(io.BytesIO(artifact_bytes), "r") as zf: + names = zf.namelist() + assert "test_tool.json" in names + assert "requirements.txt" in names + assert "bundle-format" in names + + +# --------------------------------------------------------------------------- +# WxOCredentials repr masking +# --------------------------------------------------------------------------- + + +def test_wxo_credentials_repr_does_not_expose_authenticator_data(): + class _FakeAuthenticator: + pass + + creds = WxOCredentials(instance_url="https://example.com", authenticator=_FakeAuthenticator()) + repr_str = repr(creds) + assert repr_str == "WxOCredentials(instance_url='https://example.com', authenticator=_FakeAuthenticator)" + + +# --------------------------------------------------------------------------- +# NotImplementedError stubs +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_redeploy_raises_operation_not_supported(): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + with pytest.raises(OperationNotSupportedError, match="Redeployment is not supported"): + await service.redeploy(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_duplicate_raises_operation_not_supported(): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + with pytest.raises(OperationNotSupportedError, match="duplication is not supported"): + await service.duplicate(user_id="user-1", deployment_id="dep-1", db=object()) + + +@pytest.mark.anyio +async def test_teardown_succeeds(): + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + await service.teardown() + + +# --------------------------------------------------------------------------- +# Tests for review fixes +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_get_agent_run_empty_response_raises(monkeypatch): + """get_agent_run raises DeploymentError when provider returns empty payload.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import get_agent_run + + async def fake_to_thread(fn, *args, **kwargs): # noqa: ARG001 + return None + + import asyncio + + monkeypatch.setattr(asyncio, "to_thread", fake_to_thread) + + fake_client = SimpleNamespace(get_run=lambda _run_id: None) + with pytest.raises(DeploymentError, match="empty response"): + await get_agent_run(fake_client, run_id="run-123") + + +def test_retry_rollback_uses_retryable_filter(): + """retry_rollback should use is_retryable_create_exception to skip non-retryable errors. + + Validates that the filter correctly identifies non-retryable HTTP status codes + (via HTTPException, which is checked by is_retryable_create_exception). + """ + from langflow.services.adapters.deployment.watsonx_orchestrate.core.retry import is_retryable_create_exception + + # Non-retryable status codes should not be retried + for code in [400, 401, 403, 404, 409, 422]: + assert not is_retryable_create_exception(HTTPException(status_code=code)), ( + f"HTTPException with status {code} should NOT be retryable" + ) + + # Retryable status codes should be retried + for code in [500, 502, 503, 504]: + assert is_retryable_create_exception(HTTPException(status_code=code)), ( + f"HTTPException with status {code} should be retryable" + ) + + # Domain exceptions that are non-retryable + from lfx.services.adapters.deployment.exceptions import DeploymentConflictError, InvalidContentError + + assert not is_retryable_create_exception(DeploymentConflictError()) + assert not is_retryable_create_exception(InvalidContentError()) + + # Generic exceptions are retryable (e.g. transient network errors) + assert is_retryable_create_exception(RuntimeError("transient")) + + +@pytest.mark.anyio +async def test_credential_resolution_catches_arbitrary_exceptions(monkeypatch): + """resolve_wxo_client_credentials wraps unexpected exceptions as CredentialResolutionError.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.client import resolve_wxo_client_credentials + from lfx.services.adapters.deployment.exceptions import CredentialResolutionError + + class FakeSQLAlchemyError(Exception): + pass + + async def mock_get_provider(*args, **kwargs): # noqa: ARG001 + error_message = "connection refused" + raise FakeSQLAlchemyError(error_message) + + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.client.get_provider_account_by_id", + mock_get_provider, + ) + + with pytest.raises(CredentialResolutionError, match="unexpected error"): + await resolve_wxo_client_credentials( + user_id="user-1", + db=object(), + provider_id=UUID("00000000-0000-0000-0000-000000000001"), + ) + + +def test_wxo_client_eagerly_constructs_sub_clients(): + """WxOClient eagerly builds tool/connections/agent from instance_url and authenticator.""" + types_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.types") + wxo_client_cls = types_module.WxOClient + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + client = wxo_client_cls(instance_url="https://test.example.com", authenticator=NoAuthAuthenticator()) + + # Sub-clients should be eagerly created at construction + assert client.tool is not None + assert client.connections is not None + assert client.agent is not None + assert client.base is not None + + +def test_wxo_client_is_frozen(): + """WxOClient is frozen and rejects post-construction mutation.""" + types_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.types") + wxo_client_cls = types_module.WxOClient + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + client = wxo_client_cls(instance_url="https://test.example.com", authenticator=NoAuthAuthenticator()) + with pytest.raises(AttributeError): + client.instance_url = "https://evil.example.com" + + +def test_wxo_client_strips_trailing_slash(): + """WxOClient normalizes instance_url by stripping trailing slashes.""" + types_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.types") + wxo_client_cls = types_module.WxOClient + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + client = wxo_client_cls(instance_url="https://test.example.com/", authenticator=NoAuthAuthenticator()) + assert client.instance_url == "https://test.example.com" + + +def test_wxo_client_rejects_empty_url(): + """WxOClient rejects empty instance_url at construction.""" + types_module = importlib.import_module("langflow.services.adapters.deployment.watsonx_orchestrate.types") + wxo_client_cls = types_module.WxOClient + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + with pytest.raises(ValueError, match="non-empty"): + wxo_client_cls(instance_url="", authenticator=NoAuthAuthenticator()) + + +def test_wxo_credentials_is_frozen(): + """WxOCredentials is frozen and rejects post-construction mutation.""" + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + creds = WxOCredentials(instance_url="https://test.example.com", authenticator=NoAuthAuthenticator()) + with pytest.raises(AttributeError): + creds.instance_url = "https://evil.example.com" + + +def test_wxo_credentials_rejects_empty_url(): + """WxOCredentials rejects empty instance_url at construction.""" + from ibm_cloud_sdk_core.authenticators import NoAuthAuthenticator + + with pytest.raises(ValueError, match="non-empty"): + WxOCredentials(instance_url="", authenticator=NoAuthAuthenticator()) + + +def test_raise_for_status_separates_status_codes_from_string_heuristics(): + """Status code dispatch and string heuristic dispatch are now separate cascading checks. + + Previously, status_code == 404 and 'not found' in detail were combined with `or`, + meaning ANY status code with 'not found' text would raise DeploymentNotFoundError. + Now, the 404 status code check is standalone, and 'not found' string heuristic only + fires as a fallback for unmapped status codes. + """ + from lfx.services.adapters.deployment.exceptions import raise_for_status_and_detail + + # status_code=404 raises DeploymentNotFoundError regardless of detail text + with pytest.raises(DeploymentNotFoundError): + raise_for_status_and_detail(status_code=404, detail="anything", message_prefix="test") + + # status_code=409 raises DeploymentConflictError regardless of detail text + with pytest.raises(DeploymentConflictError): + raise_for_status_and_detail(status_code=409, detail="anything", message_prefix="test") + + # String heuristics still work as fallback for unmapped/None status codes + with pytest.raises(DeploymentNotFoundError): + raise_for_status_and_detail(status_code=None, detail="agent not found", message_prefix="test") + with pytest.raises(DeploymentConflictError): + raise_for_status_and_detail(status_code=None, detail="resource already exists", message_prefix="test") + + +# --------------------------------------------------------------------------- +# Test Coverage Gap #1: create — 409 conflict via ClientAPIException +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_create_maps_409_conflict_to_deployment_conflict_error(monkeypatch): + """Create raises DeploymentConflictError when retry_create gets a 409 from the provider.""" + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_process_config(*args, **kwargs): # noqa: ARG001 + response = SimpleNamespace(status_code=409, text='{"detail":"already exists"}') + raise ClientAPIException(response=response) + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.service.process_config", + mock_process_config, + ) + + with pytest.raises(DeploymentConflictError, match="already exist"): + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate( + spec=BaseDeploymentData( + name="my_deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_test_"}, + ), + ), + ) + + +@pytest.mark.anyio +async def test_create_maps_422_to_invalid_content_error(monkeypatch): + """Create raises InvalidContentError when retry_create gets a 422 from the provider.""" + from ibm_watsonx_orchestrate_clients.tools.tool_client import ClientAPIException + + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": []}), + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_process_config(*args, **kwargs): # noqa: ARG001 + response = SimpleNamespace(status_code=422, text='{"detail":"validation error"}') + raise ClientAPIException(response=response) + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + monkeypatch.setattr( + "langflow.services.adapters.deployment.watsonx_orchestrate.service.process_config", + mock_process_config, + ) + + with pytest.raises(InvalidContentError, match="unprocessable"): + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate( + spec=BaseDeploymentData( + name="my_deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_test_"}, + ), + ), + ) + + +# --------------------------------------------------------------------------- +# Test Coverage Gap #2: create — unsupported deployment type rejection +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_create_rejects_unsupported_deployment_type(monkeypatch): + """Create raises DeploymentSupportError for non-AGENT deployment types.""" + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + # Simulate a deployment type that is not in SUPPORTED_ADAPTER_DEPLOYMENT_TYPES + # by creating a spec with AGENT type but monkeypatching the check. + spec = BaseDeploymentData( + name="my_deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_test_"}, + ) + # Override the type to a fake unsupported value after construction + monkeypatch.setattr(spec, "type", SimpleNamespace(value="unsupported_type")) + + with pytest.raises(DeploymentSupportError, match="not supported"): + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate(spec=spec), + ) + + +# --------------------------------------------------------------------------- +# Test Coverage Gap #3: update — empty provider_data + no spec → InvalidContentError +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_update_rejects_empty_provider_data_with_no_spec_changes(monkeypatch): + """Update raises InvalidContentError when provider_data is None and spec produces no update fields.""" + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + fake_clients = SimpleNamespace( + agent=FakeAgentClient({"id": "dep-1", "tools": ["tool-1"]}), + tool=FakeToolClient([{"id": "tool-1", "binding": {}}]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + # Construct a payload that bypasses Pydantic validation to reach the guard + payload = DeploymentUpdate.model_construct( + spec=None, + snapshot=None, + config=None, + provider_data=None, + ) + + with pytest.raises(InvalidContentError, match="provider_data is required"): + await service.update( + user_id="user-1", + deployment_id="dep-1", + payload=payload, + db=object(), + ) + + +# --------------------------------------------------------------------------- +# Test Coverage Gap #4: extract_langflow_artifact_from_zip — all error paths +# --------------------------------------------------------------------------- + + +def test_extract_langflow_artifact_from_zip_success(): + """extract_langflow_artifact_from_zip returns parsed JSON from a valid zip.""" + import json + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + extract_langflow_artifact_from_zip, + ) + + flow_data = {"name": "test_flow", "nodes": []} + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("flow.json", json.dumps(flow_data)) + result = extract_langflow_artifact_from_zip(buf.getvalue(), snapshot_id="snap-1") + assert result == flow_data + + +def test_extract_langflow_artifact_from_zip_no_json(): + """extract_langflow_artifact_from_zip raises InvalidContentError when no JSON in zip.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + extract_langflow_artifact_from_zip, + ) + + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("readme.txt", "hello") + with pytest.raises(InvalidContentError, match="does not include a flow JSON"): + extract_langflow_artifact_from_zip(buf.getvalue(), snapshot_id="snap-1") + + +def test_extract_langflow_artifact_from_zip_bad_zip(): + """extract_langflow_artifact_from_zip raises InvalidContentError for invalid zip data.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + extract_langflow_artifact_from_zip, + ) + + with pytest.raises(InvalidContentError, match="not a valid zip"): + extract_langflow_artifact_from_zip(b"not a zip file", snapshot_id="snap-1") + + +def test_extract_langflow_artifact_from_zip_invalid_utf8(): + """extract_langflow_artifact_from_zip raises InvalidContentError for non-UTF-8 content.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + extract_langflow_artifact_from_zip, + ) + + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("flow.json", b"\xff\xfe invalid utf-8") + with pytest.raises(InvalidContentError, match="not valid UTF-8"): + extract_langflow_artifact_from_zip(buf.getvalue(), snapshot_id="snap-1") + + +def test_extract_langflow_artifact_from_zip_invalid_json(): + """extract_langflow_artifact_from_zip raises InvalidContentError for malformed JSON.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import ( + extract_langflow_artifact_from_zip, + ) + + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + zf.writestr("flow.json", "not valid json {{{") + with pytest.raises(InvalidContentError, match="invalid JSON"): + extract_langflow_artifact_from_zip(buf.getvalue(), snapshot_id="snap-1") + + +# --------------------------------------------------------------------------- +# Test Coverage Gap #5: validate_connection — all negative paths +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_validate_connection_missing_connection(): + """validate_connection raises InvalidContentError when connection not found.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import validate_connection + + connections_client = FakeConnectionsClient() # no existing connections + + with pytest.raises(InvalidContentError, match="not found"): + await validate_connection(connections_client, app_id="missing_app") + + +@pytest.mark.anyio +async def test_validate_connection_missing_config(monkeypatch): + """validate_connection raises InvalidContentError when config not found.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import validate_connection + + connections_client = FakeConnectionsClient(existing_app_id="my_app") + + def get_config_none(app_id, env): # noqa: ARG001 + return None + + monkeypatch.setattr(connections_client, "get_config", get_config_none) + + with pytest.raises(InvalidContentError, match="missing draft config"): + await validate_connection(connections_client, app_id="my_app") + + +@pytest.mark.anyio +async def test_validate_connection_wrong_security_scheme(monkeypatch): + """validate_connection raises InvalidContentError for non-key-value security scheme.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import validate_connection + + connections_client = FakeConnectionsClient(existing_app_id="my_app") + + def get_config_wrong_scheme(app_id, env): # noqa: ARG001 + return SimpleNamespace(security_scheme="oauth2") + + monkeypatch.setattr(connections_client, "get_config", get_config_wrong_scheme) + + with pytest.raises(InvalidContentError, match="key-value credentials"): + await validate_connection(connections_client, app_id="my_app") + + +@pytest.mark.anyio +async def test_validate_connection_missing_credentials(monkeypatch): + """validate_connection raises InvalidContentError when credentials are missing.""" + from ibm_watsonx_orchestrate_core.types.connections import ConnectionSecurityScheme + from langflow.services.adapters.deployment.watsonx_orchestrate.core.config import validate_connection + + connections_client = FakeConnectionsClient(existing_app_id="my_app") + + def get_config_ok(app_id, env): # noqa: ARG001 + return SimpleNamespace(security_scheme=ConnectionSecurityScheme.KEY_VALUE) + + def get_credentials_none(app_id, env, *, use_app_credentials): # noqa: ARG001 + return None + + monkeypatch.setattr(connections_client, "get_config", get_config_ok) + monkeypatch.setattr(connections_client, "get_credentials", get_credentials_none) + + with pytest.raises(InvalidContentError, match="missing draft runtime credentials"): + await validate_connection(connections_client, app_id="my_app") + + +# --------------------------------------------------------------------------- +# Additional coverage: create_agent_run_result raises on missing run_id +# --------------------------------------------------------------------------- + + +def test_create_agent_run_result_raises_on_missing_run_id(): + """create_agent_run_result raises DeploymentError when response has no run_id.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import create_agent_run_result + + with pytest.raises(DeploymentError, match="did not return a run_id"): + create_agent_run_result({"status": "accepted"}) + + +def test_create_agent_run_result_extracts_run_id(): + """create_agent_run_result successfully extracts run_id from response.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import create_agent_run_result + + result = create_agent_run_result({"status": "accepted", "run_id": "run-123"}) + assert result["run_id"] == "run-123" + assert result["status"] == "accepted" + + +def test_create_agent_run_result_falls_back_to_id_field(): + """create_agent_run_result uses 'id' field when 'run_id' is absent.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.core.execution import create_agent_run_result + + result = create_agent_run_result({"status": "running", "id": "id-456"}) + assert result["run_id"] == "id-456" + + +# --------------------------------------------------------------------------- +# Additional coverage: _require_single_deployment_id — multiple IDs +# --------------------------------------------------------------------------- + + +def test_require_single_deployment_id_rejects_multiple_ids(): + """_require_single_deployment_id raises InvalidContentError for multiple IDs.""" + from langflow.services.adapters.deployment.watsonx_orchestrate.utils import _require_single_deployment_id + + params = ConfigListParams(deployment_ids=["id-1", "id-2"]) + with pytest.raises(InvalidContentError, match="exactly one deployment_id"): + _require_single_deployment_id(params, resource_label="config") + + +# --------------------------------------------------------------------------- +# Additional coverage: exception chain preserved (from exc, not from None) +# --------------------------------------------------------------------------- + + +@pytest.mark.anyio +async def test_create_preserves_exception_chain_on_unexpected_error(monkeypatch): + """Create preserves exception chain with 'from exc' instead of 'from None'.""" + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + original_error = RuntimeError("unexpected db error") + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + raise original_error + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentError) as exc_info: + await service.create( + user_id="user-1", + db=object(), + payload=DeploymentCreate( + spec=BaseDeploymentData( + name="my_deployment", + description="desc", + type=DeploymentType.AGENT, + provider_spec={"resource_name_prefix": "lf_test_"}, + ), + ), + ) + + assert exc_info.value.__cause__ is original_error + + +@pytest.mark.anyio +async def test_delete_preserves_exception_chain_on_unexpected_error(monkeypatch): + """Delete preserves exception chain with 'from exc' instead of 'from None'.""" + service = WatsonxOrchestrateDeploymentService(DummySettingsService()) + + original_error = RuntimeError("unexpected error") + + fake_agent = FakeAgentClient({"id": "dep-1", "tools": []}) + fake_agent.delete = lambda _deployment_id: (_ for _ in ()).throw(original_error) + + fake_clients = SimpleNamespace( + agent=fake_agent, + tool=FakeToolClient([]), + connections=FakeConnectionsClient(), + ) + + async def mock_get_provider_clients(*, user_id, db): # noqa: ARG001 + return fake_clients + + monkeypatch.setattr(service, "_get_provider_clients", mock_get_provider_clients) + + with pytest.raises(DeploymentError) as exc_info: + await service.delete( + user_id="user-1", + deployment_id="dep-1", + db=object(), + ) + + assert exc_info.value.__cause__ is original_error + + +# --------------------------------------------------------------------------- +# Additional coverage: _ensure_dict logs warning on non-dict replacement +# --------------------------------------------------------------------------- + + +def test_ensure_dict_logs_warning_on_non_dict(caplog): + """_ensure_dict logs a warning when replacing a non-dict value.""" + import logging + + from langflow.services.adapters.deployment.watsonx_orchestrate.core.tools import _ensure_dict + + parent = {"binding": "not a dict"} + with caplog.at_level(logging.WARNING): + result = _ensure_dict(parent, "binding") + assert result == {} + assert parent["binding"] == {} + assert "Expected dict" in caplog.text + assert "str" in caplog.text diff --git a/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate_update_schema.py b/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate_update_schema.py new file mode 100644 index 0000000000..b27aac87ce --- /dev/null +++ b/src/backend/tests/unit/services/deployment/test_watsonx_orchestrate_update_schema.py @@ -0,0 +1,249 @@ +"""Unit tests for Watsonx deployment update provider schema.""" + +from __future__ import annotations + +from uuid import UUID + +import pytest + +try: + import langflow.services.adapters.deployment.watsonx_orchestrate # noqa: F401 +except ModuleNotFoundError: + pytest.skip( + "Skipping Watsonx deployment tests: optional IBM SDK dependencies not available.", + allow_module_level=True, + ) + +from langflow.services.adapters.deployment.watsonx_orchestrate.payloads import WatsonxDeploymentUpdatePayload +from langflow.services.adapters.deployment.watsonx_orchestrate.service import WatsonxOrchestrateDeploymentService +from lfx.services.adapters.payload import AdapterPayloadValidationError + + +def _raw_tool(name: str, suffix: int) -> dict: + return { + "id": str(UUID(int=suffix)), + "name": name, + "description": "desc", + "data": {"nodes": [], "edges": []}, + "tags": [], + "provider_data": {"project_id": "project-1"}, + } + + +def _raw_connection(app_id: str) -> dict: + return { + "app_id": app_id, + "environment_variables": { + "API_KEY": {"source": "raw", "value": "secret"}, + }, + } + + +def test_payload_schema_slot_registered_for_deployment_update() -> None: + slot = WatsonxOrchestrateDeploymentService.payload_schemas + assert slot is not None + assert slot.deployment_update is not None + assert slot.deployment_update.adapter_model is WatsonxDeploymentUpdatePayload + + +def test_update_schema_accepts_raw_tool_pool_and_shared_connection_refs() -> None: + slot = WatsonxOrchestrateDeploymentService.payload_schemas + assert slot is not None + assert slot.deployment_update is not None + + payload = { + "resource_name_prefix": "lf_pref_", + "tools": { + "existing_ids": ["tool-existing-1"], + "raw_payloads": [_raw_tool("tool-new-1", 1)], + }, + "connections": { + "existing_app_ids": ["app-existing-1"], + "raw_payloads": [_raw_connection("app-new-1")], + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "tool-new-1"}, + "app_ids": ["app-new-1"], + }, + { + "op": "bind", + "tool": {"reference_id": "tool-existing-1"}, + "app_ids": ["app-existing-1"], + }, + ], + } + + applied = slot.deployment_update.apply(payload) + assert applied["operations"][0]["tool"]["name_of_raw"] == "tool-new-1" + assert applied["operations"][1]["tool"]["reference_id"] == "tool-existing-1" + + +def test_update_schema_rejects_prefixed_app_id_collisions() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "resource_name_prefix": "lf_", + "tools": {"existing_ids": ["tool-existing-1"]}, + "connections": { + "existing_app_ids": ["dup"], + "raw_payloads": [_raw_connection("dup")], + }, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-existing-1"}, + "app_ids": ["dup"], + } + ], + } + ) + assert "collides with raw app ids" in str(exc.value.error) + + +def test_update_schema_rejects_missing_raw_tool_reference() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "connections": { + "existing_app_ids": ["app-existing-1"], + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "missing-tool"}, + "app_ids": ["app-existing-1"], + } + ], + } + ) + assert "name_of_raw not found" in str(exc.value.error) + + +def test_update_schema_rejects_reference_id_when_existing_ids_not_declared() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "connections": {"existing_app_ids": ["app-existing-1"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-existing-1"}, + "app_ids": ["app-existing-1"], + } + ], + } + ) + assert "reference_id not found in tools.existing_ids" in str(exc.value.error) + + +def test_update_schema_rejects_bind_app_id_not_in_declared_pools() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "tools": { + "raw_payloads": [_raw_tool("tool-new-1", 2)], + }, + "connections": { + "existing_app_ids": ["app-existing-1"], + }, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "tool-new-1"}, + "app_ids": ["app-not-declared"], + } + ], + } + ) + assert "operation app_ids must be declared in" in str(exc.value.error) + + +def test_update_schema_rejects_unused_existing_app_ids() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "tools": {"existing_ids": ["tool-existing-1"]}, + "connections": {"existing_app_ids": ["app-existing-1", "app-unused"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-existing-1"}, + "app_ids": ["app-existing-1"], + } + ], + } + ) + assert "existing_app_ids contains ids not referenced by operations" in str(exc.value.error) + + +def test_update_schema_rejects_unused_raw_connection_app_ids() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "resource_name_prefix": "lf_pref_", + "tools": {"raw_payloads": [_raw_tool("tool-new-1", 3)]}, + "connections": {"raw_payloads": [_raw_connection("app-new-1"), _raw_connection("app-unused")]}, + "operations": [ + { + "op": "bind", + "tool": {"name_of_raw": "tool-new-1"}, + "app_ids": ["app-new-1"], + } + ], + } + ) + assert "raw_payloads contains app_id values not referenced by operations" in str(exc.value.error) + + +def test_update_schema_dedupes_bind_app_ids() -> None: + slot = WatsonxOrchestrateDeploymentService.payload_schemas + assert slot is not None + assert slot.deployment_update is not None + + payload = { + "tools": {"existing_ids": ["tool-existing-1"]}, + "connections": {"existing_app_ids": ["app-existing-1", "app-existing-2"]}, + "operations": [ + { + "op": "bind", + "tool": {"reference_id": "tool-existing-1"}, + "app_ids": ["app-existing-1", "app-existing-1", "app-existing-2"], + } + ], + } + + applied = slot.deployment_update.apply(payload) + assert applied["operations"][0]["app_ids"] == ["app-existing-1", "app-existing-2"] + + +def test_update_schema_requires_unbind_app_ids() -> None: + with pytest.raises(AdapterPayloadValidationError): + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "operations": [ + { + "op": "unbind", + "tool_id": "tool-1", + } + ] + } + ) + + +def test_update_schema_rejects_unbind_raw_app_ids() -> None: + with pytest.raises(AdapterPayloadValidationError) as exc: + WatsonxOrchestrateDeploymentService.payload_schemas.deployment_update.apply( # type: ignore[union-attr] + { + "connections": {"raw_payloads": [_raw_connection("app-new-1")]}, + "operations": [ + { + "op": "unbind", + "tool_id": "tool-1", + "app_ids": ["app-new-1"], + } + ], + } + ) + assert "must reference connections.existing_app_ids only" in str(exc.value.error) diff --git a/src/lfx/src/lfx/_assets/component_index.json b/src/lfx/src/lfx/_assets/component_index.json index 2388627d45..79c55ce20e 100644 --- a/src/lfx/src/lfx/_assets/component_index.json +++ b/src/lfx/src/lfx/_assets/component_index.json @@ -118539,6 +118539,6 @@ "num_components": 359, "num_modules": 97 }, - "sha256": "13f65a29f6c2ff9456219241d124739b669f2d58199107d961e515d1da6a73e4", - "version": "0.3.1" + "sha256": "594e0427eb317645ab26b635d9910dab39775c531fe06fe1e225f60f18e20a5e", + "version": "0.4.0" } \ No newline at end of file diff --git a/src/lfx/src/lfx/services/adapters/deployment/exceptions.py b/src/lfx/src/lfx/services/adapters/deployment/exceptions.py index 449bb2d89d..8c23c20873 100644 --- a/src/lfx/src/lfx/services/adapters/deployment/exceptions.py +++ b/src/lfx/src/lfx/services/adapters/deployment/exceptions.py @@ -6,12 +6,17 @@ regardless of adapter. from __future__ import annotations +from typing import NoReturn + +from fastapi import status + class DeploymentServiceError(Exception): """Root exception for all deployment service failures. Both operational errors (:class:`DeploymentError`) and authentication - errors (:class:`AuthenticationError`) inherit from this class. + or authorization errors (:class:`AuthenticationError`, :class:`AuthorizationError`) + inherit from this class. Catch this only when you truly want to handle *every* deployment-related failure uniformly. """ @@ -41,6 +46,18 @@ class AuthenticationError(DeploymentServiceError): """ +class AuthorizationError(DeploymentServiceError): + """Base exception for authorization failures (authenticated but forbidden). + + Intentionally a sibling of :class:`DeploymentError`, not a child. + This ensures ``except DeploymentError`` will not accidentally swallow + authorization failures, allowing API layers to return status 403 + separately from deployment errors (400/404/409/422). + + Please ensure that the message does not leak sensitive information. + """ + + class CredentialResolutionError(AuthenticationError): """Raised when credentials resolution fails. @@ -84,6 +101,27 @@ class DeploymentSupportError(DeploymentError): super().__init__(message, error_code="unsupported_deployment_type", cause=cause) +class RateLimitError(DeploymentError): + """Raised when provider/API rate limits deployment operations.""" + + def __init__(self, message: str = "Deployment operation rate limited", *, cause: Exception | None = None): + super().__init__(message, error_code="deployment_rate_limited", cause=cause) + + +class DeploymentTimeoutError(DeploymentError): + """Raised when deployment operation times out.""" + + def __init__(self, message: str = "Deployment operation timed out", *, cause: Exception | None = None): + super().__init__(message, error_code="deployment_timeout", cause=cause) + + +class ServiceUnavailableError(DeploymentError): + """Raised when upstream provider is temporarily unavailable.""" + + def __init__(self, message: str = "Deployment provider is unavailable", *, cause: Exception | None = None): + super().__init__(message, error_code="deployment_provider_unavailable", cause=cause) + + class AuthSchemeError(AuthenticationError): """Raised when no matching authentication scheme was found. @@ -101,7 +139,25 @@ class InvalidDeploymentTypeError(DeploymentError): super().__init__(message, error_code="invalid_deployment_type", cause=cause) -class DeploymentNotFoundError(DeploymentError): +class ResourceNotFoundError(DeploymentError): + """Raised when a requested deployment-related resource is not found.""" + + def __init__( + self, + message: str | None = None, + *, + resource_id: str | None = None, + cause: Exception | None = None, + ): + self.resource_id = resource_id + if message is None: + message = "Resource not found" + if resource_id: + message = f"Resource not found: {resource_id}" + super().__init__(message, error_code="resource_not_found", cause=cause) + + +class DeploymentNotFoundError(ResourceNotFoundError): """Raised when a deployment is not found.""" def __init__( @@ -116,7 +172,25 @@ class DeploymentNotFoundError(DeploymentError): message = "Deployment not found" if deployment_id: message = f"Deployment not found: {deployment_id}" - super().__init__(message, error_code="deployment_not_found", cause=cause) + super().__init__(message=message, resource_id=deployment_id, cause=cause) + self.error_code = "deployment_not_found" + + +class OperationNotSupportedError(DeploymentError): + """Raised when a deployment operation is not supported by the current adapter. + + Use this for operations that are valid in the abstract contract but + are not implemented by the active adapter (e.g. ``redeploy`` or + ``duplicate`` on a provider that does not support them). + """ + + def __init__( + self, + message: str = "This operation is not supported by the current deployment adapter.", + *, + cause: Exception | None = None, + ): + super().__init__(message, error_code="operation_not_supported", cause=cause) class DeploymentNotConfiguredError(DeploymentError): @@ -142,3 +216,74 @@ class DeploymentNotConfiguredError(DeploymentError): "Register one to enable deployment operations." ) super().__init__(message, error_code="deployment_not_configured", cause=cause) + + +# lru+ttl cache? +def raise_for_status_and_detail( + *, + status_code: int | None, + detail: str, + message_prefix: str | None = None, +) -> NoReturn: + """Raise domain-specific deployment exceptions based on HTTP-like status/detail.""" + detail_text = str(detail) + detail_lower = detail_text.lower() + prefix = str(message_prefix or "").strip() + message = f"{prefix} error details: {detail_text}" if prefix else detail_text + + if status_code == status.HTTP_401_UNAUTHORIZED: + raise AuthenticationError(message=message, error_code="authentication_error") from None + if status_code == status.HTTP_403_FORBIDDEN: + raise AuthorizationError(message=message, error_code="authorization_error") from None + if status_code == status.HTTP_422_UNPROCESSABLE_CONTENT: + raise InvalidContentError(message=message) from None + if status_code == status.HTTP_400_BAD_REQUEST: + raise InvalidDeploymentOperationError(message=message) from None + if status_code == status.HTTP_405_METHOD_NOT_ALLOWED: + raise InvalidDeploymentOperationError(message=message) from None + if status_code == status.HTTP_413_REQUEST_ENTITY_TOO_LARGE: + raise InvalidContentError(message=message) from None + if status_code == status.HTTP_415_UNSUPPORTED_MEDIA_TYPE: + raise InvalidContentError(message=message) from None + if status_code == status.HTTP_404_NOT_FOUND: + raise DeploymentNotFoundError(message) from None + if status_code == status.HTTP_410_GONE: + raise DeploymentNotFoundError(message) from None + if status_code == status.HTTP_409_CONFLICT: + raise DeploymentConflictError(message=message) from None + if status_code == status.HTTP_429_TOO_MANY_REQUESTS: + raise RateLimitError(message=message) from None + if status_code in {status.HTTP_408_REQUEST_TIMEOUT, status.HTTP_504_GATEWAY_TIMEOUT}: + raise DeploymentTimeoutError(message=message) from None + if status_code in {status.HTTP_502_BAD_GATEWAY, status.HTTP_503_SERVICE_UNAVAILABLE}: + raise ServiceUnavailableError(message=message) from None + if "not found" in detail_lower: + raise DeploymentNotFoundError(message) from None + if "already exists" in detail_lower or "conflict" in detail_lower: + raise DeploymentConflictError(message=message) from None + if "unprocessable" in detail_lower: + raise InvalidContentError(message=message) from None + if "too many requests" in detail_lower or "rate limit" in detail_lower: + raise RateLimitError(message=message) from None + if "timed out" in detail_lower or "timeout" in detail_lower: + raise DeploymentTimeoutError(message=message) from None + if ( + "service unavailable" in detail_lower + or "temporarily unavailable" in detail_lower + or "bad gateway" in detail_lower + ): + raise ServiceUnavailableError(message=message) from None + if "unauthorized" in detail_lower or "authentication" in detail_lower: + raise AuthenticationError(message=message, error_code="authentication_error") from None + if "forbidden" in detail_lower or "permission" in detail_lower or "not allowed" in detail_lower: + raise AuthorizationError(message=message, error_code="authorization_error") from None + if "bad request" in detail_lower: + raise InvalidDeploymentOperationError(message=message) from None + if ( + "invalid" in detail_lower + or "missing" in detail_lower + or "required" in detail_lower + or "malformed" in detail_lower + ): + raise InvalidContentError(message=message) from None + raise DeploymentError(message=message, error_code="deployment_error") from None diff --git a/src/lfx/tests/unit/services/deployment/test_deployment_exceptions.py b/src/lfx/tests/unit/services/deployment/test_deployment_exceptions.py index 3525c84a51..6964c9da89 100644 --- a/src/lfx/tests/unit/services/deployment/test_deployment_exceptions.py +++ b/src/lfx/tests/unit/services/deployment/test_deployment_exceptions.py @@ -10,14 +10,21 @@ from lfx.services.adapters.deployment import ( ) from lfx.services.adapters.deployment.exceptions import ( AuthenticationError, + AuthorizationError, AuthSchemeError, CredentialResolutionError, DeploymentConflictError, DeploymentNotFoundError, DeploymentSupportError, + DeploymentTimeoutError, InvalidContentError, InvalidDeploymentOperationError, InvalidDeploymentTypeError, + OperationNotSupportedError, + RateLimitError, + ResourceNotFoundError, + ServiceUnavailableError, + raise_for_status_and_detail, ) from lfx.services.interfaces import DeploymentServiceProtocol @@ -26,10 +33,13 @@ def test_exception_hierarchy_is_preserved() -> None: # DeploymentServiceError is the common root assert issubclass(DeploymentError, DeploymentServiceError) assert issubclass(AuthenticationError, DeploymentServiceError) + assert issubclass(AuthorizationError, DeploymentServiceError) # AuthenticationError is a sibling of DeploymentError, NOT a child assert not issubclass(AuthenticationError, DeploymentError) + assert not issubclass(AuthorizationError, DeploymentError) assert not issubclass(DeploymentError, AuthenticationError) + assert not issubclass(DeploymentError, AuthorizationError) # Auth subtypes assert issubclass(CredentialResolutionError, AuthenticationError) @@ -39,18 +49,27 @@ def test_exception_hierarchy_is_preserved() -> None: assert issubclass(DeploymentConflictError, DeploymentError) assert issubclass(InvalidContentError, DeploymentError) assert issubclass(InvalidDeploymentOperationError, DeploymentError) + assert issubclass(ResourceNotFoundError, DeploymentError) + assert issubclass(DeploymentNotFoundError, ResourceNotFoundError) assert issubclass(DeploymentNotConfiguredError, DeploymentError) + assert issubclass(OperationNotSupportedError, DeploymentError) def test_exception_error_codes_are_set() -> None: + assert AuthorizationError("forbidden", error_code="authorization_error").error_code == "authorization_error" assert CredentialResolutionError().error_code == "credentials_resolution_error" assert DeploymentConflictError().error_code == "deployment_conflict" + assert RateLimitError().error_code == "deployment_rate_limited" + assert DeploymentTimeoutError().error_code == "deployment_timeout" + assert ServiceUnavailableError().error_code == "deployment_provider_unavailable" assert DeploymentSupportError().error_code == "unsupported_deployment_type" + assert ResourceNotFoundError().error_code == "resource_not_found" assert InvalidDeploymentTypeError().error_code == "invalid_deployment_type" assert InvalidContentError().error_code == "unprocessable_content_error" assert InvalidDeploymentOperationError().error_code == "invalid_deployment_operation" assert AuthSchemeError().error_code == "unsupported_auth_type" assert DeploymentNotConfiguredError().error_code == "deployment_not_configured" + assert OperationNotSupportedError().error_code == "operation_not_supported" def test_deployment_type_exceptions_have_distinct_default_messages() -> None: @@ -71,6 +90,13 @@ def test_deployment_not_found_default_message() -> None: assert err.deployment_id is None +def test_resource_not_found_defaults() -> None: + err = ResourceNotFoundError(resource_id="r1") + assert str(err) == "Resource not found: r1" + assert err.resource_id == "r1" + assert err.error_code == "resource_not_found" + + def test_deployment_not_found_preserves_deployment_id_with_custom_message() -> None: err = DeploymentNotFoundError(message="Custom error", deployment_id="dep_1") assert str(err) == "Custom error" @@ -156,6 +182,20 @@ def test_auth_errors_not_caught_by_deployment_error() -> None: raise_auth_error() +def test_authorization_errors_not_caught_by_deployment_error() -> None: + """Ensure except DeploymentError does NOT catch authorization failures.""" + + def raise_authorization_error() -> None: + error_message = "forbidden" + try: + raise AuthorizationError(error_message, error_code="authorization_error") + except DeploymentError: + pytest.fail("DeploymentError should not catch AuthorizationError") + + with pytest.raises(AuthorizationError): + raise_authorization_error() + + def test_deployment_service_error_catches_both_hierarchies() -> None: """DeploymentServiceError catches both deployment and auth errors.""" with pytest.raises(DeploymentServiceError): @@ -164,6 +204,52 @@ def test_deployment_service_error_catches_both_hierarchies() -> None: raise DeploymentNotFoundError +def test_raise_for_status_and_detail_maps_known_http_statuses() -> None: + with pytest.raises(AuthenticationError): + raise_for_status_and_detail(status_code=401, detail="unauthorized", message_prefix="x") + with pytest.raises(AuthorizationError): + raise_for_status_and_detail(status_code=403, detail="forbidden", message_prefix="x") + with pytest.raises(DeploymentNotFoundError): + raise_for_status_and_detail(status_code=404, detail="missing", message_prefix="x") + with pytest.raises(DeploymentConflictError): + raise_for_status_and_detail(status_code=409, detail="conflict", message_prefix="x") + with pytest.raises(InvalidContentError): + raise_for_status_and_detail(status_code=422, detail="unprocessable", message_prefix="x") + with pytest.raises(InvalidDeploymentOperationError): + raise_for_status_and_detail(status_code=400, detail="bad request", message_prefix="x") + with pytest.raises(InvalidDeploymentOperationError): + raise_for_status_and_detail(status_code=405, detail="method not allowed", message_prefix="x") + with pytest.raises(InvalidContentError): + raise_for_status_and_detail(status_code=413, detail="payload too large", message_prefix="x") + with pytest.raises(InvalidContentError): + raise_for_status_and_detail(status_code=415, detail="unsupported media type", message_prefix="x") + with pytest.raises(DeploymentNotFoundError): + raise_for_status_and_detail(status_code=410, detail="gone", message_prefix="x") + with pytest.raises(RateLimitError): + raise_for_status_and_detail(status_code=429, detail="too many requests", message_prefix="x") + with pytest.raises(DeploymentTimeoutError): + raise_for_status_and_detail(status_code=408, detail="timeout", message_prefix="x") + with pytest.raises(DeploymentTimeoutError): + raise_for_status_and_detail(status_code=504, detail="gateway timeout", message_prefix="x") + with pytest.raises(ServiceUnavailableError): + raise_for_status_and_detail(status_code=502, detail="bad gateway", message_prefix="x") + with pytest.raises(ServiceUnavailableError): + raise_for_status_and_detail(status_code=503, detail="service unavailable", message_prefix="x") + + +def test_raise_for_status_and_detail_uses_detail_heuristics_without_status() -> None: + with pytest.raises(AuthorizationError): + raise_for_status_and_detail(status_code=None, detail="permission denied") + with pytest.raises(InvalidContentError): + raise_for_status_and_detail(status_code=None, detail="invalid payload") + with pytest.raises(RateLimitError): + raise_for_status_and_detail(status_code=None, detail="rate limit exceeded") + with pytest.raises(DeploymentTimeoutError): + raise_for_status_and_detail(status_code=None, detail="request timed out") + with pytest.raises(ServiceUnavailableError): + raise_for_status_and_detail(status_code=None, detail="service unavailable") + + def test_package_exports_base_and_error() -> None: from lfx.services.adapters.deployment import BaseDeploymentService, DeploymentError, DeploymentService diff --git a/uv.lock b/uv.lock index cbb12f7fa7..8df646a7ca 100644 --- a/uv.lock +++ b/uv.lock @@ -4240,6 +4240,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c4/f7/5cc291d701094754a1d327b44d80a44971e13962881d9a400235726171da/hypothesis-6.151.9-py3-none-any.whl", hash = "sha256:7b7220585c67759b1b1ef839b1e6e9e3d82ed468cfc1ece43c67184848d7edd9", size = 529307, upload-time = "2026-02-16T22:59:20.443Z" }, ] +[[package]] +name = "ibm-cloud-sdk-core" +version = "3.24.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyjwt" }, + { name = "python-dateutil" }, + { name = "requests" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4a/0c/965f4f4b34f1a9a2210bec224ac489c631153f18ef75c6e694750efda654/ibm_cloud_sdk_core-3.24.4.tar.gz", hash = "sha256:20498959ce0177a58938e1855ee09f08b9712e43cc3209e95010c046e8c6d2a6", size = 81778, upload-time = "2026-02-03T08:44:42.217Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/8d/90cf967468fd81f9fd22bd42dbf3b1fbb1234485d1f81b6f17c8de138fff/ibm_cloud_sdk_core-3.24.4-py3-none-any.whl", hash = "sha256:4c3487ed5eb5180770ea2d07d95cfed1e69002ed249bbbc7d155226985c0b785", size = 75786, upload-time = "2026-02-03T08:44:41.094Z" }, +] + [[package]] name = "ibm-cos-sdk" version = "2.14.3" @@ -4333,6 +4348,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/99/89/f6a113d08bd8796c8ed82b151009479fe7e8a7b4792a12c98918b806c0e6/ibm_watsonx_ai-1.5.3-py3-none-any.whl", hash = "sha256:a7a1af3bebd8271e0fb7a04b17792131ad7f3e697cd4aea0aae0687695cb884f", size = 1080316, upload-time = "2026-02-26T09:18:01.778Z" }, ] +[[package]] +name = "ibm-watsonx-orchestrate-clients" +version = "2.3.2.dev5117" +source = { registry = "https://test.pypi.org/simple" } +dependencies = [ + { name = "ibm-cloud-sdk-core", marker = "python_full_version >= '3.11'" }, + { name = "ibm-watsonx-orchestrate-core", marker = "python_full_version >= '3.11'" }, +] +sdist = { url = "https://test-files.pythonhosted.org/packages/77/95/f202c3578199261afa69d525c7c8212fa2d328fe8ef189deac426685c6a4/ibm_watsonx_orchestrate_clients-2.3.2.dev5117.tar.gz", hash = "sha256:c0d1ce2831dc54ad578afd8fd6f586b160fc5ee9e6bfdfc0b7938c8a1ec0aa16", size = 1255, upload-time = "2026-02-11T13:47:57.775Z" } +wheels = [ + { url = "https://test-files.pythonhosted.org/packages/33/11/278075261cdaee932d4844a7ab0a422cdccb055c205b6a9043830d17c43c/ibm_watsonx_orchestrate_clients-2.3.2.dev5117-py3-none-any.whl", hash = "sha256:1248b6f0118043e4021fc9b759adc23a376e8ad806cc0f13f6aecb51b312914b", size = 22247, upload-time = "2026-02-11T13:47:52.677Z" }, +] + +[[package]] +name = "ibm-watsonx-orchestrate-core" +version = "2.3.2.dev5117" +source = { registry = "https://test.pypi.org/simple" } +dependencies = [ + { name = "pydantic", marker = "python_full_version >= '3.11'" }, +] +sdist = { url = "https://test-files.pythonhosted.org/packages/f0/93/37d2a3b8374e0a8c9550c7b22574659ed6f871ebd9db1719c6952894ffac/ibm_watsonx_orchestrate_core-2.3.2.dev5117.tar.gz", hash = "sha256:4a2cdce1fc762839ce3502bbae2e3236b80bf53341622056b307b8137a5da4b9", size = 1195, upload-time = "2026-02-11T13:47:58.316Z" } +wheels = [ + { url = "https://test-files.pythonhosted.org/packages/79/9e/e6cea9daf819637807d722f3c00513dd34d31efe50481669928615298167/ibm_watsonx_orchestrate_core-2.3.2.dev5117-py3-none-any.whl", hash = "sha256:5bd64699ae3dad05683bffe2438e38dcffd2b58d2270893d5a4ac3f177b58339", size = 25175, upload-time = "2026-02-11T13:47:53.72Z" }, +] + [[package]] name = "identify" version = "2.6.17" @@ -5873,6 +5913,9 @@ all = [ { name = "google-search-results" }, { name = "graph-retriever" }, { name = "huggingface-hub", extra = ["inference"] }, + { name = "ibm-cloud-sdk-core" }, + { name = "ibm-watsonx-orchestrate-clients", marker = "python_full_version >= '3.11'" }, + { name = "ibm-watsonx-orchestrate-core", marker = "python_full_version >= '3.11'" }, { name = "jigsawstack" }, { name = "json-repair" }, { name = "kubernetes" }, @@ -5927,10 +5970,12 @@ all = [ { name = "pypdf" }, { name = "python-docx" }, { name = "pytube" }, + { name = "pyyaml" }, { name = "qdrant-client" }, { name = "qianfan" }, { name = "ragstack-ai-knowledge-store" }, { name = "redis" }, + { name = "rich" }, { name = "scrapegraph-py" }, { name = "smolagents" }, { name = "spider-client" }, @@ -6039,6 +6084,9 @@ complete = [ { name = "google-search-results" }, { name = "graph-retriever" }, { name = "huggingface-hub", extra = ["inference"] }, + { name = "ibm-cloud-sdk-core" }, + { name = "ibm-watsonx-orchestrate-clients", marker = "python_full_version >= '3.11'" }, + { name = "ibm-watsonx-orchestrate-core", marker = "python_full_version >= '3.11'" }, { name = "jigsawstack" }, { name = "json-repair" }, { name = "kubernetes" }, @@ -6093,10 +6141,12 @@ complete = [ { name = "pypdf" }, { name = "python-docx" }, { name = "pytube" }, + { name = "pyyaml" }, { name = "qdrant-client" }, { name = "qianfan" }, { name = "ragstack-ai-knowledge-store" }, { name = "redis" }, + { name = "rich" }, { name = "scrapegraph-py" }, { name = "smolagents" }, { name = "spider-client" }, @@ -6190,6 +6240,13 @@ groq = [ huggingface = [ { name = "huggingface-hub", extra = ["inference"] }, ] +ibm-watsonx = [ + { name = "ibm-cloud-sdk-core" }, + { name = "ibm-watsonx-orchestrate-clients", marker = "python_full_version >= '3.11'" }, + { name = "ibm-watsonx-orchestrate-core", marker = "python_full_version >= '3.11'" }, + { name = "pyyaml" }, + { name = "rich" }, +] jigsawstack = [ { name = "jigsawstack" }, ] @@ -6490,7 +6547,10 @@ requires-dist = [ { name = "gunicorn", specifier = ">=22.0.0,<23.0.0" }, { name = "httpx", extras = ["http2"], specifier = ">=0.27,<1.0.0" }, { name = "huggingface-hub", extras = ["inference"], marker = "extra == 'huggingface'", specifier = ">=0.23.2,<1.0.0" }, + { name = "ibm-cloud-sdk-core", marker = "extra == 'ibm-watsonx'", specifier = "~=3.24.4" }, { name = "ibm-watsonx-ai", specifier = ">=1.3.1,<2.0.0" }, + { name = "ibm-watsonx-orchestrate-clients", marker = "python_full_version >= '3.11' and extra == 'ibm-watsonx'", specifier = "==2.3.2.dev5117", index = "https://test.pypi.org/simple" }, + { name = "ibm-watsonx-orchestrate-core", marker = "python_full_version >= '3.11' and extra == 'ibm-watsonx'", specifier = "==2.3.2.dev5117", index = "https://test.pypi.org/simple" }, { name = "jaraco-context", specifier = ">=6.1.0" }, { name = "jigsawstack", marker = "extra == 'jigsawstack'", specifier = "==0.2.7" }, { name = "jq", marker = "sys_platform != 'win32'", specifier = ">=1.7.0,<2.0.0" }, @@ -6569,6 +6629,7 @@ requires-dist = [ { name = "langflow-base", extras = ["graph-retriever"], marker = "extra == 'complete'", editable = "src/backend/base" }, { name = "langflow-base", extras = ["groq"], marker = "extra == 'complete'", editable = "src/backend/base" }, { name = "langflow-base", extras = ["huggingface"], marker = "extra == 'complete'", editable = "src/backend/base" }, + { name = "langflow-base", extras = ["ibm-watsonx"], marker = "extra == 'complete'", editable = "src/backend/base" }, { name = "langflow-base", extras = ["jigsawstack"], marker = "extra == 'complete'", editable = "src/backend/base" }, { name = "langflow-base", extras = ["json-repair"], marker = "extra == 'complete'", editable = "src/backend/base" }, { name = "langflow-base", extras = ["kubernetes"], marker = "extra == 'complete'", editable = "src/backend/base" }, @@ -6685,11 +6746,13 @@ requires-dist = [ { name = "python-docx", marker = "extra == 'docx'", specifier = ">=1.1.0" }, { name = "python-multipart", specifier = ">=0.0.12,<1.0.0" }, { name = "pytube", marker = "extra == 'pytube'", specifier = "==15.0.0" }, + { name = "pyyaml", marker = "extra == 'ibm-watsonx'" }, { name = "qdrant-client", marker = "extra == 'qdrant'", specifier = "==1.9.2" }, { name = "qianfan", marker = "extra == 'qianfan'", specifier = "==0.3.5" }, { name = "ragstack-ai-knowledge-store", marker = "extra == 'ragstack'", specifier = "==0.2.1" }, { name = "redis", marker = "extra == 'redis'", specifier = ">=5.2.1,<6.0.0" }, { name = "rich", specifier = ">=13.7.0,<14.0.0" }, + { name = "rich", marker = "extra == 'ibm-watsonx'" }, { name = "scipy", specifier = ">=1.15.2,<2.0.0" }, { name = "scrapegraph-py", marker = "extra == 'scrapegraph'", specifier = ">=1.12.0" }, { name = "sentence-transformers", marker = "extra == 'local'", specifier = ">=2.0.0" }, @@ -6724,7 +6787,7 @@ requires-dist = [ { name = "youtube-transcript-api", marker = "extra == 'youtube'", specifier = ">=1.0.0,<2.0.0" }, { name = "zep-python", marker = "extra == 'zep'", specifier = "==2.0.2" }, ] -provides-extras = ["audio", "postgresql", "local", "couchbase", "cassandra", "clickhouse", "mongodb", "supabase", "redis", "elasticsearch", "faiss", "qdrant", "weaviate", "chroma", "upstash", "pinecone", "milvus", "mongodb-vectors", "astradb", "openai", "anthropic", "cohere", "google", "ollama", "nvidia", "mistral", "sambanova", "groq", "llama-cpp", "sentence-transformers", "ctransformers", "langfuse", "langwatch", "langsmith", "arize", "pypdf", "docx", "pytube", "dspy", "smolagents", "beautifulsoup", "serpapi", "google-api", "wikipedia", "fake-useragent", "duckduckgo", "yfinance", "wolframalpha", "pyarrow", "fastavro", "gitpython", "nltk", "lark", "huggingface", "metaphor", "metal", "markup", "aws", "numexpr", "qianfan", "pgvector", "litellm", "zep", "youtube", "markdown", "kubernetes", "json-repair", "astra", "composio", "ragstack", "opensearch", "atlassian", "mem0", "needle", "sseclient", "openinference", "mcp", "ag2", "scrapegraph", "pydantic-ai", "apify", "spider", "altk", "langchain-huggingface", "langchain-unstructured", "graph-retriever", "langchain-mcp-adapters", "opik", "traceloop", "openlayer", "docling", "easyocr", "cleanlab", "twelvelabs", "jigsawstack", "assemblyai", "datasets", "fastparquet", "vlmrun", "cuga", "mlx", "fastmcp", "aioboto3", "astrapy", "gassist", "complete", "all"] +provides-extras = ["audio", "postgresql", "local", "couchbase", "cassandra", "clickhouse", "mongodb", "supabase", "redis", "elasticsearch", "faiss", "qdrant", "weaviate", "chroma", "upstash", "pinecone", "milvus", "mongodb-vectors", "astradb", "openai", "anthropic", "cohere", "google", "ollama", "nvidia", "mistral", "sambanova", "groq", "llama-cpp", "sentence-transformers", "ctransformers", "langfuse", "langwatch", "langsmith", "arize", "pypdf", "docx", "pytube", "dspy", "smolagents", "beautifulsoup", "serpapi", "google-api", "wikipedia", "fake-useragent", "duckduckgo", "yfinance", "wolframalpha", "pyarrow", "fastavro", "gitpython", "nltk", "lark", "huggingface", "metaphor", "metal", "markup", "aws", "numexpr", "qianfan", "pgvector", "litellm", "zep", "youtube", "markdown", "kubernetes", "json-repair", "astra", "composio", "ragstack", "opensearch", "atlassian", "mem0", "needle", "sseclient", "openinference", "mcp", "ag2", "scrapegraph", "pydantic-ai", "apify", "spider", "altk", "langchain-huggingface", "langchain-unstructured", "graph-retriever", "langchain-mcp-adapters", "opik", "traceloop", "openlayer", "docling", "easyocr", "cleanlab", "twelvelabs", "jigsawstack", "assemblyai", "datasets", "fastparquet", "vlmrun", "cuga", "mlx", "fastmcp", "aioboto3", "astrapy", "gassist", "ibm-watsonx", "complete", "all"] [package.metadata.requires-dev] dev = [