mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-24 15:19:46 +08:00
5a8c17b22d
* ci: stop publishing -nightly bundle variants; release bundles on demand (#13206) Bundles (lfx-arxiv, lfx-duckduckgo) change infrequently enough that a nightly cadence is overkill. Drop the rename-to-`-nightly` and bundle publish lanes from the nightly pipeline and add a dedicated workflow_dispatch-only release_bundles.yml for purposeful releases. - nightly_build.yml: drop update_bundle_versions.py invocation and the src/bundles/*/pyproject.toml git-add guard. - release_nightly.yml: remove build-nightly-bundles and publish-nightly-bundles jobs; untether test-cross-platform and publish-nightly-main from them. - release_bundles.yml (new): build all src/bundles/* wheels, run a cross-OS install smoke test, then publish to PyPI under their stable names. Tolerates already-published versions so re-runs without a version bump are no-ops. release.yml still owns bundle publishing as part of main releases. Manual follow-up (PyPI admin): delete the lfx-arxiv-nightly and lfx-duckduckgo-nightly projects from PyPI. * ci(release_bundles): build lfx wheel locally for smoke test (#13207) The release_bundles.yml smoke test installs bundle wheels into a fresh venv, which makes uv resolve their `lfx>=X.Y,<Z` pin against PyPI. When bundles are released ahead of a matching lfx (typical case — bundles ship more often than lfx bumps land on PyPI), the resolve fails: Because only lfx<=0.4.3 is available and lfx-arxiv==0.1.0 depends on lfx>=0.5.0,<0.6.0, we can conclude that lfx-arxiv==0.1.0 cannot be used. Build the lfx wheel from the current ref in the build-bundles job and install it alongside the bundle wheels in the smoke test. The lfx wheel ships as a separate `dist-lfx-smoketest` artifact and is NOT included in the publish-bundles step — only the `dist-bundles` artifact gets pushed to PyPI. * ci(nightly): re-pin bundle lfx deps to lfx-nightly during nightly build (#13208) The nightly pipeline renames the workspace `lfx` package to `lfx-nightly` in `src/lfx/pyproject.toml` and the root workspace dep, but leaves each `src/bundles/*/pyproject.toml`'s `"lfx>=X.Y,<Z"` pin unchanged. With the workspace `lfx` gone and PyPI still on lfx 0.4.3, the create-nightly-tag job's `uv lock` fails: × No solution found when resolving dependencies for split [...] ╰─▶ Because only lfx<=0.4.3 is available and lfx-arxiv depends on lfx>=0.5.0,<0.6.0, we can conclude that lfx-arxiv's requirements are unsatisfiable. `update_lfx_version.py` now also rewrites the `lfx` (or already-rewritten `lfx-nightly`) specifier in every `src/bundles/*/pyproject.toml` to `lfx-nightly==<dev version>`, mirroring how `update_lf_base_dependency` re-pins the lfx dep inside `langflow-base`. The lookahead in the regex prevents matching sibling packages like `lfx-arxiv` / `lfx-duckduckgo`. No-op when no bundle pyprojects exist (e.g. on main today), so this is safe to merge ahead of the bundle extraction landing on main. Restored the guarded `git add src/bundles/*/pyproject.toml` step so the modified bundle pyprojects ride the same commit/tag as the rest of the nightly version bumps. Cherry-pick to release-1.10.0 to unblock nightlies cut from that branch.
412 lines
16 KiB
YAML
412 lines
16 KiB
YAML
name: Nightly Build
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
runs_on:
|
|
description: "Runner to use for tests (use self-hosted for safe/release code)"
|
|
required: false
|
|
type: choice
|
|
options:
|
|
- ubuntu-latest
|
|
- self-hosted
|
|
- '["self-hosted", "linux", "ARM64", "langflow-ai-arm64-40gb-ephemeral"]'
|
|
default: ubuntu-latest
|
|
skip_frontend_tests:
|
|
description: "Skip frontend tests. Only do this for testing purposes."
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
skip_backend_tests:
|
|
description: "Skip backend tests. Only do this for testing purposes."
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
skip_slack:
|
|
description: "Skip slack message. Only do this for testing purposes."
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
push_to_registry:
|
|
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
|
required: false
|
|
type: boolean
|
|
default: true
|
|
schedule:
|
|
# Run job at 00:00 UTC (4:00 PM PST / 5:00 PM PDT)
|
|
- cron: "0 0 * * *"
|
|
|
|
env:
|
|
POETRY_VERSION: "1.8.3"
|
|
PYTHON_VERSION: "3.13"
|
|
|
|
jobs:
|
|
validate-inputs:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Validate inputs
|
|
if: inputs.push_to_registry && (inputs.skip_frontend_tests || inputs.skip_backend_tests)
|
|
run: |
|
|
echo "Cannot skip tests while push_to_registry is true."
|
|
exit 1
|
|
resolve-release-branch:
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
branch: ${{ steps.get_branch.outputs.branch }}
|
|
steps:
|
|
- name: Find latest release branch
|
|
id: get_branch
|
|
run: |
|
|
git ls-remote --heads https://github.com/${{ github.repository }} 'refs/heads/release-*' \
|
|
| awk '{print $2}' \
|
|
| sed 's|refs/heads/||' \
|
|
| grep -E '^release-[0-9]+\.[0-9]+\.[0-9]+$' \
|
|
| sort -V \
|
|
| tail -n 1 > branch.txt
|
|
|
|
BRANCH=$(cat branch.txt)
|
|
if [ -z "$BRANCH" ]; then
|
|
echo "No release-* branch found in ${{ github.repository }}"
|
|
exit 1
|
|
fi
|
|
echo "branch=$BRANCH" >> $GITHUB_OUTPUT
|
|
echo "Using release branch: $BRANCH"
|
|
create-nightly-tag:
|
|
if: github.repository == 'langflow-ai/langflow'
|
|
needs: [validate-inputs, resolve-release-branch]
|
|
runs-on: ubuntu-latest
|
|
defaults:
|
|
run:
|
|
shell: bash -ex -o pipefail {0}
|
|
permissions:
|
|
# Required to create tag
|
|
contents: write
|
|
outputs:
|
|
release_tag: ${{ steps.generate_release_tag.outputs.release_tag }}
|
|
base_tag: ${{ steps.set_base_tag.outputs.base_tag }}
|
|
lfx_tag: ${{ steps.generate_lfx_tag.outputs.lfx_tag }}
|
|
sdk_tag: ${{ steps.generate_sdk_tag.outputs.sdk_tag }}
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ needs.resolve-release-branch.outputs.branch }}
|
|
persist-credentials: true
|
|
- name: Confirm branch
|
|
run: git branch --show-current
|
|
- name: "Setup Environment"
|
|
uses: astral-sh/setup-uv@v6
|
|
with:
|
|
enable-cache: true
|
|
cache-dependency-glob: "uv.lock"
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
prune-cache: false
|
|
- name: Install the project
|
|
run: uv sync
|
|
- name: Check script location
|
|
run: |
|
|
echo "PWD: $(pwd)"
|
|
find . -name "pypi_nightly_tag.py"
|
|
- name: Generate main nightly tag
|
|
id: generate_release_tag
|
|
run: |
|
|
# NOTE: This outputs the tag with the `v` prefix.
|
|
RELEASE_TAG="$(uv run ./scripts/ci/pypi_nightly_tag.py main)"
|
|
echo "release_tag=$RELEASE_TAG" >> $GITHUB_OUTPUT
|
|
echo "release_tag=$RELEASE_TAG"
|
|
|
|
- name: Delete existing tag if it exists
|
|
id: check_release_tag
|
|
run: |
|
|
git fetch --tags
|
|
git tag -d ${{ steps.generate_release_tag.outputs.release_tag }} || true
|
|
git push --delete origin ${{ steps.generate_release_tag.outputs.release_tag }} || true
|
|
echo "release_tag_exists=false" >> $GITHUB_OUTPUT
|
|
|
|
- name: Generate base nightly tag
|
|
id: generate_base_tag
|
|
run: |
|
|
# NOTE: This outputs the tag with the `v` prefix.
|
|
BASE_TAG="$(uv run ./scripts/ci/pypi_nightly_tag.py base)"
|
|
echo "base_tag=$BASE_TAG" >> $GITHUB_OUTPUT
|
|
echo "base_tag=$BASE_TAG"
|
|
|
|
- name: Generate LFX nightly tag
|
|
id: generate_lfx_tag
|
|
run: |
|
|
# NOTE: This outputs the tag with the `v` prefix.
|
|
LFX_TAG="$(uv run ./scripts/ci/lfx_nightly_tag.py)"
|
|
echo "lfx_tag=$LFX_TAG" >> $GITHUB_OUTPUT
|
|
echo "lfx_tag=$LFX_TAG"
|
|
|
|
- name: Generate SDK nightly tag
|
|
id: generate_sdk_tag
|
|
run: |
|
|
# NOTE: This outputs the tag with the `v` prefix.
|
|
SDK_TAG="$(uv run ./scripts/ci/sdk_nightly_tag.py)"
|
|
echo "sdk_tag=$SDK_TAG" >> $GITHUB_OUTPUT
|
|
echo "sdk_tag=$SDK_TAG"
|
|
|
|
- name: Commit tag
|
|
id: commit_tag
|
|
run: |
|
|
# If the main tag does not exist in GH, we create the base tag from the existing codebase.
|
|
|
|
git config --global user.email "bot-nightly-builds@langflow.org"
|
|
git config --global user.name "Langflow Bot"
|
|
|
|
RELEASE_TAG="${{ steps.generate_release_tag.outputs.release_tag }}"
|
|
BASE_TAG="${{ steps.generate_base_tag.outputs.base_tag }}"
|
|
LFX_TAG="${{ steps.generate_lfx_tag.outputs.lfx_tag }}"
|
|
SDK_TAG="${{ steps.generate_sdk_tag.outputs.sdk_tag }}"
|
|
echo "Updating SDK project version to $SDK_TAG"
|
|
uv run --no-sync ./scripts/ci/update_sdk_version.py $SDK_TAG
|
|
echo "Updating LFX project version to $LFX_TAG"
|
|
uv run --no-sync ./scripts/ci/update_lfx_version.py $LFX_TAG $SDK_TAG
|
|
echo "Updating base project version to $BASE_TAG and updating main project version to $RELEASE_TAG"
|
|
uv run --no-sync ./scripts/ci/update_pyproject_combined.py main $RELEASE_TAG $BASE_TAG $LFX_TAG
|
|
|
|
uv lock
|
|
cd src/backend/base && uv lock && cd ../../..
|
|
cd src/lfx && uv lock && cd ../..
|
|
|
|
git add pyproject.toml src/backend/base/pyproject.toml src/lfx/pyproject.toml src/sdk/pyproject.toml uv.lock src/backend/base/uv.lock
|
|
# update_lfx_version.py re-pins each bundle's `lfx` dep to
|
|
# lfx-nightly==<dev>, so any modified bundle pyprojects need to
|
|
# ride the same commit/tag as the rest of the version bumps.
|
|
if compgen -G "src/bundles/*/pyproject.toml" > /dev/null; then
|
|
git add src/bundles/*/pyproject.toml
|
|
fi
|
|
git commit -m "Update version and project name"
|
|
|
|
echo "Tagging main with $RELEASE_TAG"
|
|
if ! git tag -a $RELEASE_TAG -m "Langflow nightly $RELEASE_TAG"; then
|
|
echo "Tag creation failed. Exiting the workflow."
|
|
exit 1
|
|
fi
|
|
|
|
echo "Pushing main tag $RELEASE_TAG"
|
|
if ! git push origin $RELEASE_TAG; then
|
|
echo "Tag push failed. Check if the tag already exists. Exiting the workflow."
|
|
exit 1
|
|
fi
|
|
# TODO: notify on failure
|
|
|
|
- name: Checkout main nightly tag
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ steps.generate_release_tag.outputs.release_tag }}
|
|
persist-credentials: true
|
|
|
|
- name: Retrieve Base Tag
|
|
id: retrieve_base_tag
|
|
working-directory: src/backend/base
|
|
run: |
|
|
# If the main tag already exists, we need to retrieve the base version from the main tag codebase.
|
|
version=$(uv tree | grep 'langflow-base' | awk '{print $3}' | head -n 1)
|
|
echo "base_tag=$version" >> $GITHUB_OUTPUT
|
|
echo "base_tag=$version"
|
|
|
|
- name: Set Base Tag
|
|
id: set_base_tag
|
|
run: |
|
|
if [ "${{ steps.retrieve_base_tag.conclusion }}" != "skipped" ] && [ "${{ steps.retrieve_base_tag.outputs.base_tag }}" ]; then
|
|
BASE_TAG="${{ steps.retrieve_base_tag.outputs.base_tag }}"
|
|
echo "base_tag=$BASE_TAG" >> $GITHUB_OUTPUT
|
|
echo "base_tag=$BASE_TAG"
|
|
elif [ "${{ steps.commit_tag.conclusion }}" != "skipped" ] && [ "${{ steps.generate_base_tag.outputs.base_tag }}" ]; then
|
|
BASE_TAG="${{ steps.generate_base_tag.outputs.base_tag }}"
|
|
echo "base_tag=$BASE_TAG" >> $GITHUB_OUTPUT
|
|
echo "base_tag=$BASE_TAG"
|
|
else
|
|
echo "No base tag found. Exiting the workflow."
|
|
exit 1
|
|
fi
|
|
|
|
frontend-tests-linux:
|
|
if: github.repository == 'langflow-ai/langflow' && !inputs.skip_frontend_tests
|
|
name: Run Frontend Tests - Linux
|
|
needs: [resolve-release-branch, create-nightly-tag]
|
|
uses: ./.github/workflows/typescript_test.yml
|
|
with:
|
|
tests_folder: "tests"
|
|
release: true
|
|
runs-on: ubuntu-latest
|
|
ref: ${{ needs.resolve-release-branch.outputs.branch }}
|
|
secrets:
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
STORE_API_KEY: ${{ secrets.STORE_API_KEY }}
|
|
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
|
|
|
|
frontend-tests-windows:
|
|
if: github.repository == 'langflow-ai/langflow' && !inputs.skip_frontend_tests
|
|
name: Run Frontend Tests - Windows
|
|
needs: [resolve-release-branch, create-nightly-tag]
|
|
# Windows tests are non-blocking - the release-nightly-build job only checks
|
|
# frontend-tests-linux.result, allowing Windows tests to fail without blocking the build.
|
|
# This gives us visibility into Windows-specific issues while we stabilize the tests.
|
|
uses: ./.github/workflows/typescript_test.yml
|
|
with:
|
|
tests_folder: "tests"
|
|
release: true
|
|
runs-on: windows-latest
|
|
ref: ${{ needs.resolve-release-branch.outputs.branch }}
|
|
secrets:
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
STORE_API_KEY: ${{ secrets.STORE_API_KEY }}
|
|
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
|
|
|
|
backend-unit-tests:
|
|
if: github.repository == 'langflow-ai/langflow' && !inputs.skip_backend_tests
|
|
name: Run Backend Unit Tests
|
|
needs: [resolve-release-branch, create-nightly-tag]
|
|
uses: ./.github/workflows/python_test.yml
|
|
with:
|
|
python-versions: '["3.10", "3.11", "3.12", "3.13", "3.14"]'
|
|
runs-on: ${{ inputs['runs_on'] || github.event.inputs['runs_on'] || 'ubuntu-latest' }}
|
|
ref: ${{ needs.resolve-release-branch.outputs.branch }}
|
|
secrets:
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
|
|
# Not making nightly builds dependent on integration test success
|
|
# due to inherent flakiness of 3rd party integrations
|
|
# Revisit when https://github.com/langflow-ai/langflow/pull/3607 is merged.
|
|
# backend-integration-tests:
|
|
# name: Run Backend Integration Tests
|
|
# needs: create-nightly-tag
|
|
# uses: ./.github/workflows/integration_tests.yml
|
|
# with:
|
|
# python-versions: '["3.10", "3.11", "3.12", "3.13"]'
|
|
# ref: ${{ needs.create-nightly-tag.outputs.tag }}
|
|
|
|
release-nightly-build:
|
|
if: github.repository == 'langflow-ai/langflow' && always() && needs.frontend-tests-linux.result != 'failure' && needs.backend-unit-tests.result != 'failure'
|
|
name: Run Nightly Langflow Build
|
|
needs:
|
|
[validate-inputs, create-nightly-tag, frontend-tests-linux, frontend-tests-windows, backend-unit-tests]
|
|
uses: ./.github/workflows/release_nightly.yml
|
|
with:
|
|
build_docker_base: true
|
|
build_docker_main: true
|
|
build_docker_ep: true
|
|
build_lfx: true
|
|
nightly_tag_release: ${{ needs.create-nightly-tag.outputs.release_tag }}
|
|
nightly_tag_base: ${{ needs.create-nightly-tag.outputs.base_tag }}
|
|
nightly_tag_lfx: ${{ needs.create-nightly-tag.outputs.lfx_tag }}
|
|
nightly_tag_sdk: ${{ needs.create-nightly-tag.outputs.sdk_tag }}
|
|
# When triggered by schedule, inputs.push_to_registry is not set, so default to true
|
|
# When triggered manually, use the provided value (default is also true)
|
|
push_to_registry: ${{ github.event_name == 'schedule' || inputs.push_to_registry != false }}
|
|
secrets: inherit
|
|
|
|
slack-notification:
|
|
name: Send Slack Notification
|
|
needs: [frontend-tests-linux, frontend-tests-windows, backend-unit-tests, release-nightly-build]
|
|
if: ${{ github.repository == 'langflow-ai/langflow' && !inputs.skip_slack && always() && (needs.release-nightly-build.result == 'failure' || needs.frontend-tests-linux.result == 'failure' || needs.frontend-tests-windows.result == 'failure' || needs.backend-unit-tests.result == 'failure' || needs.release-nightly-build.result == 'success') }}
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Send failure notification to Slack
|
|
if: ${{ needs.release-nightly-build.result == 'failure' || needs.frontend-tests-linux.result == 'failure' || needs.frontend-tests-windows.result == 'failure' || needs.backend-unit-tests.result == 'failure' }}
|
|
run: |
|
|
# Determine which job failed
|
|
FAILED_JOB="unknown"
|
|
if [ "${{ needs.release-nightly-build.result }}" == "failure" ]; then
|
|
FAILED_JOB="release-nightly-build"
|
|
elif [ "${{ needs.frontend-tests-linux.result }}" == "failure" ]; then
|
|
FAILED_JOB="frontend-tests-linux"
|
|
elif [ "${{ needs.frontend-tests-windows.result }}" == "failure" ]; then
|
|
FAILED_JOB="frontend-tests-windows (non-blocking)"
|
|
elif [ "${{ needs.backend-unit-tests.result }}" == "failure" ]; then
|
|
FAILED_JOB="backend-unit-tests"
|
|
fi
|
|
|
|
curl -X POST -H 'Content-type: application/json' \
|
|
--data "{
|
|
\"blocks\": [
|
|
{
|
|
\"type\": \"section\",
|
|
\"text\": {
|
|
\"type\": \"mrkdwn\",
|
|
\"text\": \"❌ *Nightly Build Failed*\"
|
|
}
|
|
},
|
|
{
|
|
\"type\": \"section\",
|
|
\"fields\": [
|
|
{
|
|
\"type\": \"mrkdwn\",
|
|
\"text\": \"*Failed Job:*\\n$FAILED_JOB\"
|
|
},
|
|
{
|
|
\"type\": \"mrkdwn\",
|
|
\"text\": \"*Status:*\\nfailure\"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
\"type\": \"section\",
|
|
\"text\": {
|
|
\"type\": \"mrkdwn\",
|
|
\"text\": \"*Note:* Frontend tests now run on both Linux and Windows\"
|
|
}
|
|
},
|
|
{
|
|
\"type\": \"context\",
|
|
\"elements\": [
|
|
{
|
|
\"type\": \"mrkdwn\",
|
|
\"text\": \"<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View full logs on GitHub>\"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}" ${{ secrets.LANGFLOW_ENG_SLACK_WEBHOOK_URL }}
|
|
|
|
- name: Send success notification to Slack
|
|
if: ${{ !inputs.skip_slack && needs.release-nightly-build.result == 'success' }}
|
|
run: |
|
|
curl -X POST -H 'Content-type: application/json' \
|
|
--data '{
|
|
"blocks": [
|
|
{
|
|
"type": "section",
|
|
"text": {
|
|
"type": "mrkdwn",
|
|
"text": "✅ *Nightly Build Successful*"
|
|
}
|
|
},
|
|
{
|
|
"type": "section",
|
|
"fields": [
|
|
{
|
|
"type": "mrkdwn",
|
|
"text": "*Job:*\nrelease-nightly-build"
|
|
},
|
|
{
|
|
"type": "mrkdwn",
|
|
"text": "*Status:*\n`${{ needs.release-nightly-build.result }}`"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"type": "section",
|
|
"text": {
|
|
"type": "mrkdwn",
|
|
"text": "*Platforms:* Linux & Windows Playwright tests passed"
|
|
}
|
|
},
|
|
{
|
|
"type": "context",
|
|
"elements": [
|
|
{
|
|
"type": "mrkdwn",
|
|
"text": "<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View full logs on GitHub>"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}' ${{ secrets.LANGFLOW_ENG_SLACK_WEBHOOK_URL }}
|