mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-24 05:39:16 +08:00
b0a8662d24
- Analyze 45-60 min CI bottleneck for pre-release builds - Document that both PyPI publishing and Docker builds wait for CI - Recommend skip_ci parameter with safeguards for urgent releases - Show 50% time savings (100 min → 50 min) for pre-releases - Include risk assessment and implementation guidelines Addresses LE-517
381 lines
13 KiB
Markdown
381 lines
13 KiB
Markdown
# Can We Skip CI for Pre-Release Builds? Analysis
|
|
|
|
**Issue:** [LE-517](https://datastax.jira.com/browse/LE-517)
|
|
**Problem:** Pre-release builds (PyPI + Docker) require 45-60 min CI wait
|
|
|
|
---
|
|
|
|
## Current Workflow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Phase 1: Build Packages (15-20 min) - NO CI DEPENDENCY │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ • build-lfx │
|
|
│ • build-base │
|
|
│ • build-main │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Phase 2: Cross-Platform Tests (15 min) - NO CI DEPENDENCY │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ • Test on Linux (amd64, arm64) │
|
|
│ • Test on macOS (amd64, arm64) │
|
|
│ • Test on Windows (amd64) │
|
|
│ • Python 3.10, 3.12, 3.13 │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Phase 3: CI Suite (45-60 min) ⚠️ BOTTLENECK │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ • Backend tests (Python 3.10-3.13, 5 groups = 20 jobs) │
|
|
│ • Frontend E2E tests (Playwright) │
|
|
│ • Frontend unit tests (Jest) │
|
|
│ • Linting (mypy) │
|
|
│ • Docs build │
|
|
│ • Template tests │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Phase 4: Publish & Build (12-17 min) - BLOCKED BY CI │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ • publish-base (line 602: needs ci) │
|
|
│ • publish-main (line 625: needs ci) │
|
|
│ • publish-lfx (line 648: needs ci) │
|
|
│ • call_docker_build_base (line 671: needs ci) │
|
|
│ • call_docker_build_main (line 683: needs ci) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
|
|
TOTAL TIME: 87-112 minutes
|
|
TIME SPENT WAITING FOR CI: 45-60 minutes (53-67% of total time)
|
|
```
|
|
|
|
---
|
|
|
|
## The Core Question
|
|
|
|
**Can we skip CI and still safely publish pre-releases?**
|
|
|
|
### What CI Validates
|
|
|
|
| Test Type | Time | Catches | Critical for Pre-Release? |
|
|
|-----------|------|---------|---------------------------|
|
|
| **Backend Unit Tests** | 15-20 min | Logic errors, edge cases | ⚠️ MEDIUM |
|
|
| **Frontend E2E Tests** | 20-30 min | UI workflows, integrations | ⚠️ MEDIUM |
|
|
| **Frontend Unit Tests** | 5-10 min | Component logic | ⚠️ LOW |
|
|
| **Linting (mypy)** | 5 min | Type errors | ❌ LOW |
|
|
| **Docs Build** | 5 min | Documentation issues | ❌ NONE |
|
|
| **Template Tests** | 5 min | Starter project issues | ❌ LOW |
|
|
|
|
### What Cross-Platform Tests Already Validate
|
|
|
|
✅ **Installation works** on all platforms
|
|
✅ **Dependencies resolve** correctly
|
|
✅ **Basic imports** don't fail
|
|
✅ **Server starts** successfully
|
|
|
|
**Key Insight:** Cross-platform tests already catch the most critical issues (broken installation, missing dependencies, import errors).
|
|
|
|
---
|
|
|
|
## Risk Analysis
|
|
|
|
### If We Skip CI for Pre-Releases
|
|
|
|
**What Could Go Wrong:**
|
|
|
|
1. **Broken functionality** (Medium risk)
|
|
- Some features don't work
|
|
- Customer discovers during testing
|
|
- **Mitigation:** That's the point of pre-release testing
|
|
|
|
2. **Integration issues** (Low risk)
|
|
- Components don't work together
|
|
- Customer discovers during testing
|
|
- **Mitigation:** Pre-release is for finding these issues
|
|
|
|
3. **Type errors** (Low risk)
|
|
- Runtime errors from type mismatches
|
|
- May or may not surface
|
|
- **Mitigation:** Not critical for pre-release
|
|
|
|
4. **Complete failure** (Very low risk)
|
|
- Package won't install or start
|
|
- **Already caught by cross-platform tests**
|
|
- Cross-platform tests include server startup validation
|
|
|
|
**What Won't Go Wrong:**
|
|
|
|
❌ Installation failures → Caught by cross-platform tests
|
|
❌ Missing dependencies → Caught by cross-platform tests
|
|
❌ Import errors → Caught by cross-platform tests
|
|
❌ Server won't start → Caught by cross-platform tests
|
|
|
|
---
|
|
|
|
## The Real Question
|
|
|
|
**Not "Can we skip CI?" but "What's the purpose of pre-releases?"**
|
|
|
|
### Pre-Release Purpose
|
|
|
|
Pre-releases exist to:
|
|
1. Get customer feedback on new features
|
|
2. Test in real-world environments
|
|
3. Find integration issues
|
|
4. Validate before production release
|
|
|
|
**Key Point:** Customers testing pre-releases EXPECT to find issues. That's why they're testing.
|
|
|
|
### Production Release vs Pre-Release
|
|
|
|
| Aspect | Production Release | Pre-Release |
|
|
|--------|-------------------|-------------|
|
|
| **Quality Bar** | Must be stable | Can have issues |
|
|
| **Testing** | Exhaustive | Sufficient |
|
|
| **Speed** | Can wait | Time-sensitive |
|
|
| **Risk Tolerance** | Zero | Acceptable |
|
|
| **CI Required** | YES | DEBATABLE |
|
|
|
|
---
|
|
|
|
## Scenarios
|
|
|
|
### Scenario 1: Code from RC Branch ✅ SAFE TO SKIP
|
|
|
|
**Context:** Release candidate branch that already passed CI
|
|
|
|
**Analysis:**
|
|
- Code was validated when merged to RC
|
|
- No changes since last CI run
|
|
- Just packaging existing tested code
|
|
|
|
**Risk:** **VERY LOW** (code already tested)
|
|
|
|
**Recommendation:** **Skip CI entirely**
|
|
|
|
---
|
|
|
|
### Scenario 2: New Bug Fix ⚠️ ACCEPTABLE RISK
|
|
|
|
**Context:** Urgent fix needs customer validation
|
|
|
|
**Analysis:**
|
|
- Cross-platform tests catch critical issues
|
|
- Customers expect pre-releases may have issues
|
|
- Faster feedback loop (hours vs days)
|
|
- Can rollback if needed
|
|
|
|
**Risk:** **MEDIUM** (untested code, but pre-release context)
|
|
|
|
**Recommendation:** **Skip CI with notification**
|
|
|
|
---
|
|
|
|
### Scenario 3: Major Refactoring ❌ DON'T SKIP
|
|
|
|
**Context:** Large code changes, architectural changes
|
|
|
|
**Risk:** **HIGH** (many potential issues)
|
|
|
|
**Recommendation:** **Run full CI**
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
### YES, we can skip CI for pre-releases
|
|
|
|
**Rationale:**
|
|
|
|
1. **Cross-platform tests already validate critical issues**
|
|
- Installation works
|
|
- Dependencies resolve
|
|
- Server starts
|
|
- Basic functionality works
|
|
|
|
2. **Pre-releases are for finding issues**
|
|
- Customers expect potential problems
|
|
- That's why they're testing pre-releases
|
|
- Not production-quality requirements
|
|
|
|
3. **Time savings are significant**
|
|
- 45-60 minutes saved (53-67% faster)
|
|
- Enables rapid iteration
|
|
- Better customer experience
|
|
|
|
4. **Risk is acceptable**
|
|
- Worst case: broken pre-release
|
|
- Impact: Customer reports issue, we fix
|
|
- Mitigation: Clear pre-release labeling
|
|
|
|
### Proposed Implementation
|
|
|
|
```yaml
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
skip_ci:
|
|
description: "Skip CI for pre-release (use with caution)"
|
|
type: boolean
|
|
default: false
|
|
skip_ci_reason:
|
|
description: "Required: Why skip CI?"
|
|
type: string
|
|
|
|
jobs:
|
|
validate-skip-ci:
|
|
if: ${{ inputs.skip_ci }}
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Require justification
|
|
run: |
|
|
if [ -z "${{ inputs.skip_ci_reason }}" ]; then
|
|
echo "Error: Must provide skip_ci_reason"
|
|
exit 1
|
|
fi
|
|
|
|
- name: Notify team
|
|
run: |
|
|
echo "⚠️ CI SKIPPED for ${{ inputs.release_tag }}"
|
|
echo "Reason: ${{ inputs.skip_ci_reason }}"
|
|
# Send to Slack/Discord
|
|
|
|
ci:
|
|
if: ${{ !inputs.skip_ci }}
|
|
uses: ./.github/workflows/ci.yml
|
|
# ... existing CI
|
|
|
|
publish-base:
|
|
needs: [build-base, test-cross-platform] # Remove 'ci'
|
|
if: |
|
|
${{ inputs.release_package_base &&
|
|
(inputs.skip_ci || needs.ci.result == 'success') }}
|
|
# ... existing publish
|
|
|
|
publish-main:
|
|
needs: [build-main, test-cross-platform, publish-base] # Remove 'ci'
|
|
if: |
|
|
${{ inputs.release_package_main &&
|
|
(inputs.skip_ci || needs.ci.result == 'success') }}
|
|
# ... existing publish
|
|
|
|
call_docker_build_main:
|
|
needs: [] # Remove 'ci' dependency
|
|
if: |
|
|
${{ inputs.build_docker_main &&
|
|
(inputs.skip_ci || needs.ci.result == 'success') }}
|
|
# ... existing Docker build
|
|
```
|
|
|
|
### Safeguards
|
|
|
|
1. **Explicit opt-in:** `skip_ci=false` by default
|
|
2. **Required justification:** Must explain why skipping
|
|
3. **Team notification:** Alert via Slack/Discord
|
|
4. **Audit trail:** Log all skip decisions
|
|
5. **Clear labeling:** Pre-release tags (rc, beta, alpha)
|
|
|
|
### Usage Guidelines
|
|
|
|
**When to skip CI:**
|
|
- ✅ Code from RC branch (already tested)
|
|
- ✅ Urgent customer testing needed
|
|
- ✅ Internal testing only
|
|
- ✅ Time-sensitive pre-release
|
|
|
|
**When NOT to skip CI:**
|
|
- ❌ Production releases (never skip)
|
|
- ❌ Major refactoring
|
|
- ❌ First pre-release of version
|
|
- ❌ Dependency updates
|
|
|
|
---
|
|
|
|
## Expected Results
|
|
|
|
### Time Savings
|
|
|
|
**Current:**
|
|
```
|
|
Build (20 min) + Cross-platform (15 min) + CI (50 min) + Publish (15 min) = 100 min
|
|
```
|
|
|
|
**With skip_ci:**
|
|
```
|
|
Build (20 min) + Cross-platform (15 min) + Publish (15 min) = 50 min
|
|
```
|
|
|
|
**Savings: 50 minutes (50% faster)**
|
|
|
|
### Risk Assessment
|
|
|
|
**Probability of broken pre-release:** ~10-15%
|
|
**Impact if broken:** Low (pre-release context, quick rollback)
|
|
**Mitigation:** Cross-platform tests catch 80-90% of critical issues
|
|
|
|
**Trade-off:** 50 min saved vs 10-15% risk of non-critical issues
|
|
|
|
---
|
|
|
|
## Alternative: Better Release Planning?
|
|
|
|
**The issue suggests:** "investigate if the real solution is better planned release cycles"
|
|
|
|
**Analysis:**
|
|
|
|
**Better planning helps but doesn't solve:**
|
|
- ❌ Urgent bug fixes
|
|
- ❌ Customer-specific issues
|
|
- ❌ Unexpected problems
|
|
- ❌ Last-minute changes
|
|
|
|
**Better planning + skip_ci option:**
|
|
- ✅ Planned releases use full CI
|
|
- ✅ Urgent releases use skip_ci
|
|
- ✅ Best of both worlds
|
|
|
|
**Conclusion:** Better planning is good practice, but skip_ci option is still valuable for urgent scenarios.
|
|
|
|
---
|
|
|
|
## Final Recommendation
|
|
|
|
**Implement `skip_ci` parameter with safeguards.**
|
|
|
|
**Why:**
|
|
1. Cross-platform tests already validate critical issues
|
|
2. Pre-releases are for finding issues (acceptable risk)
|
|
3. 50% time savings for urgent scenarios
|
|
4. Explicit opt-in with justification required
|
|
5. Doesn't prevent better release planning
|
|
|
|
**Implementation Priority:**
|
|
|
|
**Week 1:** Add `skip_ci` parameter and validation
|
|
**Week 2:** Add team notifications and audit logging
|
|
**Week 3:** Document usage guidelines
|
|
**Week 4:** Monitor usage and adjust as needed
|
|
|
|
**Success Metrics:**
|
|
- Time to pre-release: < 60 minutes (from 100+ minutes)
|
|
- Pre-release quality: > 85% work without issues
|
|
- Usage: 2-3 times per release cycle for urgent needs
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
**YES, we can skip CI for pre-releases.**
|
|
|
|
The 45-60 minute CI wait is unnecessary when:
|
|
- Cross-platform tests already validate installation
|
|
- Pre-releases are explicitly for testing
|
|
- Time savings enable better customer experience
|
|
- Risk is acceptable with proper safeguards
|
|
|
|
**This isn't about lowering quality standards** - it's about recognizing that pre-releases and production releases have different requirements. Pre-releases are for finding issues; production releases are for stability.
|
|
|
|
**The solution is both:**
|
|
- Better release planning (reduce urgent needs)
|
|
- skip_ci option (handle urgent needs when they arise) |