Files
langflow/scripts/gp/gp_client.py
Ram Gopal Srikar Katakam 30f351766b feat: IBM Globalization Pipeline integration and i18n setup (#12226)
* feat: add IBM Globalization Pipeline integration and i18n setup

- Add GP REST API client with GP-HMAC authentication (scripts/gp/gp_client.py)
- Add upload/download scripts for syncing strings with GP
- Set up i18next + react-i18next with 7 language support (en, fr, ja, es, de, pt, zh-Hans)
- Add 48 UI strings in flat dot notation to en.json (from alerts_constants)
- Add GitHub Actions for automated upload on en.json changes and daily translation download

* [autofix.ci] apply automated fixes

* fix: resolve ruff linting issues in GP scripts

- Fix quote style, import ordering, docstring format
- Use Path instead of os.makedirs/open/os.path.join
- Add missing timeouts to requests calls
- Move noqa comments to correct lines for S501

* feat: migrate all frontend UI strings to i18n for GP localization

- Replace all hardcoded user-facing strings with t() calls across 80+ components, pages, modals, and utilities
- Add 326 translation keys to en.json covering errors, alerts, dialogs, chat, flow, settings, store, table, output, nav, auth, and more
- Add translated locale files for fr, es, de, pt, ja, zh-Hans (downloaded from Globalization Pipeline)
- Migrate alerts_constants.tsx and constants.ts string usages to react-i18next
- Support non-React files (stores, utils) via i18n.t() direct calls

* feat: migrate sidebar strings to i18n (Phase 4)

- Add 56 new translation keys for sidebar categories, nav items, labels, buttons, and empty states
- Convert SIDEBAR_CATEGORIES display_names to i18n key references in styleUtils.ts
- Update 13 sidebar components to use t() for all user-facing strings
- Refresh all 6 locale files from GP (382 total keys)
- Add BACKEND_STRINGS_STRATEGY.md documenting plan for component display_name i18n

* [autofix.ci] apply automated fixes

* fix: add react-i18next mock and fix sidebar nav tests for i18n migration

- Add global react-i18next mock in jest.setup.js using en.json lookups so
  t(key) returns English strings instead of raw keys in test environment
- Fix sidebarSegmentedNav tests: update NAV_ITEMS expectations to use i18n
  keys, and use en.json lookups for tooltip/accessibility label assertions

* feat: add LanguageSelector dropdown to globalization pipeline

Cherry-picked frontend-only changes from feat/gp-backend-i18n (6ab21559db, 3e274a84e3):
- Add LanguageSelector component with dropdown to switch UI language
- Wire LanguageSelector into AccountMenu header
- Persist language preference to localStorage in i18n.ts
- Send Accept-Language header via useCustomApiHeaders (reactive via useTranslation)

No backend files included.

* [autofix.ci] apply automated fixes

* ci: add GP_TEST environment to gp-download and gp-upload workflows

* temp: add feature branch push trigger for workflow testing

* temp: trigger gp-upload workflow test

* fix: use correct GP_test environment name in workflows

* fix: use vars for GP_INSTANCE and GP_BUNDLE

* [autofix.ci] apply automated fixes

* fix: correct environment name to GP-test (hyphen not underscore)

* chore: update translations from Globalization Pipeline [skip ci]

* [autofix.ci] apply automated fixes

* chore: update translations from Globalization Pipeline [skip ci]

* temp: trigger gp-upload workflow test

* ci: add download-gp-bundle job to nightly build pipeline

Downloads frontend translations from Globalization Pipeline after
create-nightly-tag and before frontend tests run, ensuring tests
always validate the latest translated locale files.

- Frontend tests (Linux/Windows) now depend on download-gp-bundle
- Backend unit tests are unaffected (no frontend locale dependency)
- GP failure blocks release-nightly-build (same severity as linux tests)

* chore: remove temporary dev files from gp scripts directory

Remove BACKEND_STRINGS_STRATEGY.md (internal planning doc), test_auth.py
(GP auth debugging script), and en.json (dev-time string snapshot).
Also add venv/ to .gitignore to prevent accidental commits of the local virtualenv.

* [autofix.ci] apply automated fixes

* chore: update translations from Globalization Pipeline [skip ci]

* [autofix.ci] apply automated fixes

* chore: update translations from Globalization Pipeline [skip ci]

* feat(i18n): move language selector from account menu to Settings > General

- Add LanguageForm card component in GeneralPage following the existing
  Card pattern (ProfilePictureForm), with immediate language switching
- English marked as "(Recommended)" via i18n key
- Remove Language row from AccountMenu dropdown
- Add settings.languageTitle/Description/Recommended keys to en.json
  for upload to Globalization Pipeline

* feat(i18n): refactor language constants and improve GP pipeline

- Extract SUPPORTED_LANGUAGES into src/frontend/src/constants/languages.ts
- Update LanguageForm to import from shared constants, add aria-label
- Add settings.languageSelectAriaLabel key to en.json
- Update LanguageSelector component to use shared SUPPORTED_LANGUAGES
- Fix gp_client and download_translations scripts
- Update GP upload/download/nightly GitHub Actions workflows
- Add GP script tests

* [autofix.ci] apply automated fixes

* ci(gp): trigger upload on release branches, download via dispatch only

- Upload now triggers on push to release-* branches (instead of main)
  so translators get strings once they're finalized on the release branch
- Download removes the daily cron schedule; workflow_dispatch only
  since the nightly build already handles scheduled runs

* chore: update translations from Globalization Pipeline

* fix(i18n): address PR review comments — tests, nightly build decoupling, GP test workflow

- Add pytest tests for gp_client.py (HMAC auth, HTTP errors, timeout)
- Add pytest tests for upload_strings.py and download_translations.py
- Add conftest.py to set sys.path so GP scripts can be imported from repo root
- Add scripts/gp/__init__.py to make gp a proper package for test imports
- Add gp-test.yml workflow to run GP script tests on PRs touching scripts/gp/**
- Decouple download-gp-bundle from nightly build chain:
  - Add continue-on-error: true so GP failure never blocks the build
  - Remove download-gp-bundle from needs/if of frontend-tests-linux,
    frontend-tests-windows, and release-nightly-build
  - Add warning annotation step that fires on GP download failure

* feat(i18n): lazy-load non-English locale files via Vite dynamic imports

- Remove 6 static locale imports from i18n.ts; only en.json is bundled
- Export loadLanguage() which dynamically imports a locale chunk on demand
  and caches it via i18n.hasResourceBundle (no-op on repeat calls)
- Update index.tsx to await loadLanguage(detectedLang) before rendering,
  preventing an English flash for non-English users
- Make handleChange async in LanguageForm and LanguageSelector to await
  loadLanguage before calling i18n.changeLanguage
- Add i18n.test.ts covering loadLanguage: en no-op, new language load,
  cache hit, and multiple independent languages
- Update LanguageForm.test.tsx to mock @/i18n and assert loadLanguage
  is called before changeLanguage on language switch

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* fix: add ruff ignores for scripts/gp/tests

- Add per-file ignores for scripts/gp/tests/* to fix CI failures
- Ignore S101 (assert usage), TRY003, EM101 for test files
- Follows existing pattern for other test directories

* [autofix.ci] apply automated fixes

* fix(ci): run GP translation download before nightly tag creation

Previously download-gp-bundle ran after create-nightly-tag, so fresh
translations were committed to the branch after the tag was already
created. The nightly Docker image builds from the tag, meaning it never
included the day's translations.

Fix: move download-gp-bundle to run before create-nightly-tag so the
tag is created on a commit that already includes the latest translations.

* fix(ui): replace native select with Radix UI Select in LanguageForm

Swaps the native <select> element for the existing Radix UI Select
component to get consistent padding on both sides of the chevron,
removing reliance on browser-rendered chevron positioning.

* fix(ci): rearchitect GP translation download to use PRs instead of direct push

Replace direct branch commits in gp-download.yml and nightly_build.yml with
a PR-based flow using peter-evans/create-pull-request, respecting release
branch protection policies.

- gp-download.yml: dynamically resolves latest release-* branch, closes stale
  translation PRs, opens a new PR via peter-evans/create-pull-request@v8, and
  enables auto-merge (squash); scheduled at 23:00 UTC (1h before nightly)
- nightly_build.yml: remove download-gp-bundle job and its dependency from
  create-nightly-tag — nightly build now uses translations already in the branch

* fix(ci): skip GP upload when triggered from an outdated release branch

Add a branch guard step that resolves the latest release-* branch at
runtime and skips the upload (with a notice annotation) if the triggering
branch is not the latest. Prevents stale en.json from an old release branch
overwriting current source strings in the Globalization Pipeline.

* fix(tests): fix LanguageForm test suite failures

- Add `...jest.requireActual` to @tanstack/react-query mock so QueryClient
  constructor remains available (used in contexts/index.tsx at module load)
- Mock @/components/ui/select with a native <select> shim so test assertions
  using selectOptions() and querySelectorAll("option") work correctly

* fix: add trailing newline to component_index.json

The build_component_index.py script was generating component_index.json
without a trailing newline, causing CI failures when autofix.ci tried to
add the missing newline. This fix ensures the generated JSON file always
ends with a newline character, following POSIX text file standards.

Fixes the 'No newline at end of file' error in the update-component-index
workflow job.

* [autofix.ci] apply automated fixes

* fix: add trailing newline to component_index.json

The build_component_index.py script was generating component_index.json
without a trailing newline, causing CI failures when autofix.ci tried to
add the missing newline. This fix ensures the generated JSON file always
ends with a newline character, following POSIX text file standards.

Fixes the 'No newline at end of file' error in the update-component-index
workflow job.

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: vijay kumar katuri <vijay.katuri@ibm.com>
2026-04-08 17:51:18 +00:00

101 lines
3.1 KiB
Python

"""GP (Globalization Pipeline) REST API client.
Handles HMAC authentication and common API operations.
"""
import base64
import hashlib
import hmac
import json
import os
from datetime import datetime, timezone
import requests
from dotenv import load_dotenv
load_dotenv()
BASE_URL = "https://g11n-pipeline-api.straker.global/translate/rest"
GP_USER_ID = os.getenv("GP_ADMIN_USER_ID")
GP_PASSWORD = os.getenv("GP_ADMIN_PASSWORD")
GP_INSTANCE = os.getenv("GP_INSTANCE", "langflow-test")
GP_BUNDLE = os.getenv("GP_BUNDLE", "langflow-ui")
TARGET_LANGS = ["fr", "ja", "es", "de", "pt", "zh-Hans"]
REQUEST_TIMEOUT = 30
VERIFY_SSL = os.getenv("GP_VERIFY_SSL", "true").lower() != "false"
def get_headers(url, method, body=None):
"""Generate GP-HMAC auth headers. url must be the full URL."""
date = datetime.now(timezone.utc)
date_string = date.strftime("%a, %d %b %Y %H:%M:%S %Z").replace("UTC", "GMT")
if method.upper() == "GET":
msg = "GET" + "\n" + url + "\n" + date_string + "\n"
else:
msg = method.upper() + "\n" + url + "\n" + date_string + "\n" + json.dumps(body)
message = bytes(msg, "ISO-8859-1")
password = bytes(GP_PASSWORD, "ISO-8859-1")
signature = hmac.new(password, msg=message, digestmod=hashlib.sha1).digest()
hmac_header = "GP-HMAC " + GP_USER_ID + ":" + base64.b64encode(signature).decode()
headers = {
"Authorization": hmac_header,
"GP-Date": date_string,
"accept": "application/json",
}
if method.upper() == "PATCH":
headers["Content-Type"] = "application/merge-patch+json"
elif method.upper() in ("PUT", "POST"):
headers["Content-Type"] = "application/json"
return headers
def list_bundles():
"""List all bundles in the GP instance."""
url = f"{BASE_URL}/{GP_INSTANCE}/v2/bundles"
response = requests.get(url, headers=get_headers(url, "GET"), verify=VERIFY_SSL, timeout=REQUEST_TIMEOUT)
response.raise_for_status()
return response.json()
def create_bundle(source_lang="en"):
"""Create a new bundle in GP."""
url = f"{BASE_URL}/{GP_INSTANCE}/v2/bundles/{GP_BUNDLE}"
body = {"sourceLanguage": source_lang, "targetLanguages": TARGET_LANGS}
response = requests.put(
url,
headers=get_headers(url, "PUT", body),
json=body,
verify=VERIFY_SSL,
timeout=REQUEST_TIMEOUT,
)
response.raise_for_status()
return response.json()
def upload_strings(strings, lang="en"):
"""Upload resource strings for a language to GP."""
url = f"{BASE_URL}/{GP_INSTANCE}/v2/bundles/{GP_BUNDLE}/{lang}"
response = requests.put(
url,
headers=get_headers(url, "PUT", strings),
json=strings,
verify=VERIFY_SSL,
timeout=REQUEST_TIMEOUT,
)
response.raise_for_status()
return response.json()
def get_strings(lang):
"""Download translated strings for a language from GP."""
url = f"{BASE_URL}/{GP_INSTANCE}/v2/bundles/{GP_BUNDLE}/{lang}"
response = requests.get(url, headers=get_headers(url, "GET"), verify=VERIFY_SSL, timeout=REQUEST_TIMEOUT)
response.raise_for_status()
return response.json()