mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-24 05:39:16 +08:00
feat: Add SSRF protection with DNS rebinding prevention (#13016)
* feat: Add SSRF protection with DNS rebinding prevention - Implement DNS pinning to prevent DNS rebinding attacks - Enable SSRF protection by default (ssrf_protection_enabled = True) - Add validate_and_resolve_url() function for DNS pinning - Create SSRFProtectedTransport for custom HTTP transport - Add comprehensive test suite for DNS rebinding protection - Change from warn_only=True to warn_only=False for enforcement This fixes a HIGH severity SSRF vulnerability (CVSS 8.6) that could allow attackers to bypass SSRF protection using DNS rebinding attacks to access: - Internal services and private networks - Cloud metadata endpoints (AWS, GCP, Azure) - Localhost services The DNS pinning implementation ensures that the IP address validated during security checks is the same IP used for the actual HTTP request, eliminating the TOCTOU (Time-of-Check to Time-of-Use) vulnerability. Changes: - src/lfx/src/lfx/components/data_source/api_request.py: Integrate DNS pinning - src/lfx/src/lfx/utils/ssrf_protection.py: Add validate_and_resolve_url() - src/lfx/src/lfx/utils/ssrf_transport.py: Custom transport with DNS pinning - src/lfx/src/lfx/services/settings/base.py: Enable SSRF protection by default - src/backend/tests/unit/components/data_source/test_dns_rebinding.py: Test suite Tested and verified working in GUI with DNS pinning logs visible. * fix(security): implement network-level DNS pinning for SSRF protection with HTTPS support This commit implements proper DNS rebinding protection that works with both HTTP and HTTPS by using network-level DNS pinning instead of URL rewriting. ## Changes ### Security Fixes - **ssrf_protection.py**: Fixed critical vulnerability where IP validation returned early on first allowlisted IP, skipping validation of remaining IPs. Now validates ALL resolved IPs before making decisions. - Blocks hostname if ANY resolved IP is blocked (even if others are allowlisted) - Supports partial allowlisting (uses only allowlisted IPs for DNS pinning) - Updated documentation to reflect that SSRF protection is now enabled by default - Changed warn_only default from True to False for stricter security ### DNS Pinning Implementation - **ssrf_transport.py**: Created DNSPinningNetworkBackend class that implements network-level DNS pinning - Extends httpcore.AsyncNetworkBackend to intercept TCP connections - Connects to pinned IP at network layer while preserving hostname for TLS - Works with both HTTP and HTTPS (fixes HTTPS regression from URL rewriting approach) - Properly handles TLS SNI and certificate verification ### Testing - **test_dns_rebinding.py**: Updated all DNS rebinding tests to verify network-level behavior - Tests now patch AutoBackend.connect_tcp to capture actual IP connections - Verifies DNS pinning prevents rebinding attacks - Verifies hostname preservation for TLS - Tests IPv6 support - All 7 tests passing ## Technical Details The previous URL rewriting approach (replacing hostname with IP in URL) broke HTTPS because: 1. TLS certificate is issued for hostname, not IP address 2. Certificate verification fails when connecting to IP 3. This was a regression - HTTPS worked before but was vulnerable The new network-level approach: 1. Resolves DNS during validation and pins IPs 2. Uses custom AsyncNetworkBackend to intercept TCP connections 3. Connects to pinned IP at network layer 4. Preserves original hostname for TLS SNI and certificate verification 5. Works transparently with both HTTP and HTTPS ## Function Clarification Two validation functions exist for different purposes: - validate_url_for_ssrf(): Simple validation for URL component (no HTTP requests) - validate_and_resolve_url(): Validation + DNS pinning for API Request component (makes HTTP requests) Fixes DNS rebinding vulnerability while maintaining HTTPS compatibility. * fix(security): support multiple IPs for dual-stack and load-balanced hosts This commit fixes DNS pinning to properly handle hosts that resolve to multiple IP addresses (dual-stack IPv4/IPv6, load balancing, etc.). ## Changes ### DNS Pinning Enhancement - **ssrf_transport.py**: Updated to accept and use list of validated IPs - Changed pinned_ips from dict[str, str] to dict[str, list[str]] - DNSPinningNetworkBackend now tries IPs in order with fallback - Supports dual-stack (IPv4+IPv6) and load-balanced hosts - Falls back to next IP if connection fails - **api_request.py**: Pass all validated IPs instead of just first one - Changed from validated_ips[0] to validated_ips - Enables proper failover for unreachable IPs ### Testing - **test_dns_rebinding.py**: Added test_dns_pinning_with_multiple_ips_fallback - Verifies system tries multiple IPs when first fails - Tests dual-stack scenario (IPv4 fails, IPv6 succeeds) - Confirms IPs are tried in order - All 8 tests passing ## Technical Details **Previous behavior:** - Only used first validated IP (validated_ips[0]) - Failed if that single IP was unreachable - Broke dual-stack and load-balanced hosts **New behavior:** - Accepts full list of validated IPs - Tries each IP in order until one succeeds - Properly supports: - Dual-stack hosts (IPv4 + IPv6) - Load-balanced hosts (multiple IPs) - Failover scenarios This maintains security (all IPs validated during SSRF check) while improving reliability and compatibility with modern networking configurations. * fix: improve SSRF protection with DNS pinning and fix resource leak - Implement network-level DNS pinning to prevent DNS rebinding attacks - Fix resource leak in SSRFProtectedTransport by not calling super().__init__() - Simplify error messages in SSRF protection (remove verbose explanations) - Restore warn_only parameter in validate_url_for_ssrf() for backward compatibility - Remove warn_only from validate_and_resolve_url() (not needed in API Request) - Switch from private AutoBackend to public httpcore.AnyIOBackend() API - Add noqa comments for unavoidable FBT and B008 warnings (matching parent class signature) - Support dual-stack (IPv4/IPv6) and load-balanced hosts with multiple IPs - Move imports to top of file for better code organization * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * test: fix SSRF protection test mocks to use validate_and_resolve_url - Updated test mocks from validate_url_for_ssrf to validate_and_resolve_url - Fixed test_ssrf_protection_disabled_by_default to explicitly disable protection - All 12 SSRF protection tests now passing * fix: read SSRF allowlist directly from environment for test compatibility - Modified get_allowed_hosts() to read from os.getenv() first - Fixes test isolation issue where settings service cached old values - Ensures patch.dict() in tests properly overrides allowlist - Maintains backward compatibility with settings service fallback * test: fix test_ssrf_protection_enforcement_mode to properly mock validation The test was failing because it wasn't mocking validate_and_resolve_url, so the actual SSRF protection wasn't being triggered. Updated the test to: - Mock validate_and_resolve_url to raise SSRFProtectionError - Follow the same pattern as other SSRF protection tests - Properly test that API Request component enforces blocking (no warn-only mode) * refactor: rewrite SSRF protection tests to test real implementation BREAKING CHANGE: Completely rewrote SSRF protection tests to properly test actual security behavior instead of mocking core functions. Changes: - Removed all mocks of validate_and_resolve_url() - tests now verify real SSRF blocking - Added tests that verify DNS pinning actually prevents rebinding attacks - Test real blocking of private IPs (127.0.0.1, 192.168.x.x, 10.x.x.x, 172.16.x.x) - Test real blocking of cloud metadata endpoints (169.254.169.254) - Verify allowlist functionality with actual hostnames, IPs, and CIDR ranges - Test that custom DNS pinning transport is used when protection is enabled - Test that normal httpx client is used when protection is disabled - Fixed environment variable caching in is_ssrf_protection_enabled() Why this matters: The old tests were mocking validate_and_resolve_url(), which meant they weren't actually testing if SSRF protection works. They were just testing error handling. Real SSRF vulnerabilities could have slipped through. The new tests: 1. Let the real SSRF protection code run 2. Verify actual private IPs are blocked 3. Verify public URLs are allowed 4. Verify allowlist bypasses work correctly 5. Verify DNS pinning transport is actually used All 41 tests pass (37 passed, 4 skipped for version checks). --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
This commit is contained in:
Janardan Singh Kavia
committed by
GitHub
GitHub
parent
60627bec3a
commit
6993a19bbe
@ -1327,7 +1327,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
}
|
||||
],
|
||||
"total_dependencies": 4
|
||||
|
||||
@ -1781,7 +1781,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
}
|
||||
],
|
||||
"total_dependencies": 7
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -949,7 +949,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
}
|
||||
],
|
||||
"total_dependencies": 4
|
||||
|
||||
@ -2641,7 +2641,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
}
|
||||
],
|
||||
"total_dependencies": 4
|
||||
|
||||
@ -1643,7 +1643,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
}
|
||||
],
|
||||
"total_dependencies": 4
|
||||
|
||||
@ -274,7 +274,7 @@
|
||||
},
|
||||
{
|
||||
"name": "googleapiclient",
|
||||
"version": "2.194.0"
|
||||
"version": "2.195.0"
|
||||
},
|
||||
{
|
||||
"name": "lfx",
|
||||
|
||||
@ -363,7 +363,15 @@ class TestAPIRequestComponent(ComponentTestBaseWithoutClient):
|
||||
|
||||
|
||||
class TestAPIRequestSSRFProtection:
|
||||
"""Test SSRF protection in API Request component."""
|
||||
"""Rewritten SSRF Protection Tests for API Request Component.
|
||||
|
||||
These tests properly test the actual SSRF protection implementation without mocking
|
||||
the core security functions. They verify:
|
||||
1. Real SSRF blocking with actual private IPs
|
||||
2. DNS pinning actually prevents rebinding
|
||||
3. Allowlist functionality works correctly
|
||||
4. Custom transport is used when protection is enabled.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def component_class(self):
|
||||
@ -376,10 +384,10 @@ class TestAPIRequestSSRFProtection:
|
||||
return {
|
||||
"url_input": "https://example.com/api/test",
|
||||
"method": "GET",
|
||||
"headers": [{"key": "User-Agent", "value": "test-agent"}],
|
||||
"headers": [],
|
||||
"body": [],
|
||||
"timeout": 30,
|
||||
"follow_redirects": False, # Changed default for SSRF security
|
||||
"follow_redirects": False,
|
||||
"save_to_file": False,
|
||||
"include_httpx_metadata": False,
|
||||
"mode": "URL",
|
||||
@ -392,73 +400,93 @@ class TestAPIRequestSSRFProtection:
|
||||
"""Return a component instance."""
|
||||
return component_class(**default_kwargs)
|
||||
|
||||
async def test_ssrf_protection_disabled_by_default(self, component):
|
||||
"""Test that SSRF protection is disabled by default (warn-only mode)."""
|
||||
# Even with protection disabled, this should not raise
|
||||
async def test_ssrf_protection_disabled_allows_all_urls(self, component):
|
||||
"""Test that when SSRF protection is disabled, all URLs are allowed."""
|
||||
component.url_input = "http://127.0.0.1:8080"
|
||||
|
||||
with respx.mock:
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://127.0.0.1:8080").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
# Should not raise (protection is off by default)
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["result"]["status"] == "ok"
|
||||
|
||||
async def test_ssrf_protection_enabled_blocks_localhost(self, component):
|
||||
"""Test that SSRF protection blocks localhost when enabled."""
|
||||
async def test_ssrf_protection_blocks_localhost_127_0_0_1(self, component):
|
||||
"""Test that SSRF protection blocks 127.0.0.1 (localhost)."""
|
||||
component.url_input = "http://127.0.0.1:8080/admin"
|
||||
|
||||
# Enable SSRF protection in enforcement mode
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch("lfx.components.data_source.api_request.validate_url_for_ssrf") as mock_validate,
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
from lfx.utils.ssrf_protection import SSRFProtectionError
|
||||
await component.make_api_request()
|
||||
|
||||
# Make it raise in enforcement mode
|
||||
mock_validate.side_effect = SSRFProtectionError("Access to 127.0.0.1 blocked")
|
||||
async def test_ssrf_protection_blocks_localhost_0_0_0_0(self, component):
|
||||
"""Test that SSRF protection blocks 0.0.0.0."""
|
||||
component.url_input = "http://0.0.0.0:8080/admin"
|
||||
|
||||
with pytest.raises(ValueError, match="SSRF Protection"):
|
||||
await component.make_api_request()
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
await component.make_api_request()
|
||||
|
||||
async def test_ssrf_protection_enabled_blocks_private_networks(self, component):
|
||||
"""Test that SSRF protection blocks private network IPs when enabled."""
|
||||
private_ips = [
|
||||
"http://192.168.1.1/config",
|
||||
"http://10.0.0.1/admin",
|
||||
"http://172.16.0.1/internal",
|
||||
]
|
||||
async def test_ssrf_protection_blocks_private_network_192_168(self, component):
|
||||
"""Test that SSRF protection blocks 192.168.x.x private network."""
|
||||
component.url_input = "http://192.168.1.1/config"
|
||||
|
||||
with patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}):
|
||||
for url in private_ips:
|
||||
component.url_input = url
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
await component.make_api_request()
|
||||
|
||||
with patch("lfx.components.data_source.api_request.validate_url_for_ssrf") as mock_validate:
|
||||
from lfx.utils.ssrf_protection import SSRFProtectionError
|
||||
async def test_ssrf_protection_blocks_private_network_10_0(self, component):
|
||||
"""Test that SSRF protection blocks 10.x.x.x private network."""
|
||||
component.url_input = "http://10.0.0.1/admin"
|
||||
|
||||
mock_validate.side_effect = SSRFProtectionError(f"Access to {url} blocked")
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
await component.make_api_request()
|
||||
|
||||
with pytest.raises(ValueError, match="SSRF Protection"):
|
||||
await component.make_api_request()
|
||||
async def test_ssrf_protection_blocks_private_network_172_16(self, component):
|
||||
"""Test that SSRF protection blocks 172.16.x.x private network."""
|
||||
component.url_input = "http://172.16.0.1/internal"
|
||||
|
||||
async def test_ssrf_protection_enabled_blocks_metadata_endpoint(self, component):
|
||||
"""Test that SSRF protection blocks cloud metadata endpoints when enabled."""
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
await component.make_api_request()
|
||||
|
||||
async def test_ssrf_protection_blocks_cloud_metadata_endpoint(self, component):
|
||||
"""Test that SSRF protection blocks AWS/GCP metadata endpoint."""
|
||||
component.url_input = "http://169.254.169.254/latest/meta-data/"
|
||||
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch("lfx.components.data_source.api_request.validate_url_for_ssrf") as mock_validate,
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
from lfx.utils.ssrf_protection import SSRFProtectionError
|
||||
await component.make_api_request()
|
||||
|
||||
mock_validate.side_effect = SSRFProtectionError("Access to 169.254.169.254 blocked")
|
||||
async def test_ssrf_protection_blocks_link_local_169_254(self, component):
|
||||
"""Test that SSRF protection blocks link-local addresses."""
|
||||
component.url_input = "http://169.254.1.1/api"
|
||||
|
||||
with pytest.raises(ValueError, match="SSRF Protection"):
|
||||
await component.make_api_request()
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
pytest.raises(ValueError, match="SSRF Protection"),
|
||||
):
|
||||
await component.make_api_request()
|
||||
|
||||
@respx.mock
|
||||
async def test_ssrf_protection_allows_public_urls(self, component):
|
||||
"""Test that SSRF protection allows public URLs."""
|
||||
"""Test that SSRF protection allows legitimate public URLs."""
|
||||
public_urls = [
|
||||
"https://api.openai.com/v1/chat/completions",
|
||||
"https://api.github.com/repos/langflow-ai/langflow",
|
||||
@ -470,56 +498,133 @@ class TestAPIRequestSSRFProtection:
|
||||
component.url_input = url
|
||||
respx.get(url).mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
# Should not raise - these are public URLs
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["result"]["status"] == "ok"
|
||||
|
||||
async def test_ssrf_protection_allowlist_bypass(self, component):
|
||||
"""Test that allowlisted hosts bypass SSRF protection."""
|
||||
async def test_ssrf_allowlist_hostname(self, component):
|
||||
"""Test that allowlisted hostnames bypass SSRF protection."""
|
||||
component.url_input = "http://internal.company.local/api"
|
||||
|
||||
with (
|
||||
patch.dict(
|
||||
os.environ,
|
||||
{"LANGFLOW_SSRF_PROTECTION_ENABLED": "true", "LANGFLOW_SSRF_ALLOWED_HOSTS": "internal.company.local"},
|
||||
{
|
||||
"LANGFLOW_SSRF_PROTECTION_ENABLED": "true",
|
||||
"LANGFLOW_SSRF_ALLOWED_HOSTS": "internal.company.local",
|
||||
},
|
||||
),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://internal.company.local/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
# Should not raise - host is in allowlist
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["result"]["status"] == "ok"
|
||||
|
||||
async def test_ssrf_protection_allowlist_cidr(self, component):
|
||||
async def test_ssrf_allowlist_ip_address(self, component):
|
||||
"""Test that allowlisted IP addresses bypass SSRF protection."""
|
||||
component.url_input = "http://192.168.1.100/api"
|
||||
|
||||
with (
|
||||
patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"LANGFLOW_SSRF_PROTECTION_ENABLED": "true",
|
||||
"LANGFLOW_SSRF_ALLOWED_HOSTS": "192.168.1.100",
|
||||
},
|
||||
),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://192.168.1.100/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["result"]["status"] == "ok"
|
||||
|
||||
async def test_ssrf_allowlist_cidr_range(self, component):
|
||||
"""Test that CIDR ranges in allowlist work correctly."""
|
||||
component.url_input = "http://192.168.1.5/api"
|
||||
|
||||
with (
|
||||
patch.dict(
|
||||
os.environ,
|
||||
{"LANGFLOW_SSRF_PROTECTION_ENABLED": "true", "LANGFLOW_SSRF_ALLOWED_HOSTS": "192.168.1.0/24"},
|
||||
{
|
||||
"LANGFLOW_SSRF_PROTECTION_ENABLED": "true",
|
||||
"LANGFLOW_SSRF_ALLOWED_HOSTS": "192.168.1.0/24",
|
||||
},
|
||||
),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://192.168.1.5/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["result"]["status"] == "ok"
|
||||
|
||||
async def test_ssrf_allowlist_multiple_entries(self, component):
|
||||
"""Test that multiple allowlist entries work correctly."""
|
||||
component.url_input = "http://192.168.1.5/api"
|
||||
|
||||
with (
|
||||
patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"LANGFLOW_SSRF_PROTECTION_ENABLED": "true",
|
||||
"LANGFLOW_SSRF_ALLOWED_HOSTS": "localhost,192.168.1.0/24,internal.local",
|
||||
},
|
||||
),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://192.168.1.5/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
# Should not raise - IP is in allowlisted CIDR range
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
|
||||
async def test_ssrf_protection_warn_only_mode(self, component):
|
||||
"""Test that warn_only mode logs warnings instead of blocking."""
|
||||
component.url_input = "http://127.0.0.1:8080/admin"
|
||||
async def test_dns_pinning_is_used_when_protection_enabled(self, component):
|
||||
"""Test that DNS pinning (custom transport) is used when SSRF protection is enabled."""
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
with patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}), respx.mock:
|
||||
respx.get("http://127.0.0.1:8080/admin").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
component.url_input = "https://example.com/api"
|
||||
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch("lfx.components.data_source.api_request.create_ssrf_protected_client") as mock_create_client,
|
||||
respx.mock,
|
||||
):
|
||||
# Mock the context manager returned by create_ssrf_protected_client
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__.return_value = mock_client
|
||||
mock_client.__aexit__.return_value = None
|
||||
mock_create_client.return_value = mock_client
|
||||
|
||||
# Mock the make_request to return a Data object
|
||||
component.make_request = AsyncMock(return_value=Data(data={"status": "ok"}))
|
||||
|
||||
await component.make_api_request()
|
||||
|
||||
# Verify that create_ssrf_protected_client was called (DNS pinning is used)
|
||||
mock_create_client.assert_called_once()
|
||||
call_kwargs = mock_create_client.call_args[1]
|
||||
assert call_kwargs["hostname"] == "example.com"
|
||||
assert len(call_kwargs["validated_ips"]) > 0 # Should have validated IPs
|
||||
|
||||
async def test_normal_client_used_when_protection_disabled(self, component):
|
||||
"""Test that normal httpx client is used when SSRF protection is disabled."""
|
||||
component.url_input = "https://example.com/api"
|
||||
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
patch("lfx.components.data_source.api_request.create_ssrf_protected_client") as mock_create_client,
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("https://example.com/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
# In warn_only mode (default), should not raise but should log
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
|
||||
# TODO: In next major version, this should raise instead of just warning
|
||||
# Verify that create_ssrf_protected_client was NOT called
|
||||
mock_create_client.assert_not_called()
|
||||
assert isinstance(result, Data)
|
||||
|
||||
async def test_follow_redirects_security_warning(self, component):
|
||||
"""Test that enabling follow_redirects logs a security warning."""
|
||||
@ -527,58 +632,55 @@ class TestAPIRequestSSRFProtection:
|
||||
|
||||
component.url_input = "https://example.com/api"
|
||||
component.follow_redirects = True
|
||||
|
||||
# Mock the log method to capture what's being logged
|
||||
component.log = MagicMock()
|
||||
|
||||
with respx.mock:
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("https://example.com/api").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
await component.make_api_request()
|
||||
|
||||
# Verify log was called with security warning
|
||||
# Verify security warning was logged
|
||||
component.log.assert_called()
|
||||
log_call_args = component.log.call_args[0][0]
|
||||
assert "Security Warning" in log_call_args
|
||||
assert "SSRF bypass" in log_call_args
|
||||
assert "redirects are enabled" in log_call_args
|
||||
all_log_messages = [call[0][0] for call in component.log.call_args_list]
|
||||
security_warning_found = any("Security Warning" in msg and "SSRF bypass" in msg for msg in all_log_messages)
|
||||
assert security_warning_found, f"Security warning not found in: {all_log_messages}"
|
||||
|
||||
async def test_follow_redirects_disabled_by_default(self, component):
|
||||
"""Test that follow_redirects is disabled by default."""
|
||||
# Verify the default value is False
|
||||
assert component.follow_redirects is False
|
||||
|
||||
async def test_url_normalization(self, component):
|
||||
async def test_url_normalization_adds_https(self, component):
|
||||
"""Test that URLs without protocol get normalized to https://."""
|
||||
# Test URL without protocol
|
||||
component.url_input = "example.com"
|
||||
|
||||
with respx.mock:
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("https://example.com").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["source"] == "https://example.com"
|
||||
|
||||
async def test_url_normalization_preserves_protocol(self, component):
|
||||
"""Test that URLs with protocol are not modified."""
|
||||
# Test http:// is preserved
|
||||
async def test_url_normalization_preserves_http(self, component):
|
||||
"""Test that http:// protocol is preserved."""
|
||||
component.url_input = "http://example.com"
|
||||
|
||||
with respx.mock:
|
||||
with (
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
respx.mock,
|
||||
):
|
||||
respx.get("http://example.com").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["source"] == "http://example.com"
|
||||
|
||||
# Test https:// is preserved
|
||||
component.url_input = "https://example.com"
|
||||
async def test_invalid_url_raises_error(self, component):
|
||||
"""Test that invalid URLs raise ValueError."""
|
||||
component.url_input = "not_a_valid_url"
|
||||
|
||||
with respx.mock:
|
||||
respx.get("https://example.com").mock(return_value=Response(200, json={"status": "ok"}))
|
||||
with pytest.raises(ValueError, match="Invalid URL provided"):
|
||||
await component.make_api_request()
|
||||
|
||||
result = await component.make_api_request()
|
||||
assert isinstance(result, Data)
|
||||
assert result.data["source"] == "https://example.com"
|
||||
async def test_follow_redirects_disabled_by_default(self, component):
|
||||
"""Test that follow_redirects is disabled by default for security."""
|
||||
assert component.follow_redirects is False
|
||||
|
||||
@ -0,0 +1,374 @@
|
||||
"""Test DNS rebinding protection in API Request component.
|
||||
|
||||
This test suite verifies that the DNS pinning implementation prevents
|
||||
DNS rebinding attacks that could bypass SSRF protection.
|
||||
"""
|
||||
# ruff: noqa: ARG001, SIM117
|
||||
|
||||
import os
|
||||
import socket
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
import pytest
|
||||
from lfx.components.data_source.api_request import APIRequestComponent
|
||||
from lfx.schema import Data
|
||||
|
||||
|
||||
class TestDNSRebindingProtection:
|
||||
"""Test DNS rebinding attack prevention through DNS pinning."""
|
||||
|
||||
@pytest.fixture
|
||||
def component(self):
|
||||
"""Create a basic API request component."""
|
||||
return APIRequestComponent(
|
||||
url_input="http://rebinding.test:8080/api",
|
||||
method="GET",
|
||||
headers=[],
|
||||
body=[],
|
||||
timeout=30,
|
||||
follow_redirects=False,
|
||||
save_to_file=False,
|
||||
include_httpx_metadata=False,
|
||||
mode="URL",
|
||||
curl_input="",
|
||||
query_params={},
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_prevents_rebinding_attack(self, component):
|
||||
"""Test that DNS pinning prevents DNS rebinding attacks.
|
||||
|
||||
This test simulates a DNS rebinding attack where:
|
||||
1. First DNS lookup (validation): returns public IP (8.8.8.8)
|
||||
2. Second DNS lookup (httpx): would return localhost (127.0.0.1)
|
||||
|
||||
With DNS pinning, the second lookup should NOT happen - the validated
|
||||
IP from the first lookup should be used directly at the network layer.
|
||||
"""
|
||||
call_count = 0
|
||||
connected_to_ip = None
|
||||
|
||||
def mock_getaddrinfo(_hostname, _port, *_args, **_kwargs):
|
||||
"""Mock DNS resolution to simulate rebinding attack."""
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
|
||||
if call_count == 1:
|
||||
# First call (during validation): return public IP
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("8.8.8.8", 0))]
|
||||
# Second call (during httpx request): return localhost
|
||||
# This simulates the DNS rebinding attack
|
||||
# With DNS pinning, this should NOT be called
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", 0))]
|
||||
|
||||
# Mock the network backend's connect_tcp to capture the actual IP being connected to
|
||||
async def mock_connect_tcp(self, host, port, **kwargs):
|
||||
"""Capture the IP that's actually being connected to."""
|
||||
nonlocal connected_to_ip
|
||||
connected_to_ip = host
|
||||
# Return a mock stream with proper format (list of bytes)
|
||||
return httpcore.AsyncMockStream(
|
||||
[
|
||||
b"HTTP/1.1 200 OK\r\n",
|
||||
b"Content-Type: application/json\r\n",
|
||||
b"Content-Length: 15\r\n",
|
||||
b"\r\n",
|
||||
b'{"status":"ok"}',
|
||||
]
|
||||
)
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch.object(httpcore.AnyIOBackend, "connect_tcp", mock_connect_tcp),
|
||||
):
|
||||
# Execute the request
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert result is not None
|
||||
|
||||
# CRITICAL CHECK: DNS should only be called once (during validation)
|
||||
# If called twice, DNS pinning failed and the attack succeeded
|
||||
assert call_count == 1, (
|
||||
f"DNS was called {call_count} times. Expected 1 (validation only). "
|
||||
"DNS pinning failed - the component is vulnerable to DNS rebinding!"
|
||||
)
|
||||
|
||||
# Verify the connection was made to the pinned IP
|
||||
assert connected_to_ip is not None, "No TCP connection was made"
|
||||
assert connected_to_ip == "8.8.8.8", (
|
||||
f"Connection should be to pinned IP 8.8.8.8, but was to {connected_to_ip}"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_preserves_hostname_in_header(self, component):
|
||||
"""Test that DNS pinning connects to pinned IP while preserving hostname for TLS.
|
||||
|
||||
With network-level DNS pinning:
|
||||
- TCP connection goes to the pinned IP (93.184.216.34)
|
||||
- URL preserves the original hostname (rebinding.test)
|
||||
- This allows TLS SNI and certificate verification to work correctly
|
||||
|
||||
This is important for:
|
||||
- Virtual hosting (multiple sites on same IP)
|
||||
- SNI (Server Name Indication) for HTTPS
|
||||
- Certificate verification (cert is for hostname, not IP)
|
||||
"""
|
||||
connected_to_ip = None
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution."""
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
# Mock network backend to capture TCP connection
|
||||
async def mock_connect_tcp(self, host, port, **kwargs):
|
||||
"""Capture the IP that TCP connects to."""
|
||||
nonlocal connected_to_ip
|
||||
connected_to_ip = host
|
||||
return httpcore.AsyncMockStream(
|
||||
[
|
||||
b"HTTP/1.1 200 OK\r\n",
|
||||
b"Content-Type: application/json\r\n",
|
||||
b"Content-Length: 15\r\n",
|
||||
b"\r\n",
|
||||
b'{"status":"ok"}',
|
||||
]
|
||||
)
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch.object(httpcore.AnyIOBackend, "connect_tcp", mock_connect_tcp),
|
||||
):
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert result is not None
|
||||
|
||||
# Verify TCP connection was made to the pinned IP
|
||||
assert connected_to_ip is not None, "No TCP connection was made"
|
||||
assert connected_to_ip == "93.184.216.34", (
|
||||
f"TCP connection should be to pinned IP 93.184.216.34, but was to {connected_to_ip}"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_with_direct_ip_address(self, component):
|
||||
"""Test that direct IP addresses work correctly (no DNS pinning needed)."""
|
||||
component.url_input = "http://93.184.216.34:8080/api"
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution - should not be called for direct IPs."""
|
||||
# For direct IPs, socket.getaddrinfo is still called but returns the same IP
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
mock_response = httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch("httpx.AsyncClient.request", return_value=mock_response) as mock_request,
|
||||
):
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert isinstance(result, Data)
|
||||
assert mock_request.called
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_disabled_when_protection_disabled(self, component):
|
||||
"""Test that DNS pinning is skipped when SSRF protection is disabled."""
|
||||
call_count = 0
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution."""
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("8.8.8.8", 0))]
|
||||
|
||||
mock_response = httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "false"}),
|
||||
patch("httpx.AsyncClient.request", return_value=mock_response) as mock_request,
|
||||
):
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert isinstance(result, Data)
|
||||
assert mock_request.called
|
||||
|
||||
# When protection is disabled, DNS pinning is not used
|
||||
# So DNS might be called multiple times (once by httpx)
|
||||
# This is expected behavior when protection is off
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_blocks_private_ip_resolution(self, component):
|
||||
"""Test that DNS pinning blocks hostnames that resolve to private IPs."""
|
||||
component.url_input = "http://internal.example.com/api"
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution to return private IP."""
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("192.168.1.1", 0))]
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
):
|
||||
# Should raise ValueError due to SSRF protection
|
||||
with pytest.raises(ValueError, match="SSRF Protection"):
|
||||
await component.make_api_request()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_with_ipv6(self, component):
|
||||
"""Test that DNS pinning works with IPv6 addresses.
|
||||
|
||||
With network-level DNS pinning:
|
||||
- TCP connection goes to the IPv6 address
|
||||
- URL preserves the original hostname
|
||||
- IPv6 addresses don't need brackets in connect_tcp (only in URLs)
|
||||
"""
|
||||
component.url_input = "http://ipv6.example.com/api"
|
||||
connected_to_ip = None
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution to return IPv6."""
|
||||
return [(socket.AF_INET6, socket.SOCK_STREAM, 6, "", ("2001:4860:4860::8888", 0))]
|
||||
|
||||
# Mock network backend to capture TCP connection
|
||||
async def mock_connect_tcp(self, host, port, **kwargs):
|
||||
"""Capture the IP that TCP connects to."""
|
||||
nonlocal connected_to_ip
|
||||
connected_to_ip = host
|
||||
return httpcore.AsyncMockStream(
|
||||
[
|
||||
b"HTTP/1.1 200 OK\r\n",
|
||||
b"Content-Type: application/json\r\n",
|
||||
b"Content-Length: 15\r\n",
|
||||
b"\r\n",
|
||||
b'{"status":"ok"}',
|
||||
]
|
||||
)
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch.object(httpcore.AnyIOBackend, "connect_tcp", mock_connect_tcp),
|
||||
):
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert isinstance(result, Data)
|
||||
|
||||
# Verify TCP connection was made to the IPv6 address
|
||||
assert connected_to_ip is not None, "No TCP connection was made"
|
||||
assert connected_to_ip == "2001:4860:4860::8888", (
|
||||
f"TCP connection should be to IPv6 2001:4860:4860::8888, but was to {connected_to_ip}"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_with_allowlisted_host(self, component):
|
||||
"""Test that allowlisted hosts bypass DNS pinning and preserve original hostname."""
|
||||
component.url_input = "http://internal.example.com:8080/api" # Use a valid hostname format
|
||||
captured_request = None
|
||||
|
||||
def mock_getaddrinfo(hostname, port, *args, **kwargs):
|
||||
"""Mock DNS resolution."""
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("10.0.0.1", 0))]
|
||||
|
||||
# Mock the transport's handle_async_request to capture the rewritten request
|
||||
async def mock_handle_async_request(_self, request):
|
||||
"""Capture the request after transport rewrite."""
|
||||
nonlocal captured_request
|
||||
captured_request = request
|
||||
return httpx.Response(200, json={"status": "ok"}, request=request)
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"LANGFLOW_SSRF_PROTECTION_ENABLED": "true",
|
||||
"LANGFLOW_SSRF_ALLOWED_HOSTS": "internal.example.com,10.0.0.1",
|
||||
},
|
||||
),
|
||||
# Patch get_allowed_hosts to return the allowlist directly
|
||||
patch(
|
||||
"lfx.utils.ssrf_protection.get_allowed_hosts",
|
||||
return_value=["internal.example.com", "10.0.0.1"],
|
||||
),
|
||||
patch.object(httpx.AsyncHTTPTransport, "handle_async_request", mock_handle_async_request),
|
||||
):
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify the request succeeded
|
||||
assert isinstance(result, Data)
|
||||
assert captured_request is not None, "Transport did not capture request"
|
||||
|
||||
# Verify the original hostname is preserved (no DNS pinning for allowlisted hosts)
|
||||
url_str = str(captured_request.url)
|
||||
assert "internal.example.com" in url_str, f"Allowlisted host should preserve hostname: {url_str}"
|
||||
assert "10.0.0.1" not in url_str, f"Allowlisted host should not use IP: {url_str}"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dns_pinning_with_multiple_ips_fallback(self, component):
|
||||
"""Test that DNS pinning tries multiple IPs when first one fails (dual-stack/load balancing)."""
|
||||
component.url_input = "http://dual-stack.example.com/api"
|
||||
|
||||
# Track which IPs were attempted
|
||||
attempted_ips = []
|
||||
|
||||
def mock_getaddrinfo(host, port, family=0, type_=0, proto=0, flags=0):
|
||||
"""Mock DNS resolution to return multiple IPs (IPv4 and IPv6)."""
|
||||
if host == "dual-stack.example.com":
|
||||
# Return both IPv4 and IPv6 addresses (dual-stack)
|
||||
return [
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0)), # IPv4
|
||||
(socket.AF_INET6, socket.SOCK_STREAM, 6, "", ("2606:2800:220:1:248:1893:25c8:1946", 0)), # IPv6
|
||||
]
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (host, 0))]
|
||||
|
||||
async def mock_connect_tcp(self, host, port, **kwargs):
|
||||
"""Mock connection that fails on first IP, succeeds on second."""
|
||||
nonlocal attempted_ips
|
||||
attempted_ips.append(host)
|
||||
|
||||
# First IP fails (simulating IPv4 unreachable)
|
||||
if host == "93.184.216.34":
|
||||
msg = "Connection refused"
|
||||
raise OSError(msg)
|
||||
|
||||
# Second IP succeeds (IPv6 works)
|
||||
if host == "2606:2800:220:1:248:1893:25c8:1946":
|
||||
return httpcore.AsyncMockStream(
|
||||
[
|
||||
b"HTTP/1.1 200 OK\r\n",
|
||||
b"Content-Type: application/json\r\n",
|
||||
b"Content-Length: 15\r\n",
|
||||
b"\r\n",
|
||||
b'{"status":"ok"}',
|
||||
]
|
||||
)
|
||||
|
||||
msg = f"Unexpected host: {host}"
|
||||
raise OSError(msg)
|
||||
|
||||
with (
|
||||
patch("socket.getaddrinfo", side_effect=mock_getaddrinfo),
|
||||
patch.dict(os.environ, {"LANGFLOW_SSRF_PROTECTION_ENABLED": "true"}),
|
||||
patch.object(httpcore.AnyIOBackend, "connect_tcp", mock_connect_tcp),
|
||||
):
|
||||
# Execute the request
|
||||
result = await component.make_api_request()
|
||||
|
||||
# Verify both IPs were attempted in order
|
||||
assert len(attempted_ips) == 2
|
||||
assert attempted_ips[0] == "93.184.216.34" # IPv4 tried first
|
||||
assert attempted_ips[1] == "2606:2800:220:1:248:1893:25c8:1946" # IPv6 tried second
|
||||
|
||||
# Verify the result (should succeed with second IP)
|
||||
assert isinstance(result, Data)
|
||||
assert result.data is not None
|
||||
File diff suppressed because one or more lines are too long
@ -27,7 +27,10 @@ from lfx.io import (
|
||||
from lfx.schema.data import Data
|
||||
from lfx.schema.dotdict import dotdict
|
||||
from lfx.utils.component_utils import set_current_fields, set_field_advanced, set_field_display
|
||||
from lfx.utils.ssrf_protection import SSRFProtectionError, validate_url_for_ssrf
|
||||
|
||||
# SSRF Protection imports - for preventing Server-Side Request Forgery attacks
|
||||
from lfx.utils.ssrf_protection import SSRFProtectionError, validate_and_resolve_url
|
||||
from lfx.utils.ssrf_transport import create_ssrf_protected_client
|
||||
|
||||
# Define fields for each mode
|
||||
MODE_FIELDS = {
|
||||
@ -422,7 +425,22 @@ class APIRequestComponent(Component):
|
||||
return {}
|
||||
|
||||
async def make_api_request(self) -> Data:
|
||||
"""Make HTTP request with optimized parameter handling."""
|
||||
"""Make HTTP request with SSRF protection and DNS pinning.
|
||||
|
||||
This method implements comprehensive SSRF (Server-Side Request Forgery) protection
|
||||
using DNS pinning to prevent DNS rebinding attacks. The protection works by:
|
||||
1. Validating the URL and resolving DNS during security check
|
||||
2. Pinning the validated IP address
|
||||
3. Forcing the HTTP client to use the pinned IP for the actual request
|
||||
4. Ignoring any subsequent DNS changes (prevents rebinding attacks)
|
||||
|
||||
Returns:
|
||||
Data: Response data from the HTTP request
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is invalid or blocked by SSRF protection
|
||||
"""
|
||||
# Extract request parameters
|
||||
method = self.method
|
||||
url = self.url_input.strip() if isinstance(self.url_input, str) else ""
|
||||
headers = self.headers or {}
|
||||
@ -432,7 +450,8 @@ class APIRequestComponent(Component):
|
||||
save_to_file = self.save_to_file
|
||||
include_httpx_metadata = self.include_httpx_metadata
|
||||
|
||||
# Security warning when redirects are enabled
|
||||
# Security warning: HTTP redirects can bypass SSRF protection
|
||||
# A public URL could redirect to an internal resource
|
||||
if follow_redirects:
|
||||
self.log(
|
||||
"Security Warning: HTTP redirects are enabled. This may allow SSRF bypass attacks "
|
||||
@ -440,51 +459,122 @@ class APIRequestComponent(Component):
|
||||
"Only enable this if you trust the target server."
|
||||
)
|
||||
|
||||
# if self.mode == "cURL" and self.curl_input:
|
||||
# self._build_config = self.parse_curl(self.curl_input, dotdict())
|
||||
# # After parsing curl, get the normalized URL
|
||||
# url = self._build_config["url_input"]["value"]
|
||||
|
||||
# Normalize URL before validation
|
||||
# Normalize URL (add https:// if no protocol specified)
|
||||
url = self._normalize_url(url)
|
||||
|
||||
# Validate URL
|
||||
# Basic URL format validation
|
||||
if not validators.url(url):
|
||||
msg = f"Invalid URL provided: {url}"
|
||||
raise ValueError(msg)
|
||||
|
||||
# SSRF Protection: Validate URL to prevent access to internal resources
|
||||
# TODO: In next major version (2.0), remove warn_only=True to enforce blocking
|
||||
# ============================================================================
|
||||
# SSRF Protection with DNS Pinning
|
||||
# ============================================================================
|
||||
# This prevents DNS rebinding attacks by:
|
||||
# 1. Resolving DNS and validating IPs during security check
|
||||
# 2. Pinning the validated IP address
|
||||
# 3. Using a custom HTTP transport that forces use of the pinned IP
|
||||
# 4. Ignoring any new DNS resolutions (prevents rebinding)
|
||||
#
|
||||
# Without DNS pinning, an attacker could:
|
||||
# - First DNS lookup: returns public IP (passes validation)
|
||||
# - Second DNS lookup: returns internal IP (bypasses protection)
|
||||
# - Attack succeeds: accesses internal services
|
||||
#
|
||||
# With DNS pinning:
|
||||
# - First DNS lookup: returns public IP (passes validation)
|
||||
# - IP is pinned: "example.com = 93.184.216.34"
|
||||
# - HTTP request: uses pinned IP directly (no new DNS lookup)
|
||||
# - Attack fails: even if DNS changes, we use the validated IP
|
||||
# ============================================================================
|
||||
|
||||
try:
|
||||
validate_url_for_ssrf(url, warn_only=True)
|
||||
# Validate URL and get validated IPs for DNS pinning
|
||||
_validated_url, validated_ips = validate_and_resolve_url(url)
|
||||
|
||||
# Log DNS pinning information for security auditing
|
||||
if validated_ips:
|
||||
self.log(f"SSRF Protection: Using DNS pinning with {len(validated_ips)} validated IP(s)")
|
||||
|
||||
except SSRFProtectionError as e:
|
||||
# This will only raise if SSRF protection is enabled and warn_only=False
|
||||
# SSRF protection blocked the request (private IP, internal network, etc.)
|
||||
msg = f"SSRF Protection: {e}"
|
||||
raise ValueError(msg) from e
|
||||
|
||||
# Process query parameters
|
||||
# Process query parameters (from string or Data object)
|
||||
if isinstance(self.query_params, str):
|
||||
query_params = dict(parse_qsl(self.query_params))
|
||||
else:
|
||||
query_params = self.query_params.data if self.query_params else {}
|
||||
|
||||
# Process headers and body
|
||||
# Process headers and body into proper format
|
||||
headers = self._process_headers(headers)
|
||||
body = self._process_body(body)
|
||||
url = self.add_query_params(url, query_params)
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
result = await self.make_request(
|
||||
client,
|
||||
method,
|
||||
url,
|
||||
headers,
|
||||
body,
|
||||
timeout,
|
||||
follow_redirects=follow_redirects,
|
||||
save_to_file=save_to_file,
|
||||
include_httpx_metadata=include_httpx_metadata,
|
||||
)
|
||||
# ============================================================================
|
||||
# Create HTTP Client with DNS Pinning (if SSRF protection enabled)
|
||||
# ============================================================================
|
||||
from lfx.utils.ssrf_protection import is_ssrf_protection_enabled
|
||||
|
||||
if is_ssrf_protection_enabled() and validated_ips:
|
||||
# SSRF protection is enabled and DNS pinning is needed
|
||||
# Extract hostname from the final URL (after query params added)
|
||||
hostname = urlparse(url).hostname
|
||||
|
||||
if hostname:
|
||||
# Create client with DNS pinning to prevent rebinding attacks
|
||||
# The custom transport will try validated IPs in order (supports dual-stack/load balancing)
|
||||
# while preserving the Host header for virtual hosting/SNI
|
||||
async with create_ssrf_protected_client(
|
||||
hostname=hostname,
|
||||
validated_ips=validated_ips, # Pass all validated IPs
|
||||
) as client:
|
||||
result = await self.make_request(
|
||||
client,
|
||||
method,
|
||||
url,
|
||||
headers,
|
||||
body,
|
||||
timeout,
|
||||
follow_redirects=follow_redirects,
|
||||
save_to_file=save_to_file,
|
||||
include_httpx_metadata=include_httpx_metadata,
|
||||
)
|
||||
else:
|
||||
# Hostname extraction failed - fallback to normal client
|
||||
# This should rarely happen as URL was already validated
|
||||
async with httpx.AsyncClient() as client:
|
||||
result = await self.make_request(
|
||||
client,
|
||||
method,
|
||||
url,
|
||||
headers,
|
||||
body,
|
||||
timeout,
|
||||
follow_redirects=follow_redirects,
|
||||
save_to_file=save_to_file,
|
||||
include_httpx_metadata=include_httpx_metadata,
|
||||
)
|
||||
else:
|
||||
# No DNS pinning needed - use normal client
|
||||
# This happens when SSRF protection is disabled or host is allowlisted
|
||||
# - SSRF protection is disabled
|
||||
# - Host is in the allowlist (e.g., localhost for Ollama)
|
||||
# - Direct IP address was used (no DNS to pin)
|
||||
async with httpx.AsyncClient() as client:
|
||||
result = await self.make_request(
|
||||
client,
|
||||
method,
|
||||
url,
|
||||
headers,
|
||||
body,
|
||||
timeout,
|
||||
follow_redirects=follow_redirects,
|
||||
save_to_file=save_to_file,
|
||||
include_httpx_metadata=include_httpx_metadata,
|
||||
)
|
||||
|
||||
self.status = result
|
||||
return result
|
||||
|
||||
|
||||
@ -398,12 +398,12 @@ class Settings(BaseSettings):
|
||||
use hardware-level isolation to restrict access."""
|
||||
|
||||
# SSRF Protection
|
||||
ssrf_protection_enabled: bool = False
|
||||
ssrf_protection_enabled: bool = True
|
||||
"""If set to True, Langflow will enable SSRF (Server-Side Request Forgery) protection.
|
||||
When enabled, blocks requests to private IP ranges, localhost, and cloud metadata endpoints.
|
||||
When False (default), no URL validation is performed, allowing requests to any destination
|
||||
When False, no URL validation is performed, allowing requests to any destination
|
||||
including internal services, private networks, and cloud metadata endpoints.
|
||||
Default is False for backward compatibility. In v2.0, this will be changed to True.
|
||||
Default is True to protect against SSRF attacks including DNS rebinding.
|
||||
|
||||
Note: When ssrf_protection_enabled is disabled, the ssrf_allowed_hosts setting is ignored and has no effect."""
|
||||
ssrf_allowed_hosts: list[str] = []
|
||||
|
||||
@ -13,15 +13,9 @@ IMPORTANT: HTTP Redirects
|
||||
See: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
|
||||
|
||||
Configuration:
|
||||
LANGFLOW_SSRF_PROTECTION_ENABLED: Enable/disable SSRF protection (default: false)
|
||||
TODO: Change default to true in next major version (2.0)
|
||||
LANGFLOW_SSRF_PROTECTION_ENABLED: Enable/disable SSRF protection (default: true)
|
||||
LANGFLOW_SSRF_ALLOWED_HOSTS: Comma-separated list of allowed hosts/CIDR ranges
|
||||
Examples: "192.168.1.0/24,internal-api.company.local,10.0.0.5"
|
||||
|
||||
TODO: In next major version (2.0):
|
||||
- Change LANGFLOW_SSRF_PROTECTION_ENABLED default to "true"
|
||||
- Remove warning-only mode and enforce blocking
|
||||
- Update documentation to reflect breaking change
|
||||
"""
|
||||
|
||||
import functools
|
||||
@ -83,6 +77,16 @@ def is_ssrf_protection_enabled() -> bool:
|
||||
Returns:
|
||||
bool: True if SSRF protection is enabled, False otherwise.
|
||||
"""
|
||||
# Read directly from environment variable to support test mocking with patch.dict()
|
||||
# This ensures tests can override the protection state without settings service caching issues
|
||||
import os
|
||||
|
||||
env_value = os.getenv("LANGFLOW_SSRF_PROTECTION_ENABLED")
|
||||
if env_value is not None:
|
||||
# Environment variable is set - use it (supports test mocking)
|
||||
return env_value.lower() in ("true", "1", "yes", "on")
|
||||
|
||||
# Fall back to settings service for non-test scenarios
|
||||
return get_settings_service().settings.ssrf_protection_enabled
|
||||
|
||||
|
||||
@ -92,11 +96,23 @@ def get_allowed_hosts() -> list[str]:
|
||||
Returns:
|
||||
list[str]: Stripped hostnames or CIDR blocks from settings, or empty list if unset.
|
||||
"""
|
||||
allowed_hosts = get_settings_service().settings.ssrf_allowed_hosts
|
||||
if not allowed_hosts:
|
||||
return []
|
||||
# ssrf_allowed_hosts is already a list[str], just clean and filter entries
|
||||
return [host.strip() for host in allowed_hosts if host and host.strip()]
|
||||
# Read directly from environment variable to support test mocking with patch.dict()
|
||||
# This ensures tests can override the allowlist without settings service caching issues
|
||||
import os
|
||||
|
||||
env_value = os.getenv("LANGFLOW_SSRF_ALLOWED_HOSTS", "")
|
||||
if env_value:
|
||||
# Parse comma-separated list from environment variable
|
||||
return [host.strip() for host in env_value.split(",") if host.strip()]
|
||||
|
||||
# Fall back to settings service for non-test scenarios
|
||||
settings_service = get_settings_service()
|
||||
if settings_service:
|
||||
allowed_hosts = settings_service.settings.ssrf_allowed_hosts
|
||||
if allowed_hosts:
|
||||
return [host.strip() for host in allowed_hosts if host and host.strip()]
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def is_host_allowed(hostname: str, ip: str | None = None) -> bool:
|
||||
@ -271,7 +287,6 @@ def _validate_direct_ip_address(hostname: str) -> bool:
|
||||
if is_ip_blocked(ip_obj):
|
||||
msg = (
|
||||
f"Access to IP address {hostname} is blocked by SSRF protection. "
|
||||
"Requests to private/internal IP ranges are not allowed for security reasons. "
|
||||
"To allow this IP, add it to LANGFLOW_SSRF_ALLOWED_HOSTS environment variable."
|
||||
)
|
||||
raise SSRFProtectionError(msg)
|
||||
@ -313,15 +328,12 @@ def _validate_hostname_resolution(hostname: str) -> None:
|
||||
if blocked_ips:
|
||||
msg = (
|
||||
f"Hostname {hostname} resolves to blocked IP address(es): {', '.join(blocked_ips)}. "
|
||||
"Requests to private/internal IP ranges are not allowed for security reasons. "
|
||||
"This protection prevents access to internal services, cloud metadata endpoints "
|
||||
"(e.g., AWS 169.254.169.254), and other sensitive internal resources. "
|
||||
"To allow this hostname, add it to LANGFLOW_SSRF_ALLOWED_HOSTS environment variable."
|
||||
)
|
||||
raise SSRFProtectionError(msg)
|
||||
|
||||
|
||||
def validate_url_for_ssrf(url: str, *, warn_only: bool = True) -> None:
|
||||
def validate_url_for_ssrf(url: str, *, warn_only: bool = False) -> None:
|
||||
"""Validate a URL to prevent SSRF attacks.
|
||||
|
||||
This function performs the following checks:
|
||||
@ -333,8 +345,7 @@ def validate_url_for_ssrf(url: str, *, warn_only: bool = True) -> None:
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
warn_only: If True, only log warnings instead of raising errors (default: True)
|
||||
TODO: Change default to False in next major version (2.0)
|
||||
warn_only: If True, only log warnings instead of raising errors (default: False)
|
||||
|
||||
Raises:
|
||||
SSRFProtectionError: If the URL is blocked due to SSRF protection (only if warn_only=False)
|
||||
@ -382,3 +393,187 @@ def validate_url_for_ssrf(url: str, *, warn_only: bool = True) -> None:
|
||||
)
|
||||
return
|
||||
raise
|
||||
|
||||
|
||||
def validate_and_resolve_url(url: str) -> tuple[str, list[str]]:
|
||||
"""Validate URL for SSRF and return validated IP addresses for DNS pinning.
|
||||
|
||||
This function is the core of DNS pinning-based SSRF protection. It performs
|
||||
comprehensive validation and returns the validated IP addresses that should
|
||||
be used for the actual HTTP request, preventing DNS rebinding attacks.
|
||||
|
||||
DNS Rebinding Attack Prevention:
|
||||
Without DNS pinning, an attacker can exploit the time gap between validation
|
||||
and the actual HTTP request:
|
||||
1. Validation: DNS returns public IP (8.8.8.8) → passes security check
|
||||
2. [Attacker changes DNS with TTL=0]
|
||||
3. HTTP request: DNS returns internal IP (127.0.0.1) → bypasses protection
|
||||
|
||||
With DNS pinning (this function):
|
||||
1. Validation: DNS returns public IP (8.8.8.8) → passes security check
|
||||
2. Function returns: (url, ['8.8.8.8']) → IP is pinned
|
||||
3. HTTP request: Uses pinned IP directly → no new DNS lookup → secure
|
||||
|
||||
Args:
|
||||
url: URL to validate (e.g., "http://example.com/api")
|
||||
|
||||
Returns:
|
||||
Tuple of (original_url, list_of_validated_ips):
|
||||
- original_url: The input URL unchanged
|
||||
- list_of_validated_ips: List of validated IP addresses to use for DNS pinning
|
||||
Returns empty list if:
|
||||
- SSRF protection is disabled
|
||||
- Host is in the allowlist (e.g., localhost for Ollama)
|
||||
- URL scheme is not http/https
|
||||
|
||||
Raises:
|
||||
SSRFProtectionError: If URL is blocked by SSRF protection
|
||||
ValueError: If URL format is invalid
|
||||
|
||||
Example:
|
||||
>>> # Public domain - returns validated IPs for pinning
|
||||
>>> url, ips = validate_and_resolve_url("http://example.com")
|
||||
>>> print(ips) # ['93.184.216.34']
|
||||
|
||||
>>> # Localhost (if in allowlist) - returns empty list (no pinning needed)
|
||||
>>> url, ips = validate_and_resolve_url("http://localhost:8080")
|
||||
>>> print(ips) # []
|
||||
|
||||
>>> # Private IP - raises SSRFProtectionError
|
||||
>>> url, ips = validate_and_resolve_url("http://192.168.1.1")
|
||||
# Raises: SSRFProtectionError("Access to IP address 192.168.1.1 is blocked...")
|
||||
"""
|
||||
# ============================================================================
|
||||
# Step 1: Check if SSRF protection is enabled
|
||||
# ============================================================================
|
||||
if not is_ssrf_protection_enabled():
|
||||
# Protection is disabled - return empty list (no DNS pinning)
|
||||
return url, []
|
||||
|
||||
# ============================================================================
|
||||
# Step 2: Parse and validate URL format
|
||||
# ============================================================================
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
except Exception as e:
|
||||
msg = f"Invalid URL format: {e}"
|
||||
raise ValueError(msg) from e
|
||||
|
||||
try:
|
||||
# ============================================================================
|
||||
# Step 3: Validate URL scheme (only http/https allowed)
|
||||
# ============================================================================
|
||||
_validate_url_scheme(parsed.scheme)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
# Non-HTTP schemes (ftp, file, etc.) are not subject to SSRF protection
|
||||
return url, []
|
||||
|
||||
# ============================================================================
|
||||
# Step 4: Extract and validate hostname
|
||||
# ============================================================================
|
||||
hostname = _validate_hostname_exists(parsed.hostname)
|
||||
|
||||
# ============================================================================
|
||||
# Step 5: Check allowlist (early return for trusted hosts)
|
||||
# ============================================================================
|
||||
# Allowlisted hosts bypass all SSRF checks and DNS pinning
|
||||
# This is used for legitimate internal services like Ollama (localhost)
|
||||
if is_host_allowed(hostname):
|
||||
logger.debug(f"Hostname {hostname} is in allowlist, bypassing SSRF checks and DNS pinning")
|
||||
return url, []
|
||||
|
||||
# ============================================================================
|
||||
# Step 6: Handle direct IP addresses
|
||||
# ============================================================================
|
||||
# Check if the hostname is already an IP address (no DNS resolution needed)
|
||||
try:
|
||||
ip_obj = ipaddress.ip_address(hostname)
|
||||
|
||||
# Check if this specific IP is in the allowlist
|
||||
if is_host_allowed(hostname, str(ip_obj)):
|
||||
logger.debug(f"IP {hostname} is in allowlist")
|
||||
return url, []
|
||||
|
||||
# Check if IP is in blocked ranges (private IPs, localhost, etc.)
|
||||
if is_ip_blocked(ip_obj):
|
||||
msg = (
|
||||
f"Access to IP address {hostname} is blocked by SSRF protection. "
|
||||
"To allow this IP, add it to LANGFLOW_SSRF_ALLOWED_HOSTS environment variable."
|
||||
)
|
||||
raise SSRFProtectionError(msg)
|
||||
# Direct IP is public and allowed - return it for DNS pinning
|
||||
# (Even though it's already an IP, we return it for consistency)
|
||||
logger.debug(f"Direct IP {hostname} validated, will use for DNS pinning")
|
||||
return url, [hostname] # noqa: TRY300
|
||||
|
||||
except ValueError:
|
||||
# Not an IP address, it's a hostname - continue to DNS resolution
|
||||
pass
|
||||
|
||||
# ============================================================================
|
||||
# Step 7: Resolve hostname to IP addresses
|
||||
# ============================================================================
|
||||
# This is the critical step for DNS pinning - we resolve DNS here during
|
||||
# validation, and the returned IPs will be used for the actual HTTP request
|
||||
resolved_ips = resolve_hostname(hostname)
|
||||
blocked_ips = []
|
||||
|
||||
# ============================================================================
|
||||
# Step 8: Validate all resolved IPs
|
||||
# ============================================================================
|
||||
# Security: We must check ALL resolved IPs before making any decisions.
|
||||
# A hostname might resolve to multiple IPs (e.g., [8.8.8.8, 192.168.1.1]).
|
||||
# If we return early on the first allowlisted IP, we skip validation of
|
||||
# remaining IPs, which could include blocked/internal addresses.
|
||||
#
|
||||
# Strategy:
|
||||
# 1. Collect all allowlisted IPs and all blocked IPs
|
||||
# 2. If ANY IP is blocked → block the entire hostname (security first)
|
||||
# 3. If some IPs are allowlisted but others are not → use only allowlisted IPs for pinning
|
||||
# 4. If all IPs are public (none blocked, none allowlisted) → use all for pinning
|
||||
allowed_ips = []
|
||||
for ip in resolved_ips:
|
||||
# Check if this resolved IP is in the allowlist
|
||||
if is_host_allowed(hostname, ip):
|
||||
allowed_ips.append(ip)
|
||||
# Check if IP is in blocked ranges
|
||||
elif is_ip_blocked(ip):
|
||||
blocked_ips.append(ip)
|
||||
|
||||
# ============================================================================
|
||||
# Step 9: Block if any resolved IPs are private/internal
|
||||
# ============================================================================
|
||||
# Security: If ANY resolved IP is blocked, we block the entire hostname.
|
||||
# This prevents attacks where a hostname resolves to both safe and unsafe IPs.
|
||||
if blocked_ips:
|
||||
msg = (
|
||||
f"Hostname {hostname} resolves to blocked IP address(es): {', '.join(blocked_ips)}. "
|
||||
"To allow this hostname, add it to LANGFLOW_SSRF_ALLOWED_HOSTS environment variable."
|
||||
)
|
||||
raise SSRFProtectionError(msg)
|
||||
|
||||
# ============================================================================
|
||||
# Step 9b: Handle partially allowlisted IPs
|
||||
# ============================================================================
|
||||
# If some (but not all) IPs are allowlisted, use only the allowlisted ones for pinning
|
||||
if allowed_ips:
|
||||
logger.debug(
|
||||
f"Hostname {hostname} has {len(allowed_ips)} allowlisted IP(s) out of {len(resolved_ips)} total. "
|
||||
f"Using allowlisted IPs for DNS pinning: {allowed_ips}"
|
||||
)
|
||||
return url, allowed_ips
|
||||
# ============================================================================
|
||||
# Step 10: Return validated IPs for DNS pinning
|
||||
# ============================================================================
|
||||
# All IPs are public and safe - return them for DNS pinning
|
||||
# The HTTP client will use these IPs directly, preventing DNS rebinding
|
||||
logger.debug(f"Hostname {hostname} validated, resolved to {resolved_ips}, will use for DNS pinning")
|
||||
return url, resolved_ips # noqa: TRY300
|
||||
|
||||
except SSRFProtectionError:
|
||||
# Re-raise SSRF errors as-is
|
||||
raise
|
||||
except Exception as e:
|
||||
# Wrap unexpected errors in SSRFProtectionError
|
||||
msg = f"Error validating URL: {e}"
|
||||
raise SSRFProtectionError(msg) from e
|
||||
|
||||
242
src/lfx/src/lfx/utils/ssrf_transport.py
Normal file
242
src/lfx/src/lfx/utils/ssrf_transport.py
Normal file
@ -0,0 +1,242 @@
|
||||
"""Custom httpx transport with DNS pinning for SSRF protection.
|
||||
|
||||
This module provides a custom httpx transport that pins DNS resolution to validated
|
||||
IP addresses, preventing DNS rebinding attacks that could bypass SSRF protection.
|
||||
|
||||
The implementation uses a custom AsyncNetworkBackend to intercept TCP connections
|
||||
and connect to the pinned IP address while preserving the original hostname for
|
||||
TLS SNI (Server Name Indication) and certificate verification.
|
||||
"""
|
||||
|
||||
import ssl
|
||||
from collections.abc import Iterable
|
||||
|
||||
import httpcore
|
||||
import httpx
|
||||
from httpx import URL, Proxy
|
||||
from httpx._config import create_ssl_context
|
||||
|
||||
from lfx.logging import logger
|
||||
|
||||
|
||||
class DNSPinningNetworkBackend(httpcore.AsyncNetworkBackend):
|
||||
"""Network backend that pins DNS resolution to validated IP addresses.
|
||||
|
||||
This backend intercepts TCP connection attempts and redirects them to pinned
|
||||
IP addresses while preserving the original hostname for TLS SNI and certificate
|
||||
verification. This prevents DNS rebinding attacks without breaking HTTPS.
|
||||
|
||||
How it works:
|
||||
1. When httpcore tries to connect to a hostname, we intercept the connect_tcp() call
|
||||
2. If the hostname has a pinned IP, we connect to that IP instead
|
||||
3. The original hostname is preserved for TLS handshake (SNI) and cert verification
|
||||
4. This prevents DNS rebinding while maintaining full HTTPS compatibility
|
||||
"""
|
||||
|
||||
def __init__(self, pinned_ips: dict[str, list[str]], backend: httpcore.AsyncNetworkBackend | None = None):
|
||||
"""Initialize the DNS pinning backend.
|
||||
|
||||
Args:
|
||||
pinned_ips: Dictionary mapping hostnames to list of validated IP addresses
|
||||
backend: Underlying network backend (defaults to AnyIOBackend for asyncio)
|
||||
"""
|
||||
self.pinned_ips = pinned_ips
|
||||
# Use httpcore's default async backend (AnyIOBackend) if none provided
|
||||
# This is the public API recommended in httpcore documentation
|
||||
if backend is None:
|
||||
backend = httpcore.AnyIOBackend()
|
||||
self._backend = backend
|
||||
logger.debug(f"Created DNS pinning network backend with pinned IPs: {pinned_ips}")
|
||||
|
||||
async def connect_tcp(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float | None = None,
|
||||
local_address: str | None = None,
|
||||
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
|
||||
) -> httpcore.AsyncNetworkStream:
|
||||
"""Connect to TCP socket, using pinned IP if available.
|
||||
|
||||
This method intercepts connection attempts and redirects to pinned IPs
|
||||
while preserving the original hostname for TLS.
|
||||
|
||||
Args:
|
||||
host: Hostname to connect to (may be replaced with pinned IP)
|
||||
port: Port number
|
||||
timeout: Connection timeout
|
||||
local_address: Local address to bind to
|
||||
socket_options: Socket options
|
||||
|
||||
Returns:
|
||||
Network stream for the connection
|
||||
"""
|
||||
# Check if this hostname has pinned IPs
|
||||
if host in self.pinned_ips:
|
||||
pinned_ips = self.pinned_ips[host]
|
||||
|
||||
# Security: If host is in pinned_ips but list is empty, fail rather than bypass
|
||||
if not pinned_ips:
|
||||
msg = f"DNS pinning: Host {host} is marked for pinning but has no pinned IPs"
|
||||
logger.error(msg)
|
||||
raise RuntimeError(msg)
|
||||
|
||||
logger.debug(f"DNS pinning: Connecting to pinned IPs {pinned_ips} for hostname {host}")
|
||||
|
||||
# Try each pinned IP in order (supports dual-stack and load balancing)
|
||||
# The TLS layer will still use the original hostname for SNI and cert verification
|
||||
last_error = None
|
||||
for pinned_ip in pinned_ips:
|
||||
try:
|
||||
logger.debug(f"DNS pinning: Attempting connection to {pinned_ip}")
|
||||
return await self._backend.connect_tcp(
|
||||
host=pinned_ip,
|
||||
port=port,
|
||||
timeout=timeout,
|
||||
local_address=local_address,
|
||||
socket_options=socket_options,
|
||||
)
|
||||
except (OSError, TimeoutError) as e:
|
||||
last_error = e
|
||||
logger.debug(f"DNS pinning: Failed to connect to {pinned_ip}: {e}")
|
||||
continue
|
||||
|
||||
# All pinned IPs failed, raise the last error
|
||||
# This should never be None since we checked for empty list above
|
||||
if last_error is None:
|
||||
msg = f"DNS pinning: All pinned IPs failed for {host} but no error was captured"
|
||||
raise RuntimeError(msg)
|
||||
raise last_error
|
||||
|
||||
# No pinned IP, use normal connection
|
||||
return await self._backend.connect_tcp(
|
||||
host=host,
|
||||
port=port,
|
||||
timeout=timeout,
|
||||
local_address=local_address,
|
||||
socket_options=socket_options,
|
||||
)
|
||||
|
||||
async def connect_unix_socket(
|
||||
self,
|
||||
path: str,
|
||||
timeout: float | None = None,
|
||||
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
|
||||
) -> httpcore.AsyncNetworkStream:
|
||||
"""Connect to Unix socket (pass through to underlying backend)."""
|
||||
return await self._backend.connect_unix_socket(
|
||||
path=path,
|
||||
timeout=timeout,
|
||||
socket_options=socket_options,
|
||||
)
|
||||
|
||||
async def sleep(self, seconds: float) -> None:
|
||||
"""Sleep for specified duration (pass through to underlying backend)."""
|
||||
await self._backend.sleep(seconds)
|
||||
|
||||
|
||||
class SSRFProtectedTransport(httpx.AsyncHTTPTransport):
|
||||
"""HTTP transport that pins DNS resolution to validated IPs.
|
||||
|
||||
This transport prevents DNS rebinding attacks by using a custom network backend
|
||||
that connects to pinned IP addresses while preserving the original hostname for
|
||||
TLS SNI and certificate verification.
|
||||
|
||||
Unlike the naive approach of rewriting URLs (which breaks HTTPS), this implementation
|
||||
works at the network layer to ensure both security and compatibility.
|
||||
|
||||
Example:
|
||||
>>> pinned_ips = {"example.com": "93.184.216.34"}
|
||||
>>> transport = SSRFProtectedTransport(pinned_ips=pinned_ips)
|
||||
>>> async with httpx.AsyncClient(transport=transport) as client:
|
||||
... # Request to example.com will connect to 93.184.216.34
|
||||
... # But TLS will still verify against example.com certificate
|
||||
... response = await client.get("https://example.com/path")
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
pinned_ips: dict[str, list[str]],
|
||||
verify: bool | str | ssl.SSLContext = True, # noqa: FBT001, FBT002
|
||||
cert: tuple[str, str] | tuple[str, str, str] | str | None = None,
|
||||
trust_env: bool = True, # noqa: FBT001, FBT002
|
||||
http1: bool = True, # noqa: FBT001, FBT002
|
||||
http2: bool = False, # noqa: FBT001, FBT002
|
||||
limits: httpx.Limits = httpx.Limits(), # noqa: B008
|
||||
proxy: httpx._types.ProxyTypes | None = None,
|
||||
uds: str | None = None,
|
||||
local_address: str | None = None,
|
||||
retries: int = 0,
|
||||
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
|
||||
):
|
||||
"""Initialize transport with pinned DNS mappings.
|
||||
|
||||
Args:
|
||||
pinned_ips: Dictionary mapping hostnames to list of validated IP addresses.
|
||||
Example: {"example.com": ["93.184.216.34", "2606:2800:220:1:248:1893:25c8:1946"]}
|
||||
verify: SSL verification settings
|
||||
cert: Client certificate
|
||||
trust_env: Whether to trust environment variables for proxy config
|
||||
http1: Enable HTTP/1.1
|
||||
http2: Enable HTTP/2
|
||||
limits: Connection pool limits
|
||||
proxy: Proxy configuration
|
||||
uds: Unix domain socket path
|
||||
local_address: Local address to bind to
|
||||
retries: Number of retries
|
||||
socket_options: Socket options
|
||||
"""
|
||||
# Create custom network backend with DNS pinning
|
||||
network_backend = DNSPinningNetworkBackend(pinned_ips=pinned_ips)
|
||||
|
||||
# Create SSL context (same as parent class)
|
||||
ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env)
|
||||
|
||||
# Handle proxy (same as parent class)
|
||||
if proxy is not None:
|
||||
proxy = Proxy(url=proxy) if isinstance(proxy, (str, URL)) else proxy
|
||||
|
||||
# Create pool with our custom network backend
|
||||
# We replicate the parent's logic but add network_backend parameter
|
||||
if proxy is None:
|
||||
self._pool = httpcore.AsyncConnectionPool(
|
||||
ssl_context=ssl_context,
|
||||
max_connections=limits.max_connections,
|
||||
max_keepalive_connections=limits.max_keepalive_connections,
|
||||
keepalive_expiry=limits.keepalive_expiry,
|
||||
http1=http1,
|
||||
http2=http2,
|
||||
uds=uds,
|
||||
local_address=local_address,
|
||||
retries=retries,
|
||||
socket_options=socket_options,
|
||||
network_backend=network_backend, # Our custom backend!
|
||||
)
|
||||
else:
|
||||
# For proxy scenarios, we'd need to handle HTTPProxy/SOCKSProxy
|
||||
# For now, raise an error as DNS pinning with proxies needs special handling
|
||||
msg = "DNS pinning with proxies is not currently supported"
|
||||
raise NotImplementedError(msg)
|
||||
|
||||
self.pinned_ips = pinned_ips
|
||||
logger.debug(f"Created SSRF protected transport with pinned IPs: {pinned_ips}")
|
||||
|
||||
|
||||
def create_ssrf_protected_client(
|
||||
hostname: str, validated_ips: list[str] | tuple[str, ...], **client_kwargs
|
||||
) -> httpx.AsyncClient:
|
||||
"""Create an httpx client with DNS pinning for SSRF protection.
|
||||
|
||||
Args:
|
||||
hostname: The hostname to pin
|
||||
validated_ips: List of validated IP addresses to use for this hostname.
|
||||
IPs will be tried in order for dual-stack/load-balanced hosts.
|
||||
**client_kwargs: Additional arguments for AsyncClient (e.g., timeout, headers)
|
||||
|
||||
Returns:
|
||||
Configured AsyncClient with DNS pinning
|
||||
"""
|
||||
# Convert to list if tuple
|
||||
ip_list = list(validated_ips) if isinstance(validated_ips, tuple) else validated_ips
|
||||
transport = SSRFProtectedTransport(pinned_ips={hostname: ip_list})
|
||||
return httpx.AsyncClient(transport=transport, **client_kwargs)
|
||||
Reference in New Issue
Block a user