mirror of
https://github.com/langflow-ai/langflow.git
synced 2026-07-26 09:36:07 +08:00
docs: configure s3 for file storage backend (#10678)
* configure-file-storage-s3 * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com> * clarify-s3-credentials * add-storage-tags-and-cleanup-creds-seciton * role-link-name * fix-parse-error --------- Co-authored-by: April I. Murphy <36110273+aimurphy@users.noreply.github.com>
This commit is contained in:
Mendon Kissling
@ -7,8 +7,7 @@ import Icon from "@site/src/components/icon";
|
||||
|
||||
Each Langflow server has a file management system where you can store files that you want to use in your flows.
|
||||
|
||||
Files uploaded to Langflow file management are stored locally in your [Langflow configuration directory](/memory), and they are available to all of your flows.
|
||||
Local storage is set by `LANGFLOW_STORAGE_TYPE`, which has only one allowed value (`local`).
|
||||
Files uploaded to Langflow file management are stored in Langflow's [storage backend (local or AWS S3)](/concepts-file-management#configure-file-storage), and they are available to all of your flows.
|
||||
|
||||
Uploading files to Langflow file management keeps your files in a central location, and allows you to reuse files across flows without repeated manual uploads.
|
||||
|
||||
@ -126,6 +125,81 @@ For more specialized image processing, browse <Icon name="Blocks" aria-hidden="t
|
||||
|
||||
For videos, see the **Twelve Labs** and **YouTube** <Icon name="Blocks" aria-hidden="true" /> [**Bundles**](/components-bundle-components).
|
||||
|
||||
## Configure file storage
|
||||
|
||||
Langflow supports two storage backends for file management:
|
||||
|
||||
* **Local storage**: Langflow's default storage backend. Files are stored locally in your [Langflow configuration directory](/memory). Set `LANGFLOW_STORAGE_TYPE=local` or leave it unset to use local storage.
|
||||
|
||||
* **S3 storage**: Files are stored in an AWS S3 bucket.
|
||||
Langflow uses the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) library to interact with S3.
|
||||
|
||||
To use S3 as your file storage backend, add the following configuration to your `.env` file:
|
||||
|
||||
```text
|
||||
# S3 Storage Configuration
|
||||
LANGFLOW_STORAGE_TYPE=s3
|
||||
LANGFLOW_OBJECT_STORAGE_BUCKET_NAME=S3_BUCKET_NAME
|
||||
LANGFLOW_OBJECT_STORAGE_PREFIX=S3_BUCKET_DIRECTORY
|
||||
|
||||
# AWS Credentials (required for S3)
|
||||
AWS_ACCESS_KEY_ID=S3_ACCESS_KEY
|
||||
AWS_SECRET_ACCESS_KEY=S3_ACCESS_SECRET_KEY
|
||||
AWS_DEFAULT_REGION=S3_REGION
|
||||
```
|
||||
|
||||
Replace the following placeholders with the actual values for your S3 instance:
|
||||
|
||||
* `S3_BUCKET_NAME`: The name of your S3 bucket.
|
||||
* `S3_BUCKET_DIRECTORY`: An optional folder path within the bucket where files are stored, such as `s3://S3_BUCKET_NAME/S3_BUCKET_DIRECTORY`.
|
||||
* `S3_ACCESS_KEY`: Your AWS Access Key ID.
|
||||
* `S3_ACCESS_SECRET_KEY`: Your AWS Secret Access Key.
|
||||
* `S3_REGION`: The AWS region where your bucket is located, such as `us-east-2`.
|
||||
|
||||
Your AWS credentials must have the necessary permissions to perform the required S3 operations for your use case, such as reading, writing, and deleting files in S3.
|
||||
This example policy allows basic CRUD operations on S3 objects.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "LangflowS3StorageAccess",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:PutObject",
|
||||
"s3:GetObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:ListBucket",
|
||||
"s3:PutObjectTagging",
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::S3_BUCKET_NAME",
|
||||
"arn:aws:s3:::S3_BUCKET_NAME/S3_BUCKET_DIRECTORY/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Replace the following placeholders with the actual values for your IAM policy and S3 instance:
|
||||
|
||||
* `S3_BUCKET_NAME`: The name of your S3 bucket.
|
||||
* `S3_BUCKET_DIRECTORY`: An optional folder path within the bucket where files are stored, such as `s3://S3_BUCKET_NAME/S3_BUCKET_DIRECTORY`.
|
||||
|
||||
For more information, see the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html).
|
||||
|
||||
## File storage environment variables {#file-storage-environment-variables}
|
||||
|
||||
The following environment variables configure file storage backends for Langflow's file management system:
|
||||
|
||||
| Variable | Format | Default | Description |
|
||||
|----------|--------|---------|-------------|
|
||||
| `LANGFLOW_STORAGE_TYPE` | String | `local` | Set the file storage backend. Supported values: `local` (files stored in the Langflow configuration directory) or `s3` (files stored in AWS S3). For S3 storage, you must also configure AWS credentials and bucket settings. |
|
||||
| `LANGFLOW_OBJECT_STORAGE_BUCKET_NAME` | String | Not set | The name of the S3 bucket to use for file storage. Required when `LANGFLOW_STORAGE_TYPE=s3`. |
|
||||
| `LANGFLOW_OBJECT_STORAGE_PREFIX` | String | Not set | Optional prefix/folder path within the S3 bucket where files will be stored. If not set, files are stored at the bucket root. |
|
||||
| `LANGFLOW_OBJECT_STORAGE_TAGS` | JSON object | Not set | Optional S3 object tags applied to stored files when `LANGFLOW_STORAGE_TYPE=s3`. Ignored for local storage. Provided as a JSON map of string keys to string values, such as `{"env": "prod", "owner": "data-team"}`. |
|
||||
|
||||
## See also
|
||||
|
||||
* [Components reference](/concepts-components)
|
||||
@ -462,7 +462,9 @@ For more information about deploying Langflow servers, see [Langflow deployment
|
||||
|
||||
### Storage
|
||||
|
||||
See [Memory management options](/memory) and [Manage files](/concepts-file-management).
|
||||
For file storage environment variables, see [File storage environment variables](/concepts-file-management#file-storage-environment-variables).
|
||||
|
||||
For database environment variables, including PostgreSQL configuration, see [Memory management options](/memory#configure-external-memory).
|
||||
|
||||
### Telemetry
|
||||
|
||||
|
||||
@ -100,6 +100,8 @@ To fine-tune your database connection pool and timeout settings, you can set the
|
||||
Don't use the deprecated environment variables `LANGFLOW_DB_POOL_SIZE` or `LANGFLOW_DB_MAX_OVERFLOW`.
|
||||
Instead, use `pool_size` and `max_overflow` in `LANGFLOW_DB_CONNECTION_SETTINGS`.
|
||||
|
||||
* `LANGFLOW_MIGRATION_LOCK_NAMESPACE`: Optional namespace for PostgreSQL advisory locks used during database migrations. This is useful when running multiple Langflow instances that share the same PostgreSQL database. Each instance should use a unique namespace to avoid conflicts. If not set, Langflow uses a default namespace. This setting only applies when using PostgreSQL as your database backend.
|
||||
|
||||
## Configure cache memory
|
||||
|
||||
The default Langflow caching behavior is an asynchronous, in-memory cache:
|
||||
|
||||
Reference in New Issue
Block a user