docs/daprdocs/content/en/reference/components-reference/supported-bindings/s3.md at 34075d053cf082a5e3917827877379dc89f15a84 · dapr/docs

type

docs

title

AWS S3 binding spec

linkTitle

AWS S3

description

Detailed documentation on the AWS S3 binding component

aliases

/operations/components/setup-bindings/supported-bindings/s3/

Component format

To setup an AWS S3 binding create a component of type bindings.aws.s3. This binding works with other S3-compatible services, such as Minio. See [this guide]({{% ref "howto-bindings.md#1-create-a-binding" %}}) on how to create and apply a binding configuration.

See [Authenticating to AWS]({{% ref authenticating-aws.md %}}) for information about authentication-related attributes.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: <NAME>
spec:
  type: bindings.aws.s3
  version: v1
  metadata:
  - name: bucket
    value: "mybucket"
  - name: region
    value: "us-west-2"
  - name: endpoint
    value: "s3.us-west-2.amazonaws.com"
  - name: accessKey
    value: "*****************"
  - name: secretKey
    value: "*****************"
  - name: sessionToken
    value: "mysession"
  - name: decodeBase64
    value: "<bool>"
  - name: encodeBase64
    value: "<bool>"
  - name: forcePathStyle
    value: "<bool>"
  - name: disableSSL
    value: "<bool>"
  - name: insecureSSL
    value: "<bool>"
  - name: storageClass
    value: "<string>"

{{% alert title="Warning" color="warning" %}} The above example uses secrets as plain strings. It is recommended to use a secret store for the secrets as described [here]({{% ref component-secrets.md %}}). {{% /alert %}}

Spec metadata fields

Field	Required	Binding support	Details	Example
`bucket`	Y	Output	The name of the S3 bucket to write to	`"bucket"`
`region`	Y	Output	The specific AWS region	`"us-east-1"`
`endpoint`	N	Output	The specific AWS endpoint	`"s3.us-east-1.amazonaws.com"`
`accessKey`	Y	Output	The AWS Access Key to access this resource	`"key"`
`secretKey`	Y	Output	The AWS Secret Access Key to access this resource	`"secretAccessKey"`
`sessionToken`	N	Output	The AWS session token to use	`"sessionToken"`
`forcePathStyle`	N	Output	Currently Amazon S3 SDK supports virtual hosted-style and path-style access. `"true"` is path-style format like `"https://<endpoint>/<your bucket>/<key>"`. `"false"` is hosted-style format like `"https://<your bucket>.<endpoint>/<key>"`. Defaults to `"false"`	`"true"`, `"false"`
`decodeBase64`	N	Output	Configuration to decode base64 file content before saving to bucket storage. (In case of saving a file with binary content). `"true"` is the only allowed positive value. Other positive variations like `"True", "1"` are not acceptable. Defaults to `false`	`"true"`, `"false"`
`encodeBase64`	N	Output	Configuration to encode base64 file content before return the content. (In case of opening a file with binary content). `"true"` is the only allowed positive value. Other positive variations like `"True", "1"` are not acceptable. Defaults to `"false"`	`"true"`, `"false"`
`disableSSL`	N	Output	Allows to connect to non `https://` endpoints. Defaults to `"false"`	`"true"`, `"false"`
`insecureSSL`	N	Output	When connecting to `https://` endpoints, accepts invalid or self-signed certificates. Defaults to `"false"`	`"true"`, `"false"`
`storageClass`	N	Output	The desired storage class for objects during the create operation. Valid aws storage class types can be found here	`STANDARD_IA`

{{% alert title="Important" color="warning" %}} When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you must not provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using.
{{% /alert %}}

S3 Bucket Creation

Using with Minio

Minio is a service that exposes local storage as S3-compatible block storage, and it's a popular alternative to S3 especially in development environments. You can use the S3 binding with Minio too, with some configuration tweaks:

Set endpoint to the address of the Minio server, including protocol (http:// or https://) and the optional port at the end. For example, http://minio.local:9000 (the values depend on your environment).
forcePathStyle must be set to true
The value for region is not important; you can set it to us-east-1.
Depending on your environment, you may need to set disableSSL to true if you're connecting to Minio using a non-secure connection (using the http:// protocol). If you are using a secure connection (https:// protocol) but with a self-signed certificate, you may need to set insecureSSL to true.

{{% tab "LocalStack" %}} For local development, the LocalStack project is used to integrate AWS S3. Follow these instructions to run LocalStack.

To run LocalStack locally from the command line using Docker, use a docker-compose.yaml similar to the following:

version: "3.8"

services:
  localstack:
    container_name: "cont-aws-s3"
    image: localstack/localstack:1.4.0
    ports:
      - "127.0.0.1:4566:4566"
    environment:
      - DEBUG=1
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "<PATH>/init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh"  # init hook
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

To use the S3 component, you need to use an existing bucket. The example above uses a LocalStack Initialization Hook to setup the bucket.

To use LocalStack with your S3 binding, you need to provide the endpoint configuration in the component metadata. The endpoint is unnecessary when running against production AWS.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
    name: aws-s3
    namespace: default
spec:
  type: bindings.aws.s3
  version: v1
  metadata:
    - name: bucket
      value: conformance-test-docker
    - name: endpoint
      value: "http://localhost:4566"
    - name: accessKey
      value: "my-access"
    - name: secretKey
      value: "my-secret"
    - name: region
      value: "us-east-1"

To use the S3 component, you need to use an existing bucket. Follow the AWS documentation for creating a bucket.

Binding support

This component supports output binding with the following operations:

create : Create object
get : Get object
delete : Delete object
list: List objects
bulkGet: Bulk get objects
bulkCreate: Bulk create objects
bulkDelete: Bulk delete objects

Create object

To perform a create operation, invoke the AWS S3 binding with a POST method and the following JSON body:

Note: by default, a random UUID is generated. See below for Metadata support to set the name

{
  "operation": "create",
  "data": "YOUR_CONTENT",
  "metadata": { 
    "storageClass": "STANDARD_IA",
    "tags": "project=sashimi,year=2024",
  }
}

For example you can provide a storage class or tags while using the create operation with a Linux curl command

curl -d '{ "operation": "create", "data": "YOUR_BASE_64_CONTENT", "metadata": { "storageClass": "STANDARD_IA", "project=sashimi,year=2024" } }' /
http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Share object with a presigned URL

To presign an object with a specified time-to-live, use the presignTTL metadata key on a create request. Valid values for presignTTL are Go duration strings.

curl -d "{ \"operation\": \"create\", \"data\": \"Hello World\", \"metadata\": { \"presignTTL\": \"15m\" } }" \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "create", "data": "Hello World", "metadata": { "presignTTL": "15m" } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains the following example JSON:

{
    "location":"https://<your bucket>.s3.<your region>.amazonaws.com/<key>",
    "versionID":"<version ID if Bucket Versioning is enabled>",
    "presignURL": "https://<your bucket>.s3.<your region>.amazonaws.com/image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJJWZ7B6WCRGMKFGQ%2F20180210%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20180210T171315Z&X-Amz-Expires=1800&X-Amz-Signature=12b74b0788aa036bc7c3d03b3f20c61f1f91cc9ad8873e3314255dc479a25351&X-Amz-SignedHeaders=host"
}

Examples

Save text to a random generated UUID file

{{< tabpane text=true >}} {{% tab "Windows" %}} On Windows, utilize cmd prompt (PowerShell has different escaping mechanism)

curl -d "{ \"operation\": \"create\", \"data\": \"Hello World\" }" http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "create", "data": "Hello World" }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Save text to a specific file

curl -d "{ \"operation\": \"create\", \"data\": \"Hello World\", \"metadata\": { \"key\": \"my-test-file.txt\" } }" \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "create", "data": "Hello World", "metadata": { "key": "my-test-file.txt" } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Save a file to a object

To upload a file, encode it as Base64 and let the Binding know to deserialize it:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: <NAME>
spec:
  type: bindings.aws.s3
  version: v1
  metadata:
  - name: bucket
    value: mybucket
  - name: region
    value: us-west-2
  - name: endpoint
    value: s3.us-west-2.amazonaws.com
  - name: accessKey
    value: *****************
  - name: secretKey
    value: *****************
  - name: sessionToken
    value: mysession
  - name: decodeBase64
    value: <bool>
  - name: forcePathStyle
    value: <bool>

Then you can upload it as you would normally:

curl -d "{ \"operation\": \"create\", \"data\": \"YOUR_BASE_64_CONTENT\", \"metadata\": { \"key\": \"my-test-file.jpg\" } }" http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "create", "data": "YOUR_BASE_64_CONTENT", "metadata": { "key": "my-test-file.jpg" } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Upload from file path

To upload a file from a supplied path (relative or absolute), use the filepath metadata key on a create request that contains empty data fields.

curl -d '{ \"operation\": \"create\", \"metadata\": { \"filePath\": \"my-test-file.txt\" }}' http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "create", "metadata": { "filePath": "my-test-file.txt" }}' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body will contain the following JSON:

{
    "location":"https://<your bucket>.s3.<your region>.amazonaws.com/<key>",
    "versionID":"<version ID if Bucket Versioning is enabled"
}

Presign an existing object

To presign an existing S3 object with a specified time-to-live, use the presignTTL and key metadata keys on a presign request. Valid values for presignTTL are Go duration strings.

curl -d "{ \"operation\": \"presign\", \"metadata\": { \"presignTTL\": \"15m\", \"key\": \"my-test-file.txt\" } }" \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "presign", "metadata": { "presignTTL": "15m", "key": "my-test-file.txt" } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains the following example JSON:

{
    "presignURL": "https://<your bucket>.s3.<your region>.amazonaws.com/image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJJWZ7B6WCRGMKFGQ%2F20180210%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20180210T171315Z&X-Amz-Expires=1800&X-Amz-Signature=12b74b0788aa036bc7c3d03b3f20c61f1f91cc9ad8873e3314255dc479a25351&X-Amz-SignedHeaders=host"
}

Get object

To perform a get file operation, invoke the AWS S3 binding with a POST method and the following JSON body:

{
  "operation": "get",
  "metadata": {
    "key": "my-test-file.txt"
  }
}

The metadata parameters are:

key - the name of the object

Example

curl -d '{ \"operation\": \"get\", \"metadata\": { \"key\": \"my-test-file.txt\" }}' http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "get", "metadata": { "key": "my-test-file.txt" }}' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains the value stored in the object.

Delete object

To perform a delete object operation, invoke the AWS S3 binding with a POST method and the following JSON body:

{
  "operation": "delete",
  "metadata": {
    "key": "my-test-file.txt"
  }
}

The metadata parameters are:

key - the name of the object

Examples

Delete object

curl -d '{ \"operation\": \"delete\", \"metadata\": { \"key\": \"my-test-file.txt\" }}' http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "delete", "metadata": { "key": "my-test-file.txt" }}' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

An HTTP 204 (No Content) and empty body will be returned if successful.

List objects

To perform a list object operation, invoke the S3 binding with a POST method and the following JSON body:

{
  "operation": "list",
  "data": {
    "maxResults": 10,
    "prefix": "file",
    "marker": "hvlcCQFSOD5TD",
    "delimiter": "i0FvxAn2EOEL6"
  }
}

The data parameters are:

maxResults - (optional) sets the maximum number of keys returned in the response. By default the action returns up to 1,000 key names. The response might contain fewer keys but will never contain more.
prefix - (optional) limits the response to keys that begin with the specified prefix.
marker - (optional) marker is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. Marker can be any key in the bucket. The marker value may then be used in a subsequent call to request the next set of list items.
delimiter - (optional) A delimiter is a character you use to group keys.

Response

The response body contains the list of found objects.

The list of objects will be returned as JSON array in the following form:

{
	"CommonPrefixes": null,
	"Contents": [
		{
			"ETag": "\"7e94cc9b0f5226557b05a7c2565dd09f\"",
			"Key": "hpNdFUxruNuwm",
			"LastModified": "2021-08-16T06:44:14Z",
			"Owner": {
				"DisplayName": "owner name",
				"ID": "owner id"
			},
			"Size": 6916,
			"StorageClass": "STANDARD"
		}
	],
	"Delimiter": "",
	"EncodingType": null,
	"IsTruncated": true,
	"Marker": "hvlcCQFSOD5TD",
	"MaxKeys": 1,
	"Name": "mybucketdapr",
	"NextMarker": "hzaUPWjmvyi9W",
	"Prefix": ""
}

Bulk get objects

To download multiple objects in parallel, invoke the AWS S3 binding with a POST method and the following JSON body:

{
  "operation": "bulkGet",
  "data": {
    "items": [
      { "key": "file1.txt" },
      { "key": "file2.txt" },
      { "key": "file3.txt", "filePath": "/tmp/file3.txt" }
    ],
    "concurrency": 5
  }
}

The data parameters are:

items - (required) an array of objects to download. Each item has:
- key - (required) the name of the S3 object.
- filePath - (optional) if provided, the object is streamed directly to this local file path instead of being returned in the response body.
concurrency - (optional) the maximum number of concurrent downloads. Defaults to 10. Must be greater than 0.

The component-level encodeBase64 metadata option applies to bulk get. When set to "true", object contents are base64-encoded before being returned in the response body. When filePath is specified, the file will contain base64-encoded text rather than raw bytes.

Example

curl -d "{ \"operation\": \"bulkGet\", \"data\": { \"items\": [{ \"key\": \"file1.txt\" }, { \"key\": \"file2.txt\" }], \"concurrency\": 5 } }" http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "bulkGet", "data": { "items": [{ "key": "file1.txt" }, { "key": "file2.txt" }], "concurrency": 5 } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains a JSON array with a result for each requested item. Each result includes a per-item error field for partial failure semantics — the operation succeeds overall, and individual failures are reported per key:

[
  { "key": "file1.txt", "data": "contents of file1" },
  { "key": "file2.txt", "data": "contents of file2" },
  { "key": "missing.txt", "error": "object not found" }
]

When filePath is specified for an item, the data field is omitted and the content is written to the specified file.

Bulk create objects

To upload multiple objects in parallel, invoke the AWS S3 binding with a POST method and the following JSON body:

{
  "operation": "bulkCreate",
  "data": {
    "items": [
      { "key": "file1.txt", "data": "Hello World" },
      { "key": "file2.txt", "data": "Another file", "contentType": "text/plain" },
      { "key": "file3.txt", "filePath": "/path/to/local/file.txt" }
    ],
    "concurrency": 5
  }
}

The data parameters are:

items - (required) an array of objects to upload. Each item has:
- key - (required) the name for the S3 object.
- data - the content to upload. Either data or filePath must be provided.
- filePath - a local file path to upload from. Either data or filePath must be provided.
- contentType - (optional) the MIME type of the object.
concurrency - (optional) the maximum number of concurrent uploads. Defaults to 10. Must be greater than 0.

The component-level decodeBase64 metadata option applies to bulk create. When set to "true", the input is decoded from base64 before uploading to S3. This applies to both inline data fields and content read from filePath — use filePath with decodeBase64 only when the file itself contains base64-encoded content.

Example

curl -d "{ \"operation\": \"bulkCreate\", \"data\": { \"items\": [{ \"key\": \"file1.txt\", \"data\": \"Hello World\" }, { \"key\": \"file2.txt\", \"data\": \"Another file\" }] } }" http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "bulkCreate", "data": { "items": [{ "key": "file1.txt", "data": "Hello World" }, { "key": "file2.txt", "data": "Another file" }] } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains a JSON array with a result for each item, including the location of the created object or a per-item error:

[
  { "key": "file1.txt", "location": "https://mybucket.s3.us-west-2.amazonaws.com/file1.txt" },
  { "key": "file2.txt", "location": "https://mybucket.s3.us-west-2.amazonaws.com/file2.txt" },
  { "key": "bad-file.txt", "error": "either data or filePath is required" }
]

Bulk delete objects

To delete multiple objects in a single batch operation, invoke the AWS S3 binding with a POST method and the following JSON body:

{
  "operation": "bulkDelete",
  "data": {
    "keys": ["file1.txt", "file2.txt", "file3.txt"]
  }
}

The data parameters are:

keys - (required) an array of object key names to delete. Must contain at least one key.

Bulk delete uses the S3 DeleteObjects batch API. If more than 1,000 keys are provided, the request is automatically split into batches of 1,000.

Example

curl -d "{ \"operation\": \"bulkDelete\", \"data\": { \"keys\": [\"file1.txt\", \"file2.txt\"] } }" http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

curl -d '{ "operation": "bulkDelete", "data": { "keys": ["file1.txt", "file2.txt"] } }' \
      http://localhost:<dapr-port>/v1.0/bindings/<binding-name>

Response

The response body contains a JSON array with a result for each key, including a per-item error field for partial failure semantics:

[
  { "key": "file1.txt" },
  { "key": "file2.txt" },
  { "key": "no-permission.txt", "error": "Access Denied" }
]

An empty error field (or its absence) indicates successful deletion.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Component format

Spec metadata fields

S3 Bucket Creation

Using with Minio

Binding support

Create object

Share object with a presigned URL

Response

Examples

Save text to a random generated UUID file

Save text to a specific file

Save a file to a object

Upload from file path

Response

Presign an existing object

Response

Get object

Example

Response

Delete object

Examples

Delete object

Response

List objects

Response

Bulk get objects

Example

Response

Bulk create objects

Example

Response

Bulk delete objects

Example

Response

Related links

FilesExpand file tree

s3.md

Latest commit

History

s3.md

File metadata and controls

Component format

Spec metadata fields

S3 Bucket Creation

Using with Minio

Binding support

Create object

Share object with a presigned URL

Response

Examples

Save text to a random generated UUID file

Save text to a specific file

Save a file to a object

Upload from file path

Response

Presign an existing object

Response

Get object

Example

Response

Delete object

Examples

Delete object

Response

List objects

Response

Bulk get objects

Example

Response

Bulk create objects

Example

Response

Bulk delete objects

Example

Response

Related links