rust-s3 [docs]

Rust library for working with Amazon S3 or arbitrary S3 compatible APIs, fully compatible with async/await and futures ^0.3. All async features can be turned off and sync only implementations can be used. All requests are automatically retried once, can be further modified with a set_retries call.

🙌 Support further maintenance and development

As many have noted my attention to the this library oscialtes, the best way to ensure I keep maintaining and adding features to it is to donate som BTC, or become a sponsor. Thank you for the support!

💰 BTC 👉 bc1q7ukqe09zplg2sltgfrkukghpelfaz7qja8pw6u

Intro

Modest interface towards Amazon S3, as well as S3 compatible object storage APIs such as Backblaze B2, Wasabi, Yandex, Minio or Google Cloud Storage. Supports: put, get, list, delete, operations on tags and location, as well as head.

Additionally, a dedicated presign_get Bucket method is available. This means you can upload to S3, and give the link to select people without having to worry about publicly accessible files on S3. This also means that you can give people a PUT presigned URL, meaning they can upload to a specific key in S3 for the duration of the presigned URL.

Quick Start

[dependencies]
rust-s3 = "0.37.2"

Read and run examples from the examples folder, make sure you have valid credentials for the variant you're running.

# tokio, default
cargo run --example tokio

# async-std
cargo run --example async-std --no-default-features --features async-std-native-tls

# sync
cargo run --example sync --no-default-features --features sync-native-tls

# minio
cargo run --example minio

# r2
cargo run --example r2

# google cloud
cargo run --example google-cloud

Features

There are a lot of various features that enable a wide variety of use cases, refer to s3/Cargo.toml for an exhaustive list. Below is a table of various useful features as well as a short description for each.

• default - tokio runtime and a native-tls implementation
• blocking - generates *_blocking variant of all Bucket methods, otherwise only async versions are available
• fail-on-err - return Result::Err for HTTP errors

With default-features = false

• with-async-std - async-std runtime, surf client used
• async-std-native-tls and async-std-rustls-tls open a fresh HTTP/1 connection for each request to honor server Connection: close responses. This adds TCP/TLS setup overhead and disables Surf's HTTP/1 pool, so its implicit per-host connection cap no longer applies.
• sync - no async runtime, attohttpc is used for HTTP requests
• tags - required for Bucket::get_object_tagging

All runtimes support either native-tls or rustls-tls, there are features for all combinations, refer to s3/Cargo.toml for a complete list.

A download returns the bytes the server sends and reports their Content-Encoding as metadata. The tokio client never asks for a compressed transfer, and its decoders stay off even when another crate in the build enables reqwest/gzip, brotli, zstd or deflate. attohttpc's decoder cannot be switched off, and it asks servers to compress on its own: when another crate enables one of attohttpc's compression features (compress is in its defaults), the sync backend returns an error for an object download whose Content-Encoding is gzip or deflate, whether stored with the object or applied on the way, rather than its decoded bytes; cargo tree -e features -i attohttpc names the crate.

To build with Tokio and rustls on ring, disable the default native-TLS feature:

cargo build -p rust-s3 --no-default-features --features tokio-rustls-tls

tokio-rustls-tls-ring adds fetching credentials over HTTP (STS web identity, ECS, instance metadata), also with rustls on ring and the bundled Mozilla roots.

Unless tokio-native-tls is enabled too, the tokio-rustls-tls S3 client uses the rustls CryptoProvider the program installed process-wide before creating the Bucket, or ring if there is none, without installing one. It checks server certificates against the Mozilla roots bundled by webpki-roots; the system certificate store is not read. reqwest itself is built with rustls-no-provider, so a program that builds its own reqwest 0.13 clients on rustls, including through s3::reqwest, has to install a rustls CryptoProvider first or enable reqwest's rustls feature.

The default feature set remains Tokio with native TLS.

Path or subdomain style URLs and headers

Bucket struct provides constructors for path-style paths, subdomain style is the default. Bucket exposes methods for configuring and accessing path-style configuration. blocking feature will generate a *_blocking variant of all the methods listed below.

LocalStack Compatibility

When using LocalStack, you may need to skip sending the location constraint in bucket creation requests. LocalStack doesn't support location constraints in the request body and will return InvalidLocationConstraint errors. Set this environment variable to skip the constraint:

export RUST_S3_SKIP_LOCATION_CONSTRAINT=true
# or
export RUST_S3_SKIP_LOCATION_CONSTRAINT=1

This may also be needed for other S3-compatible services that don't support AWS-style location constraints.

Buckets

create	async
delete	async
list	async
exists	async

Presign

POST	presign_post
PUT	presign_put
GET	presign_get
DELETE	presign_delete

GET

There are a few different options for getting an object. sync and async methods are generic over std::io::Write, while tokio methods are generic over tokio::io::AsyncWriteExt.

async/sync/async-blocking	get_object
async	get_object_stream
async	get_object_range_stream
async/sync/async-blocking	get_object_to_writer

PUT

Each GET method has a PUT companion sync and async methods are generic over std::io::Read. async stream methods are generic over futures_io::AsyncReadExt, while tokio methods are generic over tokio::io::AsyncReadExt.

async/sync/async-blocking	put_object
async/sync/async-blocking	put_object_with_content_type
async/sync/async-blocking	put_object_stream

List

async/sync/async-blocking	list

DELETE

async/sync/async-blocking	delete_object
async/sync/async-blocking	delete_objects

Location

async/sync/async-blocking	location

Tagging

async/sync/async-blocking	put_object_tagging
async/sync/async-blocking	get_object_tagging

Head

async/sync/async-blocking	head_object