From f24f4eb41cd4c9aa10746f27bca008e655f373d1 Mon Sep 17 00:00:00 2001 From: alessio94 Date: Mon, 23 Mar 2026 14:13:07 +0100 Subject: [PATCH 1/6] Add S3 storage and multi-hop transfer tutorial section Addresses feedback from PR #744: - Remove Jupyter notebook - Inline script content directly as code blocks - Replace inline script comments with prose descriptions - Remove docker exec wrapping; assume reader is in a Rucio admin environment - Narrow scope to S3 RSE setup only; remove environment initialization steps --- docs/operator/setting_up_demo.md | 204 +++++++++++++++++++++++++++++++ 1 file changed, 204 insertions(+) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 8191a811b8d..293f2a70ac0 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -426,3 +426,207 @@ $ rucio list-file-replicas test:mynewdataset | test | file4 | 10.486 MB | 65786e49 | XRD3: root://xrd3:1096//rucio/... | +-------+-------+-----------+----------+-----------------------------------+ ``` + +# Configuring S3 Storage and Multi-Hop Transfers in Rucio + +This tutorial covers how to register S3-compatible storage (MinIO) as Rucio Storage Elements (RSEs), configure credentials for both Rucio and FTS, and set up RSE distances to enable multi-hop transfers between S3 and XRootD endpoints. + +The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already operating within a Rucio admin environment with the `rucio` and `rucio-admin` CLI tools available. + +## Enabling HTTPS on XRD3 for Multi-Hop + +XRD3 acts as the intermediate hop between S3 and XRootD storage. To allow it to communicate with S3 backends, add an HTTPS protocol entry to the XRD3 RSE: + +```bash +rucio rse protocol add XRD3 \ + --host xrd3 \ + --scheme https \ + --prefix //rucio \ + --port 1096 \ + --impl rucio.rse.protocols.gfal.Default \ + --domain-json '{"wan": {"read": 2, "write": 2, "delete": 2, "third_party_copy_read": 2, "third_party_copy_write": 2}, "lan": {"read": 2, "write": 2, "delete": 2}}' +``` + +The priority values (`"read": 2` etc.) ensure that the existing XRootD protocol remains preferred for direct transfers, while HTTPS is available for multi-hop routing. + +## Creating Buckets on MinIO + +Before registering MinIO instances as RSEs, create the `rucio` bucket on each. This uses the MinIO Client (`mc`) from within each MinIO container: + +```bash +# On MINIO1 +export MC_INSECURE=true +mc alias set local https://localhost:9001 admin password +mc mb local/rucio + +# On MINIO2 +export MC_INSECURE=true +mc alias set local https://localhost:9002 admin password +mc mb local/rucio +``` + +## Registering MinIO RSEs + +Register both MinIO instances as RSEs with S3 protocol configuration. The `gfal.NoRename` implementation is used because S3 does not support server-side rename operations. + +```bash +rucio rse add MINIO1 +rucio rse protocol add MINIO1 \ + --host minio1 \ + --port 9001 \ + --scheme https \ + --prefix /rucio/ \ + --impl rucio.rse.protocols.gfal.NoRename \ + --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' +rucio rse attribute add MINIO1 --key sign_url --value s3 +rucio rse attribute add MINIO1 --key s3_url_style --value path +rucio rse attribute add MINIO1 --key verify_checksum --value False +rucio rse attribute add MINIO1 --key skip_upload_stat --value True +rucio rse attribute add MINIO1 --key strict_copy --value True +rucio rse attribute add MINIO1 --key fts --value https://fts:8446 +rucio account limit add root --rse MINIO1 --bytes infinity + +rucio rse add MINIO2 +rucio rse protocol add MINIO2 \ + --host minio2 \ + --port 9002 \ + --scheme https \ + --prefix /rucio/ \ + --impl rucio.rse.protocols.gfal.NoRename \ + --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' +rucio rse attribute add MINIO2 --key sign_url --value s3 +rucio rse attribute add MINIO2 --key s3_url_style --value path +rucio rse attribute add MINIO2 --key verify_checksum --value False +rucio rse attribute add MINIO2 --key skip_upload_stat --value True +rucio rse attribute add MINIO2 --key strict_copy --value True +rucio rse attribute add MINIO2 --key fts --value https://fts:8446 +rucio account limit add root --rse MINIO2 --bytes infinity +``` + +### Setting RSE Credentials + +Rucio needs S3 credentials to generate presigned URLs for transfers. These are stored in `rse-accounts.cfg`, keyed by RSE ID: + +```bash +ID1=$(rucio rse show MINIO1 | grep '^ id:' | awk '{print$2}') +ID2=$(rucio rse show MINIO2 | grep '^ id:' | awk '{print$2}') +cat >/opt/rucio/etc/rse-accounts.cfg < Date: Mon, 23 Mar 2026 13:22:43 +0000 Subject: [PATCH 2/6] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/operator/setting_up_demo.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 293f2a70ac0..5a36595292a 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -629,4 +629,3 @@ Monitor the rule status with: ```bash rucio rule list --account root ``` - From 21ad8a77908618ea0b54aeb38441d47cd2df2628 Mon Sep 17 00:00:00 2001 From: alessio94 Date: Fri, 29 May 2026 16:58:26 +0200 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: voetberg --- docs/operator/setting_up_demo.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 5a36595292a..702fa23c12f 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -431,7 +431,7 @@ $ rucio list-file-replicas test:mynewdataset This tutorial covers how to register S3-compatible storage (MinIO) as Rucio Storage Elements (RSEs), configure credentials for both Rucio and FTS, and set up RSE distances to enable multi-hop transfers between S3 and XRootD endpoints. -The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already operating within a Rucio admin environment with the `rucio` and `rucio-admin` CLI tools available. +The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already have an rucio instance with an admin account. ## Enabling HTTPS on XRD3 for Multi-Hop @@ -530,7 +530,8 @@ JSON ### Configuring RSE Distances for Multi-Hop -RSE distances tell Rucio which transfer paths are available and their relative cost. Setting a distance of 1 between MINIO RSEs and XRD3 establishes the multi-hop path: transfers from MinIO to XRD1 or XRD2 will route through XRD3 as an intermediate. +RSE distances establish transfer paths between RSEs. Setting a distance of 1 between the source and an intermediate will ensure the intermediate transfer will always be preferred over longer direct transfers. + ```bash rucio rse distance add XRD3 MINIO1 --distance 1 From 69b148d38380968fccca4d5fcbb8f849dc81cd8b Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 29 May 2026 14:58:38 +0000 Subject: [PATCH 4/6] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/operator/setting_up_demo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 702fa23c12f..86ba3d7ed6f 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -431,7 +431,7 @@ $ rucio list-file-replicas test:mynewdataset This tutorial covers how to register S3-compatible storage (MinIO) as Rucio Storage Elements (RSEs), configure credentials for both Rucio and FTS, and set up RSE distances to enable multi-hop transfers between S3 and XRootD endpoints. -The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already have an rucio instance with an admin account. +The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already have an rucio instance with an admin account. ## Enabling HTTPS on XRD3 for Multi-Hop @@ -530,7 +530,7 @@ JSON ### Configuring RSE Distances for Multi-Hop -RSE distances establish transfer paths between RSEs. Setting a distance of 1 between the source and an intermediate will ensure the intermediate transfer will always be preferred over longer direct transfers. +RSE distances establish transfer paths between RSEs. Setting a distance of 1 between the source and an intermediate will ensure the intermediate transfer will always be preferred over longer direct transfers. ```bash From c11e12235110947f25446cd627fd2501182d8ce9 Mon Sep 17 00:00:00 2001 From: alessio94 Date: Fri, 29 May 2026 17:24:59 +0200 Subject: [PATCH 5/6] Address review comments and add RustFS compatibility note I have addressed the inline review comments. In the meanwhile Mario asked me to look into trying the same setup with another S3-compatible backend other than MinIO, RustFS so I added a short RustFS compatibility section at the end. --- docs/operator/setting_up_demo.md | 155 +++++++++++++++++++++++-------- 1 file changed, 117 insertions(+), 38 deletions(-) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 86ba3d7ed6f..32435513bc5 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -431,7 +431,7 @@ $ rucio list-file-replicas test:mynewdataset This tutorial covers how to register S3-compatible storage (MinIO) as Rucio Storage Elements (RSEs), configure credentials for both Rucio and FTS, and set up RSE distances to enable multi-hop transfers between S3 and XRootD endpoints. -The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already have an rucio instance with an admin account. +The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you already have a rucio instance with an admin account. ## Enabling HTTPS on XRD3 for Multi-Hop @@ -468,48 +468,44 @@ mc mb local/rucio ## Registering MinIO RSEs Register both MinIO instances as RSEs with S3 protocol configuration. The `gfal.NoRename` implementation is used because S3 does not support server-side rename operations. - -```bash -rucio rse add MINIO1 -rucio rse protocol add MINIO1 \ - --host minio1 \ - --port 9001 \ - --scheme https \ - --prefix /rucio/ \ - --impl rucio.rse.protocols.gfal.NoRename \ - --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' -rucio rse attribute add MINIO1 --key sign_url --value s3 -rucio rse attribute add MINIO1 --key s3_url_style --value path -rucio rse attribute add MINIO1 --key verify_checksum --value False -rucio rse attribute add MINIO1 --key skip_upload_stat --value True -rucio rse attribute add MINIO1 --key strict_copy --value True -rucio rse attribute add MINIO1 --key fts --value https://fts:8446 -rucio account limit add root --rse MINIO1 --bytes infinity - -rucio rse add MINIO2 -rucio rse protocol add MINIO2 \ - --host minio2 \ - --port 9002 \ - --scheme https \ - --prefix /rucio/ \ - --impl rucio.rse.protocols.gfal.NoRename \ - --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' -rucio rse attribute add MINIO2 --key sign_url --value s3 -rucio rse attribute add MINIO2 --key s3_url_style --value path -rucio rse attribute add MINIO2 --key verify_checksum --value False -rucio rse attribute add MINIO2 --key skip_upload_stat --value True -rucio rse attribute add MINIO2 --key strict_copy --value True -rucio rse attribute add MINIO2 --key fts --value https://fts:8446 -rucio account limit add root --rse MINIO2 --bytes infinity -``` +Each RSE is configured with the following attributes: + +- `sign_url`: enables S3 presigned URL generation for this RSE. +- `s3_url_style`: uses path-style S3 URLs, which are suitable for the local MinIO endpoints. +- `verify_checksum`: disables checksum verification for this local demonstration setup. +- `skip_upload_stat`: avoids relying on an upload stat call for the S3 endpoint. +- `strict_copy`: enables stricter transfer handling for this RSE. +- `fts`: points Rucio to the FTS endpoint used for third-party transfers. + +```bash +for i in 1 2; do + RSE="MINIO${i}" + HOST="minio${i}" + PORT="900${i}" + rucio rse add "${RSE}" + rucio rse protocol add "${RSE}" \ + --host "${HOST}" \ + --port "${PORT}" \ + --scheme https \ + --prefix /rucio/ \ + --impl rucio.rse.protocols.gfal.NoRename \ + --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' + rucio rse attribute add "${RSE}" --key sign_url --value s3 + rucio rse attribute add "${RSE}" --key s3_url_style --value path + rucio rse attribute add "${RSE}" --key verify_checksum --value False + rucio rse attribute add "${RSE}" --key skip_upload_stat --value True + rucio rse attribute add "${RSE}" --key strict_copy --value True + rucio rse attribute add "${RSE}" --key fts --value https://fts:8446 + rucio account limit add root --rse "${RSE}" --bytes infinity +done +``` ### Setting RSE Credentials Rucio needs S3 credentials to generate presigned URLs for transfers. These are stored in `rse-accounts.cfg`, keyed by RSE ID: - ```bash -ID1=$(rucio rse show MINIO1 | grep '^ id:' | awk '{print$2}') -ID2=$(rucio rse show MINIO2 | grep '^ id:' | awk '{print$2}') +ID1=$(rucio rse show MINIO1 | grep '^ id:' | awk '{print $2}') +ID2=$(rucio rse show MINIO2 | grep '^ id:' | awk '{print $2}') cat >/opt/rucio/etc/rse-accounts.cfg </opt/rucio/etc/rse-accounts.cfg <": { + "access_key": "admin", + "secret_key": "password", + "signature_version": "s3v4", + "region": "us-east-1" + }, + "": { + "access_key": "admin", + "secret_key": "password", + "signature_version": "s3v4", + "region": "us-east-1" + } +} +``` ### Configuring RSE Distances for Multi-Hop RSE distances establish transfer paths between RSEs. Setting a distance of 1 between the source and an intermediate will ensure the intermediate transfer will always be preferred over longer direct transfers. +```mermaid +graph TD + MINIO[MINIO RSE] + XRD1[XRD1 RSE] + XRD2[XRD2 RSE] + XRD3[XRD3 RSE] + MINIO -.->|distance=1| XRD3 + XRD3 -.->|distance=1| XRD1 + XRD3 -.->|distance=1| XRD2 +``` ```bash rucio rse distance add XRD3 MINIO1 --distance 1 @@ -630,3 +654,58 @@ Monitor the rule status with: ```bash rucio rule list --account root ``` +## RustFS: Validating Storage Portability + +RustFS can be used as an alternative S3-compatible backend for the same workflow. +To evaluate the portability of the workflow, the testbed was additionally deployed against +RustFS. The transition required only minimal configuration changes, confirming that the +Rucio integration is largely independent of the underlying storage backend. + +### Starting RustFS + +Generate a self-signed TLS certificate and add RustFS to the stack as a Docker Compose override service, mounting the certificate into the container: + +```yaml +services: + rustfs: + image: rustfs/rustfs:latest + hostname: rustfs + environment: + RUSTFS_ACCESS_KEY: admin + RUSTFS_SECRET_KEY: password + RUSTFS_DOMAIN: "" # enforce path-style URLs + command: server --address :9003 --certs-dir /certs /data + ports: + - "9003:9003" +``` + +### Trusting the Certificate + +After the containers are running, inject the self-signed certificate into the Rucio CA bundle. Without this, GFAL2 rejects the certificate on upload. + +```bash +docker cp rustfs/public.crt \ + dev-rucio-1:/etc/pki/ca-trust/source/anchors/rustfs.pem +docker exec dev-rucio-1 update-ca-trust +``` + +### Registering the RSE + +The RSE is registered similarly to MinIO, using `--scheme https` and the same set of S3 attributes: + +```bash +rucio rse add RUSTFS_EU +rucio rse protocol add RUSTFS_EU \ + --host rustfs --port 9003 --scheme https --prefix /rucio/ \ + --impl rucio.rse.protocols.gfal.NoRename \ + --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, + "wan": {"read": 1, "write": 1, "delete": 1, + "third_party_copy_read": 1, "third_party_copy_write": 1}}' +rucio rse attribute add RUSTFS_EU --key sign_url --value s3 +rucio rse attribute add RUSTFS_EU --key s3_url_style --value path +rucio rse attribute add RUSTFS_EU --key verify_checksum --value False +rucio rse attribute add RUSTFS_EU --key skip_upload_stat --value True +rucio rse attribute add RUSTFS_EU --key fts --value https://fts:8446 +``` + +Add credentials to `rse-accounts.cfg` and register the FTS cloud-storage entry and GFAL2 `s3.conf` section following the same pattern as for MinIO, replacing the hostname with `rustfs`. From 492f2ed401a44aedf1accc614b5a7f0797d911a2 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 29 May 2026 15:25:13 +0000 Subject: [PATCH 6/6] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/operator/setting_up_demo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md index 32435513bc5..e19e295855d 100644 --- a/docs/operator/setting_up_demo.md +++ b/docs/operator/setting_up_demo.md @@ -498,7 +498,7 @@ for i in 1 2; do rucio rse attribute add "${RSE}" --key fts --value https://fts:8446 rucio account limit add root --rse "${RSE}" --bytes infinity done -``` +``` ### Setting RSE Credentials @@ -656,7 +656,7 @@ rucio rule list --account root ``` ## RustFS: Validating Storage Portability -RustFS can be used as an alternative S3-compatible backend for the same workflow. +RustFS can be used as an alternative S3-compatible backend for the same workflow. To evaluate the portability of the workflow, the testbed was additionally deployed against RustFS. The transition required only minimal configuration changes, confirming that the Rucio integration is largely independent of the underlying storage backend.