Introduced support for S3ObjectStore

[steph-ieffam]

Add S3 support to the Replication Task Suite

Resolves #82

Summary

Adds a new ObjectStore implementation, S3ObjectStore, that allows the Replication Task
Suite to store and retrieve AIPs directly in an Amazon S3 bucket, alongside the existing
LocalObjectStore, MountableObjectStore and DuraCloudObjectStore backends.

The implementation is based on the AWS SDK for Java v2 (the same SDK already used by
dspace-api for S3BitStoreService) and uses S3TransferManager so that AIP uploads and
downloads are not limited by the 5 GB single-PutObject cap: multipart transfers are
handled automatically based on file size.

Refs: CST-27589

Changes

src/main/java/org/dspace/ctask/replicate/store/S3ObjectStore.java (new)

Full ObjectStore implementation:

• init() – builds the sync and async S3 clients plus the S3TransferManager, resolves the
  target bucket and creates it if it does not exist yet.
• objectExists() / objectAttribute() – HeadObject-based lookups for existence and
  sizebytes; checksum is computed as MD5 over a temporary local copy of the object.
• transferObject() / fetchObject() – uploads and downloads via S3TransferManager,
  with automatic multipart handling for large AIPs and a synchronous fallback for downloads.
• removeObject() / moveObject() – delete, and copy-then-delete to emulate a move between
  groups (e.g. from the AIP store to the delete/replica group).
• Objects are keyed as <group>/<id>, mirroring the folder layout of the existing stores.
• A dedicated logger (org.dspace.ctask.replicate.store.S3ObjectStore.operations) is used
  for all S3 operations so that S3 traffic can be traced independently of the rest of the
  curation logging.

Three authentication modes are supported, resolved in this order:

1. Static credentials (replicate.s3.access-key / replicate.s3.secret-key) – kept only
   for backwards compatibility with existing/legacy buckets.
2. AssumeRole for cross-account access (replicate.s3.assume-role-arn), with optional
   ExternalId (replicate.s3.assume-role-external-id) to protect against confused-deputy
   attacks. Credentials are refreshed automatically.
3. The default AWS credentials provider chain, i.e. the IAM role attached to the instance /
   task / pod, which is the recommended setup and requires only the region to be configured.

config/modules/replicate.cfg

• S3ObjectStore added to the list of documented ObjectStore implementations.
• New ### S3 Replicate configuration ### section documenting
  replicate.s3.bucket-name, replicate.s3.region-name,
  replicate.s3.assume-role-arn, replicate.s3.assume-role-external-id and the
  commented-out legacy replicate.s3.access-key / replicate.s3.secret-key.
  All properties default to empty; if replicate.s3.bucket-name is not set the store logs a
  warning and skips initialization, so existing installations are unaffected.

pom.xml

dspace-api only declares software.amazon.awssdk:s3, so the two artifacts required by this
class had to be added explicitly:

• software.amazon.awssdk:sts – StsClient, StsAssumeRoleCredentialsProvider,
  AssumeRoleRequest
• software.amazon.awssdk:s3-transfer-manager – S3TransferManager, UploadFileRequest,
  DownloadFileRequest

Both are declared with provided scope (the JARs are expected in the DSpace installation at
runtime) and with the same netty-nio-client / apache-client exclusions that dspace-api
applies to s3. The version is pinned via a new aws.sdk.version property set to 2.43.2,
which is the AWS SDK version pulled in by dspace-api 9.3 — keeping them aligned is required
to satisfy the DependencyConvergence enforcer rule.

Configuration example

plugin.single.org.dspace.ctask.replicate.ObjectStore = \
    org.dspace.ctask.replicate.store.S3ObjectStore

replicate.s3.bucket-name = my-dspace-aip-bucket
replicate.s3.region-name = eu-west-1
# optional, for cross-account access
replicate.s3.assume-role-arn = arn:aws:iam::123456789012:role/dspace-replication
replicate.s3.assume-role-external-id = some-shared-secret

[steph-ieffam]

@tdonohue & @nwoodward I have managed to open a new PR for introducing support to replicate data using a S3 bucket

[nwoodward]

@steph-ieffam Thanks for this much needed PR! I'll set aside some time to test it.

[nwoodward]

Hi @steph-ieffam. I apologize for not getting to this sooner. The code looks very good, but I keep getting the error below when running curation tasks such as transmitaip with the legacy S3 access and secret keys. I made sure that my AWS SDK version (2.49.4) matched the version in dspace-api. Any ideas?

2026-09-17T18:42:49.956405Z INFO curate - 40 @ The script has started
2026-09-17T18:42:50.184914Z ERROR curate - 40 @ Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.
2026-09-17T18:42:50.194091Z ERROR curate - 40 @ software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
	at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.lambda$buildWithDefaults$1(DefaultSdkHttpClientBuilder.java:51)
	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
	at software.amazon.awssdk.core.internal.http.loader.DefaultSdkHttpClientBuilder.buildWithDefaults(DefaultSdkHttpClientBuilder.java:45)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveSyncHttpClient$18(SdkDefaultClientBuilder.java:480)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveSyncHttpClient(SdkDefaultClientBuilder.java:480)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$finalizeSyncConfiguration$5(SdkDefaultClientBuilder.java:316)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
	at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
	at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:224)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeSyncConfiguration(SdkDefaultClientBuilder.java:318)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:207)
	at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:37)
	at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:26)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:173)
	at org.dspace.ctask.replicate.store.S3ObjectStore.initializeS3Client(S3ObjectStore.java:377)
	at org.dspace.ctask.replicate.store.S3ObjectStore.init(S3ObjectStore.java:74)
	at org.dspace.ctask.replicate.ReplicaManager.<init>(ReplicaManager.java:79)
	at org.dspace.ctask.replicate.ReplicaManager.instance(ReplicaManager.java:102)
	at org.dspace.ctask.replicate.TransmitAIP.perform(TransmitAIP.java:69)
	at org.dspace.curate.ResolvedTask.perform(ResolvedTask.java:79)
	at org.dspace.curate.Curator$TaskRunner.run(Curator.java:565)
	at org.dspace.curate.Curator.doCollection(Curator.java:516)
	at org.dspace.curate.Curator.curate(Curator.java:295)
	at org.dspace.curate.Curator.curate(Curator.java:258)
	at org.dspace.curate.Curation.handleCurationTask(Curation.java:152)
	at org.dspace.curate.Curation.internalRun(Curation.java:82)
	at org.dspace.scripts.DSpaceRunnable.run(DSpaceRunnable.java:150)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

[steph-ieffam]

Hey @nwoodward no worries.
I pushed some fixes. I think the problem was related to having this implementation directly in the DSpace code base for some of our projects while I have moved this directly to the addon and this issue seemed to be related to the AWS dependency.
I have refactored the code a bit and now the problem should be solved.
Please tell me if you're still experiencing issues while installing/testing this feature.

Thanks for your feedback