Tuning SqsAsyncClient to not face Acquire Timeout issue coming from aws sdk v2

[aki59]

We are getting Acquire timeout issue in our queue due to slowness coming in downstream processing. We are using spring cloud aws starter sqs 4.0.0 and want to know how to tune our default SqsAsyncClient to handle this. I have read this but not sure if this is the only way to do it.

Also our SqsAsynclient is being shared with multiple queues(4 queues) with each queue having default maxconcurrentmessages(10) each.

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate. Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate. Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout. If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.

[MatejNedic]

Hey @aki59 , if you have high performance app. I would recommend creating SqsMessageListenerContainerFactory for each queue. You can then create multiple sqsAsyncClient with each having connection pool of their own. You can then fine tune each client for specific need of that queue.

@Bean
    public SqsMessageListenerContainerFactory<Object> defaultSqsListenerContainerFactory() {
        return SqsMessageListenerContainerFactory
            .builder()
            .sqsAsyncClient(sqsAsyncClient())
            .build();
    }

You can also use SqsAsyncClientCustomizer to fine tune connection pool which by default will be used by all methods annotated with SqsListener.

Example:

@Bean
public SqsAsyncClientCustomizer sqsAsyncClientCustomizer() {
    return builder -> builder.httpClient(
        NettyNioAsyncHttpClient.builder()
            .maxConcurrency(200)
            .connectionAcquireTimeout(Duration.ofSeconds(15))
            .build()
    );
}

[tomazfernandes]

Hey @aki59, thanks for bringing this up, and @MatejNedic for the configuration suggestion.

For a shared client across multiple queues, tuning the HTTP client settings may be enough. If you want separate clients per listener container, SqsMessageListenerContainerFactory also supports configuring a Supplier<SqsAsyncClient>, as described in this section of the docs:

@Bean
public SqsMessageListenerContainerFactory<Object> defaultSqsListenerContainerFactory() {
    return SqsMessageListenerContainerFactory
        .builder()
        .sqsAsyncClientSupplier(() -> SqsAsyncClient.builder()
            .httpClient(NettyNioAsyncHttpClient.builder()
                .maxConcurrency(50)
                .connectionAcquireTimeout(Duration.ofSeconds(15))
                .build())
            .build())
        .build();
}

This could be better documented, happy to review a PR adding a more explicit section to the docs.