Copying images during publishing is extremely slow

[lbussell]

#2139 introduced bounded concurrency during the CopyAcrImagesCommand:

docker-tools/src/ImageBuilder/Commands/CopyAcrImagesCommand.cs

Lines 62 to 73 in 4a187e7

	// Bound concurrency so ImportImageAsync's ORAS referrer lookups don't stampede ACR's
	// rate limiter. See AcrParallelism for details.
	await Parallel.ForEachAsync(
	allTagInfos,
	AcrParallelism.CreateOptions(),
	async (tagInfo, cancellationToken) =>
	await ImportImageAsync(
	DockerHelper.TrimRegistry(tagInfo.DestinationTag, Manifest.Registry),
	Manifest.Registry,
	DockerHelper.TrimRegistry(tagInfo.SourceTag, Options.SourceRegistry),
	copyReferrers: true,
	srcRegistryName: Options.SourceRegistry));

That PR was based on the premise that bounding concurrency while making ACR HTTP Requests (through ORAS) would result in a higher overall throughput than saturating the new ACR rate limiting (#2137).

This appeared to be true in my local testing that simply hammered the ACR as quickly as possible. However, a recent pipeline run averaged about 26 requests per 60 seconds, roughly 10 times lower than the lowest rate limit.

This is because ImportImageAsync, which runs inside the same restricted parallelism, waits for each import to complete. In the previously linked pipeline, each import operation took anywhere from 0.2 to 10.0 seconds.

The actual pipeline took 72 minutes to copy all images. I have calculated some hypothetical numbers on what would happen if we ran at the rate limit with unbounded imports:

• Total ACR requests: 3,290 (1,638 manifest requests + 1,624 referrer requests + 28 oauth requests)
• Permit rate: 240 / 60s = 4 req/s

3290 requests / 4 rps = 822.5 seconds = 13.7 minutes

Additional note on ACR rate limits

From some experimentation, I learned that ACR's rate limits are different per-operation. Manifest requests and referrer lookups appear to have different rate limits. Referrer lookups have a much lower rate limits than other operations.

If we rate limit only the referrer lookups, then the theoretical floor drops to ~6.5 minutes.

Furthermore, it appears that different tiers of ACR have different rate limits. I have personally observed that Premium ACRs have a referrer rate limit of 1000/60s, while Basic ACRs have a limit of 250/60s.

Proposed solution

The changes from #2137 and #2139 should be significantly dialed back. We should:

• Revert unnecessary changes to bounded parallelism until we notice problems happening in other commands.
• Only rate limit referrer lookups. Other operations can rely on Microsoft.Extensions.Http's default resilience pipeline behavior which retries on 429s and respects Retry-After headers. The likelihood of hitting the other rate limits is highly unlikely.
• Referrer rate limiting should default to 250/60s, but allow setting a higher limit via configuration.