s3: publishing can be slow

[arges]

When publishing to an s3 bucket target, it can take a very long time. If there was a way to push files in parallel it would greatly speed things up.

[cchazalet]

Hi @arges

I've also noticed some slowdowns, but they could be caused by a variety of factors, so it's difficult to investigate.

I recommend switching to “MultiDist=True”; this greatly improves the speed of S3 publications.

[arges]

@cchazalet for the case I investigated, wtihin a single distribution I saw that every package was uploaded to the s3 bucket one at a time. The upload throughput was very low relative to what I knew was possible on that network. By making the individual files upload in parallel I was able to make this much faster in my patch: #1616.