feat(AzBatchService): Include cpu-shares and memory in docker run command

[adamrtalbot]

The Azure Batch task did not include the --cpu-shares or --memory options of the docker run command, meaning resource allocation was only controlled by slots on the node. This PR introduces the additional options to the container run command, including --cpu-shares and --memory. The behaviour will now be more similar to AWS Batch which includes a soft limit on CPUs and hard limit on memory.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	549bb8f
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67b6497fce5d2500083634ce
😎 Deploy Preview	https://deploy-preview-5799--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pditommaso]

Nice. I'd prefer readability in this code, seem the befits of string builder is negligible in this scenario.

[adamrtalbot]

I quite like the StringBuilder 😢 but I can revert it quickly enough.

[adamrtalbot]

Nice. I'd prefer readability in this code, seem the befits of string builder is negligible in this scenario.

Reverted in e0daedb

[pditommaso]

Nice @adamrtalbot, well done

[KH-NN]

Hi,

Just fyi. This PR has implications on AZCOPY. If you set memory to e.g. 4GB azcopy will crash when transferring multiple large files (50GB+). For large files we found that a minimum of 16GB is required to copy the files to the VM.

Due to code in link it will then try to generate a folder that also crashs. It took a bit of time to find this bug :)

[adamrtalbot]

Just fyi. This PR has implications on AZCOPY. If you set memory to e.g. 4GB azcopy will crash when transferring multiple large files (50GB+). For large files we found that a minimum of 16GB is required to copy the files to the VM.

Is this because azcopy requires more than 4GB? This is now a hard limit, so if azcopy tries to use more than 4GB it will cause issues. In theory, it will cause fewer issues when packing large number of tasks onto a single node.

I'm not massively familiar with azcopy memory control but we could see if we could control azcopy memory usage, e.g. take longer but stay under that limit.

[adamrtalbot]

Due to code in link it will then try to generate a folder that also crashs. It took a bit of time to find this bug :)

I've created issue #6158 for this, please drop a reproducible example there and the impact this has on your workflow there.

[adamrtalbot]

Looks like we should be able to fix this by setting AZCOPY_BUFFER_GB: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize#optimize-memory-use

#6161