aws-batch | 易学教程

AWS Batch job getting Access Denied on S3 despite user role

阅读更多关于 AWS Batch job getting Access Denied on S3 despite user role

问题 I am deploying my first batch job on AWS. When I run my docker image in an EC2 instance, the script called by the job runs fine. I have assigned an IAM role to this instance to allow S3 access. But when I run the same script as a job on AWS Batch, it fails due to Access Denied errors on S3 access. This is despite the fact that in the Job Definition, I assign an IAM role (created for Elastic Container Service Task) that has full S3 access. If I launch my batch job with a command that does not

How is a batch processing system in AWS ECS different from AWS Batch?

阅读更多关于 How is a batch processing system in AWS ECS different from AWS Batch?

问题 I have a batch processing system which runs for 5 hours daily on a fixed time. With AWS Batch, I could schedule the job which creates the required EC2 instances to do the job and terminate back the instances.But with ECS, can i launch and terminate the EC2 instance automatically as per my requirement? 回答1: This automatic downscaling can be done using AWS Batch as long as you create the Compute Environment as a Managed Compute Environment instead of unmanaged. However, using Tasks in ECS means

Use of Snakemake workflows in AWS Batch

阅读更多关于 Use of Snakemake workflows in AWS Batch

问题 I wanted to ask the Snakemake community if anybody has had success implementing Snakemake workflows in AWS Batch. Page 4 of a recent publication from Oct 2018 seems to suggest that Snakemake does not work on AWS as it cannot handle resource management. Here is the publication: Tibanna: software for scalable execution of portable pipelines on the cloud - https://www.biorxiv.org/content/early/2019/04/29/440974.full.pdf Yes, the same paper does suggest the Snakemake works well with the Google

Create AWS Batch Managed Compute Environment passing UserData to Container Instances

阅读更多关于 Create AWS Batch Managed Compute Environment passing UserData to Container Instances

问题 I would like to create a Managed Compute Environment for AWS Batch, but use EC2 User Data to configure the instances as they are brought into the ECS fleet that Batch is scheduling jobs onto. It shouldn't matter, but the purpose of the User Data script is to pull down large data files onto an InstanceStore that the Docker containers will reference. This is possible in ECS, but I have found no way to pass User Data to a Managed Batch Compute Environment. At most, I can specify the AMI. But

AWS batch - how to limit number of concurrent jobs

阅读更多关于 AWS batch - how to limit number of concurrent jobs

问题 I am looking for a way to limit the number of batch jobs that are running by holding the remaining jobs in the queue. Is it possible with aws batch? 回答1: Limiting the maximum number of vcpus of the managed compute environment the queue is tied to will effectively limit the number of batch jobs running concurrently on that queue. However, this comes with the caveat that, if you have other queues sharing this compute environment, they would also be limited accordingly. Moreover, if you have

How do I view AWS Batch Compute Environment Errors?

阅读更多关于 How do I view AWS Batch Compute Environment Errors?

问题 We setup a batch compute environment, job queue, and job definition. The min CPUs for the compute environment is set to 16, so it should always have at least one EC2 instance running. It's a MANAGED environment. It is not starting any, yet everything is still reporting healthy. I've looked at the troubleshooting page and nothing useful has come of it yet. Where can I go to see what is going wrong? Is this completely a black box and if I make a mistake somewhere in my config (Probable some

Why are AWS Batch Jobs stuck in RUNNABLE?

阅读更多关于 Why are AWS Batch Jobs stuck in RUNNABLE?

问题 I use a computing environment of 0-256 m3.medium on demand instances. My Job definition requires 1 CPU and 3 GB of Ram, which m3.medium has. What are possible reasons why AWS Batch Jobs are stuck in state RUNNABLE ? AWS says: A job that resides in the queue, has no outstanding dependencies, and is therefore ready to be scheduled to a host. Jobs in this state are started as soon as sufficient resources are available in one of the compute environments that are mapped to the job’s queue. However

Why are AWS Batch Jobs stuck in RUNNABLE?

阅读更多关于 Why are AWS Batch Jobs stuck in RUNNABLE?

I use a computing environment of 0-256 m3.medium on demand instances. My Job definition requires 1 CPU and 3 GB of Ram, which m3.medium has. What are possible reasons why AWS Batch Jobs are stuck in state RUNNABLE ? AWS says: A job that resides in the queue, has no outstanding dependencies, and is therefore ready to be scheduled to a host. Jobs in this state are started as soon as sufficient resources are available in one of the compute environments that are mapped to the job’s queue. However, jobs can remain in this state indefinitely when sufficient resources are unavailable. but that does