Why flink container vcore size is always 1

Question

I am running flink on yarn(more precisely in AWS EMR yarn cluster).

I read flink document and source code that by default for each task manager container, flink will

Answer 1

An answer from Kien Truong

Hi,

You have to enable CPU scheduling in YARN, otherwise, it always shows that only 1 CPU is allocated for each container, regardless of how many Flink try to allocate. So you should add (edit) the following property in capacity-scheduler.xml:

 yarn.scheduler.capacity.resource-calculator
 
 org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

TaskManager memory is, for example, 1400MB, but Flink reserves some amount for off-heap memory, so the actual heap size is smaller.

This is controlled by 2 settings:

containerized.heap-cutoff-min: default 600MB

containerized.heap-cutoff-ratio: default 15% of TM's memory

That's why your TM's heap size is limitted to ~800MB (1400 - 600)

Regards,

Kien