How to set request_cpu globally for airflow worker pods using the kubernetes executor?

问题

I'm trying to set the request_cpu parameter in the Kubernetes executor for Airflow but haven't been able to find where I can do that. In the default airflow config I found default_cpus but according to this answer there is nowhere that that is used, and nowhere else in the Kubernetes section could I find a reference to the CPU request.

How can I set the request_cpu parameter in the Airflow Kubernetes executor?

EDIT: Ideally, what I would like to be able to do is set this as a global default rather than on a per-operator basis, even though in general I think that it does make more sense to set it per-operator/task

回答1:

You can set this executor_config at task level for KubernetesExecutor as follows.

    exmaple_task = PythonOperator(
        task_id="exmaple_task",
        python_callable=print_stuff,
        executor_config={
            "KubernetesExecutor": {"request_cpu": "1",
                                   "request_memory": "128Mi",
                                   "limit_memory": "128Mi"}}
    )

You can define the following in executor_config :

image
image_pull_policy
request_memory
request_cpu
limit_memory
limit_cpu
limit_gpu
node_selectors
affinity
tolerations
annotations
volumes
volume_mounts
volume_mounts

Doc: https://airflow.apache.org/docs/1.10.9/_api/airflow/contrib/executors/kubernetes_executor/index.html#airflow.contrib.executors.kubernetes_executor.KubernetesExecutorConfig

来源：https://stackoverflow.com/questions/60443124/how-to-set-request-cpu-globally-for-airflow-worker-pods-using-the-kubernetes-exe

标签

Kubernetes

airflow