google app engine deploy a custom vm app takes a long time to deploy

Question

Here is my worker.yaml

runtime: custom #python27
api_version: 1
threadsafe: false
vm: true
service: worker
env_variables:
  PYTHON_ENV: lab
network:
  instance_tag: testing123
  name: dev


handlers:
- url: /.*
  script: Framework.Workers.PushQueues.worker.app
  login: admin

Dockerfile

FROM us.gcr.io/smiling-diode-638/basic-algo-docker-v2

and the console output:

gcloud app deploy worker.yaml  --verbosity='debug'                              ✱
DEBUG: Running gcloud.app.deploy with Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.app.deploy', usage=None, description='Deploy the local code and/or configuration of your app to App Engine.', version=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=False), account=None, authority_selector=None, authorization_token_file=None, bucket=None, calliope_command=<googlecloudsdk.calliope.backend.Command object at 0x108770290>, command_path=['gcloud', 'app', 'deploy'], configuration=None, credential_file_override=None, deployables=['worker.yaml'], docker_build=None, document=None, flatten=None, format=None, h=None, help=None, http_timeout=None, ignore_bad_certs=False, image_url=None, log_http=None, project=None, promote=None, quiet=None, server=None, skip_image_url_validation=False, skip_staging=False, stop_previous_version=None, trace_email=None, trace_log=None, trace_token=None, user_output_enabled=None, verbosity='debug', version=None).
DEBUG: API endpoint: [https://appengine.googleapis.com/], API version: [v1beta5]
You are about to deploy the following services:
 - smiling-diode-638/worker/20161109t165841 (from [/Volumes/ownhealth_dev/Backend/worker.yaml])
     Deploying to URL: [https://worker-dot-smiling-diode-638.appspot.com]

Do you want to continue (Y/n)?  Y

DEBUG: No bucket specified, retrieving default bucket.
DEBUG: Using bucket [gs://staging.smiling-diode-638.appspot.com/].
If this is your first deployment, this may take a while...DEBUG: Host: appengine.google.com
DEBUG: _Authenticate configuring auth; needs_auth=False
DEBUG: Sending request to https://appengine.google.com/api/vms/prepare?app_id=smiling-diode-638 headers={'X-appcfg-api-version': '1', 'content-length': '0', 'Content-Type': 'application/octet-stream'} body=
INFO: Attempting refresh to obtain initial access_token
INFO: Refreshing access_token
If this is your first deployment, this may take a while...\DEBUG: Got response: {bucket: vm-containers.smiling-diode-638.appspot.com, path: /containers}

If this is your first deployment, this may take a while...done.

Beginning deployment of service [worker]...
DEBUG: No staging command found for runtime [custom] and environment [MANAGED_VMS].
WARNING: Deployment of App Engine Flexible Environment apps is currently in Beta
INFO: Using Dockerfile found in /Volumes/ownhealth_dev/Backend
Building and pushing image for service [worker]


INFO: Refreshing access_token
INFO: Uploading [/var/folders/d1/lgf326px48s8w6cvj2f7ng200000gp/T/tmplQDUjR/src.tgz] to [us.gcr.io/smiling-diode-638/appengine/worker.20161109t165841:latest]

The image is 3.5GB

it's been more than an hour and it's still deploying.

Edit: I tried deploying from a local network machine and it's indeed faster to download the image but the deploy is endless and i'm getting Error Response: [4] DEADLINE_EXCEEDED

all the errors I see on that are attached ERROR: build step "gcr.io/cloud-builders/docker@sha256:1b5a8642d04ef785f7dd2dbb0296f837895f50442fdfe5ebd0b04c206ab0b1b5" failed: exit status 2

where can I check logs for that?

Edit # 2:

I ran it like this

gcloud preview app deploy worker.yaml --image-url=us.gcr.io/smiling-diode-638/basic-algo-docker-v2 --verbosity='debug'

getting Error Response: [13] Timed out when starting VMs. It's possible that the application code is unhealthy

But I've added this to the worker.yaml

health_check:
  enable_health_check: False

so how do I disable the health check ??

Answer 1

A good chunk of the deployment time is often spent uploading the docker image, largely depending on the image size (in your case pretty big - 3.5GB), your internet connection upload speed and your server's "closeness" to the GAE region datacenter.

You might get a significant upload speed boost by performing the deployment from a cloud-hosted VM (ideally a GCE machine hosted in the same region as your GAE app).

Building the image also takes time, some of it unavoidable. You might be able to speed it up a bit, see How can I speed up Rails Docker deployments on Google Cloud Platform?

Also, when re-deploying the app some unavoidable additional steps are required compared to the first app deployment, see why does google appengine deployment take several minutes to update service