Stackdriver logs not available for Cloud ML jobs since migration to V2

Question

Since migration to V2 logs from Cloud ML jobs are not accessible on the Stackdriver logging console anymore. The last log displayed is

Waiting for Tensorflow to         


        

Answer 1

This problem also took me two weeks to search answers online with frustration, until I came across this post. I did not see "migration to V2" as OP mentions but I simply could not get any application logs in StackDriver, only system logs of job started/completed. Following what Jeremy replies solves the problem.

To make Jeremy's reply simpler to follow, essentially you add the ML service account

cloud-ml-service@.iam.gserviceaccount.com

to your project's IAM members, with at least "Logs Writer" role.

You can get "project-id" by:

gcloud config list project --format "value(core.project)"

I also assigned Project->Editor role to allow Bucket access.