how to solve “Process terminated because the request deadline was exceeded. (Error code 123)” in google api?

Question

I have a projects deployed on Google App Engine having Google API (Python). Every request to any of API make a database connection , execute a procedure and return data and close the connection. I was not able to access any of API as it was showing

"Process terminated because the request deadline was exceeded. (Error code 123)" and "This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application." error.

Database is also on cloud (Google Cloud SQL). As I checked there was 900 connection and more than 150 instances were up but no api request was getting handled. This happens frequently. So I restart database server and deploy API code again to solve this issue. What is the issue and how can I solve this permanently ? Here is my python code for database connectivity :-

import logging
import traceback
import os
import MySQLdb
from warnings import filterwarnings

filterwarnings('ignore', category = MySQLdb.Warning)

class TalkWithDB:
    def callQueries(self,query,req_args):
        try:
            if (os.getenv('SERVER_SOFTWARE') and os.getenv('SERVER_SOFTWARE').startswith('Google App Engine/')):
                db = MySQLdb.connect(unix_socket = UNIX_SOCKET + INSTANCE_NAME, host = HOST, db = DB, user = USER ,charset='utf8',use_unicode=True)
            else:
                db = MySQLdb.connect(host = HOST, port = PORT, db = DB, user = USER, passwd = PASSWORD,charset='utf8',use_unicode=True)

            cursor = db.cursor()
            cursor.connection.autocommit(True)
        try:
        sql = query+str(req_args)
        logging.info("QUERY = "+ sql )
        cursor.execute(sql)     
        procedureResult = cursor.fetchall();
        if str(procedureResult) == '()':
            logging.info("Procedure Returned 0 Record")
            procedureResult = []
            procedureResult.append({0:"NoRecord", 1:"Error"})
            #procedureResult = (("NoRecord","Error",),)
        elif procedureResult[0][0] == 'Session Expired'.encode(encoding='unicode-escape',errors='strict'):
            procedureResult = []
            procedureResult.append({0:"SessionExpired", 1:"Error"})                     
            except Exception, err:
        logging.info("ConnectDB.py : - Error in Procedure Calling :  " + traceback.format_exc())
        #procedureResult = (('ProcedureCallError','Error',),)
        procedureResult = []

        procedureResult.append({0:"ProcedureCallError", 1:"Error"})         

        except Exception, err:
            logging.info("Error In DataBase Connection : " + traceback.format_exc())
           #procedureResult = (('DataBaseConnectionError','Error',),)
        procedureResult = []

        procedureResult.append({0:"DataBaseConnectionError", 1:"Error"})       
# disconnect from server
        finally:
            try:
                cursor.close()
                db.close()
            except Exception, err:
                logging.info("Error In Closing Connection : " + traceback.format_exc())
            return procedureResult

Answer 1

Two possible improvements :

• your startup code for instances may take too long, check what is the startup time and if possible use warmup requests to reduce startup times. Since increasing your idle instances seems to help, your startup time may take too long.
• A better approach would be to call external services (e.g. talk to Google Calendar) in a Task Queue outside of the user request scope. This gives you a 10-min deadline instead of the 60s deadline for user requests