aws-glue

I'm getting an error when running an Athena query against a Glue table created from an RDS database: HIVE_UNKNOWN_ERROR: Unable to create input format The tables are created using a crawler. The tables show up correctly in the Glue interface: However, they do not show up in the Athena interface under the database. It says: "The selected database has no tables" I do not see this behaviour when using a database created using an S3 file. Maybe this is related to the error. Does anybody have an idea? I had the same problem. This is the answer that I have got from AWS Support: I understand that you

When running a python job in AWS Glue I get the error: Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead When running this in the beginning of the script: print '--- Before Conf --' print 'spark.yarn.driver.memory', sc._conf.get('spark.yarn.driver.memory') print 'spark.yarn.driver.cores', sc._conf.get('spark.yarn.driver.cores') print 'spark.yarn.executor.memory', sc._conf.get('spark.yarn.executor.memory') print 'spark.yarn.executor.cores', sc._conf.get('spark.yarn.executor.cores') print

I created a Development Endpoint in the AWS Glue console and now I have access to SparkContext and SQLContext in gluepyspark console. How can I access the catalog and list all databases and tables? The usual sqlContext.sql("show tables").show() does not work. What might help is the CatalogConnection Class but I have no idea in which package it is. I tried importing from awsglue.context and no success. I spend several hours trying to find some info about CatalogConnection class but haven't found anything. (Even in the aws-glue-lib repository https://github.com/awslabs/aws-glue-libs ) In my case