Spark, how to print the query?

问题

I'm using pyspark

    df = self.sqlContext.read.option(
        "es.resource", indexes
    ).format("org.elasticsearch.spark.sql").load()


    df = df.filter(
        df.data.timestamp >= self.period_start
    )

I'd like to see sql query version of df if possible.

something like print(df.query) to see something like select * from my-indexes where data.timestamp > self.period_start

回答1:

You can check out this piece of documentation for pyspark.sql.DataFrame.explain . explain prints the (logical and physical) plan to the console for debugging purpose.

来源：https://stackoverflow.com/questions/56033068/spark-how-to-print-the-query