pyspark

问题 I am running spark on yarn cluster. I tried to start the history server ./start-history-server.sh but got the following errors. starting org.apache.spark.deploy.history.HistoryServer, logging to /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out failed to launch org.apache.spark.deploy.history.HistoryServer: at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:47) ... 6 more full log in

问题 I have a date string from a source in the format 'Fri May 24 00:00:00 BST 2019' that I would convert to a date and store in my dataframe as '2019-05-24' using code like my example which works for me under spark 2.0 from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime df = spark.createDataFrame([("Fri May 24 00:00:00 BST 2019",)], ['date_str']) df2 = df.select('date_str', to_date(from_unixtime(unix_timestamp('date_str', 'EEE MMM dd HH:mm:ss zzz yyyy'))).alias('date')) df2

问题 I have a date string from a source in the format 'Fri May 24 00:00:00 BST 2019' that I would convert to a date and store in my dataframe as '2019-05-24' using code like my example which works for me under spark 2.0 from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime df = spark.createDataFrame([("Fri May 24 00:00:00 BST 2019",)], ['date_str']) df2 = df.select('date_str', to_date(from_unixtime(unix_timestamp('date_str', 'EEE MMM dd HH:mm:ss zzz yyyy'))).alias('date')) df2

问题 I have a date string from a source in the format 'Fri May 24 00:00:00 BST 2019' that I would convert to a date and store in my dataframe as '2019-05-24' using code like my example which works for me under spark 2.0 from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime df = spark.createDataFrame([("Fri May 24 00:00:00 BST 2019",)], ['date_str']) df2 = df.select('date_str', to_date(from_unixtime(unix_timestamp('date_str', 'EEE MMM dd HH:mm:ss zzz yyyy'))).alias('date')) df2

问题 I have a date string from a source in the format 'Fri May 24 00:00:00 BST 2019' that I would convert to a date and store in my dataframe as '2019-05-24' using code like my example which works for me under spark 2.0 from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime df = spark.createDataFrame([("Fri May 24 00:00:00 BST 2019",)], ['date_str']) df2 = df.select('date_str', to_date(from_unixtime(unix_timestamp('date_str', 'EEE MMM dd HH:mm:ss zzz yyyy'))).alias('date')) df2

问题 I have been running a workflow on some 3 Million records x 15 columns all strings on my 4 cores 16GB machine using pyspark 1.5 in local mode. I have noticed that if I run the same workflow again without first restarting spark, memory runs out and I get Out of Memory Exceptions. Since all my caches sum up to about 1 GB I thought that the problem lies in the garbage collection. I was able to run the python garbage collector manually by calling: import gc collected = gc.collect() print "Garbage

问题 I have been running a workflow on some 3 Million records x 15 columns all strings on my 4 cores 16GB machine using pyspark 1.5 in local mode. I have noticed that if I run the same workflow again without first restarting spark, memory runs out and I get Out of Memory Exceptions. Since all my caches sum up to about 1 GB I thought that the problem lies in the garbage collection. I was able to run the python garbage collector manually by calling: import gc collected = gc.collect() print "Garbage

问题 I am looking for a way to select columns of my dataframe in pyspark. For the first row, I know I can use df.first() but not sure about columns given that they do not have column names. I have 5 columns and want to loop through each one of them. +--+---+---+---+---+---+---+ |_1| _2| _3| _4| _5| _6| _7| +--+---+---+---+---+---+---+ |1 |0.0|0.0|0.0|1.0|0.0|0.0| |2 |1.0|0.0|0.0|0.0|0.0|0.0| |3 |0.0|0.0|1.0|0.0|0.0|0.0| 回答1: Try something like this: df.select([c for c in df.columns if c in ['_2','

问题 I have a kafka producer sending large amounts of data in the format of { '1000': { '3': { 'seq': '1', 'state': '2', 'CMD': 'XOR' } }, '1001': { '5': { 'seq': '2', 'state': '2', 'CMD': 'OR' } }, '1003': { '5': { 'seq': '3', 'state': '4', 'CMD': 'XOR' } } } .... the data I want is in the final loop: {'seq': '1', 'state': '2', 'CMD': 'XOR'} and the keys in the loops above ('1000' and '3') are variable. Please note that the above values are only for example. the original dataset is huge with lots

问题 I have a kafka producer sending large amounts of data in the format of { '1000': { '3': { 'seq': '1', 'state': '2', 'CMD': 'XOR' } }, '1001': { '5': { 'seq': '2', 'state': '2', 'CMD': 'OR' } }, '1003': { '5': { 'seq': '3', 'state': '4', 'CMD': 'XOR' } } } .... the data I want is in the final loop: {'seq': '1', 'state': '2', 'CMD': 'XOR'} and the keys in the loops above ('1000' and '3') are variable. Please note that the above values are only for example. the original dataset is huge with lots