Write each row of a spark dataframe as a separate file

Question

I have Spark Dataframe with a single column, where each row is a long string (actually an xml file). I want to go through the DataFrame and save a string from each row as a

Answer 1

I would do it this way in Java and Hadoop FileSystem API. You can write similar code using Python.

List strings = Arrays.asList("file1", "file2", "file3");
JavaRDD stringrdd = new JavaSparkContext().parallelize(strings);
stringrdd.collect().foreach(x -> {
    Path outputPath = new Path(x);
    Configuration conf = getConf();
    FileSystem fs = FileSystem.get(conf);
    OutputStream os = fs.create(outputPath);
});