Append/Prepend string to a large file in Scala

末鹿安然 提交于 2019-12-08 08:03:39

问题


I've got a large file that after some spark processing has a header inside it(but not at the top) so I have to append this string header to the first line of the file.

I tried to do this since I could not find any way to prepend directly to first line (first I read all large content, then I clear the entire file and finally overwrite the file with the header and then append all the large string)

val content: String = txtOutput.contentAsString
    txtOutput.clear()
  txtOutput.overwrite(model_header)
  txtOutput.appendLine().append(content)

But I get the following problem "Requested array size exceeds VM limit" that I can't manage since I am coding in Databricks so I cannot tweak it, so I need any way to process my data

来源:https://stackoverflow.com/questions/48666423/append-prepend-string-to-a-large-file-in-scala

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!