问题
when i run "yarn logs -applicationId application_1438080928000_6932", appear this exception:
Exception in thread "main" java.io.IOException: Not a valid BCFile.
at org.apache.hadoop.io.file.tfile.BCFile$Magic.readAndVerify(BCFile.java:927)
at org.apache.hadoop.io.file.tfile.BCFile$Reader.<init>(BCFile.java:628)
at org.apache.hadoop.io.file.tfile.TFile$Reader.<init>(TFile.java:804)
at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.<init>(AggregatedLogFormat.java:358)
at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:122)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:137)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:199)
can anyone give some help?
回答1:
I get this error if I try to read the logs directly after killing the job. I would say give it a min at tops then attempt to read the logs again.
回答2:
In HDP environment if anybody gets this error this while parsing the container logs manually using some TFile reader, this answer might help...
In HDP, yarn.log-aggregation.file-formats's default value is IndexedFormat,TFile
So it will aggregate the logs in indexed file format only in /app-logs/<user>/logs-ifile
Remove the IndexedFile value for the yarn.log-aggregation.file-formats property. Only give TFile format.
The logs will get aggregated in /app-logs/<user>/logs
来源：https://stackoverflow.com/questions/34247528/java-io-ioexception-not-a-valid-bcfile