How to fix this error running Nutch 1.15 ERROR fetcher.Fetcher - Fetcher job did not succeed, job status:FAILED, reason: NA

问题

When I'm starting a crawl using Nutch 1.15 with this:

/usr/local/nutch/bin/crawl --i -s urls/seed.txt crawldb 5

Then it starts to run and I get this error when it tries to fetch:

2019-02-10 15:29:32,021 INFO  mapreduce.Job - Running job: job_local1267180618_0001
2019-02-10 15:29:32,145 INFO  fetcher.FetchItemQueues - Using queue mode : byHost
2019-02-10 15:29:32,145 INFO  fetcher.Fetcher - Fetcher: threads: 50
2019-02-10 15:29:32,145 INFO  fetcher.Fetcher - Fetcher: time-out divisor: 2
2019-02-10 15:29:32,149 INFO  fetcher.QueueFeeder - QueueFeeder finished: total 1 records hit by time limit : 0
2019-02-10 15:29:32,234 WARN  mapred.LocalJobRunner - job_local1267180618_0001
java.lang.Exception: java.lang.NullPointerException
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NullPointerException
        at org.apache.nutch.net.URLExemptionFilters.<init>(URLExemptionFilters.java:39)
        at org.apache.nutch.fetcher.FetcherThread.<init>(FetcherThread.java:154)
        at org.apache.nutch.fetcher.Fetcher$FetcherRun.run(Fetcher.java:222)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2019-02-10 15:29:33,023 INFO  mapreduce.Job - Job job_local1267180618_0001 running in uber mode : false
2019-02-10 15:29:33,025 INFO  mapreduce.Job -  map 0% reduce 0%
2019-02-10 15:29:33,028 INFO  mapreduce.Job - Job job_local1267180618_0001 failed with state FAILED due to: NA
2019-02-10 15:29:33,038 INFO  mapreduce.Job - Counters: 0
2019-02-10 15:29:33,039 ERROR fetcher.Fetcher - Fetcher job did not succeed, job status:FAILED, reason: NA
2019-02-10 15:29:33,039 ERROR fetcher.Fetcher - Fetcher: java.lang.RuntimeException: Fetcher job did not succeed, job status:FAILED, reason: NA
        at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:503)
        at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:543)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:517)

And I get this error in the console which is the command it runs:

Error running:
  /usr/local/nutch/bin/nutch fetch -D mapreduce.job.reduces=2 -D mapred.child.java.opts=-Xmx1000m -D mapreduce.reduce.speculative=false -D mapreduce.map.speculative=false -D mapreduce.map.output.compress=true -D fetcher.timelimit.mins=180 crawlsites/segments/20190210152929 -noParsing -threads 50

回答1:

I had to delete the nutch folder and do a new install and it worked after this.

来源：https://stackoverflow.com/questions/54617512/how-to-fix-this-error-running-nutch-1-15-error-fetcher-fetcher-fetcher-job-did

标签

nutch