1. 技术文章
  2. Scrapy - logging to file and stdout simultaneously, with spider names

Scrapy - logging to file and stdout simultaneously, with spider names

由 柔情痞子 提交于 2019-12-02 15:58:29

You want to use the ScrapyFileLogObserver.

import logging
from scrapy.log import ScrapyFileLogObserver

logfile = open('testlog.log', 'w')
log_observer = ScrapyFileLogObserver(logfile, level=logging.DEBUG)
log_observer.start()

I'm glad you asked this question, I've been wanting to do this myself.

It is very easy to redirect output using: scrapy some-scrapy's-args 2>&1 | tee -a logname

This way, all what scrapy ouputs into stdout and stderr, will be redirected to a logname file and also, prited to the screen.

For all those folks who came here before reading the current documentation version:

import logging
from scrapy.utils.log import configure_logging

configure_logging(install_root_handler=False)
logging.basicConfig(
    filename='log.txt',
    filemode = 'a',
    format='%(levelname)s: %(message)s',
    level=logging.DEBUG
)

I know this is old but it was a really helpful post since the class still isn't properly documented in the Scrapy docs. Also, we can skip importing logging and use scrapy logs directly. Thanks All!

from scrapy import log

logfile = open('testlog.log', 'a')
log_observer = log.ScrapyFileLogObserver(logfile, level=log.DEBUG)
log_observer.start()

ScrapyFileLogObserver is no longer supported. You may use standard python logging module. 

import logging
logging.getLogger().addHandler(logging.StreamHandler())
Shi XiuFeng

As the Scrapy Official Doc said:

Scrapy uses Python’s builtin logging system for event logging.

So you can config your logger just as a normal Python script.

First, you have to import the logging module:

import logging

You can add this line to your spider:

logging.getLogger().addHandler(logging.StreamHandler())

It adds a stream handler to log to console.

After that, you have to config logging file path.

Add a dict named custom_settings which consists of your spider-specified settings:

custom_settings = {
     'LOG_FILE': 'my_log.log',
     'LOG_LEVEL': 'INFO',
     ... # you can add more settings
 }

The whole class looks like:

import logging

class AbcSpider(scrapy.Spider):
    name: str = 'abc_spider'
    start_urls = ['you_url']
    custom_settings = {
         'LOG_FILE': 'my_log.log',
         'LOG_LEVEL': 'INFO',
         ... # you can add more settings
     }
     logging.getLogger().addHandler(logging.StreamHandler())

     def parse(self, response):
        pass
标签
python
web-crawler
scrapy
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!