Is it possible to access my django models inside of a Scrapy pipeline, so that I can save my scraped data straight to my model?
I\'ve seen this, but I don\'t really
The opposite solution (setup scrapy in a django management command):
# -*- coding: utf-8 -*-
# myapp/management/commands/scrapy.py
from __future__ import absolute_import
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def run_from_argv(self, argv):
self._argv = argv
self.execute()
def handle(self, *args, **options):
from scrapy.cmdline import execute
execute(self._argv[1:])
and in django's settings.py:
import os
os.environ['SCRAPY_SETTINGS_MODULE'] = 'scrapy_project.settings'
Then instead of scrapy foo
run ./manage.py scrapy foo
.
UPD: fixed the code to bypass django's options parsing.