Scrapy: How to export Json from script

问题

I created a web crawler with scrapy, but I've a problem with phone number because it is into a script. The script is:

<script data-n-head="true" type="application/ld+json">{"@context":"http://schema.org","@type":"LocalBusiness","name":"Clínica Dental Reina Victoria 23","description":".TU CLÍNICA DENTAL DE REFERENCIA EN MADRID","logo":"https://estaticos.qdq.com/CMS/directory/logos/c/l/clinica-dental-reina-victoria.png","image":"https://estaticos.qdq.com/coverphotos/098/535/ed1c5ffcf38241f8b83a1808af51a615.jpg","url":"https://www.clinicadental-reinavictoria.es/","hasMap":"https://www.google.com/maps/search/?api=1&query=40.4469174,-3.7087934","telephone":"+34915340309","address":{"@type":"PostalAddress","streetAddress":"Av. Reina Victoria 23","addressLocality":"MADRID","addressRegion":"Madrid","postalCode":"28003"}}</script>

This script change in diferents page, but only change the phone number

I extract the script with Xpath

data = response.xpath('/html/head/script[3]').extract()
        decoded = json.loads(data.telephone("utf-8"))
        ml_item['datos'] = decoded['telephone']

I think that I need the custom pipelines for extract the phone number

In pipelines.py i added the jsonWriter line

ITEM_PIPELINES = {'mercado.pipelines.MercadoPipeline': 500,
                    'mercado.pipelines.MercadoImagenesPipeline': 600,
                    'mercado.pipelines.JsonWriterPipeline': 800, }

But I need added some code in pipelines.py for define JsonWriterPipeline. The console return this error:

raise NameError("Module '%s' doesn't define any object named '%s'" % (module, name))
NameError: Module 'mercado.pipelines' doesn't define any object named 'JsonWriterPipeline'

I save all numbers in a CSV file with other information like Name, Web, etc...

回答1:

it's simple if you already crawled the contents inside the script tag

import re

script = '{"@context":"http://schema.org","@type":"LocalBusiness","name":"Clínica Dental Reina Victoria 23","description":".TU CLÍNICA DENTAL DE REFERENCIA EN MADRID","logo":"https://estaticos.qdq.com/CMS/directory/logos/c/l/clinica-dental-reina-victoria.png","image":"https://estaticos.qdq.com/coverphotos/098/535/ed1c5ffcf38241f8b83a1808af51a615.jpg","url":"https://www.clinicadental-reinavictoria.es/","hasMap":"https://www.google.com/maps/search/?api=1&query=40.4469174,-3.7087934","telephone":"+34915340309","address":{"@type":"PostalAddress","streetAddress":"Av. Reina Victoria 23","addressLocality":"MADRID","addressRegion":"Madrid","postalCode":"28003"}}'

phone_number = re.search(r'"telephone":"(.*?)","address"', script).group(1)

print(phone_number)

回答2:

The easiest and quick option will be and I also prefer this one.

import json

json.loads(response.css('script:contains("LocalBusiness") ::text').re_first('(.*)'))

来源：https://stackoverflow.com/questions/58589037/scrapy-how-to-export-json-from-script

标签

python

json

scrapy