Scrapy Script called via shell_exec doesn't perform

问题

I have a scrapy spider on this path:

define("SPIDER_PATH", "C:\\Users\\[USERNAME]\\test1\\test1\\spiders\\test.py");

Now I try to launch the script via php:

if (is_numeric(filter_input(INPUT_POST, "reload"))) {
    $additional = " -a check=" . filter_input(INPUT_POST, "reload");

}
echo shell_exec("scrapy runspider " . SPIDER_PATH . $additional);

But nothing happens and there is nothing echoed from shell_exec.

I've tested it on a local machine using wamp.

Can anyone help me?

The enviroment variables are set correctly (at least I can call exact the same command via Windows cmd.exe

回答1:

You cannot run scrapy through php the way you do it.

What you need is scrapyd.

https://scrapyd.readthedocs.org/en/latest/install.html

After you install it. Go your scrapy project dir: C:\Users\[USERNAME]\test1\

Create/edit a scrapy.cfg file with contents:

[settings]
default = crawler.settings

[deploy]
url = http://localhost:6800/
project = crawler

run a command

scrapyd-deploy -l

which will list your available targets:

default              http://localhost:6800/

now you need to deploy a project:

scrapyd-deploy default -p test1

More information on deploying a project: https://scrapyd.readthedocs.org/en/latest/deploy.html

When the project is deployed you can shedule a spider with curl request:

curl http://localhost:6800/schedule.json -d project=test1 -d spider=test

More on scrapyd API: https://scrapyd.readthedocs.org/en/latest/api.html

回答2:

You need to call chdir() first.

chdir("C:\\Users\\[USERNAME]\\test1\\test1\\spiders\\test.py");
echo shell_exec("scrapy runspider " . $additional);

来源：https://stackoverflow.com/questions/28852258/scrapy-script-called-via-shell-exec-doesnt-perform

标签

php

python

shell

scrapy