Inserting scraped data using php curl into MySQL

问题

I have been working on this script for the last couple of days and cannot seem to find a way to insert the data into MySQL. I am a beginner when it comes to PHP/MYSQL and have only written a couple of simple scripts before. I am able to echo out the scraped data and get no error messages, but when I check phpmyadmin the query isn't working (the results aren't being input to the database).

Here is the code that I have been working on

require ("mysqli_connect.php"); 
include('../simple_html_dom.php');

ini_set('user_agent', 
  'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3');

// get DOM from URL or file
$html = file_get_html('http://www.asos.com/Women/Jeans/Cat/pgecategory.aspx?cid=3630&via=lhn');
// find all images

foreach($html->find('#items') as $a)       
    echo $a->innertext .'<br>';

foreach($html->find('span.price') as $p)
    echo $p->innertext .',';    

$q = "INSERT INTO jeans (`image`, `price`) VALUES ('$a', '$p')";
$r = @mysqli_query ($dbc, $q) or die ("Update query failed : " . mysql_error());;  //Run the Query.

回答1:

In your sample code, $a and $p are objects, try this instead:

$a = '';
foreach($html->find('#items') as $item) {
    $a.= $item->innertext .'<br>';
}

$p = '';
foreach($html->find('span.price') as $price) {
    $p.= $price->innertext.',';    
}

Next, remove the @ from @mysqli_query, try not to use that, ever, try to catch/handle errors properly instead.

Next, please take a few minutes to research paramaterized queries and PDO, don't accept unknown input (from 3rd parties no less) and inject them right in to your sql:

$q = "INSERT INTO jeans (`image`, `price`) VALUES ('$a', '$p')";

ie: Don't do that ^

Finally, you probably want to validate the response from the get.

Hope that helps!

回答2:

You are suppressing the error by putting an @ before mysqli_query

Try changing this:

$q = "INSERT INTO jeans ('image', 'price') VALUES ('$a', '$p')";
mysqli_query ($dbc, $q) or die ("Update query failed : " . mysql_error());;  //Run the Query.

回答3:

I' think I know what is wrong.

If you just use

echo file_get_contents("http://www.asos.com/Women/Jeans/Cat/pgecategory.aspx?cid=3630");

(or just asos.com, doesn't matter), you will notice that you'll always get back:

Sorry, we can't find that page or something has gone wrong...

go back or try here:

women men home help desk

Because none of the elements you specified are on this page, you're code will return nothing.

My guess is that asos.com prevents scraping from their site.

回答4:

Remember to use mysql_real_escape_string function to avoid troubles saving your contents. Example below.

$query2 = sprintf("INSERT INTO jeans(image, price)  
                 VALUES ('%s','%s')",mysql_real_escape_string($a),
                                     mysql_real_escape_string($p))
mysql_query($query2) or die(mysql_error()."<br />".$query2);

来源：https://stackoverflow.com/questions/11143108/inserting-scraped-data-using-php-curl-into-mysql

标签

php

mysql

curl

insert

screen-scraping