How do I create a CSV file from all of these elements?

问题

I am trying to get the text from both of these sections and turn it into a CSV list from puppeteer:

item number: (Item 1055688)

price: ( $16.59)

here's what I tried but it doesn't seem to work to find the SKU for example:

let elements = await.self.page.$$('div[class="row item-row"]');
for (let element of elements) {
    let sku = await element.$eval(('div[class="body-copy custom-body- 
copy"]'), node => node.innerText.trim());
}

Here is the code I am trying to extract the data from:

<div class="col-xl-3 col-lg-3 col-md-6 col-sm-8 col-xs-6">
<div class="product_desc_txt">

    <a href=" https://www.costcobusinessdelivery.com/.product.1055688.html 
" class="body-copy-link">
        Pringles Snack Pack Potato Crisps, Original, 0.67 oz, 60 ct
    </a>
    <div class="body-copy custom-body-copy">
       Item&nbsp;1055688
    </div>

    <div class="margin_tp_10"></div>

    <div class="body-copy hidden visible-md visible-sm visible-xs 
visible-lg">

        <span  data-wishlist-linkfee="false" > $16.59</span>

    </div>

</div>
</div>
<div class="col-xl-2 col-lg-2 body-copy text-right hidden visible-xl ">

<span  data-wishlist-linkfee="false" > $16.59</span>


</div>

Here is my code so far:

const puppeteer = require("puppeteer-extra")

const pluginStealth = require("puppeteer-extra-plugin-stealth")
puppeteer.use(pluginStealth())

puppeteer.launch({ headless: false }).then(async browser => {
const page = await browser.newPage()
await page.setViewport({ width: 1920, height: 1080 })
await page.goto("https://www.costcobusinessdelivery.com")
await page.waitFor(5000);
await page.waitForSelector("#header_sign_in");
await page.click("#header_sign_in");
await page.waitForSelector("#logonId");

await page.type('#logonId', 'username', {delay: 20});
await page.type('#logonPassword_id', 'password', {delay: 20});
await page.type('#deliveryZipCode', 'zipcode', {delay: 20});
await page.click('#sign_in_button');

await page.waitForSelector('body > div.bd-specific > div > div > div > div > div > ul > li.set-zip-code.left-lg.colo-md-5.zipped > ul > li:nth-child(1) > a');
await page.click('body > div.bd-specific > div > div > div > div > div > ul > li.set-zip-code.left-lg.colo-md-5.zipped > ul > li:nth-child(1) > a');
await page.waitForSelector('#tiles-body-attribute > div:nth-child(2) > div.myaccount-lists > div > div:nth-child(2) > div > span > h5 > a');
await page.click('#tiles-body-attribute > div:nth-child(2) > div.myaccount-lists > div > div:nth-child(2) > div > span > h5 > a');

I am new to puppeteer so I am not sure if I am doing this right at all, any help or guidance would be appreciated. Thank you!

回答1:

I suppose the structure of your page is similar to this one

In this case you could use the following code:

// Find product descriptions
const csv = await page.$$eval('.product_desc_txt', function(products){

    // Iterate over product descriptions
    let csvLines = products.map(function(product){

        // Inside of each product find product SKU and its price
        let productId = product.querySelector(".custom-body-copy").innerText.trim();
        let productPrice = product.querySelector("span[data-wishlist-linkfee]").innerText.trim();

        // Fomrat them as a csv line
        return `${productId};${productPrice}`
    })

    // Join all lines into one file
    return csvLines.join("\n");

});

This code with the linked HTML structure produces this:

Item 1055688;$16.59
Item 1055688;$16.59
Item 1055688;$16.59
Item 1055688;$16.59

A more compact way to rewrite that with arrow functions would be the following (although I don't think it's very readable)

const csv = await page.$$eval('.product_desc_txt', products => products.map(product => product.querySelector(".custom-body-copy").innerText.trim() + ";" + product.querySelector("span[data-wishlist-linkfee]").innerText.trim()).join("\n"));

来源：https://stackoverflow.com/questions/58192028/how-do-i-create-a-csv-file-from-all-of-these-elements

标签

javascript

node.js

web-scraping

Puppeteer