Papaparse doesn't work for more than 1 MB for me, react

问题

Edit: Trying to use papaparse chunk & custom webworker multithreading now. Edit2: Sorry I could not figure this out, but I am going to render this list of cities through a web scraper on https://www.freemaptools.com/find-cities-and-towns-inside-radius.htm anyway I decided instead of from a csv with papaparse...

I am trying to render <WeatherCitySky /> for each city from a csv at a dropbox link parsed by papaparse. Inside componentDidMount the first cors-anywhere/dropbox link, commented out, is a 1.5 MB csv of eastern US cities... won't work. Wanted to do at least all of US cities at 5MB, but all I can get to work is the second corsanywhere/dropbox link at about 350bytes

Go to src>UIConainers>Map>CitiesMap.js

https://codesandbox.io/s/zen-dijkstra-1c31n?fontsize=14

the CitiesMap.js is found by bottom globe icon (after pressing the inbox icon if you're starting on the purple screen), then top left city animation

class CitiesMap extends React.Component {
  _isMounted = false;
  constructor(props) {
    super(props);
    this.updateData = this.updateData.bind(this);
    this.state = { cities: [] };
  }
  componentDidMount() {
    this._isMounted = true;
    Papa.parse(
      "https://dl.dropboxusercontent.com/s/k81s5enbamijuke/worldcitiespop_northamerica_nolonglat_few.csv",
      // this one doesn't work"https://dl.dropboxusercontent.com/s/wy9vjxczbjm796y/worldcities_usa_few.csv",
      {
        download: true,
        worker: true,
        header: true,
        skipEmptyLines: true,
        step: this.updateData,
        complete: function(results) {
        }
      }
    );
  }
  updateData(results) {
    if (this._isMounted) {
      this.setState(prevState => ({
        cities: [...prevState.cities, results.data.City]
      }));console.log(this.state.cities)
    }
  }
  componentWillUnmount() {
    this._isMounted = false;
  }


  render(props) {
    const filteredCities = this.state.cities.filter(cities => {
      return (
        cities.toUpperCase().indexOf(this.props.search.toUpperCase()) !==
        -1
      );
    });
    return (
      <div>
        <div className="Cities">
          {filteredCities.map(City => {
            return (
              <div>
                <WeatherCitySkyMap />
              </div>

I wouldn't recommend reading this question I asked for this papaparse application that user_domino solved some problems, but this problem is different evidenced by it working, but only on a small file of only 350 bytes

回答1:

Try switching step to chunk (it will process a chunk at a time vs one record at a time). You should only need to change that parameter and leave the function and everything else the same.

Another idea is setting chunkSize on Papa. So something like:

Papa.parse([your_url],
  {
    download: true,
    worker: true,
    header: true,
    skipEmptyLines: true,
    chunk: this.updateData,
    chunkSize: 1024 * 1024 * 10, // 10MB
    complete: function(results) {
    }
  }
);

来源：https://stackoverflow.com/questions/57879616/papaparse-doesnt-work-for-more-than-1-mb-for-me-react

标签

reactjs

papaparse