Writing a loop to extract Comtrade data and export them into multiple csv files

问题

I am writing a loop to extract UN Comtrade export data for each country (say country i) and will export them into individual CSV files by country for later use. So I used the "get.Comtrade" function (written by Stefan a.) as a starting point and looped it through each unique 3-digit country code in the Comtrade data. Plus, I am only interested in obtaining "commodity" "export" data (from country i to any other countries) and will have no use for all other information contained in the "list" extracted using get.Comtrade function, so I will only extract "year", "reporter", "partner", and "trade value" data for each country and assemble them into a CSV table.

To illustrate with an example, I begin with USA (country code=842) in year 2006, so I wrote

d <- get.Comtrade(r="842", p="all", ps="2006", type="C", rg="2", freq="A")

which gives me a "list" --> d that contains 2 items "validation" and "data" in which the latter (data) contains 35 variables and is the data frame of interest. However, I only need the following information (as index by their column number): [2]Year, [10]rtTitle=Reporter, [13]ptTitle=Partner, [32]Trade.Value..US..
so I entered

df <- cbind(d$data[2], d$data[10], d$data[13], d$data[32])

to make a smaller data frame and adjust "trade value" by million USD.

df$million <- as.numeric(as.character(df[,4]))/1000000

finally, I export it into a CSV file to the designated working directory

write.table(df, "~/working directory/data.csv", na = "NA", row.names = FALSE, col.names = TRUE, sep=",")

This is straightforward and indeed gives me a four-column CSV table. Building on this simple format, I went on to write a loop to obtain a CSV output for each country:

    for (i in 1:length(country_list)){  
  d <- get.Comtrade(r="i", p="all", ps="2006", type="C", rg="2", freq="A")
  df <-  cbind(d$data[2], d$data[10], d$data[13], d$data[32])
  df$million <- as.numeric(as.character(df[,4]))/1000000
  myfile <- file.path("~/working directory", paste0("_", i, ".csv"))
  write.table(df, file=myfile, na = "NA", row.names = FALSE, col.names = FALSE, quote=FALSE, append=FALSE, sep="")
  }

But it returns an error message saying that

Error in [.data.frame(d$data, 2) : undefined columns selected

indicating an error occurs on the second line of the loop, that is, the columns in d$data are undefined. So I checked with previous questions on this board pertaining to column name question in list and/or data frame format, trying different methods for literally three days but still not being able to figure out a solution for this problem.

I understand this is supposed to be a very simple loop but I just get bogged down.

What's you guys' take on this? It will be really appreciated if anyone could point out the errors in the "undefined column" issue since this pops up all the time in a number of earlier threads I've checked so far.

Thank you.

But the way,

In order to execute the code above, one will need Stefan A.'s get.Comtrade code:

get.Comtrade <- function(url="http://comtrade.un.org/api/get?"
                         ,maxrec=50000
                         ,type="C"
                         ,freq="A"
                         ,px="HS"
                         ,ps="now"
                         ,r
                         ,p
                         ,rg="all"
                         ,cc="TOTAL"
                         ,fmt="json"
)
{
  string<- paste(url
                 ,"max=",maxrec,"&" #maximum no. of records returned
                 ,"type=",type,"&" #type of trade (c=commodities)
                 ,"freq=",freq,"&" #frequency
                 ,"px=",px,"&" #classification
                 ,"ps=",ps,"&" #time period
                 ,"r=",r,"&" #reporting area
                 ,"p=",p,"&" #partner country
                 ,"rg=",rg,"&" #trade flow
                 ,"cc=",cc,"&" #classification code
                 ,"fmt=",fmt        #Format
                 ,sep = ""
  )

  if(fmt == "csv") {
    raw.data<- read.csv(string,header=TRUE)
    return(list(validation=NULL, data=raw.data))
  } else {
    if(fmt == "json" ) {
      raw.data<- fromJSON(file=string)
      data<- raw.data$dataset
      validation<- unlist(raw.data$validation, recursive=TRUE)
      ndata<- NULL
      if(length(data)> 0) {
        var.names<- names(data[[1]])
        data<- as.data.frame(t( sapply(data,rbind)))
        ndata<- NULL
        for(i in 1:ncol(data)){
          data[sapply(data[,i],is.null),i]<- NA
          ndata<- cbind(ndata, unlist(data[,i]))
        }
        ndata<- as.data.frame(ndata)
        colnames(ndata)<- var.names
      }
      return(list(validation=validation,data =ndata))
    }
  }
}

and get a list of reporters from

library(rjson)
string <- "http://comtrade.un.org/data/cache/partnerAreas.json"
reporters <- fromJSON(file=string)
reporters <- as.data.frame(t(sapply(reporters$results,rbind)))

来源：https://stackoverflow.com/questions/40661460/writing-a-loop-to-extract-comtrade-data-and-export-them-into-multiple-csv-files

标签

csv

for-loop

dataframe

undefined