Store Uploaded File in MongoDB GridFS Using mgo without Saving to Memory

Question

noob Golang and Sinatra person here. I have hacked a Sinatra app to accept an uploaded file posted from an HTML form and save it to a hosted MongoDB database via GridFS. This seems to work fine. I am writing the same app in Golang using the mgo driver.

Functionally it works fine. However in my Golang code, I read the file into memory and then write the file from memory to the MongoDB using mgo. This appears much slower than my equivalent Sinatra app. I get the sense that the interaction between Rack and Sinatra does not execute this "middle" or "interim" step.

Here's a snippet of my Go code:

func uploadfilePageHandler(w http.ResponseWriter, req *http.Request) {
  // Capture multipart form file information
  file, handler, err := req.FormFile("filename")
  if err != nil {
    fmt.Println(err)
  }

  // Read the file into memory
  data, err := ioutil.ReadAll(file)
  // ... check err value for nil

  // Specify the Mongodb database
  my_db := mongo_session.DB("... database name...")

  // Create the file in the Mongodb Gridfs instance
  my_file, err := my_db.GridFS("fs").Create(unique_filename)
  // ... check err value for nil

  // Write the file to the Mongodb Gridfs instance
  n, err := my_file.Write(data)
  // ... check err value for nil

  // Close the file
  err = my_file.Close()
  // ... check err value for nil

  // Write a log type message
  fmt.Printf("%d bytes written to the Mongodb instance\n", n)

  // ... other statements redirecting to rest of user flow...
}

Question:

• Is this "interim" step needed (data, err := ioutil.ReadAll(file))?
• If so, can I execute this step more efficiently?
• Are there other accepted practices or approaches I should be considering?

Thanks...

Answer 1

No, you should not read the file entirely in memory at once, as that will break when the file is too large. The second example in the documentation for GridFS.Create avoids this problem:

file, err := db.GridFS("fs").Create("myfile.txt")
check(err)
messages, err := os.Open("/var/log/messages")
check(err)
defer messages.Close()
err = io.Copy(file, messages)
check(err)
err = file.Close()
check(err)

As for why it's slower than something else, hard to tell without diving into the details of the two approaches used.

Answer 2

Once you have the file from multipartForm, it can be saved into GridFs using below function. I tested this against huge files as well ( upto 570MB).

//....code inside the handlerfunc
for _, fileHeaders := range r.MultipartForm.File {
        for _, fileHeader := range fileHeaders {
            file, _ := fileHeader.Open()            
            if gridFile, err := db.GridFS("fs").Create(fileHeader.Filename); err != nil {
                //errorResponse(w, err, http.StatusInternalServerError)
                return
            } else {
                gridFile.SetMeta(fileMetadata)
                gridFile.SetName(fileHeader.Filename)
                if err := writeToGridFile(file, gridFile); err != nil {
                    //errorResponse(w, err, http.StatusInternalServerError)
                    return
                }

func writeToGridFile(file multipart.File, gridFile *mgo.GridFile) error {
    reader := bufio.NewReader(file)
    defer func() { file.Close() }()
    // make a buffer to keep chunks that are read
    buf := make([]byte, 1024)
    for {
        // read a chunk
        n, err := reader.Read(buf)
        if err != nil && err != io.EOF {
            return errors.New("Could not read the input file")
        }
        if n == 0 {
            break
        }
        // write a chunk
        if _, err := gridFile.Write(buf[:n]); err != nil {
            return errors.New("Could not write to GridFs for "+ gridFile.Name())
        }
    }
    gridFile.Close()
    return nil
}