MarshalJSON without having all objects in memory at once

Question

I want to use json.Encoder to encode a large stream of data without loading all of it into memory at once.

// I want to marshal this
t := struct         


        

Answer 1

Unfortunately the encoding/json package doesn't have a way to do this yet. What you're doing now (manually) is the best way to do it, without modifying the built-in package.

If you were to patch encoding/json, you could modify the reflectValueQuoted function in encoding/json/encode.go

You would want to focus on the Array case (Slice has a fallthrough):

// Inside switch:
case reflect.Array:
    e.WriteByte('[')
    n := v.Len()
    for i := 0; i < n; i++ {
        if i > 0 {
            e.WriteByte(',')
        }
        e.reflectValue(v.Index(i))
    }
    e.WriteByte(']')

I'm assuming you'd want to treat channel the same way. It would look something like this:

// Inside switch:
case reflect.Chan:
    e.WriteByte('[')
    i := 0
    for {
        x, ok := v.Recv()
        if !ok {
            break
        }
        if i > 0 {
            e.WriteByte(',')
        }
        e.reflectValue(x)
        i++
    }
    e.WriteByte(']')

I haven't done much with channels in reflect, so the above may need other checks.

If you do end up going this route, you could always submit a patch.

Answer 2

You can unpack the channel in the MarshalJSON method in your struct like this:

type S struct {
    Foo string
    Bar chan string 
}

func (s *S) MarshalJSON() (b []byte, err error) {
    b, err := json.Marshal(s.Foo)

    if err != nil { return nil, err }

    for x := range s.Bar {
        tmp, err := json.Marshal(x)

        if err != nil { return nil, err }

        b = append(b, tmp...)
    }

    return
}