Go, tcp too many open files debug

Here's a straightforward Go http (tcp) connection test script

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    for i := 0; i < 2000; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            resp, err := http.Get(ts.URL)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%s", i, greeting)
        }(i)
    }
    wg.Wait()
}

And If I run this in Ubuntu I get:

panic: Get http://127.0.0.1:33202: dial tcp 127.0.0.1:33202: too many open files

Other posts say to make sure Close the connection, which I am doing it all here. And others say to increase the limit of maximum connection with ulimit or try sudo sysctl -w fs.inotify.max_user_watches=100000 but still does not work.

How do I run millions of tcp connection goroutines in a single server? It crashes only with 2,000 connections.

Thanks,

I think you need to change your max file descriptors. I have run into the same problem on one of my development VMs before and needed to change the file descriptors max, not anything with inotify settings.

FWIW, your program runs fine on my VM.

·> ulimit -n
120000

But after I run

·> ulimit -n 500
·> ulimit -n
500

I get:

panic: Get http://127.0.0.1:51227: dial tcp 127.0.0.1:51227: socket: too many open files

If you want to run millions of go routines that open/read/close a socket, well you better up your ulimit, or open/read/close the socket and pass the value read in to the go-routine, but I would use a buffered channel to control how many file descriptors you want to be able to open.

const (
    // this is where you can specify how many maxFileDescriptors
    // you want to allow open
    maxFileDescriptors = 100
)

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    maxChan := make(chan bool, maxFileDescriptors)
    for i := 0; i < 1000; i++ {
        maxChan <- true
        go func(url string, i int, maxChan chan bool, wg *sync.WaitGroup) {
            wg.Add(1)
            defer wg.Done()
            defer func(maxChan chan bool) { <-maxChan }(maxChan)
            resp, err := http.Get(url)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            if err != nil {
                panic(err)
            }
            err = resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%d: %s", i, string(greeting))
        }(ts.URL, i, maxChan, &wg)
    }
    wg.Wait()
}

Go’s http package doesn’t specify request timeouts by default. You should always include a timeout in your service. What if a client doesn't close their session? Your process will keep alive old sessions hitting ulimits. A bad actor could intentionally open thousands of sessions, DOSing your server. Heavy load services should adjust ulimits as well but timeouts for backstop.

Ensure you specify a timeout:

http.DefaultClient.Timeout = time.Minute * 10

You can validate before and after by monitoring files opened by your process:

lsof -p [PID_ID]

too may goruntine in your function,try this https://github.com/leenanxi/nasync

 //it has a simple usage
 nasync.Do(yourAsyncTask)

in your code

for i := 0; i < 2000; i++ {
    nasync.Do(func() {
        resp, err := http.Get("https://www.baidu.com")
        ...
    })
}

the default max go goruntine in nasync lib is 1000

change the ulimit to avoid the error "too many open files" by default max ulimit is 4096 for linux and 1024 for mac, u can change ulimit to 4096 by typing ulimit -n 4096 for beyond 4096 you need to modify limits.conf in etc/security folder for linux and set hard limit to 100000 by adding this line "* hard core 100000"

HTTP/1.1 uses persistent connections by default:
A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
The solution was to inform the server that the client wants to close the connection after the transaction is complete. This can be done by setting the Connection header,
req.Header.Set("Connection", "close") or by setting the Close property to true on the http.Request:
req.Close = true After doing that, the “too many open files” issue went away as the program was no longer keeping HTTP connections open and thus not using up file descriptors.

I solved this by adding req.Close = true and req.Header.Set("Connection", "close"). I think it's better than changing ulimit.

source: http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

I had to also to manually set the close connection header to avoid the file descriptor issue:

r, _ := http.NewRequest(http.MethodDelete, url, nil)
r.Close = true
res, err := c.Do(r)
res.Body.Close();

Without r.Close = true and res.Body.Close() I hit the file descriptor limit. With both I could fire off as many as I needed.

来源：https://stackoverflow.com/questions/32325343/go-tcp-too-many-open-files-debug

标签

sockets

tcp

goroutine