去，tcp太多打开文件调试

Question

这是一个简单的Go http（tcp）连接测试脚本

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    for i := 0; i < 2000; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            resp, err := http.Get(ts.URL)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%s", i, greeting)
        }(i)
    }
    wg.Wait()
}

如果我在Ubuntu中运行它，我得到：

panic: Get http://127.0.0.1:33202: dial tcp 127.0.0.1:33202: too many open files

其他帖子说，以确保Close连接，我在这里做这一切。其他人说要增加与ulimit的最大连接限制或尝试sudo sysctl -w fs.inotify.max_user_watches=100000但仍然无效。

如何在单个服务器中运行数百万个tcp连接goroutine？它仅与2,000个连接崩溃。

谢谢，

Answer 1

你的功能太多goroutine，试试这个qazxsw poi

https://github.com/leenanxi/nasync

在你的代码中

 //it has a simple usage
 nasync.Do(yourAsyncTask)

nasync lib中的默认max go goruntine为1000

Answer 2

更改ulimit以避免错误“太多打开文件”默认情况下，max ulimit对于linux为4096，对于mac为1024，u可以通过输入ulimit -n 4096将ulimit更改为4096，超过4096你需要修改limits.conf等等/ security的安全文件夹，并通过添加此行“* hard core 100000”将硬限制设置为100000

Answer 3

for i := 0; i < 2000; i++ {
    nasync.Do(func() {
        resp, err := http.Get("https://www.baidu.com")
        ...
    })
}

我通过添加req.Close = true和req.Header.Set（“Connection”，“close”）来解决这个问题。我认为这比改变ulimit更好。

来源：

HTTP/1.1 uses persistent connections by default:
A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
The solution was to inform the server that the client wants to close the connection after the transaction is complete. This can be done by setting the Connection header,
req.Header.Set("Connection", "close") or by setting the Close property to true on the http.Request:
req.Close = true After doing that, the “too many open files” issue went away as the program was no longer keeping HTTP connections open and thus not using up file descriptors.

Answer 4

Go的http包默认不指定请求超时。您应该始终在服务中包含超时。如果客户没有关闭会话怎么办？你的过程将保持旧的会话达到ulimits。一个坏的演员可以故意打开数千个会话，DOS服务器。重载服务也应该调整ulimits，但是需要超时支持。

确保指定超时：

http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

您可以通过监视进程打开的文件来验证之前和之后：

http.DefaultClient.Timeout = time.Minute * 10

Answer 5

我还必须手动设置关闭连接头以避免文件描述符问题：

lsof -p [PID_ID]

没有r.Close = true和res.Body.Close（）我达到了文件描述符限制。有了这两个，我可以根据需要开火。