Not sure sockets are being closed correctly...

[Altonymous]

I'm not sure if this is a me, web.go, or go issue... but I was doing a little light load testing and discovered that under a little load I started getting Connection reset by peer errors.

Here's a gist of my setup:
https://gist.github.com/Altonymous/7bab9ee8caa6da368883

Some things to note. the machine doing the siege is different than the machine serving the requests.

The limits setup on the machine serving the requests are as follows:
$ launchctl limit
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 1064 1064
maxfiles 10240 120000

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10240
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1064
virtual memory (kbytes, -v) unlimited

[hoisie]

Can you try using Go's http package (the code below should work).

I get similar errors when running on my mac. I can try running this benchmark on linux tomorrow.

As far as I know, unless you add a Connection: close header, Go's http server will keep the connection open. My guess is that it's hitting the file descriptor limit.

package main

import (
  "encoding/json"
  "net/http"
)

// structs
type Reading struct {
  Id   string `json:"id"`
  Name string `json:"name"`
}

func main() {
http.HandleFunc("/machines/", func(w http.ResponseWriter, r *http.Request) {
  // Setup readings
  readings := prepareReadings()

  // return readings
  w.Write([]byte(readingsToString(readings)))
})
  http.ListenAndServe(":3000", nil)
}

func readingsToString(readings []Reading) string {
  data, err := json.Marshal(readings)
  if err != nil {
    panic(err)
  }

  return string(data)
}

func prepareReadings() []Reading {
  var readings []Reading
  for i := 1; i <= 1; i++ {
    readings = append(readings, Reading{Name: "Thing"})
  }

  return readings
}

[Altonymous]

It may be hitting the limit. That's why I included my settings. Unless its another setting I'm missing. If you are getting the same error using their libraries I'm sure ill get more of the same. If it is a core library issue do you want to open an issue since you will probably be able to explain it better?

[Altonymous]

I swapped over to running it on linux, and don't seem to have the same problems. It looks like I may have found an issue related to Mac's only. :(

[Altonymous]

Oh I forgot, I also updated to 1.1beta of golang. So it could be that as well.

[hoisie]

This seems related:

https://code.google.com/p/go/issues/detail?id=66

[Altonymous]

Well I've moved all my stuff to ec2 to continue my testing and I'm running ubuntu there as well. I'm getting the error again.... :(

[Altonymous]

I used the code you generated above, installed rc3, and dropped it out onto my aws instance.

The problem still appears to be happening. I'm not sure how to determine what the cause of the issue is at this point.

Here are the limits for my process...

[api #3312 -- limits]
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             59479                59479                processes
Max open files            4999999              4999999              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       59479                59479                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

When I 'lsof | wc -l', I dont' get above 1000. Generally in the ~800-850 range.

When I 'watch --interval=2 'netstat -tuna |grep "SYN_RECV"|wc -l'', I am generally in the ~130-250 range.