Was load testing a puppet forge implementation today and was hitting some odd errors when I ran each load test immediately after the previous one finished. The errors would not happen if I waited a few (maybe 5 minutes) between tests.
This was odd behaviour but digging through google with the client error of Failed to open TCP connection to (Cannot assign request, I landed on this stack overflow entry (https://stackoverflow.com/a/31877033/14784297) which implied the error mostly came from a lack of ephemeral ports (correct) and more interestingly that after a TCP connection is closed, the connection is still up in a TIME_WAIT state for about 2 minutes:
The reason of this problem is that for opening a TCP connection, the operating system allocates an ephemeral port (for the source port). It binds the socket to the allocated port. After the TCP connection is closed, the connection is left in TIME_WAIT state, typically for 2 minutes, due to historical reasons
Count the number of open connections:
netstat -naptu | grep -c TIME_WAIT
No comments:
Post a Comment