Join GitHub today
We provide all the Latest Tech. News, How-To Tips, Guides, Products Reviews, Products Buying Guides & much more wise things.
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.
Sign upHave a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
commented May 7, 2018
This is unlikely to be a problem with this library, but I'm documenting my observations here for future readers and users of this library. I'm experiencing an issue where the 'server' sees an RST but I'm pretty sure it's not coming from the client(s) but something in-between. My suspicion is either the Traefik load balancer I'm using (unlikely) or the Overlay networking in Docker Swarm or some docker networking internals.Server logs: I'm able to reproduce this by:
So this is going from Mac (over WIFI) to my server(s) to msgbus running as a service in a Docker Swarm cluster.msgbus.yml : |
changed the title websocket: close 1000 (normal): read tcp 10.255.0.13:443->10.255.0.3:54633: read: connection reset by peer'May 7, 2018
changed the title websocket: close 1000 (normal): read tcp ...: read: connection reset by peer'May 7, 2018
commented May 7, 2018
I'm doing packet captures to understand this as I write this (not done yet). Expectations: To improve documentation (if at all). I'd prefer to keep this open until I get to the bottom of it; There is no known code client-side that would cause this. |
commented May 7, 2018
@dottyjones I would agree with you except that the client(s) never send a close control message. |
commented May 7, 2018
@dottyjones The weird part is that the client-side doesn't seem to get anerror from the conn.ReadJSON call. Following the code path; I expect to getthe same error client-side so I know to reconnect. I've also (by process ofelimination) ruled out any issues with gorilla/websocket (I never suspectedany in the first place). If I run msgbus without using the swarm overlaynetworking (IPVS stuff) it works fine. docker run -p 8000:8000prologic/msgbusJames Mills / prologicE: [email protected]: prologic.shortcircuit.net.au …On Mon, May 7, 2018 at 7:04 AM, dottyjones ***@***.***> wrote: The log message ***@***.*** | time='2018-05-07T00:39:20Z' level=error msg='error reading from 10.0.2.17:46442: websocket: close 1000 (normal): read tcp 10.255.0.13:443->10.255.0.3:54633: read: connection reset by peer is generated by this application code <https://github.com/prologic/msgbus/blob/0215f6f9a85580202a0caf10eafd6b04d9174c97/msgbus.go#L524> : log.Errorf('error reading from %s: %s', c.id, err) The log.Errorf function calls the Error() string <https://godoc.org/builtin#error.Error> method to convert err to the string websocket: close 1000 (normal): read tcp 10.255.0.13:443->10.255.0.3:54633: read: connection reset by peer This is the format used by websocket.CloseError <https://github.com/gorilla/websocket/blob/21ab95fa12b9bdd8fecf5fa3586aad941cc98785/conn.go#L112>. The string read tcp 10.255.0.13:443->10.255.0.3:54633: read: connection reset by peer comes from the CloseError.Text <https://github.com/gorilla/websocket/blob/21ab95fa12b9bdd8fecf5fa3586aad941cc98785/conn.go#L141-L143/21ab95fa12b9bdd8fecf5fa3586aad941cc98785/conn.go%23L112> field. The CloseError.Text field is set using data read from the peer here <https://github.com/gorilla/websocket/blob/21ab95fa12b9bdd8fecf5fa3586aad941cc98785/conn.go#L892-L907> . It does seem odd for an endpoint to send normal close (code 1000) with what looks like an error message, but that's what the client did. As far as the use of this package in your application is concerned, nothing is out of the ordinary. I suggest following up on the load balancer and the client. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#378 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOv-gPH0Udw_VNCvllNiFqL8CWs_jQjks5twFRvgaJpZM4T0RFV> . |
commented May 7, 2018
@dottyjones I'm not sure actually (how would I know?) -- I'm using Traefik I've been digging into this further; and after implementing ping/pong in both directions (was only doing this server->client) I haven't been able to repro this behavior so far. How familiar are you with Docker's IPVS load balancing and Overlay networking? If one side of the TCP session sends no data for a period of time does this affect IPVS in some way that would drop the session? FWIW there are numerous issues on this subject in the moby and swarm project issue trackers but I haven't been able to pinpoint this to a particular thing yet. |
commented May 7, 2018
@dottyjones Agreed; after many hours of testing, I can no longer repro this behavior with two-way ping/pong in both directions So I'll close this. This has to be some kind of IPVS / Overlay issue in Docker Swarm mode services (running docker run -d -p 8000:8000 prologic/msgbus directly which doesn't use IPVS or Overlay networking also doesn't exhibit this behavior) |
added a commit to prologic/msgbus that referenced this issue May 7, 2018
Add support for two-way ping/pong in both directions server<->client …
This commit was signed with a verified signature.
GPG key ID: AC4C014F1440EBD6Learn about signing commits
![Write Write](/uploads/1/2/3/9/123926501/856479705.png)
locked and limited conversation to collaborators Aug 29, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.