Fix an issue where sockets could end up being closed twice if they were in the poller, the connector is stopped and then closeSocket() was called.