GitList

Browse code

Fix issue where node promotion could fail

If a node is promoted right after another node is demoted, there exists
the possibility of a race, by which the newly promoted manager attempts
to connect to the newly demoted manager for its initial Raft membership.
This connection fails, and the whole swarm Node object exits.

At this point, the daemon nodeRunner sees the exit and restarts the
Node.

However, if the address of the no-longer-manager is recorded in the
nodeRunner's config.joinAddr, the Node again attempts to connect to the
no-longer-manager, and crashes again. This repeats. The solution is to
remove the node entirely and rejoin the Swarm as a new node.

This change erases config.joinAddr from the restart of the nodeRunner,
if the node has previously become Ready. The node becoming Ready
indicates that at some point, it did successfully join the cluster, in
some fashion. If it has successfully joined the cluster, then Swarm has
its own persistent record of known manager addresses. If no joinAddr is
provided, then Swarm will choose from its persisted list of managers to
join, and will join a functioning manager.

Signed-off-by: Drew Erny <derny@mirantis.com>

Drew Erny authored on 2024/05/22 22:48:03
Showing 1 changed files

daemon/cluster/noderunner.go

History View file @ 16e5c41

@@ -282,6 +282,19 @@ func (n *nodeRunner) handleNodeExit(node *swarmnode.Node) {
                      	close(n.done)
                      	select {
                      	case <-n.ready:
                     +		// there is a case where a node can be promoted to manager while
                     +		// another node is leaving the cluster. the node being promoted, by
                     +		// random chance, picks the IP of the node being demoted as the one it
                     +		// tries to connect to. in this case, the promotion will fail, and the
                     +		// whole swarm Node object packs it in.
                     +		//
                     +		// when the Node object is relaunched by this code, because it has
                     +		// joinAddr in the config, it attempts again to connect to the same
                     +		// no-longer-manager node, and crashes again. this continues forever.
                     +		//
                     +		// to avoid this case, in this block, we remove JoinAddr from the
                     +		// config.
                     +		n.config.joinAddr = ""
                      		n.enableReconnectWatcher()
                      	default:
                      		if n.repeatedRun {