|
1
|
1
|
new file mode 100644
|
|
...
|
...
|
@@ -0,0 +1,66 @@
|
|
|
0
|
+NetworkDB
|
|
|
1
|
+=========
|
|
|
2
|
+
|
|
|
3
|
+There are two databases used in libnetwork:
|
|
|
4
|
+
|
|
|
5
|
+- A persistent database that stores the network configuration requested by the user. This is typically the SwarmKit managers' raft store.
|
|
|
6
|
+- A non-persistent peer-to-peer gossip-based database that keeps track of the current runtime state. This is NetworkDB.
|
|
|
7
|
+
|
|
|
8
|
+NetworkDB is based on the [SWIM][] protocol, which is implemented by the [memberlist][] library.
|
|
|
9
|
+`memberlist` manages cluster membership (nodes can join and leave), as well as message encryption.
|
|
|
10
|
+Members of the cluster send each other ping messages from time to time, allowing the cluster to detect when a node has become unavailable.
|
|
|
11
|
+
|
|
|
12
|
+The information held by each node in NetworkDB is:
|
|
|
13
|
+
|
|
|
14
|
+- The set of nodes currently in the cluster (plus nodes that have recently left or failed).
|
|
|
15
|
+- For each peer node, the set of networks to which that node is connected.
|
|
|
16
|
+- For each of the node's currently-in-use networks, a set of named tables of key/value pairs.
|
|
|
17
|
+ Note that nodes only keep track of tables for networks to which they belong.
|
|
|
18
|
+
|
|
|
19
|
+Updates spread through the cluster from node to node, and nodes may have inconsistent views at any given time.
|
|
|
20
|
+They will eventually converge (quickly, if the network is operating well).
|
|
|
21
|
+Nodes look up information using their local networkdb instance. Queries are not sent to remote nodes.
|
|
|
22
|
+
|
|
|
23
|
+NetworkDB does not impose any structure on the tables; they are just maps from `string` keys to `[]byte` values.
|
|
|
24
|
+Other components in libnetwork use the tables for their own purposes.
|
|
|
25
|
+For example, there are tables for service discovery and load balancing,
|
|
|
26
|
+and the [overlay](overlay.md) driver uses NetworkDB to store routing information.
|
|
|
27
|
+Updates to a network's tables are only shared between nodes that are on that network.
|
|
|
28
|
+
|
|
|
29
|
+All libnetwork nodes join the gossip cluster.
|
|
|
30
|
+To do this, they need the IP address and port of at least one other member of the cluster.
|
|
|
31
|
+In the case of a SwarmKit cluster, for example, each Docker engine will use the IP addresses of the swarm managers as the initial join addresses.
|
|
|
32
|
+The `Join` method can be used to update these bootstrap IPs if they change while the system is running.
|
|
|
33
|
+
|
|
|
34
|
+When joining the cluster, the new node will initially synchronise its cluster-wide state (known nodes and networks, but not tables) with at least one other node.
|
|
|
35
|
+The state will be mostly kept up-to-date by small UDP gossip messages, but each node will also periodically perform a push-pull TCP sync with another random node.
|
|
|
36
|
+In a push-pull sync, the initiator sends all of its cluster-wide state to the target, and the target then sends all of its own state back in response.
|
|
|
37
|
+
|
|
|
38
|
+Once part of the gossip cluster, a node will also send a `NodeEventTypeJoin` message, which is a custom message defined by NetworkDB.
|
|
|
39
|
+This is not actually needed now, but keeping it is useful for backwards compatibility with nodes running previous versions.
|
|
|
40
|
+
|
|
|
41
|
+While a node is active in the cluster, it can join and leave networks.
|
|
|
42
|
+When a node wants to join a network, it will send a `NetworkEventTypeJoin` message via gossip to the whole cluster.
|
|
|
43
|
+It will also perform a bulk-sync of the network-specific state (the tables) with every other node on the network being joined.
|
|
|
44
|
+This will allow it to get all the network-specific information quickly.
|
|
|
45
|
+The tables will mostly be kept up-to-date by UDP gossip messages between the nodes on that network, but
|
|
|
46
|
+each node in the network will also periodically do a full TCP bulk sync of the tables with another random node on the same network.
|
|
|
47
|
+
|
|
|
48
|
+Note that there are two similar, but separate, gossip-and-periodic-sync mechanisms here:
|
|
|
49
|
+
|
|
|
50
|
+1. memberlist-provided gossip and push-pull sync of cluster-wide state, involving all nodes in the cluster.
|
|
|
51
|
+2. networkdb-provided gossip and bulk sync of network tables, for each network, involving just those nodes in that network.
|
|
|
52
|
+
|
|
|
53
|
+When a node wishes to leave a network, it will send a `NetworkEventTypeLeave` via gossip. It will then delete the network's table data.
|
|
|
54
|
+When a node hears that another node is leaving a network, it deletes all table entries belonging to the leaving node.
|
|
|
55
|
+Deleting an entry in this case means marking it for deletion for a while, so that we can detect and ignore any older events that may arrive about it.
|
|
|
56
|
+
|
|
|
57
|
+When a node wishes to leave the cluster, it will send a `NodeEventTypeLeave` message via gossip.
|
|
|
58
|
+Nodes receiving this will mark the node as "left".
|
|
|
59
|
+The leaving node will then send a memberlist leave message too.
|
|
|
60
|
+If we receive the memberlist leave message without first getting the `NodeEventTypeLeave` one, we mark the node as failed (for a while).
|
|
|
61
|
+Every node periodically attempts to reconnect to failed nodes, and will do a push-pull sync of cluster-wide state on success.
|
|
|
62
|
+On success we also send the node a `NodeEventTypeJoin` and then do a bulk sync of network-specific state for all networks that we have in common.
|
|
|
63
|
+
|
|
|
64
|
+[SWIM]: http://ieeexplore.ieee.org/document/1028914/
|
|
|
65
|
+[memberlist]: https://github.com/hashicorp/memberlist
|
