Scattercast is implemented as a replacement for UDP multicast clustering in environments that do not support multicast.
|
Cluster-wide communication mode
Choosing a cluster communication mode is a cluster-wide decision. It should be made before installation begins. The cluster cannot correctly form when the cluster communication mode is inconsistent; two independent, primary clusters will form. |
How does it work?
Normally Savanna will send UDP datagrams to a well-known multicast group address / port combination to maintain cluster membership. Message transfer happens on separate multicast group address / port combinations that are allocated at runtime from a pool.
Scattercast replaces each multicast UDP datagram with multiple unicast UDP datagrams, one to each involved node. Each
node has a unique unicast address / port combination ("scattercast endpoint") used for cluster membership. A separate
unicast address is used for each message group. Another separate unicast address is used for co-ordinating state
transfer between members. This is created from membership IP address, membership port + state_distribution_port_offset
(default 100. Configured in {$NODE_HOME}/config/savanna/cluster.properties).
All nodes must a priori know the endpoint
addresses of all other nodes in the same cluster. To achieve this, a configuration file scattercast.endpoints is stored
on each node. This file is created during install and is subsequently managed using the
Scattercast Management commands.
Separate endpoints for message transfer are allocated at
runtime based on the membership address, and a port chosen from the range first_port to last_port (defaults to 46700,
46800). This is configured in {$NODE_HOME}/config/savanna/cluster.properties).
|
|
Scattercast uses separate groups for message transfer and membership. Ports used for the membership group in scattercast endpoints must not overlap with the port range used for message groups. UDP broadcast addresses are not supported in scattercast endpoints. These will not be rejected by the installer, scattercast commands, or the recovery tool, but must be avoided by users. |
Scattercast endpoint configuration is versioned, and hashed to ensure consistency. The cluster will prefer the newest version if multiple versions are detected when a node tries to join the cluster. Nodes that detect an out-of-date local version will shut down immediately. Nodes that detect a hash mismatch will also shut down immediately, as this indicates corrupt or manually modified contents.
All clustering configuration is stored per node, and must be updated on all nodes to remain in sync. It is expected that this should not be changed often, if at all.
What’s the downside?
Scattercast requires sending additional traffic. For an N-node cluster, scattercast will generate about (N-1) times as many datagrams as the equivalent multicast cluster. That is, there is no penalty for a 2-node cluster; a 3-node cluster will generate about 2x traffic; a 4-node cluster will generate 3x traffic; and so on. At high loads you may run out of network bandwidth sooner; also, there is some CPU overhead involved in sending the extra datagrams.
Scattercast cannot automatically discover nodes; you must explicitly provide endpoint information for all nodes in the cluster. To add nodes, remove nodes, or update nodes at runtime, online management commands should be used.
|
|
Manual editing of the configuration file scattercast.endpoints is not supported. Manual editing will cause edited nodes to fail to
boot.
|
Initial setup
A cluster must be seeded with an initial scattercast endpoints file containing valid mappings for all initial nodes. Without a valid scattercast endpoints file a node is unable to boot in scattercast comms mode. This initial endpoints set may be generated by the Rhino installer. When choosing to install in scattercast mode, the installer script must be provided with an initial endpoints set. Details can be found in Unpack and Gather Information.
If the initial cluster size is known at installation time, providing the full endpoint set here is recommended, as there is no manual step required when this is done.
Troubleshooting guide
A troubleshooting guide for scattercast can be found in Scattercast Clustering.
