ScyllaDB Hinted Handoff¶

A typical write in Scylla works according to the scenarios described in our Fault Tolerance documentation.

But what happens when a write request is sent to a Scylla node that is unresponsive due to reasons including heavy write load on a node, network issues, or even hardware failure? To ensure availability and consistency, Scylla implements hinted handoff.

Hint = target replica ID + mutation data

In other words, Scylla saves a copy of the writes intended for down nodes, and replays them to the nodes when they are up later. Thus, the write operation flow, when a node is down, looks like this:

1. The co-ordinator determines all the replica nodes;

2. Based on the replication factor (RF) , the co-ordinator attempts to write to RF nodes;

3. If one node is down, acknowledgments are only returned from two nodes:

4. If the consistency level does not require responses from all replicas, the co-ordinator, V in this case, will respond to the client that the write was successful. The co-ordinator will write and store a hint for the missing node:

5. Once the down node comes up, the co-ordinator will replay the hint for that node. After the co-ordinator receives an acknowledgement of the write, the hint is deleted.

A co-ordinator stores hints for a handoff under the following conditions:

1. For down nodes;

2. If the replica doesn’t respond within write_request_timeout_in_ms.

The co-ordinator will stop creating any hints for a dead node if the node’s downtime is greater than max_hint_window_in_ms.

Hinted handoff is enabled and managed by these settings in scylla.yaml:

• hinted_handoff_enabled: enables or disables the hinted handoff feature completely or enumerates data centers where hints are allowed. By default, “true” enables hints to all nodes.

• max_hint_window_in_ms: do not generate hints if the destination node has been down for more than this value. If a node is down longer than this period, new hints are not created. Hint generation resumes once the destination node is back up. By default, this is set to 3 hours.

• hints_directory: the directory where Scylla will store hints. By default this is $SCYLLA_HOME/hints.

Storing of the hint can also fail. Enabling hinted handoff therefore does not eliminate the need for repair; a user must recurrently run a full repair to ensure data consistency across the cluster nodes.

Hinted handoff was released as production-ready in Scylla Open Source 3.0 and Scylla Enterprise 2019.1.

Copyright

© 2016, The Apache Software Foundation.

Apache®, Apache Cassandra®, Cassandra®, the Apache feather logo and the Apache Cassandra® Eye logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.