ScyllaDB University LIVE, FREE Virtual Training Event | March 21
Register for Free
ScyllaDB Documentation Logo Documentation
  • Server
  • Cloud
  • Tools
    • ScyllaDB Manager
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
  • Resources
    • ScyllaDB University
    • Community Forum
    • Tutorials
Download
ScyllaDB Docs ScyllaDB Open Source ScyllaDB for Administrators Procedures Cluster Management Procedures Adding a New Data Center Into an Existing ScyllaDB Cluster

Caution

You're viewing documentation for a previous version. Switch to the latest stable version.

Adding a New Data Center Into an Existing ScyllaDB Cluster¶

The following procedure specifies how to add a Data Center (DC) to a live Scylla Cluster, in a single data center, multi-availability zone, or multi-datacenter. Adding a DC out-scales the cluster and provides higher availability (HA).

The procedure includes:

  • Install nodes on the new DC.

  • Add the new nodes one by one to the cluster.

  • Update the replication strategy of the selected keyspace/keyspaces to use with the new DC.

  • Rebuild new nodes

  • Run full cluster repair

  • Update the Monitoring stack

Note

Make sure to complete the full procedure before starting to read from the new datacenter. You should also be aware that the new DC by default will be used for reads.

Warning

The node/nodes you add must be clean (no data). Otherwise, you risk data loss. See Clean Data from Nodes below.

Prerequisites¶

  1. Log in to one of the nodes in the cluster, collect the following info from the node:

    • cluster_name - grep cluster_name /etc/scylla/scylla.yaml

    • seeds - grep seeds: /etc/scylla/scylla.yaml

    • endpoint_snitch - grep endpoint_snitch /etc/scylla/scylla.yaml

    • Scylla version - scylla --version

    • Authenticator - grep authenticator /etc/scylla/scylla.yaml

    • consistent_cluster_management - grep consistent_cluster_management /etc/scylla/scylla.yaml

    Note

    If authenticator is set to PasswordAuthenticator, increase the replication factor of the system_auth keyspace. For example:

    ALTER KEYSPACE system_auth WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : <new_replication_factor>};

    Ensure you run repair after you alter the keyspace. See How to Safely Increase the Replication Factor.

    It is recommended to set system_auth replication factor to the number of nodes in each DC.

  2. On all client applications, switch the consistency level to LOCAL_* (LOCAL_ONE, LOCAL_QUORUM,etc.) to prevent the coordinators from accessing the data center you’re adding.

  3. Install the new clean Scylla nodes (See Clean Data from Nodes below) on the new datacenter, see Getting Started for further instructions, create as many nodes that you need. Follow the Scylla install procedure up to scylla.yaml configuration phase. In the case that the node starts during the installation process follow these instructions.

Clean Data from Nodes¶

sudo rm -rf /var/lib/scylla/data
sudo find /var/lib/scylla/commitlog -type f -delete
sudo find /var/lib/scylla/hints -type f -delete
sudo find /var/lib/scylla/view_hints -type f -delete

Add New DC¶

Procedure

Warning

Make sure all your keyspaces are using NetworkTopologyStrategy. If this is not the case, follow the these instructions to fix that.

  1. For each node in the existing data-center(s) edit the scylla.yaml file to use either

    • Ec2MultiRegionSnitch - for AWS cloud-based, multi-data-center deployments.

    • GossipingPropertyFileSnitch - for bare metal and cloud (other than AWS) deployments.

  2. Set the DC and Rack Names. In the cassandra-rackdc.properties file edit the parameters listed below. The file can be found under /etc/scylla/.

    Ec2MultiRegionSnitch

    Ec2MultiRegionSnitch gives each DC and rack default names, the region name defined as datacenter name, and availability zones are defined as racks within a datacenter.

    For example:

    A node in the us-east-1 region, us-east is the data center name, and 1 is the rack location.

    To change the DC names, do the following:

    Edit the cassandra-rackdc.properties file with the preferred datacenter name. The file can be found under /etc/scylla/

    The dc_suffix defines a suffix added to the datacenter name as described below.

    For example:

    Region us-east

    dc_suffix=_1_scylla will be us-east_1_scylla

    Or

    Region us-west

    dc_suffix=_1_scylla will be us-west_1_scylla

    GossipingPropertyFileSnitch

    • dc - Set the datacenter name

    • rack - Set the rack name

    For example:

    # cassandra-rackdc.properties
    #
    # The lines may include white spaces at the beginning and the end.
    # The rack and data center names may also include white spaces.
    # All trailing and leading white spaces will be trimmed.
    #
    dc=thedatacentername
    rack=therackname
    # prefer_local=<false | true>
    # dc_suffix=<Data Center name suffix, used by EC2SnitchXXX snitches>
    
  3. In the existing datacenter(s) restart Scylla nodes one by one.

    sudo systemctl restart scylla-server
    
    docker exec -it some-scylla supervisorctl restart scylla
    

    (without restarting some-scylla container)

  4. For each node in the new datacenter edit the scylla.yaml file parameters listed below, the file can be found under /etc/scylla/.

    • cluster_name - Set the selected cluster_name.

    • seeds - IP address of an existing node (or nodes).

    • listen_address - IP address that Scylla used to connect to the other Scylla nodes in the cluster.

    • endpoint_snitch - Set the selected snitch.

    • rpc_address - Address for client connections (Thrift, CQL).

    • consistent_cluster_management - set to the same value as used by your existing nodes.

    The parameters seeds, cluster_name and endpoint_snitch need to match the existing cluster.

  5. In the new datacenter, set the DC and Rack Names (see step number three for more details).

  6. In the new datacenter, start Scylla nodes one by one using.

    sudo systemctl start scylla-server
    
    docker exec -it some-scylla supervisorctl start scylla
    

    (with some-scylla container already running)

  7. Verify that the nodes were added to the cluster using nodetool status.

    For example:

    $ nodetool status
    
    Datacenter: US-DC
    =========================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --   Address         Load            Tokens  Owns            Host ID                                 Rack
    UN   54.191.2.121    120.97 KB       256     ?               c84b80ea-cb60-422b-bc72-fa86ede4ac2e    RACK1
    UN   54.191.72.56    109.54 KB       256     ?               129087eb-9aea-4af6-92c6-99fdadb39c33    RACK1
    UN   54.187.25.99    104.94 KB       256     ?               0540c7d7-2622-4f1f-a3f0-acb39282e0fc    RACK1
    
    Datacenter: ASIA-DC
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --   Address         Load            Tokens  Owns            Host ID                                 Rack
    UN   54.160.174.243  109.54 KB       256     ?               c7686ffd-7a5b-4124-858e-df2e61130aaa    RACK1
    UN   54.235.9.159    109.75 KB       256     ?               39798227-9f6f-4868-8193-08570856c09a    RACK1
    UN   54.146.228.25   128.33 KB       256     ?               7a4957a1-9590-4434-9746-9c8a6f796a0c    RACK1
    
  8. When all nodes are up and running ALTER the following Keyspaces in the new nodes:

    • Keyspace created by the user (which needed to replicate to the new DC).

    • System: system_auth, system_distributed, system_traces For example, replicate the data to three nodes in the new DC.

    For example:

    Before

    DESCRIBE KEYSPACE mykeyspace;
    
    CREATE KEYSPACE mykeyspace WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3};
    

    ALTER Command

    ALTER KEYSPACE mykeyspace WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    ALTER KEYSPACE system_auth WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    ALTER KEYSPACE system_distributed WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    ALTER KEYSPACE system_traces WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    

    After

    DESCRIBE KEYSPACE mykeyspace;
    CREATE KEYSPACE mykeyspace WITH REPLICATION = {'class’: 'NetworkTopologyStrategy', <exiting_dc>:3, <new_dc>: 3};
    CREATE KEYSPACE system_auth WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    CREATE KEYSPACE system_distributed WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    CREATE KEYSPACE system_traces WITH replication = { 'class' : 'NetworkTopologyStrategy', '<exiting_dc>' : 3, <new_dc> : 3};
    
  9. Run nodetool rebuild on each node in the new datacenter, specify the existing datacenter name in the rebuild command.

    For example:

    nodetool rebuild <existing_data_center_name>

    The rebuild ensures that the new nodes that were just added to the cluster will recognize the existing datacenters in the cluster.

  10. Run a full cluster repair, using nodetool repair -pr on each node, or using Scylla Manager ad-hoc repair

  11. If you are using Scylla Monitoring, update the monitoring stack to monitor it. If you are using Scylla Manager, make sure you install the Manager Agent and Manager can access the new DC.

Handling Failures¶

If one of the new nodes starts bootstrapping but then fails in the middle e.g. due to a power loss, you can retry bootstrap (by restarting the node). If you don’t want to retry, or the node refuses to boot on subsequent attempts, consult the Handling Membership Change Failures document.

Configure the Client not to Connect to the New DC¶

This procedure will help your clients not to connect to the DC you just added.

The example below is for clients using the Java driver. Modify this example to suit your needs.

  1. One way to prevent clients from connecting to the new DC is to temporarily restrict them to only use the nodes which are in the same DC as the clients. This can be done by ensuring the operations are performed with CL=LOCAL_* (for example LOCAL_QUORUM) and using DCAwareRoundRobinPolicy created by calling withLocalDc("dcLocalToTheClient) and withUsedHostsPerRemoteDc(0) on DCAwareRoundRobinPolicy.Builder.

    Example of DCAwareRoundRobinPolicy creation:

    variable = DCAwareRoundRobinPolicy.builder()
        .withLocalDc("<name of DC local to the client being configured>")
        .withUsedHostsPerRemoteDc(0)
        .build();
    
  2. Once the new DC is operational you can remove “withUsedHostsPerRemoteDc(0)” from the configuration and change CL to the previous value.

Additional Resources for Java Clients¶

  • DCAwareRoundRobinPolicy.Builder

  • DCAwareRoundRobinPolicy

Was this page helpful?

PREVIOUS
Adding a New Node Into an Existing ScyllaDB Cluster (Out Scale)
NEXT
Upscale a Cluster
  • Create an issue
  • Edit this page

On this page

  • Adding a New Data Center Into an Existing ScyllaDB Cluster
    • Prerequisites
    • Clean Data from Nodes
    • Add New DC
      • Handling Failures
    • Configure the Client not to Connect to the New DC
      • Additional Resources for Java Clients
ScyllaDB Open Source
  • 5.4
    • master
    • 6.2
    • 6.1
    • 6.0
    • 5.4
    • 5.2
    • 5.1
  • Getting Started
    • Install ScyllaDB
      • Launch ScyllaDB on AWS
      • Launch ScyllaDB on GCP
      • Launch ScyllaDB on Azure
      • ScyllaDB Web Installer for Linux
      • Install ScyllaDB Linux Packages
      • Install ScyllaDB Without root Privileges
      • Air-gapped Server Installation
      • ScyllaDB Housekeeping and how to disable it
      • ScyllaDB Developer Mode
    • Configure ScyllaDB
    • ScyllaDB Configuration Reference
    • ScyllaDB Requirements
      • System Requirements
      • OS Support by Linux Distributions and Version
      • Cloud Instance Recommendations
      • ScyllaDB in a Shared Environment
    • Migrate to ScyllaDB
      • Migration Process from Cassandra to Scylla
      • Scylla and Apache Cassandra Compatibility
      • Migration Tools Overview
    • Integration Solutions
      • Integrate Scylla with Spark
      • Integrate Scylla with KairosDB
      • Integrate ScyllaDB with Presto
      • Integrate Scylla with Elasticsearch
      • Integrate Scylla with Kubernetes
      • Integrate Scylla with the JanusGraph Graph Data System
      • Integrate Scylla with DataDog
      • Integrate Scylla with Kafka
      • Integrate Scylla with IOTA Chronicle
      • Integrate Scylla with Spring
      • Shard-Aware Kafka Connector for Scylla
      • Install Scylla with Ansible
      • Integrate Scylla with Databricks
      • Integrate Scylla with Jaeger Server
      • Integrate Scylla with MindsDB
    • Tutorials
  • ScyllaDB for Administrators
    • Administration Guide
    • Procedures
      • Cluster Management
      • Backup & Restore
      • Change Configuration
      • Maintenance
      • Best Practices
      • Benchmarking Scylla
      • Migrate from Cassandra to Scylla
      • Disable Housekeeping
    • Security
      • ScyllaDB Security Checklist
      • Enable Authentication
      • Enable and Disable Authentication Without Downtime
      • Creating a Custom Superuser
      • Generate a cqlshrc File
      • Reset Authenticator Password
      • Enable Authorization
      • Grant Authorization CQL Reference
      • Certificate-based Authentication
      • Role Based Access Control (RBAC)
      • Encryption: Data in Transit Client to Node
      • Encryption: Data in Transit Node to Node
      • Generating a self-signed Certificate Chain Using openssl
      • Configure SaslauthdAuthenticator
    • Admin Tools
      • Nodetool Reference
      • CQLSh
      • REST
      • Tracing
      • Scylla SStable
      • Scylla Types
      • SSTableLoader
      • cassandra-stress
      • SSTabledump
      • SSTable2json
      • SSTableMetadata
      • Scylla Logs
      • Seastar Perftune
      • Virtual Tables
      • SELECT * FROM MUTATION_FRAGMENTS() Statement
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
    • ScyllaDB Manager
    • Upgrade Procedures
      • ScyllaDB Versioning
      • ScyllaDB Open Source Upgrade
      • ScyllaDB Open Source to ScyllaDB Enterprise Upgrade
      • ScyllaDB Image
      • ScyllaDB Enterprise
    • System Configuration
      • System Configuration Guide
      • scylla.yaml
      • ScyllaDB Snitches
    • Benchmarking ScyllaDB
    • ScyllaDB Diagnostic Tools
  • ScyllaDB for Developers
    • Learn To Use ScyllaDB
      • ScyllaDB University
      • Course catalog
      • ScyllaDB Essentials
      • Basic Data Modeling
      • Advanced Data Modeling
      • MMS - Learn by Example
      • Care-Pet an IoT Use Case and Example
    • Scylla Alternator
    • Scylla Features
      • Scylla Open Source Features
      • Scylla Enterprise Features
    • Scylla Drivers
      • Scylla CQL Drivers
      • Scylla DynamoDB Drivers
    • Workload Attributes
  • CQL Reference
    • CQLSh: the CQL shell
    • Appendices
    • Compaction
    • Consistency Levels
    • Consistency Level Calculator
    • Data Definition
    • Data Manipulation
      • SELECT
      • INSERT
      • UPDATE
      • DELETE
      • BATCH
    • Data Types
    • Definitions
    • Global Secondary Indexes
    • Expiring Data with Time to Live (TTL)
    • Functions
    • Wasm support for user-defined functions
    • JSON Support
    • Materialized Views
    • Non-Reserved CQL Keywords
    • Reserved CQL Keywords
    • ScyllaDB CQL Extensions
  • ScyllaDB Architecture
    • ScyllaDB Ring Architecture
    • ScyllaDB Fault Tolerance
    • Consistency Level Console Demo
    • ScyllaDB Anti-Entropy
      • Scylla Hinted Handoff
      • Scylla Read Repair
      • Scylla Repair
    • SSTable
      • ScyllaDB SSTable - 2.x
      • ScyllaDB SSTable - 3.x
    • Compaction Strategies
    • Raft Consensus Algorithm in ScyllaDB
  • Troubleshooting ScyllaDB
    • Errors and Support
      • Report a Scylla problem
      • Error Messages
      • Change Log Level
    • ScyllaDB Startup
      • Ownership Problems
      • Scylla will not Start
      • Scylla Python Script broken
    • Upgrade
      • Inaccessible configuration files after ScyllaDB upgrade
    • Cluster and Node
      • Failed Decommission Problem
      • Cluster Timeouts
      • Node Joined With No Data
      • SocketTimeoutException
      • NullPointerException
      • Failed Schema Sync
    • Data Modeling
      • Scylla Large Partitions Table
      • Scylla Large Rows and Cells Table
      • Large Partitions Hunting
    • Data Storage and SSTables
      • Space Utilization Increasing
      • Disk Space is not Reclaimed
      • SSTable Corruption Problem
      • Pointless Compactions
      • Limiting Compaction
    • CQL
      • Time Range Query Fails
      • COPY FROM Fails
      • CQL Connection Table
    • ScyllaDB Monitor and Manager
      • Manager and Monitoring integration
      • Manager lists healthy nodes as down
  • Knowledge Base
    • Upgrading from experimental CDC
    • Compaction
    • Consistency in ScyllaDB
    • Counting all rows in a table is slow
    • CQL Query Does Not Display Entire Result Set
    • When CQLSh query returns partial results with followed by “More”
    • Run Scylla and supporting services as a custom user:group
    • Customizing CPUSET
    • Decoding Stack Traces
    • Snapshots and Disk Utilization
    • DPDK mode
    • Debug your database with Flame Graphs
    • How to Change gc_grace_seconds for a Table
    • Gossip in Scylla
    • Increase Permission Cache to Avoid Non-paged Queries
    • How does Scylla LWT Differ from Apache Cassandra ?
    • Map CPUs to Scylla Shards
    • Scylla Memory Usage
    • NTP Configuration for Scylla
    • Updating the Mode in perftune.yaml After a ScyllaDB Upgrade
    • POSIX networking for Scylla
    • Scylla consistency quiz for administrators
    • Recreate RAID devices
    • How to Safely Increase the Replication Factor
    • Scylla and Spark integration
    • Increase Scylla resource limits over systemd
    • Scylla Seed Nodes
    • How to Set up a Swap Space
    • Scylla Snapshots
    • Scylla payload sent duplicated static columns
    • Stopping a local repair
    • System Limits
    • How to flush old tombstones from a table
    • Time to Live (TTL) and Compaction
    • Scylla Nodes are Unresponsive
    • Update a Primary Key
    • Using the perf utility with Scylla
    • Configure Scylla Networking with Multiple NIC/IP Combinations
  • Reference
    • AWS Images
    • Configuration Parameters
    • Glossary
    • ScyllaDB Enterprise vs. Open Source Matrix
  • ScyllaDB FAQ
  • Contribute to ScyllaDB
  • Alternator: DynamoDB API in Scylla
    • Getting Started With ScyllaDB Alternator
    • ScyllaDB Alternator for DynamoDB users
Docs Tutorials University Contact Us About Us
© 2025, ScyllaDB. All rights reserved. | Terms of Service | Privacy Policy | ScyllaDB, and ScyllaDB Cloud, are registered trademarks of ScyllaDB, Inc.
Last updated on 08 May 2025.
Powered by Sphinx 7.4.7 & ScyllaDB Theme 1.8.6