Common operations#

Extend ring#

In order to provide additional capacity, new ring nodes can be incorporated by extending the ring. The process is as follows:

  1. Prepare additional instances of Strongbox store. Remember to set unique values for nodeUid in the configuration of each instance. All nodes must have access to the same relational database instance as well as the same message broker.
  2. Start the new Strongbox store instances. Each instance will start and register itself with the common database. At this point, the status of the ring node will be UNKNOWN. As such the ring node does not take part in service.
  3. Using the Strongbox CLI, the ring must be extended by up to two nodes. This process may take a while, depending on usage and load. Customer traffic is not impacted by the ring extension. The new nodes' status is set to JOINING. Joining nodes can be stopped and restarted.
  4. After the successful ring extension, the new nodes' status is ACTIVE. It is now possible to extend the ring again.
  5. At this point, clients should refresh their understanding of the ring to direct uploads and downloads to the correct ring nodes. Typically this is achieved trough a rolling restart.

Note: Currently the ring can only be extended by two nodes at any given time.

Shrink ring#

No convenient operation has been implemented so far.

Ring rebalance#

Rebalancing instructs available Strongbox store ring nodes to analyse the data stored in their SFS against the relational database as well as against the hash ring:

  • Containers and chunks that are stored in SFS but no longer exist in the database are removed from the local SFS.
  • Chunks that are found in SFS are located on the hash ring. If the result no longer includes the local ring node, it is removed from the local SFS.
  • Chunks that are found in the DB are located in the hash ring. If the result includes the local ring, but the chunk does not exist in the local SFS, it is replicated here from other ring nodes.

This process, while safe to run at any time, is particularly useful after nodes have joined or left the ring.

Ring node cleanup#

At regular intervals, it may be necessary to clean up the filesystem storage of ring nodes, removing persisted chunk payload that is not required any longer. This process can be executed by an operator. The ring node can be running, in maintenance or completely offline.

Note: The task runs in a separate JVM from the Strongbox store. It executes in the CLI process itself.

In this process, the SFS of the ring node is analysed against the database. Elements in the path that are not backed by entries in the database are removed.