Liberty leadership configuration

In a deployment with multiple Liberty servers, one server is determined to be the leader.

Liberty leadership behavior

The leader Liberty can complete actions that require exclusive access to the Information Store database. These actions include;
  • Initiating Solr indexing
  • Initiating online upgrade tasks
  • Initiating alerts for saved Visual Queries
  • Initiating deletion-by-rule jobs

Leadership state

ZooKeeper is used to determine and manage the leadership state of the Liberty servers in a deployment. Each Liberty server polls ZooKeeper for the current leadership state. If there is no leader, the Liberty attempts to become the leader. One Liberty will successfully become the leader.

Leadership poll

To become a leader, and to maintain leadership, Liberty must be able to connect to ZooKeeper, Solr, and the database management system. The connection to each component is checked whenever the leadership state is polled. If any of the components cannot be connected to, the Liberty either releases its leadership or cannot run for leader if there is not currently a leader.

The leadership poll schedule is independent on each Liberty server. If the current leader releases its leadership, there is no leader until the leadership poll runs on another Liberty that can then assume leadership status.

The leadership poll is completed on a configurable schedule. By default, the leadership state is checked every five minutes. To change the interval, update the value for the ServerLeadershipReconnectIntervalMinutes setting in DiscoServerSettingsCommon.properties.

For example:
ServerLeadershipReconnectIntervalMinutes=5

Error thresholds

The leadership state can change if a certain number of errors are encountered during indexing operations within a specified time span. These types of operations are run only on the leader Liberty. The maximum number of errors that can occur, and the time span in which they can occur, are configurable.

The MaxPermittedErrorsBeforeLeadershipRelease setting defines the maximum number of errors that can occur in the time specified by the TimeSpanForMaxPermittedErrorsBeforeLeadershipReleaseInSeconds setting. By default, up to five errors can occur within 30 seconds before the Liberty relinquishes its leadership.

For example:
MaxPermittedErrorsBeforeLeadershipRelease=5
TimeSpanForMaxPermittedErrorsBeforeLeadershipReleaseInSeconds=30

Restarting limits

If the leader Liberty releases leadership, it attempts to restart in non-leader mode. If Liberty cannot connect to each component, it fails to start. By default, it tries to start three times before failing. To start Liberty on this server again, resolve the issue that is stopping Liberty connecting to all of the components and run setup -t startLiberty.

You can change the maximum number of times that Liberty attempts to start before it fails. To change the number of attempts, update the value of the ServerLeadershipMaxFailedStarts setting in DiscoServerSettingsCommon.properties.
ServerLeadershipMaxFailedStarts=3