Managing SmarterMail High Availability

The Manage page is the cluster administrator's primary window into the health and makeup of a SmarterMail High Availability (HA) cluster. Instead of showing settings you configure, this page shows you what's actually running — every hub, every node, every domain, and every account across the cluster — along with the actions you can take on each. The page is broken down into five tabs: Server Health, Hubs, Nodes, Domains, and Accounts.

Server Health

The Server Health tab is a combined view of every server in your cluster — both hubs and nodes, whether they're currently active or sitting by as a standby. Rather than switching between the Hubs and Nodes tabs to piece together the overall state of the cluster, this tab lets a cluster administrator see everything at a glance in a single list. Each server listed displays the following information:

• Machine Name - The friendly name given to the hub or node when it was initially set up.
• Server Type - Whether the row represents a Hub or a Node.
• Status - The state the server is currently in. For hubs, this mirrors the states described under the Hubs tab below; for nodes, this mirrors the states described under the Nodes tab below (including whether it's a standby).
• Endpoint - The IP address assigned to the server.
• Server Version - The version of SmarterMail that's running on that hub or node. Because hubs and nodes are updated independently, this column is a quick way to spot a server that's fallen behind on updates — for example, if every node reports the same build except one, that one node is the first place to check before assuming a cluster-wide issue.
• Last Active - The date and time of the most recent communication between the hub you're logged into and that server. If this value stops updating for a server while every other server continues to update normally, that server has likely lost its connection to the cluster even if it hasn't yet been marked Inactive or Offline.

Hubs

The Hubs tab displays every hub configured to participate in the HA cluster. A hub doesn't store mail data itself — its job is to track cluster membership, coordinate failover decisions, and (when something like HAProxy or NGINX sits in front of the cluster) proxy protocol traffic through to whichever node is currently handling a given user or domain. Each hub listed displays:

• Machine Name - The friendly name given to the hub when it was initially set up.
• Status - The state the hub is currently in. A healthy hub generally moves through these states as follows: Initializing → Activating → Active, and later Stopping → Offline when it's shut down. These are the possible states:
  • Initializing - The hub's startup phase, before it has finished joining the cluster.
  • Activating - The hub is coming online and is almost ready to take on cluster traffic.
  • Active - The hub is fully running and participating normally in the cluster.
  • Suspended - The hub has been manually paused by a cluster administrator using the Suspend action described below.
  • Stopping - The hub is in the process of shutting down.
  • Offline - The hub is not currently connected to the cluster at all. A hub must be in this state before it can be permanently removed (see Delete, below).
• Leader - When a checkmark appears in this column, the checked hub is the cluster's Leader — the hub responsible for making cluster-wide decisions, such as approving a node failover or restoring an idle node. There can be only one Leader in an HA cluster at a time. You don't need to know or care which hub you're logged into before performing one of the node-management actions described below: if you issue an action against a hub that isn't currently the Leader, that hub automatically forwards the request to whichever hub is the Leader on your behalf.
• Last Active - The date and time of the most recent communication with the hub.

Hub Actions

Selecting a hub and using the Actions (⋮) button offers the following options:

• Restore - Brings a suspended hub back online. The hub transitions from Suspended through Activating and settles into Active once it's fully rejoined the cluster. This option is only available when the selected hub is currently Suspended.
• Suspend - Pauses a hub without removing it from the cluster's configuration. This is useful when a hub needs maintenance — for example, applying an OS patch or updating the SmarterMail build on that machine. This option is only available when the selected hub is currently Active.
• Delete - Permanently removes the hub from the cluster's configuration. This can't be undone, and the hub must already be Offline before it can be deleted; you can't delete a hub that's still Active, Suspended, or otherwise connected to the cluster.

Suspend Notes

• Suspended hubs will connect to other hubs, but other hubs cannot connect to a suspended hub.
• While suspended, no services are running and no traffic is routed through the hub.
• It is recommended that only 1 hub be suspended at a time, and should only be done when the cluster is running.
• Suspended hubs should be brought out of Suspend (Restored) before the cluster is stopped.
• If a hub gets stuck in Suspend mode, it will need to be manually cleared.

For example, if you're rolling out a SmarterMail update across a two-hub cluster, you'd Suspend the first hub, apply the update while it's paused and out of rotation, then Restore it once the update finishes and confirm it comes back Active before repeating the process on the second hub. This lets you update every hub in the cluster without ever taking the cluster itself offline.

See Adding Hubs and Nodes for information on adding a new hub.

Nodes

The Nodes tab displays every node configured to participate in the HA cluster. Unlike hubs, nodes are where mail data and domains actually live — each node runs its own full SmarterMail installation against its own mount, and the hub directs traffic to whichever node currently owns a given domain or user. Each node listed displays:

• Machine Name - The friendly name of the node. If the row represents a standby that has taken over for a primary node, the name of the primary server it's standing in for is shown in parentheses.
• Status - The state the node is currently in. These include:
  • Initializing - The node's startup phase.
  • Activating - The node is essentially "Active", but it's still coming online. (E.g., It's waiting for SmarterMail to initialize.)
  • Active - The node is running. Ideally, the State will transition from Initializing to Active.
  • Standby - The node is a Standby. That means it's ready to take over should another node go Inactive.
  • Idle - The node has come out of an Inactive state and is waiting for the cluster administrator to make it Active.
  • Inactive - The node is in a failure state. This could be a service crashing, the server was shut down, it dropped its network connection, etc.
• Standby - Indicates whether the row is a Standby node. (See above.)
• Last Active - The last time the node communicated with the hub.
• Failover - Whether failover is currently Enabled or Disabled for the node. This column is blank for standby rows, since failover is a property of the primary node slot, not the standby filling in for it.
• Users - The total number of users on the node.
• Domains - The total number of domains on the node.
• Mount Volume - The path of the mount (shared storage) the node reads and writes its mail data to.

Node Actions

The Nodes tab also contains some buttons worth mentioning.

• Refresh - This refreshes and updates the page.
• New - This allows you to configure a new node. After configuration, the node must be set up. For more information, see Hub and Node Configuration.
• Actions (⋮) - This button has a dropdown that offers a few different options:
  • Manage Node - Logs you in as the primary system administrator for that node, giving you full administrative access to the SmarterMail installation running on it. This option is only available when the selected node is currently Active.
  • Delete - Permanently removes the node's configuration from the hub. This isn't available while the node is Active; the node must be taken out of an active role first (for example, by failing it over or letting it go Inactive) before it can be deleted.
  • Restore Idle Node - Reactivates a node that has come out of an Inactive state and is sitting in the Idle state waiting for a cluster administrator's decision. This option is only available for a node's own primary entry while it's Idle — not for the standby that may have taken over for it.
  • Enable Failover - Turns failover back on for a node whose failover had previously been disabled. If you're enabling failover on a standby node specifically, this also resets that standby's internal failure count. SmarterMail automatically stops offering a standby for failover after it has failed to activate too many times in a row, precisely so the cluster doesn't keep retrying a standby that's clearly having a problem; using Enable Failover on that standby clears the slate and makes it eligible to be tried again.
  • Disable Failover - Prevents a node from being replaced by a standby if it goes Inactive. This is only available for a node's own (non-standby) entry, and only while failover is currently enabled for it.

For example, imagine a node goes Inactive because its underlying server lost power overnight. If failover is enabled and a healthy standby is available, the cluster automatically activates that standby in its place and the affected users and domains fail over with little to no interruption. When the original server comes back online, it won't automatically resume its old role; instead, it appears in the Idle state, and a cluster administrator uses Restore Idle Node to intentionally bring it back into an active role (rather than have it silently take over again and potentially interrupt users who are already being served by the standby).

See Adding Hubs and Nodes for information on adding a new node.

Domains

The Domains tab displays every domain managed by the hub, regardless of which node currently hosts it. This includes:

• Domain Name - The domain that's hosted by the cluster. If the domain has any domain aliases associated with it, a count is shown in parentheses next to the name — hover over the count to see the full list of alias domains in a tooltip.
• Machine Name - The name of the node where the domain is currently being hosted.
• Node State - This mirrors the State of the node on the Nodes tab.
• Enabled - Whether the domain is enabled or not. (No checkmark means the domain's Status is set to disabled.)
• Users - The number of users for the domain / the number of users allowed for the domain.
• Disk Space - The amount of disk space currently used by the domain / the amount of disk space allocated for the domain. If no size limit was set for the domain, this is shown as an unbounded value (∞) instead of a fixed maximum.

Domain Actions

Using the Actions (⋮) button offers you the ability to Manage a selected domain. (You can also do this by right-clicking on the domain and using the Manage Domain option from the context menu.) When you manage a domain you are essentially impersonating the primary system administrator for the domain. This allows you full control over all aspects of the domain being managed. This option is only available while the domain's node is Active and the domain itself isn't in a critically errored state.

Accounts

The Accounts tab displays every "account" across every node in the cluster. Accounts include users, aliases, mailing lists, domain resource addresses, and similar address types — so this list is broader than just user mailboxes. A cluster administrator can use this tab to search for a particular account and see, at a glance, exactly which node it lives on, which is especially useful when you need to go review that node's own logs or troubleshoot something that requires you to be on the correct machine.

• Username - The username associated to the account.
• Domain - The domain name associated to the account. As with the Domains tab, any domain aliases are summarized as a count in parentheses, with the full list available on hover.
• Enabled - Whether the account is enabled or not.
• Machine Name - The node associated to the account.
• Type - The account type: user, alias, mailing list, etc.

Account Actions

Using the Actions (⋮) button offers you the ability to Impersonate the account you've selected. (You can also do this by right-clicking on an account and using the Impersonate User option from the context menu.) When you impersonate an account you are essentially logging in to the account, or impersonating a user. This allows you full control over all aspects of the account being impersonated.

See Add Domains for information about adding domains. Domains can be added from this page or from the node itself. Regardless of how it's done, the process is the same.