Troubleshooting

A major part of troubleshooting issues is logging. By default, SmarterMail HA hubs log a number of processes and protocols within the system. Having these logs means that, when issues DO occur, cluster administrators can quickly and easily find out the what's going on and get the problems resolved. If nothing else, having access to logs makes working with SmarterTools much easier as it gives our support agents access to information that can then be used to further find and fix issues, or work with our developers to figure out what's going on so a fix can be implemented.

By default, SmarterMail sets log detail levels to Information. Use this section to adjust the log detail levels for the proxy layer logging for the protocols used with SmarterMail HA as well as the process logs. Be aware that more detailed logs require more disk space — moving a log from Information up to Debug or Trace can multiply the amount of data it writes per day. If you choose a detailed log level, you may want to shorten the “Delete Log Files After” setting on the Log Files card so the extra data doesn't accumulate indefinitely. In addition, some logging levels should only be used at the request of SmarterTools.

Cluster administrators have access to the following Troubleshooting tabs:

Options - Configure the log and indexing settings for the hubs.
View Logs - Review the hub logs to look for errors or monitor recent activity.

Options

Use this section to manage how the log files are handled and the level of logging needed for protocols and processes.

Log Files

Compress Log Files After - The number of days after which log files are automatically compressed into a .zip archive. This keeps the log's content available for later review while reducing the amount of disk space it occupies. Use the switch next to the field to enable or disable automatic compression.
Delete Log Files After - The number of days after which log files are permanently and irrecoverably deleted, whether or not they were ever compressed. Use the switch next to the field to enable or disable automatic deletion.

Both settings are enabled by default, with Compress Log Files After set to 3 days and Delete Log Files After set to 60 days. Each day, the hub's log maintenance routine runs compression first and cleanup second: it compresses any eligible log files, then removes any log file — compressed or not — that has passed the deletion threshold. Because of this order, Delete Log Files After should always be set to a larger number of days than Compress Log Files After; otherwise, a log file could be deleted before it ever gets the chance to be compressed. For example, with Compress Log Files After set to 7 days and Delete Log Files After set to 30 days, a log file is kept as plain text for its first week, automatically zipped once it reaches 7 days old to save space, and then permanently removed once it reaches 30 days old.

The same seven-level scale below is used to configure both the Protocol Logging Levels and the Process Logging Levels cards further down the page — there isn't a separate scale for protocols versus processes. Because more detailed levels can generate substantially more data, treat Debug and Trace as temporary diagnostic settings: turn them up while actively troubleshooting an issue, then return the log to Information (or lower) once you're done.

Logging levels for both Protocols and Processes include:

Trace - This level contains super "noisy" data that is only helpful in very specific cases, sort of like a general use Debug Log. Should only be set when instructed by SmarterTools.
Debug - This level provides useful information during situations like development and troubleshooting general configuration or setup issues. This will include lots of system events that may be obvious, but can be helpful to specifically see what is going on.
Information - This level is intended to be a "Normal" log level and will show system events and information that is useful in everyday operations.
Warning - This level flags potentially concerning situations that could also be normal or expected. An example from SmarterMail HA would be a Cluster Status change, which is recorded as a Warning: it may be an intentional, planned change, or it may be an indication of a problem. Warning is generally the lowest level you'd want a log set to day-to-day, unless you're intentionally trying to quiet a particularly noisy log down further.
Error - This level records failures that should be recoverable from within the application. These are intended to be highlighted and should realistically require intervention or analysis
Critical - This level is also referred to as "Fatal" in some logging paradigms. It is an unrecoverable error that terminates the application. This is considered the highest level of logging and running into these situations will require manually restarting the application.
None - This log level is used to "turn off" logs of certain types. It's essentially a no-op and just ignores any logs that meet the requirements to be logged.

Protocol Logging Levels

SmarterMail HA hubs don't process protocol traffic themselves — when something such as HAProxy sits in front of the cluster, the hub proxies incoming IMAP, LDAP, POP, SMTP, and XMPP connections through to whichever node is currently handling that user or domain. Because of this, protocol logs at the hub level only ever capture the proxy layer: which client connected, which node the connection was handed off to, and whether that hand-off succeeded or failed. They do not contain the actual protocol conversation. For example, the SMTP Proxy log on a hub might show that a connection was proxied to a particular node and that the hand-off succeeded, but it won't show the EHLO, MAIL FROM, RCPT TO, and DATA commands that were actually exchanged, or the specific rejection response a recipient's mail server returned. If you're troubleshooting an actual delivery or protocol-level problem — for example, why a specific message was rejected, or why a client can't authenticate — you'll need to review that specific node's own protocol logs, which do capture the full command/response transcript. The following proxy log files can be adjusted on the hub:

IMAP Proxy - Proxied IMAP traffic.
LDAP Proxy - Proxied LDAP traffic.
POP Proxy - Proxied POP3 traffic.
SMTP Proxy - Proxied SMTP traffic.
XMPP Proxy - Proxied XMPP (chat) traffic.

Process Logging Levels

Process Logging can help administrators in a number of ways. For example:

Cluster Service Logs can help identify why a node dropped out of the cluster, or reveal errors in hub-to-hub and hub-to-node communication.
Failover Logs can show exactly when and why the hub promoted a standby node to active status during an outage, and when a recovered node was reactivated and rejoined to the cluster.
Administrative Logs can show when a setting was changed, and which cluster administrator made the change.

The following log file types can be adjusted:

Administrative - Changes and/or modifications made by cluster administrator accounts.
Cluster Service - The log for the entire SmarterMail HA cluster, including hub-to-hub and hub-to-node cluster communication.
Failover - Logs events related to the cluster's automatic failover process: detecting that a node has become unresponsive, promoting a standby node to take its place, and reactivating and rejoining a recovered node once it's healthy again.
General - A general, overall logging available for the cluster; this is the catch-all log used for anything that doesn't map to one of the more specific logs on this list.
gRPC - Logs the internal gRPC-based communication used between the hub and its nodes, and between hubs in a multi-hub cluster. This is a very low-level, noisy log that's typically only needed at the request of SmarterTools Support when diagnosing a transport-layer connectivity issue between cluster components.
HTTP Proxy - Logs HTTP proxy traffic.
Hub Logger - The log for hub actions. This includes things such as DoS detection, blacklist checks, etc.
Lock Service - Logs the cluster's distributed locking service, which coordinates exclusive access to shared cluster resources so that only one node modifies a given piece of shared state at a time.

View Logs

Use this section to quickly view various log files. Viewing log files, especially when it's possible to narrow down the type of process or protocol that requires attention, allows cluster administrators to look for any specific errors that could cause reliability issues on the cluster or narrow down reasons why a specific behavior is being seen.

When viewing the SmarterMail HA logs, the following search fields are available:

Start and End - The date range of log entries you want to view. Both dates are required to run a search.
Hub - Narrows the search to a single hub in the cluster. Left at its default of Combined Hubs, SmarterMail instead searches every hub that has a Logs directory and merges the matching entries from all of them together into a single result, sorted chronologically — rather than simply listing each hub's matches one after another.
Type - The type of log file that you would like to view. Every adjustable protocol and process log described above is available here.
Search - An optional text string to filter for. SmarterMail returns only the log entries containing that string (the match is case-insensitive); if a single log entry spans multiple lines, such as an error with a stack trace, the entire entry is returned as long as any part of it matches.

To search a specific log, complete the date range, select the hub, select the log type, and then enter a search string. Any matching log entries will be displayed. Results are capped at 20,000 lines; if a search matches more than that, a truncation warning is shown along with the most recent matches.

Downloading Logs and/or Search Results

It's possible to download either entire logs or only the search results you receive.

To download the entire log file in a .zip format simply click the Download button. This allows you to get quick access to a hub's entire log file so that it can be reviewed more thoroughly on a local machine; if the Hub field is set to Combined Hubs, the archive contains a separate text file for each hub and date rather than one merged file. If you only need the search results, click the dropdown next to the Download button and select Download Results Only (this option isn't available until you've run a search). This downloads only the results as a text file. You can also copy the results to your clipboard, then paste those results into your favorite text editor. (We recommend Notepad ++ as it handles large files better than Notepad.)