File Share Witness in Exchange Server 2010

As we know, Exchange Server 2007 supported three different kind of high availability methods which varied from the single server to across multiple Exchange servers. And then, Exchange Server 20010 introduced Database Availability Group (DAG) which is essentially a collection of at least two and maximum 16 mailbox servers to achieve the high availability.

You should be familiar with the term, File Share Witness (FSW) if you have already implemented the CCR in the Exchange 2007 environment. As the name suggest, file share witness is a file share residing on the third server outside of the DAG for ensuring quorum availability in the cluster. We can provide the file share witness directory and share name while creating the DAG.  Unlike Exchange 2007, where Microsoft recommended hosting file share witness on the Hut Transport server, Exchange 2010 provides you full control to host file share witness on non-Exchange Server as well, but you have to add the Exchange Trusted Subsystem Universal Security Group to the local Administrator Group on the FSW Server.

You can configure the FSW and FSW directory during DAG creation as shown below:-

New-DatabaseAvailabilityGroup -Name DAG01 -WitnessServer ACENT001 -WitnessDirectory C:\FSW_DAG01

How to Design DAG environment in Exchange Server 2010

As we know, Exchange Server 2010 DAG supports 16 members with up to 1600 database; therefore, it turns into complex part to design the layout. The rule of thumb is that the more servers in the DAG provide us more option for sizing databases copies efficiently and resiliently.

  

Above scenario is designed to support a single-node failure. If more than one member is down, four databases would be offline. Just adding more nodes in the DAG does not automatically enable it to sustain multiple failures. You should be very careful while adding nodes in the DAG and keeping database copies to the relevant nodes.

As you can see in the below diagram, servers are mirrored to each other in a four-node DAG. Failure in A and B or C and D would cause a large number of databases unavailable. This design does not provide better redundancy.  

  

Follow the below two rules while designing DAG redundancy to meet your SLAs:-

1.       One-member failure requires two or more high-availability copies, two or more servers, and a witness server.

2.       Two-member failure requires three or more high-availability copies, four or more servers, and a witness server.

We can design DAGs in a variety of environments which solely depends upon your organization SLAs and recovery time/point objective for the mailbox services.  We will discuss on few DAG design which may fit in your environment:-

·         Two-Member DAG in Single Datacenter/Active Directory Site

·         Four-Member DAG in Single Datacenter/Active Directory Site

·         Four-Member DAG in Two Datacenter/Active Directory Sites

·         Two Four-Member DAGs in Two Datacenter/Active Directory Sites

Two-Member DAG in Single Datacenter/Active Directory Site

This design is suited for small offices and branch offices to meet their high availability requirements with the smallest possible DAG design taking into account the cost factor associated with adding more servers and don’t require site resiliency. This design provides redundancy for the Client Access, Hub Transport and mailbox roles by using only two servers.

Unified Messaging role can also be co-located in this scenario with other roles, but it is not recommended by the Microsoft.

It is always recommended to use load balancing for achieving high availability for Client Access and Hub Transport. Windows Network Load Balancing can’t be used, therefore other load balancing options must be used.

Two-member DAG provides three quorum votes (two member server and one witness server). Outage in one vote will not impact the services. But the loss of two of the voters (for example, a DAG member and the witness server) will result in loss of quorum and service interruption. 

Four-Member DAG in Single Datacenter/Active Directory Site

Two or three-member DAG restricts us to failure in only one vote, but four-member DAG provides greater resiliency which has five quorum voters and can sustain the loss of two voters without impacting the service.

 

Two Four-Member DAGs in Two Datacenter/Active Directory Sites

As discussed above, a four-member DAG would cause one datacenter losing voters and quorum and resulting offline. We can deploy the two four-member DAGs to overcome this problem. We can configure one witness server in each datacenters. Since both DAG contains equal number of members and one witness server, outage in WAN will maintain the service across the datacenters.

 

• For DAG1, members REDMBX1 and REDMBX2 would be in the majority and would continue to service users in the Chicago datacenter because they can communicate with the DAG1's witness server, HUB1.
• For DAG2, members BLMBX3 and BLMBX4 would be in the majority and would continue to service users in the Madrid datacenter because they can communicate with DAG2's witness server, HUB2.

Using Mailbox Servers That Don't Contain Databases in a DAG for Additional Votes

As we mentioned earlier, larger DAGs provide greater resilience because they can maintain more failures without service interruption. One more design strategy can help us to get more resilience by leveraging the Hub Transport servers. We can use the existing Hub Transport servers in our environment by adding Mailbox server role (without any databases or database copy) on them and then add that server to the DAG just for voting. More voters can sustain more voter failures in the DAG and still maintain the quorum.

 

In the above diagram, we have four-member DAG extended across two remote datacenters. We have four voters in this scenario (4 DAG members and 1 Witness server). Failure in losing two voters would still maintain the quorum. If the DAG loses a third voter, it loses quorum and would require manual administrative intervention to restore the services.

Based upon the above designing scenarios, let us try to put them in some possible designing structure for our organization. We can have either Active/Passive or Active/Active model.

 

Refer this link for more details

What is Mailbox Database Copies in Exchange Server 2010?

Database Portability was the new feature introduced in Exchange Server 2007 which allows you to move mailbox databases between servers. This feature is enhanced in Exchange Server 2010 with a new name called, Database Mobility, where all copies of a database have the same GUID, unlike Exchange Server 2007.

As we know, clustered mailbox servers and storage groups have been removed from Exchange Server 2010, continuous replication now occurs at the database level. Microsoft has recommended some key characteristics of mailbox database copies, as described below:-

·         Databases can be stored on Mailbox servers that don’t host the active copy of a database.

·          Two copies of same database can’t reside on the same server.

·         Database copies are for mailbox databases only. Use public folder replication for high availability and redundancy of public folders.

·         Exchange Server 2010 allows up to 16 mailbox database copies on multiple Mailbox servers, provided all Mailbox servers should be grouped into a DAG.

·         All Mailbox servers in the DAG should be in the same Active Directory domain, but database copies can be on created in the same or different Active Directory sites or network subnets.

·         In-built Windows Server Backup utility backup only Active copies, but not passive copies. Use
Exchange-aware, VSS-based backup application for all databases backup.

·         All copies of a database must use the same path on each server containing database copies.

·         Mailbox database copies can be created at any point of time and distributed across Mailbox servers.

Understanding Queues State in Exchange Server 2010

Messages remain in a queue when there's any problem or if they have been suspended by an administrator. Queue Viewer us used to examine the number of messages in the queues. If you see a queue with a consistent growing number of messages, there might be a problem. Normally messages come into a queue and then processed quickly. Because of this, the number of messages in a queue should gradually decrease over time as the messages are processed, provided no new messages come into the queue.

Whenever you click the Queues tab in the Queue Viewer, you get a summary of the currently available queues for the selected server. Although queue summaries provide important details for troubleshooting message flow problems, you do have to know what to look for. The connection status is the key information to look at first. This value tells you the state of the queue. States you'll see include these:

· Active queue contains messages that are being delivered.

· Ready queue is needed to allow messages to be delivered. Connection time is allocated to the queues.

· Retry state displays failed connection attempts and the server is waiting to retry.

· Suspended state displays that the queue is suspended, and none of its messages can be processed for routing.

Other summary information that you might find useful in troubleshooting includes

· Delivery Type tells what type of recipient messages are being queued for delivery.

· Next Hop Domain tells the next destination or next hop domain of a delivery queue.

· Message Count tells you the number of messages waiting in the queue. If you see a large number, you might have connectivity or routing problem.

· Next Retry Time is the time when another connection attempt will be made if connection state is Retry. You can click the Retry command to attempt a connection immediately.

· Last Retry Time is the time when the last retry attempt was made when the connection state is Retry.

· Last Error Tells you the error code and details of the last error to occur in a particular queue.

Other summary information that you might find useful in troubleshooting includes

·  Delivery Type tells what type of recipient messages are being queued for delivery.

· Next Hop Domain tells the next destination or next hop domain of a delivery queue.

· Message Count tells you the number of messages waiting in the queue. If you see a large number, you might have connectivity or routing problem.

· Next Retry Time is the time when another connection attempt will be made if connection state is Retry. You can click the Retry command to attempt a connection immediately.

· Last Retry Time is the time when the last retry attempt was made when the connection state is Retry.

· Last Error tells you the error code and details of the last error to occur in a particular queue.

Message Tracking in Exchange Server 2010

Message Tracking is one cool troubleshooting tool in message routing. Message Tracking keeps track of flow of messages into and out of an organization. This feature maintains daily log files which can be used to check the status of messages sent, received and those in queues to be delivered. We can also use Message tracking to monitor public folder usage.

By default, all Hub Transport, Mailbox servers and Edge Transport servers perform message tracking. We can enable or disable message tracking on per-server basis depending upon the requirements.

We can use below Power Shell cmdlets to enable or disable message tracking. The following example disables the message tracking feature.

Set-TransportServer -Identity "MailServer16" -MessageTrackingLogEnabled $false

Change Log File Path

By default, message tracking logs are stored in the %ExchangeInstallPath%\TransportRoles\Logs\MessageTracking directory.

We can change the default message tracking log path by running this command.

Set-TransportServer -Identity "MailServerXXX" -MessageTrackingLogPath "F:\Msg_Tracking"

Note When we change the location of the message tracking directory, Exchange Server does not copy any existing tracking logs from the old directory to the new one. We must manually copy the old logs to the new location if we want all the logs to be in the same location.

Setting Log file Size

By default, the maximum log file size is 10 megabytes (MB). We can change this behavior by setting the -MessageTrackingLogMaxFileSize parameter to the desired maximum file size. The following example sets the message log file size to 100 MB:

Set-TransportServer -Identity "MailServerXXX" -MessageTrackingLogMaxFileSize "100MB"

Configuring Subject Logging

By default, subject logging is enabled on all Hub Transport, Edge Transport, and Mailbox servers, which allows us to perform searches based on message subject lines, header information, sender, and recipient. We can disable this feature by setting -MessageTrackingLogSubjectLoggingEnabled parameter of the Set-TransportServer cmdlet to $false, as shown in the following example:

Set-TransportServer -Identity "MailServerXXX" -MessageTrackingLogSubjectLoggingEnabled $false

Configuring Maximum Log File Age

Exchange Server overwrites the oldest message tracking logs automatically when tracking logs reach a maximum age or when the maximum log directory size is reached. By default, the maximum age is 30 days and the maximum log directory size is 250 MB.

In this example, we will set the maximum log age to 60 days.(format will be DD.HH.MM.SS)

Set-TransportServer -Identity "MailServerXXX" -MessageTrackingLogMaxAge "60.00:00:00"

 You can set the maximum log directory size using the -MessageTrackingLogMaxDirectorySize parameter. As with the maximum log file size, the qualifiers are B, KB, MB, and GB. The following example sets the maximum log directory size to 1 GB:

Set-TransportServer -Identity "MailServerXXX" -MessageTrackingLogMaxDirectorySize "1GB"

Troubleshooting Basics in Exchange Server 2010

One of the primary responsibilities of Exchange Administrator is to ensure proper flow and recoverability of message data. This can only be achieved with few administration tasks instead of doing maintenance, monitoring and queue tracking in case of failure. It is better to monitor if services and processes are functioning normally or not, rather than disaster to happens and then troubleshoot.

Few administration tasks are more important than maintenance, monitoring, and queue tracking. You must maintain Microsoft Exchange Server 2010 to ensure proper flow and recoverability of message data. You need to monitor Exchange Server to ensure that services and processes are functioning normally, and you need to track Exchange Server queues to ensure that messages are being processed.

We can find several tools in the Exchange Management Console to troubleshoot messaging problems. These tools include:-

1. Mailflow Troubleshooter can help us in troubleshooting message delivery delays, non-delivery reports, problems with Edge Transport server synchronization. We can also find the lost messages from this tool.

2. Performance Troubleshooter helps in troubleshooting performance issues related to delays while using Office Outlook and Remote Procedure Call (RPC) dialog box appearing frequently in Microsoft Outlook.

Following power shell cmdlets can be used to obtain detailed information about current configuration of the Exchange Server:-

· Get-TransportServer command displays configuration details for servers with the Hub Transport Server or Edge Transport server role

· Get-UMServer command displays configuration details for servers with the Unified Messaging server role

· Get-ExchangeServer command displays the general configuration details for Exchange servers

· Get-ClientAccessServer command displays configuration details for servers with the Client Access server role

· Get-MailboxServer command displays configuration details for servers with the Mailbox server role

We can also use the Exchange Best Practices Analyzer(ExBPA) tool to gather and check Exchange configuration information. ExPBA tools scan lots of things like:-

· Permissions check Exchange BPA performs a health check and then samples the performance for over a 2-hour period.

· Organizational health check Exchange BPA performs a full scan of the organization, checking for errors, warnings, non default configurations, recent changes, and other configuration details.

· Baseline configuration checks Exchange BPA checks for configuration settings that turn from baseline values. This allows you to compare the settings on a baseline server with settings on other servers and report the differences.

· Connectivity tests Exchange BPA tests network connections and permissions on each Exchange server.

Other useful cmdlets for checking the Exchange organization include:

· Test-ActiveSyncConnectivity Performs a full synchronization against a specified mailbox to test the configuration of Exchange ActiveSync.

· Test-EcpConnectivity Verifies that the Exchange Control Panel is running as expected.

· Test-EdgeSynchronization Verifies that the subscribed Edge Transport servers have a current and accurate synchronization status.

· Test-ExchangeSearch Verifies that Exchange Search is currently enabled and is indexing new e-mail messages in a timely manner.

· Test-FederationTrust Verifies that the federation trust is properly configured and functioning as expected.

· Test-FederationTrustCertificate Verifies the status of certificates used for federation on all Hub Transport and Client Access servers.

· Test-ImapConnectivity Verifies that the IMAP4 service is running as expected.

· Test-IPAllowListProvider Verifies the configuration for a specific IP allow list provider.

· Test-IPBlockListProvider Verifies the configuration for a specific IP block list provider.

· Test-IRMConfiguration Verifies Information Rights Management (IRM) configuration and functionality.

· Test-Mailflow Verifies whether mail can be successfully sent from and delivered to the system mailbox as well as whether e-mail is sent between Mailbox servers within a defined latency threshold.

· Test-MapiConnectivity Verifies server functionality by logging on to the mailbox that you specify.

· Test-MRSHealth Verifies the health of an instance of the Microsoft Exchange Mailbox Replication Service.

· Test-OutlookConnectivity Verifies end-to-end Microsoft Outlook client connectivity and also tests for Outlook Anywhere (RPC/HTTP) and TCP-based connections.

· Test-OutlookWebServices Verifies the Autodiscover service settings for Outlook.

· Test-OwaConnectivity Verifies that Outlook Web App is running as expected.

· Test-PopConnectivity Verifies that the POP3 service is running as expected.

· Test-PowerShellConnectivity Verifies whether Windows PowerShell remoting on the target Client Access server is functioning correctly.

· Test-ReplicationHealth Verifies all aspects of the replication and replay status for a Mailbox server in a database availability group.

· Test-SenderId Verifies whether a specified IP address is the legitimate sending address for a specified SMTP address.

· Test-ServiceHealth Verifies whether all the Windows services that Exchange requires on a server have started.

· Test-SystemHealth Collects data about your Exchange system and analyzes it.

· Test-UMConnectivity Verifies the operation of a computer that has the Unified Messaging server role installed.

· Test-WebServicesConnectivity Verifies the functionality of Exchange Web Services.