While running Exchange Server 2013 setup to install or upgrade a server you may encounter the following error.

There was an error while running ‘ldifde.exe’ to import the schema file ‘C:\Windows\Temp\ExchangeSetup\Setup\Data\PostExchange2003_schema0.ldf’. The error code is: 8224.

Exchange Server 2013 installs and upgrades usually involve an update to the Active Directory schema. This error message indicates that the import of schema changes failed.

On the Schema Master in your forest you may notice the following event logged in the Directory Service log.

Log Name:      Directory Service
Source:        Microsoft-Windows-ActiveDirectory_DomainService
Date:          4/09/2014 2:08:06 PM
Event ID:      2092
Task Category: Replication
Level:         Warning
Keywords:      Classic
User:          ESPNET\administrator
Computer:      S1DC1.exchangeserverpro.net
Description:
This server is the owner of the following FSMO role, but does not consider it valid. For
the partition which contains the FSMO, this server has not replicated successfully with
any of its partners since this server has been restarted. Replication errors are
preventing validation of this role.

Operations which require contacting a FSMO operation master will fail until this
condition is corrected.

FSMO Role: CN=Schema,CN=Configuration,DC=exchangeserverpro,DC=net

The root cause is a replication issue with the domain controllers in the environment. In my experience this is usually due to a domain controller being offline, either due to a fault or due to it being decommissioned incorrectly.

After you have resolved any domain controller replication issues Exchange setup should be able to apply the schema updates without error.

C:\Admin\ex2013cu6>setup /preparead /iacceptexchangeserverlicenseterms
Welcome to Microsoft Exchange Server 2013 Cumulative Update 6 Unattended Setup
Copying Files...
File copy complete. Setup will now collect additional information needed for
installation.
Performing Microsoft Exchange Server Prerequisite Check
    Prerequisite Analysis                                     COMPLETED
Configuring Microsoft Exchange Server
    Organization Preparation                                  COMPLETED
The Exchange Server setup operation completed successfully.

This article Error Code 8224 While Running Ldifde.exe to Import Schema File is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

IT/Dev Connections starts in Las Vegas next week and I’ve been putting the final touches on the two sessions I am delivering.

The conference schedule is live now on the event website, and the Exchange track is available here.

Here’s what you can expect if you attend my sessions.

Managing Database Availability Groups

• When: Wednesday 17th September, 9:00am
• Where: Pinyon 3
• What: Tips and best practices for enjoying a good, healthy Exchange Server 2013 DAG experience, with a focus on the mantra that “good design makes good DAGs”.

Making ActiveSync Work (Instead of “It Just Works”)

• When: Thursday 18th September, 10:45am
• Where: Bluethorn 1/2
• What: Covers testing and troubleshooting connectivity, Autodiscover, Allow/Block/Quarantine, and reporting, and more.

I will also be participating in the Exchange Q&A panel at 2:45pm on Thursday 18th.

The Exchange track for this event has a bunch of other great sessions lined up as well as some terrific full day workshops. Check out the full list here.

Will I see you there?

This article Countdown to Exchange Connections 2014 in Las Vegas is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Reading the headers of an email message can reveal very useful information for Exchange Server administrators who are diagnosing problems.

Email message header information includes details such as the route that the email took (ie which email servers were involved in the transmission of the message), who sent it, who it was addressed to, and whether the email message was scanned for spam or viruses.

This is useful for both internal and external email messages. As just one real world example, I often need to use email message header information to diagnose message delivery delays.

How to Access Email Message Header Information in Outlook

Each version of Microsoft Outlook lets you access the email message headers, but they do it in slightly different ways.

To read the email message headers in Outlook 2013 click on the arrow next to Tags in the ribbon menu.

To read the email message headers in Outlook 2010 click on the arrow next to Tags in the ribbon menu.

To read the email message headers in Outlook 2007 click on the arrow next to Options in the ribbon menu.

The message options will appear with the email message header information towards the bottom.

Reading Email Message Headers in Notepad

First let’s take a look at how difficult it actually can be to read the raw message header information that you get out of a message in Outlook. If you copy the message header information into Notepad will look like a complete mess.

Even though it is is quite messy and difficult to read you can still see useful information in the message headers. First there is the basic information about the email message itself.

Then there are the email servers that the message passed through on it’s way to the destination. To follow these in order start at the bottom and read upwards.

These lines are generally in the following format:

Received: from servername (IP address) by servername (IP address) with MTA-name; timestamp

When a message passes over several hops this can get a bit confusing to read, especially when the timestamps are all from different time zones. Fortunately there are some useful tools you can use to present the email message header to you in a much easier format to read.

Reading Email Message Headers Using Header Analyzer Tools

Here are three online tools you can use analyze email message headers. For demonstration purposes I’m using the message headers from a spam email message that I recently received in a mailbox in my test lab.

Microsoft Remote Connectivity Analyzer

The Microsoft Remote Connectivity Analyzer includes a Message Analyzer tool. Paste the message headers into the field provided and click Analyze headers to produce the report.

MXToolbox

MXToolbox also has a section of the website for analyzing message headers. Again simply paste the header information into the field provided and you get a nice, graphical report out of it.

Google Apps Toolbox

Finally there is the Google Apps Toolbox which includes a Messageheader analyzer tool that has similar functionality to the others.

Summary

As you can see reading email message headers provides you with a lot of very useful information for diagnosing email problems. You can retrieve email messages easily using email clients such as Outlook, and then use any of the third party message header analyzer tools to produce an easy to read report from that message header data.

This article Email Fundamentals: How to Read Email Message Headers is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Exchange Server 2013 mailbox databases can be excluded from the automatic mailbox provisioning load balancer, which distributes new mailboxes evenly across the available databases in an Exchange environment.

This can be performed using Set-MailboxDatabase, for example:

[PS] C:\>Set-MailboxDatabase DB04 -IsExcludedFromProvisioning:$true

A bug currently exists in Exchange Server 2013 that prevents mailboxes from being manually moved to a database that is excluded from provisioning.

[PS] C:\>New-MoveRequest "mary.friel" -TargetDatabase DB04 -Verbose
The database DB04 is excluded from provisioning. Please select a different target database for the move.
    + CategoryInfo          : InvalidArgument: (mary.friel:MailboxOrMailUserIdParameter) [New-MoveRequest], DatabaseEx
   clude...ioningException
    + FullyQualifiedErrorId : [Server=EX2013SRV2,RequestId=a4e02908-2888-488c-a295-ff95fe3b23cf,TimeStamp=18/09/2014 3
   :23:27 PM] [FailureCategory=Cmdlet-DatabaseExcludedFromProvisioningException] C18F48C0,Microsoft.Exchange.Manageme
  nt.RecipientTasks.NewMoveRequest
    + PSComputerName        : ex2013srv2.exchangeserverpro.net

Normally manual mailbox moves to a database excluded from provisioning would be permitted. Microsoft has confirmed that this is currently a bug and are expecting to include a fix in an upcoming update to Exchange Server 2013.

In the meantime if you need to move mailboxes to the database you can use the “suspend” instead of “exclude” setting on the database. “Suspend” is intended for situations where an admin only temporarily wants to prevent new mailboxes from being provisioned on the database, but otherwise achieves the same thing.

[PS] C:\>Set-MailboxDatabase DB04 -IsExcludedFromProvisioning:$false
[PS] C:\>Set-MailboxDatabase DB04 -IsSuspendedFromProvisioning:$true
[PS] C:\>New-MoveRequest "mary.friel" -TargetDatabase DB04 -Verbose
DisplayName               StatusDetail              TotalMailboxSize          TotalArchiveSize         PercentComplete
-----------               ------------              ----------------          ----------------         ---------------
Mary Friel                Queued                    128.9 MB (135,205,784 ...                          0

This article Unable to Move Mailboxes to an Exchange Server 2013 Database that is Excluded from Provisioning is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

In Exchange Server 2013 and 2010 end users are able to manage their own ActiveSync mobile devices via the Exchange Control Panel. This allows them to perform tasks such as remote wiping a mobile device that is associated with their mailbox.

For some organizations this is not desirable and they need to disable this functionality for their end users.

You can disable ActiveSync management for end users by using OWA Mailbox Policies.

A default OWA Mailbox Policy is set up for Exchange organizations but it is not applied to mailboxes automatically. For example, here we can see my organization’s default policy, and that the user Vik Kirby has no policy assigned.

[PS] C:\>Get-OwaMailboxPolicy |  select name
Name
----
Default
[PS] C:\>Get-CASMailbox vik.kirby | select OWAMailboxPolicy
OwaMailboxPolicy
----------------

The first thing I need to do is disable the ActiveSync features in the OWA Mailbox Policy (or create a new policy if you don’t want to mess with the default one). Then, assign the OWA Mailbox Policy to the user.

[PS] C:\>Set-OwaMailboxPolicy "Default" -ActiveSyncIntegrationEnabled:$false
[PS] C:\>Set-CASMailbox vik.kirby -OwaMailboxPolicy "Default"

The next time the user logs in to OWA the policy should take effect and block those features.

This article Disable Exchange ActiveSync Mobile Device Management for End Users is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Current builds of Exchange Server 2013 (which is up to CU6 at this time) have a bug in the Get-MailboxDatabaseCopy cmdlet that may cause an incorrect value to be returned for the activation preference of a database copy.

The bug is harmless to the operation of the database availability group (it will still use the correct activation preference values), but if you’re relying on Get-MailboxDatabaseCopy to return those values in a script, or just during general admin, then you may be misled by the bug.

Here’s an example. In this case I’ve used Set-MailboxDatabaseCopy to set each copy of the database DB02 to an activation preference of 4.

[PS] C:\>Set-MailboxDatabaseCopy DB02\MELEX1 -ActivationPreference 4
[PS] C:\>Set-MailboxDatabaseCopy DB02\MELEX2 -ActivationPreference 4
[PS] C:\>Set-MailboxDatabaseCopy DB02\SYDEX1 -ActivationPreference 4
[PS] C:\>Set-MailboxDatabaseCopy DB02\SYDEX2 -ActivationPreference 4

There are no warnings or other indications that it isn’t working, and Get-MailboxDatabaseCopyStatus shows me a result like this.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB02 | select name,status,activationpreference | ft -auto
Name         Status ActivationPreference
----         ------ --------------------
DB02\MELEX1 Healthy                    4
DB02\MELEX2 Healthy                    4
DB02\SYDEX2 Mounted                    4
DB02\SYDEX1 Healthy                    4

Activation preference values are unique for each copy of a database, you can’t have an AP of 4 for every copy. And in this case I don’t, it is just the bug in the cmdlet.

If I use Get-MailboxDatabase to query the AP instead I see the correct values.

[PS] C:\>Get-MailboxDatabase DB02 | Select -expandproperty:ActivationPreference
Key    Value
---    -----
SYDEX2     1
MELEX1     2
MELEX2     3
SYDEX1     4

And if I use the RedistributeActiveDatabases.ps1 script to rebalance the DAG it uses the correct AP values in doing so (truncated output here just to demonstrate the point).

-----------------------
Starting Database Moves
-----------------------
Considering move of 'DB02' from 'SYDEX2' (AP = 3) to 'MELEX1' (AP = 1)...
Database 'DB02' successfully moved from 'SYDEX2' to 'melex1'.
ServerName ActiveDbs PassiveDbs
---------- --------- ----------
MELEX1             2          2
MELEX2             1          3
SYDEX1             1          3
SYDEX2             0          4 `n
----------------
Summary of Moves
----------------
Successfully moved      : 1
Moved to less preferred : 0
Failed to move          : 0
Not moved               : 3
DbName                    : DB02
ActiveOnPreferenceAtStart : 3
ActiveServerAtStart       : SYDEX2
ActiveOnPreferenceAtEnd   : 1
ActiveServerAtEnd         : melex1
IsOnMostPreferredCopy     : True
MoveStatus                : MoveSucceeded

We can expect a fix for this bug in a future update, most likely Cumulative Update 8 at the earliest. It isn’t a serious issue, as long as you’re aware of it and use the Get-MailboxDatabase cmdlet to verify your AP values are correct.

Thanks to Abram Jackson from the Microsoft Exchange PG for confirming this bug and providing the workaround.

This article Get-MailboxDatabaseCopyStatus Displays Incorrect Activation Preference Value is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

A PowerShell profile is a PowerShell script that runs automatically when a new PowerShell session is started. PowerShell profiles can be used to configure your PowerShell environment the way you like it, or to load custom functions for use in your PowerShell administration tasks.

There are actually six PowerShell profiles relating to the PowerShell console and the PowerShell ISE. You can see the four console-related profiles by running the following command:

PS C:\Users\Paul> $profile | Get-Member -MemberType NoteProperty | fl name,definition
Name       : AllUsersAllHosts
Definition : System.String AllUsersAllHosts=C:\Windows\System32\WindowsPowerShell\v1.0\profile.ps1
Name       : AllUsersCurrentHost
Definition : System.String
             AllUsersCurrentHost=C:\Windows\System32\WindowsPowerShell\v1.0\Microsoft.PowerShell_profile.ps1
Name       : CurrentUserAllHosts
Definition : System.String CurrentUserAllHosts=C:\Users\Paul\Documents\WindowsPowerShell\profile.ps1
Name       : CurrentUserCurrentHost
Definition : System.String
             CurrentUserCurrentHost=C:\Users\Paul\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1

The two additional ISE-related profiles are:

• Current User, Current Host – ISE: $Home\[My ]Documents\WindowsPowerShell\Microsoft.PowerShellISE_profile.ps1
• All Users, Current Host – ISE: $PsHome\Microsoft.PowerShellISE_profile.ps1

For the purposes of this article I will be referring to the Current User, Current Host profile.

You can check for an existing PowerShell profile by using Test-Path.

PS C:\> Test-Path $profile
False

If there is no existing profile create one with New-Item.

PS C:\> New-Item $profile -ItemType File -Force
    Directory: C:\Users\Paul\Documents\WindowsPowerShell
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        29/09/2014   2:30 PM          0 Microsoft.PowerShell_profile.ps1

You can now edit the profile script using the PowerShell ISE or your PowerShell editor of choice and add custom functions and any other customizations that you require for your PowerShell console sessions.

PS C:\> powershell_ise.exe $profile

Examples:

• PowerShell Function to Connect to Office 365

This article PowerShell Tip: Create a PowerShell Profile is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Connecting to an Office 365 tenant with PowerShell is a reasonably simple task. All you need is PowerShell on your computer, which is included by default in any recent version of Windows and Windows Server.

There are three basic steps for connecting to Office 365 with PowerShell, and you’ll find these in official help documentation on TechNet as well as on many blogs. The steps are:

• Capture credentials
• Create a new PSSession
• Import the PSSession

You can streamline this process by adding a custom function to your PowerShell profile. Here are two functions you can use for connecting and disconnecting from Office 365.

Function Connect-O365 {
    $URL = "https://ps.outlook.com/powershell"
    
    $Credentials = Get-Credential -Message "Enter your Office 365 admin credentials"
    $O365Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri $URL -Credential $Credentials -Authentication Basic -AllowRedirection -Name "Office 365"
    Import-PSSession $O365Session
}
Function Disconnect-O365 {
    
    Remove-PSSession -Name "Office 365"
}

After adding the functions to your PowerShell profile open a new console session and run Connect-O365 to connect.

PS C:\Users\Paul> Connect-O365

A dialog will popup asking for your Office 365 admin credentials. After a short wait the connection will be established and you can run cmdlets such as Get-Mailbox to list the mailbox users in your Office 365 tenant.

PS C:\Users\Paul> Connect-O365
WARNING: Your connection has been redirected to the following URI:
"https://pod51055psh.outlook.com/powershell-liveid?PSVersion=4.0 "
WARNING: The names of some imported commands from the module 'tmp_g5b5ehu0.amo' include unapproved verbs that might
make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the
Verbose parameter. For a list of approved verbs, type Get-Verb.
ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     1.0        tmp_g5b5ehu0.amo                    {Add-O365AvailabilityAddressSpace, Add-O365DistributionGro...
PS C:\Users\Paul> Get-Mailbox
Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Alan.Reid                 Alan.Reid            sixpr04mb240     49.5 GB (53,150,220,288 bytes)
Alannah.Shaw              Alannah.Shaw         sinpr04mb012     49.5 GB (53,150,220,288 bytes)
Bernadette.Hemming        Bernadette.Hemming   hkxpr04mb360     49.5 GB (53,150,220,288 bytes)
Bernadette.O'Connell      Bernadette.O'Connell sixpr04mb240     49.5 GB (53,150,220,288 bytes)
Debbie.Dalgliesh          Debbie.Dalgliesh     sinpr04mb121     49.5 GB (53,150,220,288 bytes)
Debbie.Lisa               Debbie.Lisa          sinpr04mb364     49.5 GB (53,150,220,288 bytes)
DiscoverySearchMailbox... DiscoverySearchMa... sixpr04mb015     50 GB (53,687,091,200 bytes)
Ivana.Ferrary             Ivana.Ferrary        sixpr04mb191     49.5 GB (53,150,220,288 bytes)
Jack.Hirschorn            Jack.Hirschorn       hkxpr04mb134     49.5 GB (53,150,220,288 bytes)
Kathy.Abdullahi           Kathy.Abdullahi      sinpr04mb329     49.5 GB (53,150,220,288 bytes)
Kavita.Sheikh             Kavita.Sheikh        sinpr04mb090     49.5 GB (53,150,220,288 bytes)
Melanie.Thomas            Melanie.Thomas       sixpr04mb352     49.5 GB (53,150,220,288 bytes)
Merle.Elam                Merle.Elam           sixpr04mb0382    49.5 GB (53,150,220,288 bytes)
admin                     admin                sixpr04mb048     49.5 GB (53,150,220,288 bytes)
Rowena.Khan               Rowena.Khan          hknpr04mb257     49.5 GB (53,150,220,288 bytes)
Rupinder.Zaveri           Rupinder.Zaveri      sinpr04mb267     49.5 GB (53,150,220,288 bytes)
Theresa.Bolt              Theresa.Bolt         hknpr04mb226     49.5 GB (53,150,220,288 bytes)
Tina.Miller               Tina.Miller          sixpr04mb189     49.5 GB (53,150,220,288 bytes)

To disconnect the session use the Disconnect-O365 function.

PS C:\Users\Paul> Disconnect-O365

As an additional tip, if you’re running a hybrid configuration and using an Exchange Management Shell session to connect to Office 365 you may run into issues due to the same cmdlets being available in both your on-premises Exchange and Exchange Online. In that situation you can use Jeff’s tip of adding a prefix to the Import-PSSession command.

For example:

Function Connect-O365 {
    $URL = "https://ps.outlook.com/powershell"
    
    $Credentials = Get-Credential -Message "Enter your Office 365 admin credentials"
    $O365Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri $URL -Credential $Credentials -Authentication Basic -AllowRedirection
    Import-PSSession $O365Session -Prefix O365
}

You might prefer to use a prefix of “Online” or “Cloud” instead. Really it is up to you. But when you do this the prefix is added to the cmdlets for your Office 365 session, so Get-Mailbox becomes Get-O365Mailbox instead.

PS C:\Users\Paul> Get-O365Mailbox
Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Alan.Reid                 Alan.Reid            sixpr04mb240     49.5 GB (53,150,220,288 bytes)
Alannah.Shaw              Alannah.Shaw         sinpr04mb012     49.5 GB (53,150,220,288 bytes)
Bernadette.Hemming        Bernadette.Hemming   hkxpr04mb360     49.5 GB (53,150,220,288 bytes)
Bernadette.O'Connell      Bernadette.O'Connell sixpr04mb240     49.5 GB (53,150,220,288 bytes)
Debbie.Dalgliesh          Debbie.Dalgliesh     sinpr04mb121     49.5 GB (53,150,220,288 bytes)
Debbie.Lisa               Debbie.Lisa          sinpr04mb364     49.5 GB (53,150,220,288 bytes)
DiscoverySearchMailbox... DiscoverySearchMa... sixpr04mb015     50 GB (53,687,091,200 bytes)
Ivana.Ferrary             Ivana.Ferrary        sixpr04mb191     49.5 GB (53,150,220,288 bytes)
Jack.Hirschorn            Jack.Hirschorn       hkxpr04mb134     49.5 GB (53,150,220,288 bytes)
Kathy.Abdullahi           Kathy.Abdullahi      sinpr04mb329     49.5 GB (53,150,220,288 bytes)
Kavita.Sheikh             Kavita.Sheikh        sinpr04mb090     49.5 GB (53,150,220,288 bytes)
Melanie.Thomas            Melanie.Thomas       sixpr04mb352     49.5 GB (53,150,220,288 bytes)
Merle.Elam                Merle.Elam           sixpr04mb0382    49.5 GB (53,150,220,288 bytes)
admin                     admin                sixpr04mb048     49.5 GB (53,150,220,288 bytes)
Rowena.Khan               Rowena.Khan          hknpr04mb257     49.5 GB (53,150,220,288 bytes)
Rupinder.Zaveri           Rupinder.Zaveri      sinpr04mb267     49.5 GB (53,150,220,288 bytes)
Theresa.Bolt              Theresa.Bolt         hknpr04mb226     49.5 GB (53,150,220,288 bytes)
Tina.Miller               Tina.Miller          sixpr04mb189     49.5 GB (53,150,220,288 bytes)

This article PowerShell Function to Connect to Office 365 is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Mailbox users can set their own Junk email controls using Outlook or Outlook Web App, including blocking senders.

When a mailbox user blocks a sender the sender’s email address is added to the “Blocked Senders” list, visible in Outlook’s Junk Email Options.

This list is also accessible in Outlook Web App, by navigating to the Options (by clicking the gear icon in the top right) and looking in the “block and allow” section.

When an email address is blocked like this any email messages from the sender will go to that user’s junk email folder. However, if Safelist Aggregation is being used the emails will be rejected by the server instead. The sender will receive an NDR with information similar to this:

Delivery to the following recipient failed permanently:

alan.reid@exchangeserverpro.net

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain exchangeserverpro.net by maila.locklan.com.au. [58.7.213.10].

The error that the other server returned was:
554 5.1.0 Sender denied

The user in your organization can remove the email address from their blocked sender list, and then a safelist update will stop the server from blocking those emails.

However, there are situations where the person may not be capable of removing the blocked sender, or you may simply want to do it yourself as the Exchange administrator. In these cases you can manage a mailbox’s junk email configuration using PowerShell.

Running Get-MailboxJunkEmailConfiguration will allow you to see the current trusted and blocked email addresses for that mailbox user.

[PS] C:\>Get-MailboxJunkEmailConfiguration alan.reid
RunspaceId               : 11371828-d5da-40c3-a9fc-fc287c2fcf0a
Enabled                  : True
TrustedListsOnly         : False
ContactsTrusted          : True
TrustedSendersAndDomains : {}
BlockedSendersAndDomains : {alan.oliveirabio@yahoo.com.br, exchangeserverpro@gmail.com, info@globalsales.co.cc,
                           register@ebrguide.net, service@paypal.net}
MailboxOwnerId           : exchangeserverpro.net/Company/Head Office/Users/Alan.Reid
Identity                 : exchangeserverpro.net/Company/Head Office/Users/Alan.Reid
IsValid                  : True
ObjectState              : Unchanged

To remove an item from the BlockedSendersAndDomains use Set-MailboxJunkEmailConfiguration.

[PS] C:\>Set-MailboxJunkEmailConfiguration alan.reid -BlockedSendersAndDomains @{remove="exchangeserverpro@gmail.com"}

The above command will remove exchangeserverpro@gmail.com from the BlockedSendersAndDomains but leave the other entries in place.

[PS] C:\>Get-MailboxJunkEmailConfiguration alan.reid
RunspaceId               : 11371828-d5da-40c3-a9fc-fc287c2fcf0a
Enabled                  : True
TrustedListsOnly         : False
ContactsTrusted          : True
TrustedSendersAndDomains : {}
BlockedSendersAndDomains : {alan.oliveirabio@yahoo.com.br, info@globalsales.co.cc, register@ebrguide.net,
                           service@paypal.net}
MailboxOwnerId           : exchangeserverpro.net/Company/Head Office/Users/Alan.Reid
Identity                 : exchangeserverpro.net/Company/Head Office/Users/Alan.Reid
IsValid                  : True
ObjectState              : Unchanged

Make sure you update Safelist Aggregation again after making the change by running Update-Safelist.

[PS] C:\>Update-SafeList alan.reid

When that sender sends to the mailbox they will no longer be blocked by the mailbox’s junk email configuration.

This article SMTP 554 5.1.0 Sender Denied Caused by Mailbox Junk Email Configuration is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Over on Reddit there’s a question about unexpected permissions appearing on mailboxes.

Exchange mailbox user objects inherit a number of permissions that are necessary for the day to day running of the Exchange Server environment. These are a mixture of permissions that Exchange Server computer accounts need, as well as the permissions for the various administrative roles.

Non-inherited permissions include those such as granting a user full access to another user’s mailbox.

However, sometimes when looking at the permissions on a mailbox an admin will notice users or groups that have permissions to the mailbox via inheritance.

Usually I find this is due to the lazy administrative approach of granting the permissions at the mailbox database level. For example, this command will grant Alan Reid full access to all mailboxes in the database DB01.

[PS] C:\>Get-MailboxDatabase DB01 | Add-ADPermission -User alan.reid -AccessRights GenericAll
Identity             User                 Deny  Inherited
--------             ----                 ----  ---------
DB01                 ESPNET\Alan.Reid     False False

Alan can now open any of those mailboxes in DB01, any time he wants to. This might seem convenient, but it is not a very good approach from an auditing perspective. I much prefer that admins grant themselves access to mailboxes on a case by case basis, then remove them afterwards. These actions are then logged in the admin audit log and can be correlated against things like support tickets raised by the end user, or approval emails from a manager.

If you suspect this has happened in your environment you can look for non-inherited permissions at the mailbox database level, for example:

[PS] C:\>Get-MailboxDatabase DB01 | Get-ADPermission | Where {$_.IsInherited -eq $false}
Identity             User                 Deny  Inherited
--------             ----                 ----  ---------
DB01                 ESPNET\Alan.Reid     False False

Removing the permissions is easy as well.

[PS] C:\>Get-MailboxDatabase DB01 | Remove-ADPermission -User alan.reid -AccessRights GenericAll

This article Unexpected Permissions Appearing on Exchange Server Mailboxes is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

At my IT/Dev Connections session in September I advocated for simple designs for database availability groups. This included some points about Exchange Server 2013 storage design and layout, such as:

• JBOD vs RAID
• Multiple databases per volume
• Volumes mounted in folders not drive letters
• Co-locate the database and transaction log files on the same volume

Those recommendations came with caveats of course, depending on various factors. Aside from simple designs providing ease of management they can also mean you get to leverage the terrific new feature in Exchange Server 2013 called Autoreseed.

Autoreseed in Exchange Server 2013

With Autoreseed the members of an Exchange 2013 DAG are pre-configured with one or more spare volumes. When a disk fails the Exchange server is able to automatically replace the failed disk with a spare, and then reseed the lost database copies to the new volume.

This means the recovery workflow in Exchange 2013 goes like this:

1. Disk fails (resiliency of your DAG is impacted)
2. Spare disk automatically mounted
3. Database copies reseeded (resiliency is restored automatically)
4. Manual intervention to replace failed disk replaced with a new spare

In Exchange Server 2010, which also supported JBOD storage, the recovery workflow goes like this:

1. Disk fails (resiliency of your DAG is impacted)
2. Manual intervention to replace disk
3. Manual intervention to reseed database copies (resiliency is restored)

The Exchange 2010 recovery workflow involves too many manual steps to restore the resiliency of the DAG, requires response by admins at any hour of the day, and is simply not efficient at scale.

The Exchange 2013 recovery workflow can automatically restore the resiliency of the DAG without manual intervention, requires response by admins at a lower urgency, and is far more efficient at scale.

Laying the foundation for Autoreseed involves implementing those recommendations I mentioned earlier. Let’s take a look at them in a little more detail.

RAID vs JBOD

For single datacenter deployments:

• Always use RAID for the system/OS volume
• Always use RAID when there are less than 3 database copies
• Use JBOD when there are 3 or more database copies
• Use JBOD for lagged copies only when 2 or more lagged database copies exist

For multiple datacenter deployments:

• Always use RAID for the system/OS volume
• Always use RAID when there are less than 2 database copies in a datacenter
• Use JBOD when 2 or more database copies exist in a datacenter
• Use JBOD for lagged copies as long as 2 or more lagged copies exist, or log play down is enabled

Multiple Databases Per Volume

Use multiple databases per volume when 3 or more database copies exist. Can be placed on RAID or JBOD (with preference for JBOD as I’ll explain shortly).

The number of databases per volume should equal the number of copies of the databases.

Volumes Mounted in Folders not Drive Letters

Mounting your volumes as drive letters is fine for non-DAG deployments, and works for DAG deployments as well, but is not recommended.

There is the obvious limitation of the size of the alphabet. With only 23 usable letters after A:, B:, and C: are consumed, and Exchange 2013 Enterprise capable of hosting 100 databases, you can easily run into problems or at the very least find yourself juggling a complex configuration to work around it.

Instead mount your volumes as folders, using a RAID-protected host volume (the C:\ volume for system/OS is fine for this).

Co-Locate Database and Transaction Log Files

Exchange admins are used to placing the database and transaction log files on separate volumes for recoverability from disk failures. This is still recommended for non-DAG scenarios.

For DAG scenarios the fact that you have multiple copies of each database mitigates the risk of a single disk failure taking out an entire database. So co-locating the database and transaction log files is recommended for DAG scenarios, especially when using multiple databases per volume, and also when using JBOD.

Combining the above, along with evenly distributed active, passive and lagged database copies, gives you an Exchange 2013 DAG that looks similar to this example.

This example obviously assumes that a four node DAG in two datacenters is the right solution for the environment. Your own requirements will vary of course, but this example is being used mainly to demonstrate Autoreseed.

Example Storage Layout for DAG Members

With all of the above in mind here is an example of how the storage layout would be configured for an Exchange 2013 DAG member.

We start with a RAID protected system/OS volume, and create two folders in the root of C:\.

• ExchangeDatabases
• ExchangeVolumes

These match up with the default settings of an Exchange 2013 DAG for root folder paths.

[PS] C:\>Get-DatabaseAvailabilityGroup | fl *autodag*path
AutoDagDatabasesRootFolderPath : C:\ExchangeDatabases
AutoDagVolumesRootFolderPath   : C:\ExchangeVolumes

Next, the volumes that will be hosting the databases and log files are configured. For this simple example a single volume is being configured to host active data and a single volume is being configured as a spare. These are mounted into sub-folders of C:\ExchangeVolumes named Volume1 and Volume2.

Volume1 is then mounted into additional folders for hosting the database and log files. These folder names match the names of the databases in the DAG, for example DB01, DB02, DB03 and DB04. These are created as sub-folders of the C:\ExchangeDatabases folder.

If you’re wondering what I mean by this, all I am referring to is mounting the volume into multiple paths instead of as a drive letter, just as you would normally see when first creating the volume.

Finally, create sub-folders of each database folder to host the DB and log files. These are named according to the database names again, so DB01 needs sub-folders named DB01.db and DB01.log.

These folders are then used as the paths when creating the mailbox databases themselves. For example, here are the paths for DB01 in this environment.

[PS] C:\>Get-MailboxDatabase DB01 | fl *path*
EdbFilePath             : C:\ExchangeDatabases\DB01\db01.db\DB01.edb
LogFolderPath           : C:\ExchangeDatabases\DB01\db01.log

Autoreseed in Action

When a disk fails in an Exchange Server 2013 DAG member the Autoreseed workflow begins. However, the following conditions must be met for Autoreseed to take place:

1. The database copies are not blocked from resuming replication or reseeding.
2. The logs and databases files for the database are collocated on the same volume.
3. The logs and database folder structure matches the naming convention required for Autoreseed.
4. There are no other database copies on the volume that are in an “Active” state.
5. All database copies on the volume are in a “FailedAndSuspended” state.
6. The server has no more than 8 “FailedAndSuspended” database copies.

If those conditions are met then Autoreseed can attempt to resolve the issue.

The workflow begins with detection of the failed volume. Database copies are regularly checked to see whether any of them have been at a status of “FailedAndSuspended” for 15 minutes or longer. This is the state that a database copy will be in when there is an underlying storage issue. The 15 minute threshold exists to ensure that remedial action is not taken too quickly.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 10:19:46 PM
Event ID:      1109
Task Category: Auto Reseed Manager
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager is starting repair workflow 'FailedSuspendedCopyAutoReseed' for database 'DB01'.
WorkflowLaunchReason: Database copy 'DB01\MELEX1' encountered an error during log replay. Error: The system cannot find the path specified

The server attempts to resume the FailedAndSuspended database copy 3 times.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 10:19:46 PM
Event ID:      1124
Task Category: Auto Reseed Manager
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager is beginning attempt number 1 of execution stage 'Resume' for database copy 'DB01' as part of repair workflow 'FailedSuspendedCopyAutoReseed'.
WorkflowLaunchReason: Database copy 'DB01\MELEX1' encountered an error during log replay. Error: The system cannot find the path specified

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 11:04:46 PM
Event ID:      1119
Task Category: Auto Reseed Manager
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager failed to resume database copy 'DB01' as part of repair workflow 'FailedSuspendedCopyAutoReseed' after a maximum of 3 attempts. The workflow will next attempt to assign a spare volume and reseed the database copy.
WorkflowLaunchReason: The Microsoft Exchange Replication service is unable to create required directory C:\ExchangeDatabases\DB01\db01.log for DB01\MELEX1. The database copy status will be set to Failed. Please check the file system permissions. Error: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\ExchangeDatabases\DB01\db01.log'.

The server attempts to assign a spare volume once per hour for up to 5 attempts.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 11:04:46 PM
Event ID:      1124
Task Category: Auto Reseed Manager
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager is beginning attempt number 1 of execution stage 'AssignSpare' for database copy 'DB01' as part of repair workflow 'FailedSuspendedCopyAutoReseed'.
WorkflowLaunchReason: The Microsoft Exchange Replication service is unable to create required directory C:\ExchangeDatabases\DB01\db01.log for DB01\MELEX1. The database copy status will be set to Failed. Please check the file system permissions. Error: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\ExchangeDatabases\DB01\db01.log'.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 11:04:46 PM
Event ID:      1125
Task Category: Auto Reseed Manager
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager has successfully assigned spare volume '\\?\Volume{6e77b6f8-6f83-49f1-ae48-60aa9419cd19}\' mounted at 'C:\ExchangeVolumes\Volume3\' for database copy 'DB01' as part of repair workflow 'FailedSuspendedCopyAutoReseed'. The workflow will next attempt to reseed the database copy.
WorkflowLaunchReason: The Microsoft Exchange Replication service is unable to create required directory C:\ExchangeDatabases\DB01\db01.log for DB01\MELEX1. The database copy status will be set to Failed. Please check the file system permissions. Error: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\ExchangeDatabases\DB01\db01.log'.

The server attempts to reseed the database copies to the new volume, with up to 5 attempts at 1 hour intervals.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          2/09/2014 11:19:46 PM
Event ID:      1117
Task Category: Auto Reseed Manager
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
Automatic Reseed Manager throttled repair workflow 'FailedSuspendedCopyAutoReseed' for database 'DB01'. Details: The Automatic Reseed Manager encountered an error: The automatic repair operation for database copy 'DB01\melex1' will not be run because it has been throttled by the throttling interval of '01:00:00'.
WorkflowLaunchReason: The Microsoft Exchange Replication service is unable to create required directory C:\ExchangeDatabases\DB01\db01.log for DB01\MELEX1. The database copy status will be set to Failed. Please check the file system permissions. Error: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\ExchangeDatabases\DB01\db01.log'.

Log Name:      Microsoft-Exchange-HighAvailability/Seeding
Source:        Microsoft-Exchange-HighAvailability
Date:          3/09/2014 12:10:17 AM
Event ID:      826
Task Category: Seeding Target
Level:         Information
Keywords:      
User:          SYSTEM
Computer:      MELEX1.exchange2013demo.com
Description:
DB Seeding has completed for the local copy of database 'DB01' (1b3363f6-7f82-41ca-953b-2c295c1896a9).

If the process was not successful after 5 attempts, it stops.

After 3 days, if the database copies are still “FailedAndSuspended”, the workflow begins again.

Summary

As you can see Autoreseed is quite intelligent and effective, resolving a straight-forward issue like storage failure with no manual intervention by the administrator except for replacing the failed disk with a new spare.

Just how good is Autoreseed?

In my test lab I tend to treat my servers pretty rough. To test Autoreseed I would regularly open up Server Manager on a DAG member and offline one of the volumes hosting database copies. Then I would go away and do something else for an hour or two.

Every single time Autoreseed successfully restored the resiliency of my DAG. Looking at the event logs it typically achieves this in a little over an hour. In the real world if there are delays or retries on some of the Autoreseed workflow steps, or the databases are larger and take longer to reseed, then it may take longer to recovery but I would have full confidence that it would work.

Autoreseed is a feature of a highly intelligent server application that is designed to run efficiently at scale. As with many features in Exchange Server 2013 to take full advantage of Autoreseed you design for *simpler* DAGs. This is counter-intuitive for some people who are used to adding complexity to their designs to make them more resilient.

But as you can see, by getting the right foundations in place you can easily to take advantage of the benefits of Autoreseed in your deployment.

This article Exchange Server 2013 Autoreseed in Action is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

A reader asks whether it is possible to run a PowerShell command to list the active database copies in a database availability group, and show the mailbox server that they are active on.

He’s already discovered the Get-MailboxDatabaseCopyStatus cmdlet, but the default output does not include the specific information he wants to see.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | ft -auto
Name            Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime   ContentIndexState
----            ------  --------------- ----------------- --------------------   -----------------
DB01\EX2013SRV1 Mounted 0               0                                        Healthy
DB02\EX2013SRV1 Healthy 0               0                 15/10/2014 10:31:16 AM Healthy
DB03\EX2013SRV1 Mounted 0               0                                        Healthy
DB04\EX2013SRV1 Mounted 0               0                                        Healthy
DB02\EX2013SRV2 Mounted 0               0                                        Healthy
DB01\EX2013SRV2 Healthy 0               0                 15/10/2014 10:25:33 AM Healthy
DB03\EX2013SRV2 Healthy 0               0                 15/10/2014 10:25:30 AM Healthy
DB04\EX2013SRV2 Healthy 0               0                 15/10/2014 10:27:34 AM Healthy

The information he is looking for can be retrieved using Get-MailboxDatabaseCopyStatus, we just need to use Select-Object to return different attributes than the default output displays.

Let’s say you wanted to see:

• Active mailbox database copies only, and their status for good measure
• The mailbox server they are active on (even though this is already in the name of the DB copy we’ll grab it separately as well)
• The activation preference of that database copy
• The content index state

To see all of that we can run this command:

[PS] C:\>Get-MailboxDatabaseCopyStatus * -Active | Select Name,Status,MailboxServer,ActivationPreference,ContentIndexState
Name             Status MailboxServer ActivationPreference ContentIndexState
----             ------ ------------- -------------------- -----------------
DB01\EX2013SRV1 Mounted EX2013SRV1                       1           Healthy
DB03\EX2013SRV1 Mounted EX2013SRV1                       1           Healthy
DB04\EX2013SRV1 Mounted EX2013SRV1                       1           Healthy
DB02\EX2013SRV2 Mounted EX2013SRV2                       1           Healthy

If you’re curious about other attributes that can also be viewed simply use Get-Member to list them all.

[PS] C:\>Get-MailboxDatabaseCopyStatus | Get-Member

This article PowerShell Tip: List Active Mailbox Database Copies for an Exchange Server Database Availability Group is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

I recently encountered an Exchange 2010 server that was unable to send emails to a small number of domains. The messages would queue on the outbound transport server, and a look at the queue status would return an error of “451 4.4.0 DNS query failed”.

Interestingly the domain name was resolving just fine in DNS when queried from the server, including for MX record lookups, and a telnet to port 25 was successful too. When I created a send connector just for that namespace and designated their MX as the smart host IP address delivery was successful. It just would not work if DNS lookups were being used.

There are many reports of this error on forums such as TechNet but no concrete explanations for the causes or how to solve it. I saw many references to disabling IPv6 on the server, but also follow up reports that this did not work.

In the end the solution was to enable the option to force the send connector to use the external DNS settings for the transport server.

Even though there was nothing unusual about the external DNS settings on that server. They were simply set to use the DNS settings of the network interface, which worked fine for NSLookup.

After changing the send connector config and restarting the MSExchangeTransport service the mail delivered successfully to the troublesome domains.

It’s possible this is a bug with the particular combination of Exchange 2010 and Windows Server 2008 R2 that the customer is running, although I could not find anything that specifically confirms this. It is also not an issue I’ve encountered elsewhere with this combination of server versions, nor can I reproduce it in a test lab. But if you do encounter this issue the above solution seems to work fine and is certainly very quick and easy to try.

This article Messages Queuing with Error 451 4.4.0 DNS Query Failed is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

One of my minor gripes with Exchange Server 2013 is the lack of configuration that can be performed during setup.

If you look at the TechNet documentation for unattended setup you can see a few options are available but they do not cover the range of configurations required for a new Exchange 2013 server.

Consider a scenario where a new Exchange 2013 server is being introduced into a site with existing servers. The server is going to participate in load balanced Client Access traffic, which means a variety of HTTPS workloads such as OWA, ActiveSync, and EWS, as well as potentially SMTP relay for internal application servers.

The configurations that you would need to initially apply could include:

• SSL certificates
• Virtual directory URLs and settings
• Receive connectors
• Message tracking log configurations
• Protocol log configurations

100% of those items can be scripted with PowerShell. You would need to invest some time and effort but it is certainly achievable, and your deployment process will be better for it.

The next challenge is maintaining that configuration. Many Exchange admins would be familiar with situations where some small number of the end user population is having a problem with OWA, and it turns out that one OWA virtual directory among your Client Access servers has a slightly different configuration than the others. Or a situation where SMTP connections from application servers sometimes fail, and it turns out that the remote IP ranges defined on the receive connectors of your load balanced servers are slightly different.

Again, you could write a bunch of PowerShell and implement automation and workflows that periodically run across your servers and correct any mis-configurations. That is 100% achievable and just needs time and effort applied.

But home-brewing your own configuration management solution is, quite frankly, a gigantic pain. I’ve been watching developments in the world of configuration management via the likes of Chef, Puppet, and of course PowerShell DSC with a lot of interest.

So it is with great excitement that I read the news that Microsoft has developed an Exchange module for PowerShell DSC.

The “xExchange” module has 24 DSC resources in it right now. Already they are ticking most of the boxes on my wishlist of configuration management items, such as SSL certificates, virtual directory configurations, receive connectors, and database availability groups.

It looks like Mike Hendrickson is planning a series of articles that explore the xExchange module for PowerShell DSC in a lot more detail, which will be very interesting to read.

This article Exchange Server 2013 Configuration Management with PowerShell DSC is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

One of the tools that Microsoft has provided to the community for Office 365 migrations is the Exchange2007MBtoMEU.ps1 PowerShell script.

If you’ve completed a staged Exchange migration to migrate your organization’s Exchange 2007 on-premises mailboxes to Office 365 and you want to manage cloud-based users from your on-premises organization—using Active Directory—you should convert the on-premises mailboxes to mail-enabled users (MEUs).

While using the script on a project I considered that it would be helpful if the script logged the changes that it made when it was converting the mailbox users to mail-enabled users.

With Microsoft’s permission I am publishing my modified version of the script. It functions the same way as the original, so you should follow Microsoft’s guidance here.

The only significant difference is the log file that is produced by my version of the script. It is simply written to the same folder where the script is running from, and basically contains the same information that the script outputs to the console while it is running.

Download Ex2007MBtoMEU.ps1-Customized from Github.

This article Customized Exchange2007MBtoMEU.ps1 Script for Office 365 Migrations is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

Just a quick note to let you know that I’m on the latest episode of RunAs Radio talking about Exchange Server 2013 migration and high availability. Richard Campbell and I chat for about half an hour which is hardly enough time to cover the entire topic, but I hope you pick up a few useful tips from it. Listen to the end for an announcement about what’s coming soon here on Exchange Server Pro.

You can grab the episode from the RunAs Radio website or iTunes.

RunAs Radio is one of my favorite tech podcasts, I listen to every episode and learn a lot from it. So it was a great honor to be on the show and I had a lot of fun doing it. Thanks again to Richard for the invitation.

This article Discussing Exchange Server 2013 High Availability on RunAs Radio is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

This article is an excerpt from the Deploying and Managing Exchange Server 2013 High Availability ebook.

A lagged database copy is a passive database copy in a database availability group that has a delayed log replay time configured.

Normally a passive database copy will replay the transaction log data into the database immediately, so that the passive database copy is as up to date as possible.

With a lagged database copy the administrator sets a delay on the log replay, so that the database copy “lags” behind the others in terms of the latest database changes. This lag interval specifies the amount of time between when a transaction log file is generated and when it is replayed into the passive database copy. The default lag interval is 0 and the maximum lag interval is 14 days.

The purpose of a lagged copy is to provide the ability to recover the database from an earlier point in time if some kind of database fault occurs, such as logical corruption.

Configuring Lagged Database Copies

The replay lag interval for a lagged database copy is configured using Set-MailboxDatabaseCopy, and the value is in the format “days.hours:minutes:seconds”. For example, a value of “7.0:0:0” means 7 days.

It is common to choose the least preferred database copy to be the lagged copy, so in an environment with four database copies the copy with activation preference of 4 would be set as the lagged copy.

To select all database copies that have an activation preference of 4 we can use Get-MailboxDatabaseCopyStatus.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ActivationPreference -eq "4"}
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime   ContentIndexState
----        ------  --------------- ----------------- --------------------   -----------------
DB02\SYDEX1 Healthy 0               0                 23/09/2014 10:53:37 PM Healthy
DB03\SYDEX2 Healthy 0               0                 23/09/2014 10:45:59 PM Healthy
DB04\MELEX1 Healthy 0               0                 23/09/2014 10:47:08 PM Healthy
DB01\MELEX2 Healthy 0               0                 23/09/2014 10:45:02 PM Healthy

Piping that output to Set-MailboxDatabaseCopy will configure the replay lag interval.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ActivationPreference -eq "4"} | Set-MailboxDatabaseCopy -ReplayLagTime 7.0:0:0

Real World: When you set the replay lag time you will see a warning about the SafetyNetHoldTime as well. It is always recommended to set Safety Net hold time to the same value or greater value than the replay lag time.

In addition to setting the replay lag time you can also prevent a lagged database copy from being automatically activated during a database failover scenario. To block automatic activation for a database copy use the Suspend-MailboxDatabaseCopy cmdlet.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ActivationPreference -eq "4"} | Suspend-MailboxDatabaseCopy -ActivationOnly

When automatic activation has been blocked in this manner the database copy can only become active after manual intervention by an administrator.

Real World: Blocking automatic activation for a database copy reduces the number of database copies that are available for Active Manager to attempt to mount during the Best Copy and Server Selection process. It is a decision for your organization whether the preservation of the lagged database copy by blocking automatic activation is more important than having the maximum number of available database copies available to mount in a failover scenario.

Using Lagged Database Copies in Recovery Scenarios

Lagged copies can be used for recovery in a variety of ways.

• Activate a lagged database copy by replaying all uncommitted log files. In this scenario a decision is made to replay all of the log files in the replay queue into the database and bring it online.
• Activate a lagged database copy to a specific point in time. In this scenario some of the log files are removed from the server to prevent them from being replayed into the database copy as it is brought online.
• Activate a lagged database copy and use Safety Net for recovering lost data. In this scenario the database is mounted without replaying transaction log from the replay queue, and messages are resubmitted to the database from Safety Net.

In each of these scenarios there is an optional (but recommended) step to make a copy of the lagged database files before activating it, so that you are still able to use the lagged copy at a later stage if you have a different recovery scenario arise.

Activating a Lagged Database Copy by Replaying All Log Files

A lagged database copy can be activated by replaying all of the uncommitted transaction log files (those that are in the replay queue) into the database and then mounting it. This effectively “catches up” the database copy to the others.

The amount of time it takes to commit all of the outstanding transaction log files to the database will depend on the length of the lag interval and the amount of transaction log data that has been generated in that space of time.

First, identify the lagged copy of the database. In this example of database DB01 the lagged copy is DB01\MELEX2.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB01 | Where {$_.ReplayLagStatus.Enabled -eq $true}
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime   ContentIndexState
----        ------  --------------- ----------------- --------------------   -----------------
DB01\MELEX2 Healthy 0               361               26/09/2014 11:46:31 PM Healthy

Suspend the lagged database copy using Suspend-MailboxDatabaseCopy.

[PS] C:\>Suspend-MailboxDatabaseCopy DB01\MELEX2

Make a backup of the database and log files for this database copy. You can use a backup application or simply copy them to another location.

A suspended database copy can’t be made active, so we need to resume the lagged database copy again. It may take a few minutes afterwards for the content index to return to a healthy state as well.

[PS] C:\>Resume-MailboxDatabaseCopy DB01\MELEX2

Finally, activate the lagged database copy. Note that this uses the same Move-ActiveMailboxDatabase cmdlet as a normal database switchover, with the additional –SkipLagChecks switch.

[PS] C:\>Move-ActiveMailboxDatabase DB01 –ActivateOnServer MELEX2 -SkipLagChecks

When all of the log files have been replayed the database copy will mount and become the active database copy.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB01\MELEX2
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime ContentIndexState
----        ------  --------------- ----------------- -------------------- -----------------
DB01\MELEX2 Mounted 0               0                                      Healthy

The database copy still has the same replay lag interval configure, it is just not in effect as long as the database copy is active. If the active database copy is moved to another DAG member then the replay lag interval will come into effect again and the replay queue length will begin increasing for the lagged database copy.

Activating a Lagged Database Copy to a Point in Time

Activating a lagged database copy to a specific point in time follows a different process, and is often used as a way to make a copy of the database to mount as a recovery database, rather than activate the lagged copy itself.

First, identify the lagged copy of the database. In this example of database DB02 the lagged copy is DB02\SYDEX1, so this procedure should be performed on server SYDEX1.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB02 | Where {$_.ReplayLagStatus.Enabled -eq $true} | ft -auto
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime   ContentIndexState
----        ------  --------------- ----------------- --------------------   -----------------
DB02\SYDEX1 Healthy 0               391               27/09/2014 12:16:08 AM Healthy

Suspend the lagged database copy using Suspend-MailboxDatabaseCopy.

[PS] C:\>Suspend-MailboxDatabaseCopy DB02\SYDEX1

Make a backup of the database and log files for this database copy. You can use a backup application or simply copy them to another location.

Look at the log file timestamps to determine which log files were created after the point in time you want to recover to.

Move the log files after the time that you’re recovering to into another folder. Don’t delete them, just in case you need them, but they do need to be moved to another folder.

Delete the .chk file from the transaction log folder.

Using ESEUtil in a CMD prompt we can see that the database file in a dirty shutdown state.

C:\>eseutil /mh C:\ExchangeDatabases\DB02\DB02.db\DB02.edb
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
         Database: C:\ExchangeDatabases\DB02\DB02.db\DB02.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0xe4311492
  Actual Checksum: 0xe4311492
Fields:
            State: Dirty Shutdown

So the next step is to perform a soft recovery using the log files that we’ve left in the log folder. Navigate to the log folder for the database and run ESEUtil with the /r switch and the log prefix (E00 in this example).

C:\>cd ExchangeDatabases\db02\DB02.log
C:\ExchangeDatabases\DB02\DB02.log>eseutil /r e00 /a
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating RECOVERY mode...
    Logfile base name: e00
            Log files: 
         System files: 
Performing soft recovery...
                      Restore Status (% complete)
          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................
Operation completed successfully in 31.859 seconds.

Check the database file again to verify it is in a clean shutdown state.

C:\>eseutil /mh C:\ExchangeDatabases\DB02\DB02.db\DB02.edb
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
         Database: C:\ExchangeDatabases\DB02\DB02.db\DB02.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0xf3f5ae3e
  Actual Checksum: 0xf3f5ae3e
Fields:
            State: Clean Shutdown

The database file can now be copied to another location and used as a recovery database for a restore operation.

The lagged database copy itself can be resumed as well. If you wanted to return it to the lagged state before the recovery process above was run you would need to restore the files you backed up at the start of this process before resuming the database copy.

[PS] C:\>Resume-MailboxDatabaseCopy DB02\SYDEX1

Activating a Lagged Database Copy Using Safety Net Recovery

Activating a lagged database copy using Safety Net recovery involves removing all unnecessary transaction log files from the log folder, so that only the bare minimum required for mounting the database still exist.

First, identify the lagged copy of the database. In this example of database DB03 the lagged copy is DB02\SYDEX2, so this procedure should be performed on the server SYDEX2.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB03 | Where {$_.ReplayLagStatus.Enabled -eq $true} |
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime  ContentIndexState
----        ------  --------------- ----------------- --------------------  -----------------
DB03\SYDEX2 Healthy 0               379               27/09/2014 1:11:18 AM Healthy

Suspend the lagged database copy using Suspend-MailboxDatabaseCopy.

[PS] C:\>Suspend-MailboxDatabaseCopy DB03\SYDEX2

Make a backup of the database and log files for this database copy. You can use a backup application or simply copy them to another location.

Use ESEUtil to determine which log files are required by the database.

C:\>eseutil /mh C:\ExchangeDatabases\DB03\DB03.db\DB03.edb
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
         Database: C:\ExchangeDatabases\DB03\DB03.db\DB03.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x62a178a2
  Actual Checksum: 0x62a178a2
Fields:
     Log Required: 5722-5728 (0x165a-0x1660)

There are two hexadecimal values displayed in the ESEUtil output. In this example they are “0x165a” and “0×1660”. It is the second value that is of interest, because this is the “High Generation” number.

Move all log files that have a sequence number that is higher than the “High Generation” number. For example, 0×1660 means log files E0100001661 and above should be removed from the server hosting the lagged copy, which is SYDEX2. Don’t delete the files, just move them to a temporary folder in case you need them again.

Note: Transaction log file sequence numbers use a hexadecimal numbering sequence. This means that, for example, log file E0100001699 is not followed by E0100001700, but rather E01000016A0 instead. The last log file in the E01000016xx range is E01000016FF, and then E0100001700. This is important to be aware of because sorting log files by name in Windows Explorer does not correctly sort them by log file sequence number. If you are not aware of this, and do not remove all of the correct log files, then the switchover to the lagged database copy will fail.

Next, identify the server hosting the active copy of the database. In this example it is the server SYDEX1.

[PS] C:\>Get-MailboxDatabaseCopyStatus DB03 -Active | ft -auto
Name        Status  CopyQueueLength ReplayQueueLength LastInspectedLogTime ContentIndexState
----        ------  --------------- ----------------- -------------------- -----------------
DB03\SYDEX1 Mounted 0               0                                      Healthy

On the server hosting the active copy of the database stop the Microsoft Exchange Replication service.

Perform a database switchover to activate the lagged database copy, in this example the server is SYDEX2.

[PS] C:\>Move-ActiveMailboxDatabase DB03 –ActivateOnServer SYDEX2 –MountDialOverride BestEffort –SkipActiveCopyChecks –SkipClientExperienceChecks –SkipHealthChecks -SkipLagChecks

The database will mount and make a request to Safety Net for resubmission of the mail that is missing from the database.

After Lagged Copy Activation

Activating a lagged database copy is not the end of the exercise. There are still several things that you should do to ensure that your environment is back to a healthy state. After all, some event occurred to cause you to activate a lagged copy, so you most likely still have work ahead of you to restore your environment to its original healthy condition.

Here are some steps that you should consider:

• Use Get-MailboxDatabaseCopyStatus to review the health of your database copies
• Re-seed any database copies that are failed or otherwise considered to be unusable (eg, if the reason you activated a lagged copy was due to corruption in your database)
• Move the active database copy back to a more preferred server
• Re-apply or verify that your lagged copy configuration is in effect

This article is an excerpt from the Deploying and Managing Exchange Server 2013 High Availability ebook.

This article Exchange Server 2013 Lagged Database Copies In Action is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

In a comment on my recent article on Exchange Server 2013 lagged database copies Chris wrote:

I do not see any advantage in a lagged copy of a database. If any kind of corruption happens on a database we still have daily backup of the active DB which we can “activate” very shortly.

With lagged copy of a database, you still need a passive copy for high availability, you will end up with two copies of each database + the backup the active one . For me it is a waste of storage for what it brings.

This is a point of view worth considering. Certainly for some organizations lagged database copies won’t add much value. Either the scenarios they are designed to protect against are not considered to be a high impact event, or the risks are mitigated in other ways (such as traditional backups).

For other organizations the logistics of restoring data from traditional backups make lagged copies more attractive, as the data is already available and within the control of the Exchange administrators without needing to engage with other teams or third parties.

And for organizations where traditional backups aren’t used (which includes the Exchange Online service within Office 365) lagged database copies form a part of the Native Data Protection capability of Exchange Server 2013, which eliminates traditional backups.

What do you think, are lagged database copies a waste? Or do you see them as a valuable feature that you already use (or want to use) in your environment?

This article Are Lagged Database Copies a Waste of Time? is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

High availability for Exchange Server 2013 is a complex, challenging area with many important technical details that are often difficult to discover on your own. So it’s no surprise that high availability is a topic that generates more questions in my inbox and on Twitter than any other, and that the high availability articles on this website receive some of the highest visitor numbers.

Earlier this year I teamed up with fellow Exchange MVPs Michael Van Horenbeeck and Steve Goodman to produce an ebook that helps to fill in the gaps and teach people practical skills they can apply to the design, deployment and ongoing administration of highly available Exchange Server 2013 environments.

Today I’m pleased to announce the availability of our new ebook, Deploying and Managing Exchange Server 2013 High Availability.

In this 250+ page ebook you’ll learn about:

• Client Access HA topics such as namespace planning, certificate management, and load balancing
• Mailbox HA topics such as database availability groups, site resilience, lagged copies, and autoreseed
• Transport features and how they protect mail in transit from being lost
• Managed Availability and how to use and interpret the health data it presents

Plus a whole lot more.

You can find out more about Deploying and Managing Exchange Server 2013 High Availability here.

We worked hard to write this ebook and to celebrate the launch we are offering a 20% discount for a limited time. Click here for more details.

This article Deploying and Managing Exchange Server 2013 High Availability is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com

     

One of the topics I receive a lot of questions about is Datacenter Activation Coordination Mode, or DAC Mode for short. Here is an excerpt from Deploying and Managing Exchange Server 2013 High Availability that covers this topic in more detail.

Datacenter Activation Coordination (DAC) Mode is a property of DAGs that is designed to prevent split brain conditions from occurring by enabling a protocol called Datacenter Activation Coordination Protocol (DACP).

In addition, DAC Mode enables the use of three PowerShell cmdlets for site-resilience:

• Stop-DatabaseAvailabilityGroup
• Restore-DatabaseAvailabilityGroup
• Start-DatabaseAvailabilityGroup

Without those cmdlets any datacenter switchover or failover scenario involves using other combinations of Exchange and cluster management tools. These site resilience cmdlets make datacenter switchovers and failovers much easier to manage.

A split brain condition can occur in a multi-site DAG when one datacenter goes offline entirely. It can also occur in a single-site DAG in some network failure situations. Let’s take a look at an example of a multi-site failure where the benefits of DAC mode become clear.

In this example the Sydney and Melbourne datacenters each host two DAG members, with Sydney also hosting the file share witness server. To keep this example simple a single database exists in the DAG, currently active on a Sydney DAG member.

The Sydney datacenter has a power failure that takes the entire site offline. With two DAG members and the FSW offline in Sydney, and just two DAG members online in Melbourne, quorum can’t be maintained and the database goes offline.

The administrators activate the alternate file share witness in Melbourne to restore quorum, and bring the database online in Melbourne to restore service.

Eventually the datacenter in Sydney has power restored and the Sydney DAG members and file share witness come back online. However, the WAN connection remains offline, preventing the DAG members in each site from communicating with each other.

The two Sydney DAG members and file share witness have enough votes to achieve quorum, so the database is brought online in Sydney.

At this stage the problem should be apparent. Both Sydney and Melbourne have an active copy of the same database because the DAG members in each site were not able to communicate with each other. A split brain condition has occurred.

DAC and DACP prevent this behavior by requiring a DAG member to check with other DAG members before it is allowed to bring database online.

DACP exists as a bit (a 0 or 1) that is stored in memory. When DAC mode is enabled each DAG member starts up with a DACP bit of 0. Until it can communicate with a DAG member that has a DACP bit of 1, or alternatively it can communicate with every other member of the DAG, it will not attempt to activate its database copies even if it can achieve quorum with some of the DAG members.

To demonstrate this let’s go back in the example scenario above to the stage where the Sydney datacenter was coming back online again.

When DAC Mode has been configured in advance the Sydney DAG members start up with a DACP bit of 0 and are unable to communicate with the Melbourne DAG members because the WAN link is still offline.

Therefore they do not bring the database online in Sydney, preventing a split brain condition.

When the WAN connection is restored the Sydney DAG members are able to communicate with the Melbourne DAG members. Their DACP bit is set from 0 to 1 and, because they now realize that the database is already active in Melbourne, their database copies become passive copies.

For more on DAC mode and other features of database availability groups check out the Deploying and Managing Exchange Server 2013 High Availability.

This article Datacenter Activation Coordination Mode is © 2014 ExchangeServerPro.com

Get more Exchange Server tips at ExchangeServerPro.com