Error Generating the Exchange 2007 OAB

After setting up a CCR cluster at a client site, I attempted to generate the OAB but found two errors that didn't quite add up:

Event Type: Error
Event Source: MSExchangeSA
Event Category: OAL Generator Event
ID: 9334
Date: 5/15/2008
Time: 11:00:33 AM
User: N/A
Computer:
Description:OALGen encountered error ffffffff while initializing the offline address list generation process. No offline address lists have been generated. Please check the event log for more information. - /o=/cn=addrlists/cn=oabs/cn=Exxchange 2007 OAB
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Event Type: Warning
Event Source: MSExchangeSA
Event Category: OAL Generator Event
ID: 9395
Date: 5/15/2008
Time: 11:00:33 AM
User: N/A
Computer:
Description: OALGen is running on a cluster continuous replication (CCR) node which does not have registry value 'SYSTEM\CurrentControlSet\Services\MSExchangeSA\Parameters\METROEXMAIL\EnableOabGenOnThisNode' or it is not set to this node name. Offline address book generation will not be performed.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The latter event was self-explanatory and easily resovleable. However, the first error was puzzling. I eventually found an article on DGoldman's site where he explained the permissions required for OAB generation:
http://blogs.msdn.com/dgoldman/archive/2007/02/01/exchange-2007-oab-generation-fails-with-errors-9348-and-9109.aspx

The only way around this problem, even though the event wasn't exactly the same as the ones quoted by DGoldman, was to re-apply the permissions on the objects in AD that were used by users and Exchange to access and create the OAB.

Here's what I did:

Set a variable called "$container" to contain a path to the Offline Address List object in Active Directory.

[PS] C:\>$container="CN=Offline Address List,CN=Offline Address Lists,CN=Address Lists Container,CN=,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=,DC=com"

[PS] C:\>Add-ADPermission $container -User "Authenticated Users" -AccessRights GenericRead, ListChildren -ExtendedRights Open-Address-Book

WARNING: Appropriate ACE is already present on object "CN=Offline Address List,CN=Offline Address Lists,CN=Address Lists Container,CN=,CN=MicrosoftExchange,CN=Services,CN=Configuration,DC=,DC=com" for account "NT AUTHORITY\Authenticated Users".

Identity User Deny Inherited Rights
-------- ---- ---- --------- ------
\Offline Address ... NT AUTHORITY\Auth... False False Open-Address-Book
\Offline Address ... NT AUTHORITY\Auth... False False ReadProperty
\Offline Address ... NT AUTHORITY\Auth... False False ListObject, GenericExecute
\Offline Address ... NT AUTHORITY\Auth... False False ListChildren

[PS] C:\>Add-ADPermission $container -User "Exchange Servers" -AccessRights GenericRead -ExtendedRights Open-Address-Book

Identity User Deny Inherited Rights
-------- ---- ---- --------- ------
\Offline Address ... \Exchang... False False Open-Address-Book
\Offline Address ... \Exchang... False False ReadProperty
\Offline Address ... \Exchang... False False ListObject, GenericExecute

[PS] C:\>Add-ADPermission $container -User System -AccessRights GenericAll

Identity User Deny Inherited Rights
-------- ---- ---- --------- ------
\Offline Address ... \systemnrt False False GenericAll

All the above commands do is set the required permissions for the process to work. If the OAB is published to a CAS server, it may be wise to stop and start the MSExchangeFDS (File Distribution Service) service and look for the following event in the event log:

Event Type: Information
Event Source: MSExchangeFDSEvent
Category: FileReplication
Event ID: 1008
Date: 5/15/2008
Time: 2:04:40 PM
User: N/A
Computer:
Description: Process MSExchangeFDS.exe (PID=25532). Offline Address Book data synchronization task has completed successfully. OAB name: "Exchange 2007 OAB", Guid: 8f7d2fed-187a-4246-8f1a-09e55171ed51
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

This event will confirm that replication of the OAB files is occurring.

Exchange 2007 Troubleshooting: Don't know where to start?

Exchange 2007 suppresses a host of rich information from appearing in the Event Viewer. When troubleshooting a particular Exchange 2007 component, increasing the EventLogLevel for the particular component can save a lot of time and heartbreak. To view the currently set EventLogLevel for any Exchange 2007 component, launch Exchange Shell and type the following command:

[PS] C:\>Get-EventLogLevel

You should see a list of components and their current logging level. Below is a sample of the output.

Identity EventLevel
-------- --------
MSExchange Assistants\Assistants Lowest
MSExchange Cluster\Move Lowest
MSExchange Cluster\Upgrade Lowest
MSExchange Cluster\Action Lowest
MSExchange Common\General Lowest
MSExchange Common\Configuration Lowest
MSExchange Common\Logging Lowest
MSExchange Extensibility\Transport Address Book Lowest...

To increase the EventLogLevel for any component, type the following command for the component:

[PS] C:\>Set-EventLogLevel "MSExchangeSA\OAL Generator" -Level 5

The levels can be set to Lowest, Low, Medium, High and Expert or 1,2,3,4
Some events in the event log may ask you to set a higher level in order to view the exact contents of an error.

For more information on EventLogLevel, see:

http://technet.microsoft.com/en-us/library/aa998905(EXCHG.80).aspx

Exchange 2007 "Legacy Mailbox" after migration from Exchange 200x

It's been a long week but I thought I had to post on little migration issues that I am currently seeing. OWA seems to be full of cryptic error messages. With that said, If you read the errors very carefully, especially the parts highlighted in red in the error message sample below, you should get a pretty good idea why the request is failing. Here's one I saw this week:

SCENARIO:
Exchange 2007 was installed into an existing Exchange 2000 organization. All the mailboxes are being moved manually through the EMC 2007. However, after migrating some users, they attempt to login to OWA and they get the error below:

Request

Url: https://owa.company.com/owa/lang.owa

User host address: 10.x.x.x

Exception

Exception type: Microsoft.Exchange.Data.Storage.StoragePermanentException

Exception message: There was a problem accessing Active Directory.

Call stack

Microsoft.Exchange.Data.Storage.ExchangePrincipal.Save() Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchLanguagePostLocally(OwaContext owaContext, OwaIdentity logonIdentity, CultureInfo culture, String timeZoneKeyName, Boolean isOptimized) Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchLanguagePostRequest(OwaContext owaContext) Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.PrepareRequestWithoutSession(OwaContext owaContext, UserContextCookie userContextCookie) Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.InternalDispatchRequest(OwaContext owaContext) Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchRequest(OwaContext owaContext) System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Inner Exception

Exception type: Microsoft.Exchange.Data.Directory.InvalidADObjectOperationExceptionException message: Property Languages cannot be set on this object because it requires the object to have version 0.1 (8.0.535.0) or later. Current version of the object is 0.0 (6.5.6500.0).

Call stack

Microsoft.Exchange.Data.Directory.PropertyBag.set_Item(PropertyDefinition key, Object value) Microsoft.Exchange.Data.Directory.ADObject.set_Item(PropertyDefinition propertyDefinition, Object value) Microsoft.Exchange.Data.Directory.ADObject.StampCachedCaculatedProperties(Boolean retireCachedValue) Microsoft.Exchange.Data.Directory.ADObject.ValidateWrite(List`1 errors) Microsoft.Exchange.Data.Directory.Recipient.ADRecipient.ValidateWrite(List`1 errors) Microsoft.Exchange.Data.Directory.Recipient.ADUser.ValidateWrite(List`1 errors) Microsoft.Exchange.Data.Directory.ADSession.Save(ADObject instanceToSave, IEnumerable`1 properties) Microsoft.Exchange.Data.Storage.ExchangePrincipal.Save()

PROBLEM:

The problem is that the ExchangeVersion attribute associated with the user object in active directory hasn't been set to Exchange 2007 yet.

RESOLUTION:

Open the Exchange 2007 PowerShell window and type the following commands:

[PS] C:\>Get-Mailbox UserName fl Exchangeversion

ExchangeVersion : 0.0 (6.5.6500.0)

[PS] C:\>Set-Mailbox UserName -ApplyMandatoryProperties

[PS] C:\>Get-Mailbox UserName fl Exchangeversion

ExchangeVersion : 0.1 (8.0.535.0)

These actions will update the user in the EMC interface to show up as "User Mailbox" and not "Legacy Mailbox".

For more information, see http://support.microsoft.com/kb/941146

Fix CCR replication using Update-StorageGroupCopy

During a recent migration, the log files on the Exchange 2007 server kept filling up too quickly. What engineers typically do not plan for during an Exchange 2007 migration are the amount of log files generated during the migration.

I've been caught by this issue before. So what's the cure? Plain and simple, run a full backup of the particular storage group to flush the logs.

Unfortunately, this isn't always possible since the storage group fails to continue replicating and the database dismounts because the log drive is full.

This means that you'll have to delete a few log files. In order to establish which log files are required for the database to come up and function properly, follow the instructions in this KB:
http://support.microsoft.com/default.aspx?kbid=240145

1. Dismount the database that generates the logs. (Suspend the replication first)

2. Run ESEUTIL /MH DatabaseFileName

3. Review the Log Required section.

4. If this is stated as 0-0, then all the logs can be deleted.

I usually backup (copy) the logs with the oldest time stamps to another location and then delete them. Once this is done, run the following command in powerShell on the passive node:

Update-StorageGroupCopy -Identity "VirtualNode\storage group" -DeleteExistingFiles

This command will delete all the existing log files on the CCR replica server and restart the CCR replication and mount the database.