File level Anti-Virus Exclusions for Exchange Server 2013

Many of you might have called Microsoft PSS in past and would have heard them asking you to set the AV exclusions on the exchange server. Several customers get alarmed with an idea of excluding folders and files from AV scanning. Although your company’s security officer may not like the idea, it is essential that you exclude some files and folders from AV scanning when running an exchange server installation on Windows to avoid lot of unforeseen performance and content conversion related issues.

You may ask, why  is it important to set exclusions? As a general practice several organizations deploy file level scanning on the server systems where exchange is installed. To perform a scan of a file an AV software definitely needs to put an handle on the target file. Imagine a case when an AV software has locked a huge database file that is also required to be used by information store. In this case, the amount of time taken by AV to perform a full scan of the file is certainly higher than what an information store thread can wait for. In another case processes are also sometimes locked by file level scanners. Executable files being one of the most common medium for viruses to spread across, AV pays a special attention to the .exe file types. If an executable that is supposed to work as an image for a service is locked by AV, the relevant service may also fail to start causing downtime of that service component on the server. It becomes extremely important to configure exclusion in the file level antivirus software to avoid any known or unknown issues in future.

Although exclusions are good for the health and stability of exchange servers, I have seem some people excluding entire exchange installation directory. Which is not a good practice. When you configure exclusions, you should configure them correctly.

Alright, so what folders and files should I exclude from file level scanning? Great! Seems like you were able to make your security officer happy and get an approval for setting up exclusions on server systems. I personally had to struggle to get it approved by CISOs so far :-). If you were one of those lucky guys who got an approval immediately then below is the list of files and folders to be excluded.

On a Mailbox Role

File Extensions to be excluded:

.config
.dia
.wsb
.chk
.edb
.jrs
.jsl
.log
.que
.lzx
.ci
.dir
.wid
.000
.001
.002
.cfg
.grxml
.dsc
.txt

Folder Exclusions

Type Default Location
Mailbox Database Folder %ExchangeInstallPath%\Mailbox
Log Files Folder %ExchangeInstallPath%\Mailbox
Checkpoint Files Folder %ExchangeInstallPath%\Mailbox
OAB Folder %ExchangeInstallPath%\ClientAccess\OAB
Group Metrics Folder Under %ExchangeInstallPath%\GroupMetrics
IIS System Files %SystemRoot%\System32\Inetsrv
Mailbox Database Temp Folder %ExchangeInstallPath%Mailbox\MDBTEMP
DAG FSW Folder %SystemDrive%:\DAGFileShareWitnesses\<DAGFQDN> (This folder is not hosted on the DAG members)
Cluster Quorum Database %Windir%\Cluster
Message Tracking Logs %ExchangeInstallPath%\TransportRoles\Logs
Tracing Logs %ExchangeInstallPath%\TransportRoles\Logs
Pickup and Directory %ExchangeInstallPath%\TransportRoles
Queue databases and Logs %ExchangeInstallPath%\TransportRoles\Data\Queue
Sender Reputation Files %ExchangeInstallPath%\TransportRoles\Data\SenderReputation
Content Conversion Temp Files %SystemRoot%\TEMP
OLE Content conversion folder %ExchangeInstallPath\%Working\OleConverter
Content Scanning folder %ExchangeInstallPath%\FIP-FS
Connectivity Logs %ExchangeInstallPath%\TransportRoles\Logs\Mailbox
Grammer Files %ExchangeInstallPath%\UnifiedMessaging\grammars
Voice Promots folder %ExchangeInstallPath%\UnifiedMessaging\Prompts
Voicemail files location %ExchangeInstallPath%\UnifiedMessaging\voicemail
Temp files for UM %ExchangeInstallPath%\UnifiedMessaging\temp

On a CAS Server

Folders Exclusion

Type Default Location
IIS File System %SystemRoot%\System32\Inetsrv
IIS Logs Inetpub\logs\logfiles\w3svc
IMAP4 Protocol Logs %ExchangeInstallPath%\Logging\POP3
POP3 Protocol Logs %ExchangeInstallPath%\Logging\POP4
Front End Transport Logs %ExchangeInstallPath%\TransportRoles\Logs\FrontEnd

In addition to above exclusions you should also exclude below process from scanning depending upon what server role you are excluding them on. Although it is not a mandate to do exclude the processes from scanning, some file level antivirus programs support process scanning as well. If your AV program is one them, it can cause adverse effects on exchange services.

Cdb.exe Microsoft.Exchange.Pop3service.exe MSExchangeRepl.exe
Cidaemon.exe Microsoft.Exchange.ProtectedServiceHost.exe MSExchangeSubmission.exe
Clussvc.exe Microsoft.Exchange.RPCClientAccess.Service.exe MSExchangeTransport.exe
Dsamain.exe Microsoft.Exchange.Search.Service.exe MSExchangeTransportLogSearch.exe
EdgeCredentialSvc.exe Microsoft.Exchange.Servicehost.exe MSExchangeThrottling.exe
EdgeTransport.exe Microsoft.Exchange.Store.Service.exe Msftefd.exe
ExFBA.exe Microsoft.Exchange.Store.Worker.exe Msftesql.exe
hostcontrollerservice.exe Microsoft.Exchange.TransportSyncManagerSvc.exe OleConverter.exe
Inetinfo.exe Microsoft.Exchange.UM.CallRouter.exe Powershell.exe
Microsoft.Exchange.AntispamUpdateSvc.exe MSExchangeDagMgmt.exe ScanEngineTest.exe
Microsoft.Exchange.ContentFilter.Wrapper.exe MSExchangeDelivery.exe ScanningProcess.exe
Microsoft.Exchange.Diagnostics.Service.exe MSExchangeFrontendTransport.exe TranscodingService.exe
Microsoft.Exchange.Directory.TopologyService.exe MSExchangeHMHost.exe UmService.exe
Microsoft.Exchange.EdgeSyncSvc.exe MSExchangeHMWorker.exe UmWorkerProcess.exe
Microsoft.Exchange.Imap4.exe MSExchangeLESearchWorker.exe UpdateService.exe
Microsoft.Exchange.Imap4service.exe MSExchangeMailboxAssistants.exe W3wp.exe
Microsoft.Exchange.Monitoring.exe MSExchangeMailboxReplication.exe  
Microsoft.Exchange.Pop3.exe MSExchangeMigrationWorkflow.exe  

Most of these processes can be found under the Bin directory of exchange server installation folder and some of them reside inside the sub folders. If you are not sure of what path is set for the folders to be excluded in the folder exclusion table above, a quick run Get-TransportService, Get-MailboxDatabase,  Get-UMService can give you the paths you are looking for.

Register Filter Pack IFilters with Exchange 2013

IFilter provides an interface to the Microsoft search indexing services to allow indexing of documents, file metadata, emails, email attachments, and lot other. Unless an appropriate IFilter exists search service / engine cannot index the files relevant to a missing IFilter. Exchange uses IFilters for the similar reasons and also to let some transport rules work correctly. Some transport rules can perform phrase or word based jobs and to perform it better, they should be able to read through content of an attachment. IFilters

Exchange 2013 transport rules supports multiple document formats by default. OneNote and Office Publisher are not supported by the transport rules though. That means an email going through an exchange 2013 server which contains a Onenote notebook attached to it may bounce back due to AttachmentUnsupported condition. So, if you want to support onenote and publisher file formats on exchange 2013 transport rules you should register the additional IFilters manually on an Exchange 2013 Mailbox Server.

To register IFilters with Exchange 2013 you can follow below steps:

Download Microsoft Office 2010 Filter Pack and Service Pack 1 for Microsoft Office Filter Pack 2010 (KB2460041) 64-bit Edition and install them one after another.

1. Now logon to your Exchange 2013 Mailbox Server Role, Open registry editor and locate the key -  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\CLSID

If a key named CLSID does not exist create one manually as shown in below figures. Below steps will demonstrate how to add the CLSID and filter for OneNote files.

image

2. Right click at the location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\CLSID and create another new key named {B8D12492-CE0F-40AD-83EA-099A03D493F1} as show in below figure

image

3. Double click on the string value (Default) and set the path to where Office Filter Pack is installed. Default location of the filter pack is C:\Program Files\Common Files\Microsoft Shared\Filters

image

4. Right click at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\CLSID\{B8D12492-CE0F-40AD-83EA-099A03D493F1}  and create a new String Value and rename it to ThreadingModel and set its value to Both

image

5. Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\filters. Again, if filters key does not exist create it manually.

6. Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\filters create another key .one (please note a preceding dot)

7. Change the value of (Default) string value to {A7FD8AC9-7ABF-46FC-B70B-6A5E5EC9859A}

image

In last 7 steps we registered the IFilter for OneNote file types.

8. To add an IFilter for Publisher files add a key named .pub under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\HubTransportRole\filters and set the (Default) value to {A7FD8AC9-7ABF-46FC-B70B-6A5E5EC9859A}

image

Close the registry editor and restart following services on Exchange mailbox server.

  • Restart Microsoft Exchange Transport Service
  • Restart Microsoft Filtering Management Service

To register a non Microsoft IFilter you should know the class ID of the product, it can be easily located under HKEY_Classes_Root\CLSID by searching for the file extension you want o register or referring the IFilter documentation.

Description of files inside C:\Program Files\Common Files\Microsoft Shared\Filters

File Name Description
msgfilt.dll Microsoft Message IFilter
NLHTML.dll Microsoft HTML IFilter
odfflit.dll Microsoft Filter for Open Document Format
OFFFILT.dll Microsoft Office IFilter
offfiltx.dll Microsoft Office Open XML Format IFilter
ONIFILTER.dll Microsoft OneNote IFilter
PUBFLIT.dll Microsoft Publisher IFilter
VISFIT.dll Microsoft Office Visio IFilter

Have a great time until next post. Keep reading and keep exploring!  Smile

Inside the Exchange 2013 Single Namespace – Part 1

Database Availability Group (DAG) and Client Access Array (CAS Array) brought a huge value to Exchange 2010 as a product. Both technologies compliment each other to provide high availability and seamless client experience. Switch over or failover of mailbox databases or an entire server in previous versions to Exchange 2010 were always a nightmare for the administrators to maintain the client experience. Exchange 2010 does a magnificent job to reconnecting a disconnected client back in 30-45 seconds. This is possible only because of the CAS array which works as an entry point to outlook and other native exchange clients.

Although DAG and CAS array together were an extremely successful model,  they posed a complexity of operation in a multi datacenter environment. To manage a site switchover or failover; an administrator still needs to consider a lot of factors so that the clients do not experience downtime or disconnections. In an event where entire datacenter fails, an administrator must change several things in the configuration including DAG, mailbox database properties, CAS arrays, DNS, load balancing appliances etc. I have personally seen some disaster recovery drills performed at several customer places and the effort involved to get everything working could not be reduced lesser than 15 minutes. In all cases, external clients were the most affected ones. A disconnected executive user needing to send an urgent email containing some business details to partners can be even worse.

Most of all dissatisfying experiences are due to the way CAS arrays work with DAG.

I must admit how detailed study the Microsoft Exchange Team has done to overcome the challenges faced by customers (although the primary driving factor was to make office 365 a huge success 🙂 ). Exchange 2013 overcomes the site switchover or failover challenges by introducing something new called “Single Namespace”. I am not really going to detail everything about it since there are decent articles available around this topic, a good one of those exists here. Instead, I am going to write a little more about how does it work and what happens under the hood. So, let’s get started!

Windows DNS Client 

Exchange Single Global Namespace requires more than one CAS servers and indeed more than one IP addresses to be specified for a single name that will be used by the clients to connect. Just similar to DNS load balancing using a single host name mapped to multiple IP addresses. Outlook clients will then learn these IP addresses with the help of DNS client. In order to retrieve these IP addresses DNS resolver is called and the received response is cached in the DNS resolver cache and the HTTP cache both. For example microsoft.com is the fqdn used for your exchange servers, when you query microsoft.com using nslookup, you find something like below"

image

DNS Client and HTTP Caching

While the addresses returned by querying a DNS server are cached by the DNS client, they are also cached by the HTTP client. WinInet can very well use the DNS client cache to query this cached response however, it requires a RPC call to be made to DNS cache which is still slower than querying its own cache. This way the cached addresses can be retrieved much faster and they can be used without sending a DNS query to the DNS server. Cache ignores any TTL specified by the DNS server and defaults it to 30 minutes. IE 10 and later hold up to 256 entries in the cache.

The HTTP Client

Windows clients run a built in HTTP client that is nothing but a set of DLL files sitting inside System32 folder. WinInet is an acronym used for Windows Internet and provides APIs for the application development. Internet Explorer uses this library to get you the internet resources like websites and ftp sites. WINHTTP is also provides client side APIs but its primarily used by services. HTTP is now the primary protocol of communication between clients and Exchange CAS servers, WinInet is the base for these communications. However applications which are affected by IE settings also honor and refer the WinInet for HTTP calls. Outlook is certainly one of them.

DNS Round and Robin

This is the most interesting topic for the exchange guys. This how outlook determines how to connect to the addresses returned by a DNS server. Let us take the example shown above.:

A host named microsoft.com returned two IP addresses let us call it a list of IP addresses:

64.4.11.37
65.55.58.201

The http client will attempt the connection to first address in this list, if the connection to the first IP fails, it is marked bad and next IP address is attempted. This process continues until either a successful response is received from the server, or until it reaches the end of list of IP addresses. Value of an option flag INTERNET_OPTION_CONNECT_RETRIES determines the behavior of connection attempts to each IP returned in the list of IP addresses.

INTERNET_OPTION_CONNECT_RETRIES

3

Sets or retrieves an unsigned long integer value that contains the number of times WinINet attempts to resolve and connect to a host. It only attempts once per IP address. For example, if you attempt to connect to a multihome host that has ten IP addresses and INTERNET_OPTION_CONNECT_RETRIES is set to seven, WinINet only attempts to resolve and connect to the first seven IP addresses. Conversely, given the same set of ten IP addresses, if INTERNET_OPTION_CONNECT_RETRIES is set to 20, WinINet attempts each of the ten only once. If a host has only one IP address and the first connection attempt fails, there are no further attempts. If a connection attempt still fails after the specified number of attempts, the request is canceled. The default value for INTERNET_OPTION_CONNECT_RETRIES is five attempts. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption.

Each connection attempt could take up to 21 seconds if the DNS query returns too many number of IP addresses.  This will perhaps justify why exchange team claims to have 20 seconds delay in the client reconnection. The reconnect delay may also occur due to sluggish internet connection. When you have a multi site deployment and want to use a single namespace across the sites, you can very well specify multiple IP addresses for namespace you are planning to use and the http client on the windows computers will take care of this for you

Firefox and other clients

Exchange 2013 supports multiple browsers for outlook web app. Firefox, Google Chrome, Safari are very widely used and supported web clients for OWA. NeitherI could find a lot of literature around these products nor have much interest to run 100s of tools to understand their behavior; but if you are interested to learn how firefox deals with all the stuff that this article explained so far, a good read is available at How does firefox cache DNS requests/replies?

Although Single Namespace support by Exchange 2013 leverages most of the client side features to provide a seamless experience for the outlook clients, there is a lot that servers have to do in backend when a site switchover or failover happens.

I will discuss more about what happens inside your datacenter during the site *over in the next part of this post.

Exchange Type attribute

I have a habit of spending a lot of time to understand how exchange uses AD, Windows Registry, WMI, Crypto and all related stuff. One of my favorite things to do with any new version of exchange server is to look for the AD changes it makes. When Exchange 2010 was released I was trying to see through a lot of attributes and the way their values are constructed. All other attributes could be explained with the help of MSDN documentation or spending some time to create a logical link between the attributes, schema classes, etc. but the “type attribute on the exchange server object.

image

Value of “type” attribute looks something really weird. Initially I thought it was Chinese or Japanese but it is not. 😛

So what is this “Type” attribute on the exchange server object in active directory?

This attribute and the value of this attribute contains the licensing information of the server edition that you have chosen to install. When you install an Exchange Server 2010 role only Standard Edition of exchange gets installed automatically. Edition and licensing information is stored in type attribute in an encrypted form. Based on what key you have entered during the activation, exchange edition is determined and the value of this attribute also changes accordingly. Since it is in encrypted form, there is no specific pattern in the change that can be noted but you can still observe the change in the value of type attribute.

Well, that was just a geeky finding. Nothing useful anywhere in production although.

Failed to create RHS process – Windows 2008 R2 cluster

This blog post is not related to exchange but can be useful in some cases since DAG still depends on the clustering technologies. Yesterday, one of our clients had a major issue with a cluster that runs a file server. They installed some patches on the nodes and rebooted the box. Failover cluster manager won’t connect to the cluster since then. A couple of reboots on the servers were tried in a hope that it would fix a problem but that didn’t help.

Symptoms

All cluster groups and resources in each would stay in Pending Online state for a long time and eventually fail. Cluster IP Addresses resource won’t come online either.

image

Cluster.log file was full of some errors that look like below

000013f8.00000cdc::2014/03/13-08:44:45.318 ERR   [RCM] RcmMonitor: Failed to create RHS process ‘C:\Windows\Cluster\rhs.exe -key SYSTEM\CurrentControlSet\Services\ClusSvc\Parameters\Rhs\73feb789-9b11-4be2-9354-46dba2a2419d -parentPid 5112 -initEvent c2b41299-69dd-44ff-99eb-4cc42ddb9a5b -replyEndpoint LRPC-1394a24a6375472e44’. Error ERROR_FILE_NOT_FOUND(2)
000013f8.00000cdc::2014/03/13-08:44:45.318 ERR   [RCM] rcm::RcmMonitor::StartMonitor: ERROR_FILE_NOT_FOUND(2)’ because of ‘RcmMonitor: Failed to create RHS process.’
000013f8.00000cdc::2014/03/13-08:44:46.332 WARN  [RCM] rcm::RcmMonitor::StartMonitor: Retrying…

Resolution

It took us more than 4 hours and Microsoft PSS to figure out the problem since it was really rare to happen. We relooked at the cluster logs again and again and the line that says Error ERROR_FILE_NOT_FOUND(2) gave the hint. The finding was rhs.exe was missing from the C:\Windows\Cluster directory.

image

Since the rhs.exe was missing from this location, the cluster resources could not be brought online. What deleted this file is still a mystery. But in most of the cases, an antivirus may really eat up the rhs.exe image.

To fix a deleted or missing rhs.exe, download any of the hotfixes that are applicable to the Windows Server version that you are running and fixes the issues related to rhs.exe. Some of the hotfixes like KB2907244 which replaces the rhs.exe. If the file is missing, said hotfix would recreate it.

After applying the hotfix we were able to bring up all the resources and by virtue of it; the entire cluster.

More information

RHS stands for Resource Host Subsystem in MSCS and is an extremely critical component that monitors the health of cluster resources. Microsoft core team has a great article here http://blogs.technet.com/b/askcore/archive/2009/11/23/resource-hosting-subsystem-rhs-in-windows-server-2008-failover-clusters.aspx and here http://blogs.msdn.com/b/clustering/archive/2009/06/27/9806160.aspx