Remove-ActiveSyncDevice returns an error – Couldn’t find User as a recipient

Today’s blog post comes from another interesting find about Exchange Management Shell and removal of active sync devices. A lot of customers I know prefer to keep their active sync devices clean. If an employee does not use an active sync device more than few days, they simply remove it. Removing these devices periodically is indeed done through some or the other kind of automation techniques. A whole lot of people use powershell to do that.

At one of such customers, they were seeing errors while removing old active sync devices.

Issue

Running Remove-ActiveSyncDevice returns errors stating it Couldn’t find <user identity> as a recipient or The ActiveSyncDevice <DeviceIdentity> cannot be found. Both errors would look like below:

 

Couldn’t find ‘exchange.local/New Delhi/SomeLocaion/User1’ as a recipient.

    + CategoryInfo          : InvalidArgument: (:) [Remove-ActiveSyncDevice], RecipientNotFoundException

    + FullyQualifiedErrorId : 3DAABD9F,Microsoft.Exchange.Management.Tasks.RemoveMobileDevice

and

The ActiveSyncDevice exchange.local/New Delhi/SomeLocation/User1/ExchangeActiveSyncDevices/SAMSUNGGTI9100

§SAMSUNG1818901812 cannot be found.

    + CategoryInfo          : NotSpecified: (2:Int32) [Remove-ActiveSyncDevice], ManagementObjectNotFoundException

    + FullyQualifiedErrorId : 1C3255A8,Microsoft.Exchange.Management.Tasks.RemoveMobileDevice

Cause

Assume that you have created a mailbox named User1 in an OU exchange.local/New Delhi/SomeLocation. After creation of this mailbox the user was allowed to configure his active sync device. After successful activation, the user account stayed at that location for a while.

Due to some requirements or the change in user’s location or company, you move this user account to another OU using ADUC. While user account is moved, all subsequent objects of the user object in AD are also moved along.

When an active sync device activation process starts, exchange creates an active sync device object under user object in AD and this object also gets moved along the user account when a user account movement happens.

When you run Remove-ActiveSyncDevice using EMS, EMS looks for the object at two common places. The first place is the object entry in user’s mailbox as shown in below figure. ExchangeSyncData object in user’s mailbox (inside mailbox database) contains all the active and non active EAS devices the mailbox has ever synchronized with. In this example the device name is AirSync-SAMSUNGGTN7100-SEC160xxxxx

Capture1

The second place is in AD right under the user object associated with the mailbox. You can see this association using ADSIEDIT or LDP.exe

image

Like I said, when you move a user account to another OU, these EAS device objects also get moved along with it changing the identity of the object. However, when powershell queries this device it does not really query the device object in AD but in mailbox (Show in first figure) and tries to locate the device object in AD against the path it retrieved by querying the information received from object in mailbox. Since you have already moved the user object to a different location using ADUC, exchange is not really aware of what has happened and is unable to update this data back in respective user mailbox in database and returns those errors.

Workaround

Locate the EAS object under user account in AD and remove it using ADSIEDIT and remove an associated object in database by using MFCMAPI

Important

If a user has multiple devices partnered with his mailbox it can be very difficult to find out which one to delete. A way to find out a device object that is to be deleted, you can use following steps:

1. Run Get-ActiveSyncDevice –Mailbox “User1”

2. Make a note of Identity and LastSuccessSync for all the devices.

3. Open MFCMAPI and navigate to the screen shown in first figure.

4. Expand each device or appropriate device you identified in mailbox and select SyncStatus

You should see some properties like show below:

image

PR_LOCAL_COMMIT_TIME and PR_LAST_MOFICATION_TIME are two props which should help you determining which device to delete.

 

Note: These steps are not for someone who does not know how to use MFCMAPI and ADSIEDIT and that the only reason steps are outlined in very high level. If you have questions or need help, you can feel free to drop me a note.

Powershell Password Obfuscator

While writing powershell scripts you may have needed to store the username and password inside the script. There are couple of ways to do so. Either you export the password to a text or xml file and then call it inside the script every time the script runs or generate the password combination using another script and save it inside the main script.

Second way of doing it is much easier but requires another powershell script to be run for generation of credentials.

While working on some script I needed to store the credentials inside the same powershell script. Although there was no need of doing so; someone wanted it that way.

This script generates the code that can be directly pasted inside the main script where you want to save your credentials.

Just enter the username (in the format you want) and enter the password associated with that username and click on generate button. That is all! The code you needed is ready in the text box below:

image

This really simple script can be handy in your toolbox if you are a  powershell developer or you do some scripting stuff for fun.

A download of this script is made available at Technet Gallery. You can download the script just by clicking this button 

Exchange 2010 Intermittent Password Prompts in Outlook Clients – NTLM Bottleneck

There are hundreds of articles on internet around this commonly seen issue. If you are running Exchange 2007 or later this issue occurs due to wrong certificate configuration most of the times. A wrong or missing name in certificate versus the URL defined on exchange web components like OWA, EAS, OA, OAB etc.

Exchange is a fairly complex code which runs along with or depends on several components like AD, Crypto, network components, authentication modules, etc.

This particular case I am writing about was more to do with the authentication mechanisms used by Exchange 2010. Exchange 2010 uses and supports several authentication mechanisms. Below diagram should help you understand a pretty simple looking setup that one of our customers were running:

 

image

The diagram is pretty self explanatory. It is a DAG and a CAS array with 4 domain controllers (although not all 4 are shown in diagram).

Even after verifying all certificate, url and authentication settings on OA, OWA, EAS, OAB, etc users still complained that they receive an annoying password which simply wont go away even after entering the correct user name and password.

Finally, we decided to look further into what is happening when the authentication requests is submitted to the CAS array and interestingly, we could correlate some event IDs in security log of  CAS servers which pointed towards the authentication issue. After investigating security logs carefully on the CAS server we found some entries relevant to a computer which reported a problem. The security log for this computer read as below:

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 9/5/2013 10:22:59 PM
Event ID: 4625
Task Category: Logon
Level: Information
Keywords: Audit Failure
User: N/A
Computer: cas02.exchange.local
Description:
An account failed to log on.
Subject:
  Security ID: NULL SID
  Account Name: –
  Account Domain: –
  Logon ID: 0x0
Logon Type: 3
Account For Which Logon Failed:
  Security ID: NULL SID
  Account Name: username
  Account Domain: EXCHANGE
Failure Information:
  Failure Reason: An Error occurred during Logon.
  Status: 0xc000005e
  Sub Status: 0x0
Process Information:
  Caller Process ID: 0x0
  Caller Process Name: –
Network Information:
  Workstation Name:
  Source Network Address: 178.239.86.252
  Source Port: 37109
Detailed Authentication Information:
  Logon Process: NtLmSsp
  Authentication Package: NTLM
  Transited Services: –
  Package Name (NTLM only): –
  Key Length: 0

Initially it looked like an issue described in http://support.microsoft.com/kb/2157973/en-us but that was not the case since the error code described in KB and error above do not match. Also, there was no smart card logon used. To find out what the error code 0xc000005e meant, we used err.exe and the output was

C:\Tools\Err>Err.exe 0xc000005e
# for hex 0xc000005e / decimal -1073741730 :
  STATUS_NO_LOGON_SERVERS
# There are currently no logon servers available to service

Suspecting something wrong with NTLM netlogon.log was a potential subject to be looked at. Netlogon.log on client shows

Time [LOGON] SamLogon: Network logon of EXCHANGE\UserName from WorkstationName Returns 0xC000005E

It was again little misleading since the AD servers were up and running and processing the logon requests. There was no DNS issues identified either. A lot of googling and Binging, we reached out to a conclusion that lead us to think that something was wrong with the NTLM stuff. So what was it?

You may notice that NTLM bottlenecks can be caused due to RPC/HTTPS requests. RPC/HTTPS are definitely a key contributor to large NTLM requests since the session established using RPC/HTTPS has to be authenticated twice due to two different protocol payloads. Outer layer of HTTP requires the authentication once and the tunneled RPC requires another authentication to take place generating twice the load. Moreover, HTTP is a stateless protocol which can cause multiple authentication requests to be handled by the server.

Although RPC/HTTPS generates additional NTLM authentication requests; a direct MAPI connection to CAS / CAS array can also contribute to this if the traffic is too high. MAPI supports Kerberos authentication and the default setting in Outlook 2007 and later is to negotiate the strongest authentication available when not running in Outlook Anywhere mode. Unless kerberos support is configured in the environment, outlook will fall back on NTLM by default.

Considering all the factors and research done the only conclusion derived was to look for NTLM authentication related issues. A quick network packet capture on CAS servers help determining whether it is NTLM or something else.

To capture the precise results, leave the network capture running on the CAS server until a case of password prompt is reported. You will notice that the capture reveals something like below between the CAS server and client. (Running a simultaneous capture on client and servers both can help gathering precise results

0.0000000           11198    8:13:23 PM 9/2/2013      164.8780960      OUTLOOK.EXE    ClientComputer                 198.168.36.100    MSRPC  MSRPC:c/o Request: MS Exchange Directory RFR {1544F5E0-613C-11D1-93DF-00C04FD7BD09}  Call=0x1  Opnum=0x0  Context=0x0  Hint=0xC0 Warning: Octets trailer appends to authentication token      {MSRPC:105, TCP:104, IPv4:9}     65229

0.0156250           11199    8:13:23 PM 9/2/2013      164.8937210      OUTLOOK.EXE    198.168.36.100               ClientComputer       TCP        TCP:Flags=…A…., SrcPort=6950, DstPort=3117, PayloadLen=0, Seq=3823341786, Ack=264467696, Win=63764 (scale factor 0x0) = 63764  {TCP:104, IPv4:9}               63764

0.0468750           11216    8:13:23 PM 9/2/2013      164.9405960      OUTLOOK.EXE    198.168.36.100               ClientComputer       MSRPC  MSRPC:c/o Fault:  Call=0x1  Context=0x0  Status=0x5  Cancels=0x0       {MSRPC:92, TCP:88, IPv4:9}          63364

In above capture, outlook is clearly trying to use RFR interface

Windows 2008 R2 has NTLM performance counters that can be used to find out the NTLM related issues. One of the support articles on Microsoft KB

Performance counter

Explanation

Semaphore Waiters

The number of the thread that is waiting to obtain the semaphore

Semaphore Holders

The number of the thread that is holding the semaphore

Semaphore Acquires

The total number of times that the semaphore has been obtained over the lifetime of the security channel connection, or since system startup for _Total

Semaphore Timeouts

The total number of times that a thread has timed out while it waited for the semaphore over the lifetime of the security channel connection, or since system startup for _Total

Average Semaphore Hold Time

The average time (in seconds) that the semaphore is held over the last sample.

 

In the case we were troubleshooting, the value of Semaphore Timeouts was reaching beyond 100. As you can read the explanation of the Semaphore Timeouts, this counter suggests the timeouts occurred. In this process, the threads will wait and then will expire denying logon to a requestor. This causes the authentication requests to be rejected. This is exactly what was happening on the servers.

All of these symptoms are caused by a phenomena called “NTLM Bottleneck”. To fix this issue, there are a couple of ways:

Resolution 1

First kind of resolution is increase the MaxConcurrentApi value in registry. This DWORD value can be increased to 10 on Windows Server 2003 based DCs and Member servers and up to 150 on Windows Server 2008 SP2 and later DC and member servers.

  1. Start Registry Editor.
  2. Locate the following registry subkey:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

  3. Create the following registry entry:
    Name: MaxConcurrentApi
    Type: REG_DWORD
    Value:Set the value to the larger number, which you tested (any number greater than the default value).
  4. At a command prompt, run net stop netlogon, and then run net start netlogon.

You may have to apply these settings both on the CAS servers and domain controllers depending upon the situation.

Resolution 2

Configure Exchange 2010 CAS array to use kerberos instead of NTLM using Configuring Kerberos Authentication for Load-Balanced Client Access Servers

References and Additional Reading

Is this horse dead yet: NTLM Bottlenecks and the RPC runtime

Updated: NTLM and MaxConcurrentApi Concerns

You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Services

Netlogon performance counters for Windows Server 2003

Troubleshooting SID translation failures from the obvious to the not so obvious

Script: Finding IIS Servers in Domain

One of our customers is getting ready for a security audit of their critical servers. Indeed Exchange is one of those but there are lot others running IIS on them and exposed to internet through a firewall or some other technology.

Challenge was to find out how many servers in the data center have IIS installed and not in their knowledge. Doing something like this really becomes a challenge when someone has hundreds of servers running inside that cold, noisy and windy storage room Smile with tongue out (Data Center)

Here is a simple script that can help you find the number of IIS servers in an AD domain.

$Error.Clear()
Clear-Host

#$Servers = Get-ADComputer -Filter * -ResultSetSize $null -Properties OperatingSystem | ? { ($_.OperatingSystem -like "Windows Server*") -and ($_.Name -like "BLR-*")}
Foreach ($Server in $Servers) {
Write-Host "Connecting to" $Server.DNSHostName -ForegroundColor Blue
if (Get-WmiObject -ComputerName $Server.DNSHostName -Namespace root -Class __NameSpace -Filter "name=’MicrosoftIISv2’" -ErrorAction SilentlyContinue)
{
    $Found = $Server.DNSHostName
    $Found | Out-File E:\Reports\ServersWithIIS.txt -Force -Append

}

else{
Write-host $Server.DNSHostName + "does not seem to have IIS on it" -ForegroundColor Green
}
}

Again, it is the simplest code that could come upon searching for a ready made script on internet but failing to find one. Hope this helps others too.