poor performance on people picker search in SharePoint 2010

Consider this scenario:

You go to the people picker to search for let's say: "User42". You wait for about more than 3.5 min. until the results are displayed.
You'd now check if there is a general problem and trying it again with a simple repro on the file system as follows

- chose any folder on your hard disk, right-click and chose "properties".
- "add" the wanted User as you would like to do it on granting permissions to this folder.
- Note, that this is taking less than a second(!) to resolve and adding the named User, how this?

So on setting the ULS logging to "verbose" level and retry the peoplepicker search, we will find some interesting hints like this in our logs:

[…]
01.31.2011 15:50:13.98 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq2 Verbose SearchFromGC name = corp.lan. returned. Result count = 0 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:50:13.98 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = contoso.com. start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:50:30.12 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq3 Verbose SearchFromGC name = contoso.com. Error Message: A local error has occurred. 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:50:30.12 w3wp.exe (0x1010) 0x163C SharePoint Foundation General 7fbh Verbose Exception when search "user42" from domain "contoso.com". Exception: "A local error has occurred. ", StackTrace: " at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail) at […]

01.31.2011 15:50:30.12 w3wp.exe (0x1010) 0x163C SharePoint Foundation General 72e7 Medium Error in searching user 'user42' : System.DirectoryServices.DirectoryServicesCOMException (0x8007203B): A local error has occurred. at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail) at […]

01.31.2011 15:50:30.12 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = de-corpx.com. start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:12.95 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq3 Verbose SearchFromGC name = de-corpx.com. Error Message: The server is not operational. 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:12.95 w3wp.exe (0x1010) 0x163C SharePoint Foundation General 7fbh Verbose Exception when search "user42" from domain "de-corpx.com". Exception: "The server is not operational. ", StackTrace: " at […]

01.31.2011 15:51:12.95 w3wp.exe (0x1010) 0x163C SharePoint Foundation General 72e7 Medium Error in searching user 'user42' : System.Runtime.InteropServices.COMException (0x8007203A): The server is not operational. at […]

01.31.2011 15:51:13.08 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq2 Verbose SearchFromGC name = my-group.biz. returned. Result count = 0 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.08 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = org-it.biz. start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.27 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq4 Verbose GetAccountNameFromSid "0x0105000000000005150000008AA7323F23F3F66375B9755494000400" start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.28 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq5 Verbose GetAccountNameFromSid "0x0105000000000005150000008AA7323F23F3F66375B9755494000400" returned. returnValue=True 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.28 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq2 Verbose SearchFromGC name = org-it.biz. returned. Result count = 1 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.28 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = xx-ext.biz. start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.52 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq2 Verbose SearchFromGC name = xx-ext.biz. returned. Result count = 0 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:13.52 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = ap-lan.biz. start 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:51:56.33 w3wp.exe (0x1010) 0x163C SharePoint Foundation Performance ftq3 Verbose SearchFromGC name = ap-lan.biz. Error Message: The server is not operational. 21e10f56-2f45-4a29-a53c-4fda5da9f117
01.31.2011 15:53:15.22 w3wp.exe (0x1010) 0x163C SharePoint Foundation General 72e7 Medium Error in searching user 'user42' : System.Runtime.InteropServices.COMException (0x8007203A): The server is not operational. at
[…]


CAUSE:

For any given search string, i.e. "User42" (which is NOT typed in as "Domain\username" or as UPN "user42@mydomain.com") the Query fetches the account details (SearchFromGC) for the user.

The GetAccountName() function is then used to convert the SID returned by the LDAP query.
The GetAccountName() results in LSASS calling LsarLookupSids3 (when using People Picker) OR both LsaLookupNames4 + LsarLookupSids3 (when using "Check Names").

So we see that we do get the result back from LDAP with the result set and then we use that result set's SID to get the account name in the format DOMAIN\USERLOGIN. The LDAP resultset has this information in the LDAP format, but not in the expected format for SharePoint. This is why we call GetAccountName() to resolve the SID into the Account name.

This process takes a long time and impacts the performance for People Picker / CheckNames function as well as in addition waiting for each timeout on not reachable DC's.

So by using "Isolated Account Names" on peoplepicker search, performance decreases as the number of trusted domains increases…
See more on http://support.microsoft.com/kb/818024

RESOLUTION/WORKAROUND:

Use the stsadm commands for setting the properties to be limited on a particular Domain (where the user lives) and the specific Domain under your Forest on a multi trusted AD environment.

stsadm -o setproperty -url http://<WebAppName> -pn peoplepicker-distributionlistsearchdomains -pv <domainname>

stsadm –o setproperty –pn peoplepicker-searchadforests –pv domain:<domainname> -url http://<WebAppName>

Note Note:

By default, SharePoint talks to the domain controller for the domain in which SharePoint was installed and all trusted domains for two-way trusted domains.

Remarks:

The above commands will enable a limited search against a dedicated domain where the wanted user account resides.
So when having user accounts from other domains in addition, these domains must be also set according to the above command for each needed domain name.
This setting is a per web application setting as defined by the -url parameter and must be also repeated for each web application further.
So by design, SharePoint will behave as of the above description but on forcing only to use the pure ldap results and defining the requested domain explicitly,
we can significant increase the performance on people picker search results near to less than 2 seconds!

If you're having a "one-way-trust", then you need to run additionally this command first:
stsadm –o setapppassword -password <SomeKey>

see more details here: http://technet.microsoft.com/en-us/library/cc263460(office.12).aspx