This post is interesting as it is one of the weirdest Exchange 2010/2013 performance problems I have seen in my career for a while. In writing this blog post, hopefully other organisations with the same bizarre issue can find this post hopefully resolve their Exchange performance problem.
This post will cover the following topics:
- Symptoms of the problem
- Troubleshooting Steps
- Issue Resolution
My customer was running a single Exchange 2010 SP3 CU3 multi-role server however it needs to be noted this issue can also occur for Exchange 2013 based on what I have read.
SymptomsRPC Response TimesThe Outlook RPC Response times we were seeing from clients connecting on the same subnet as the Exchange server were abnormally high. We were seeing response times in the 10,000 - 20,000 facinity consistently across multiple clients. Response times should generally be below 50 and sometimes higher for remote clients in branch offices or clients connecting using RPC over HTTP as shown in the following screenshot.
Autodiscover Requests Timing OutAutodiscover requests were randomly timing out as the Exchange server was taking to long to respond. The following error was being generated in the Test E-mail AutoConfiguration tool:
GetLastError = 12002; httpStatus=0Autodiscover internet timeout against URL https://exchangeserver.com/autodiscover/autodiscover.xmlMail Tips IssuesUsers were complaining Mail Tips was failing across multiple outlook clients. When composing new emails instead of receiving a mail tip, in the same area where the mail tips are displayed users would receive an error indicating there was a problem receiving mail tips from the server. This is due to the server not responding to the mail tips request in a timely fashion.
Outlook Web App Performance ProblemsLoading the Outlook Web App page and logging in was extremely slow taking approximately 45 seconds for the to completely load. Navigating the mailbox, composing new emails and accessing features such as Exchange Control Panel were also extremely slow and barely usable. Sometimes the server did not respond fast enough and the web page would simply timeout.
Microsoft Outlook Load TimesLaunching the Microsoft Outlook client was extremely slow and took approximately 21 seconds across all clients instead of the snappy 2-3 second load time.
Troubleshooting PerformedI was contracted to come in and resolve the performance problems for the customer. During my initial 4 hours troubleshooting the performance issues I went through and looked at a large number of items which generally contribute to Exchange performance issues.
General Server PerformanceThe general server performance was looked at including items such as physical disk, memory, cpu and network utilisation. Items examined included:
- Number of hard page faults
- Disk queue length
- Disk Time
- CPU Utilisation
- Memory utilisation
- Network number of packets sent/received
- Network bandwidth utilisation
A few other related counters in performance monitor were also examined, all within normal readings.
Network Stack TweakingFor testing the following changes were made to the Windows network stack:
netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled
netsh int tcp set global taskoffload=disabled
netsh int tcp set global autotuninglevel=disabled
These had no impact and changes were reverted.
Exchange User MonitorThe Exchange User Monitor was ran on the server to verify that no significant load was placed on the server from one user session from virus or bad Outlook profile. Readings from ExMon were normal.
Anti-Virus or Third Party ApplicationsAntiVirus or Third Party Applications were investigated. No real time file scanning AV is installed on the Exchange server (as recommended by Microsoft).
The only third party application installed is Exclaimer Outlook Signature which is found not to be causing an issue.
Network Connectivity TestA network connectivity test was performed between the Exchange server and a client on the same subnet suffering performance issues using a bandwidth measuring tool known as iPerf. The network connectivity test showed the client was almost able to achieve gigabit speeds to the Exchange 2010 server over the local network segment.
IIS Application PoolsIIS Application pools were flushed with an IISReset to rule out App Pool Recycling and memory usage/limit/leaking related issues which can significantly impact performance of web based applications on an IIS server. In the event it was related to an IISApp pool issue, an IISReset would generally restore performance for a short period of time which did not happen.
Certificate CRL CheckingWhen using public certificates, the service in which the certificate is associated with must be able to contact the public certificate revocation list. In the event it cannot, this can cause web based applications to be sluggish as we are waiting for CRL timeouts to occur on the backend. I tested this from the Exchange server and was able to contact the CRL lists meaning there was no firewall blocking this communication.
Exchange Troubleshooting AssistantThe Microsoft Exchange Troubleshooting Assistant also known as ExTRA can be used to identify performance related issues on an Exchange 2010 server. This was run and the report flagged nothing of interest.
Process MonitorMicrosoft system internals process monitor (procmon.exe) was run on the Exchange 2010 server to identify application loops or unwanted activity which may be related to slow websites loading. Output from procmon was normal.
Exchange Performance CountersSome core Exchange Performance Counters were checked on Exchange including things such as:
- MSExchangeIS\RPC Requests (should be below 30)
- MSExchangeIS\RPC Averaged Latency (should be below 50ms)
These were all normal.
Exchange Event Log LevelI increased the event log level for OWA for testing purposes. No problems could be identified relating to performance problems from OWA Event Logs.
IPv6 DisabledIPv6 can be known to cause performance issues with Microsoft Exchange. As a result for testing I advised the customer to turn of IPv6 by disabling it on the network interface adapter on the Exchange 2010 server and by putting in place the DisabledComponents registry DWORD value as required by Microsoft KB929852. This had no effect and the issue continued to occur.
Second Exchange 2010 ServerAfter all troubleshooting performed above I advised my customer to build a new Exchange 2010 server as it was they had a simple environment with only a single Exchange 2010 server. We performed the following steps:
- Performed a fresh install of Windows 2008 R2 with all latest updates
- Installed Exchange 2010 with SP1 multi role deployment
- Installed Exchange 2010 SP3 update
- Installed Exchange 2010 SP3 CU3
- Performed minor core configuration tasks such as configuring public folder replication, OAB Distribution, Enabling Outlook Anywhere, Configuring the Digital Certificate and a few other things.
- Moved a few test users to the new Exchange 2010 server for testing purposes.
The test users experienced the exact same performance issues on the new server as those on the old server despite it being a fresh installation of Windows Server and Microsoft Exchange.
ResolutionMy customer stumbled across the following webpage where another company was having a similar issue with their Exchange server performance:
http://www.outlookforums.com/threads/91195-exchange-2013-outlook-2010-slow-startups/This other company with the similar issue resolved it by Disabling IPv6. Instead of manually disabling IPv6 like I did in my step above instead used the Microsoft FitIt tool to disable IPv6 which is available on the same knowledge base article, KB929852.
http://support.microsoft.com/kb/929852After running the Microsoft FixIt tool, the customer expressed to me that the issue is resolved and Exchange is no longer under performing.
What puzzles me however is the Microsoft FixIt tool as documented simply creates the DisabledComponents registry key, something which I created manually during troubleshooting. This is documented under "For more information" in the manual section of KB929852, the same KB I used to create my manual registry key. Perhaps it makes another change which I am unaware of and is not documented on the knowledge base article? It might be worth while testing this theory by running an application such as RegSnap before and after the tool is run to identify exactly what changes it has made.
The other thing which is important to note, Exchange generally should be deployed with IPv6. I have many customers running Exchange 2010/2013 servers with IPv6 enabled and no performance issues. This particular customer did express to me that around the time the issue started occurring, they also had an upgrade on their core switch. This network infrastructure change should not be ruled out as possibly effecting the performance of clients connecting to Exchange.
This was definitely one of the most puzzling problems I have dealt with in a while. If you do have the same problem as the symptoms expressed in this post, please do try running the Microsoft FixIt tool as documented above. If this also resolves your problem, please do comment as I am looking forward to hearing your feedback.
Thanks for reading.
