I have recently worked with a server that was struggling with high CPU usage only at a certain time of the day: 4am. It sounds odd first but it made sense later. I will discuss the scenario and possible solutions in this post.
Causes and Solutions for high CPU usage around 4am
If you are troubleshooting a similar issue, I would recommend you to checking the settings of the application pool first. Look at the application pool recycle options.
In my case, I saw that the application was scheduled to be recycled at 4am everyday. Additionally, it was also be scheduled to be recycled every 24 hours. This is an obvious conflict. It’s better to keep only one of these settings.
If the issue still exists
If changing the settings doesn’t fix the issue, there might be an issue in the code. I would recommend collecting a dump file with DebugDiag for further troubleshooting. However, you may ask yourself: Why is it happening at 4am if this is a code issue? Here is what the answer might be:
- At 4am, IIS wants to recycle the app pool as per settings
- Normally (when Overlapped Recycle is enabled):
- IIS creates a new app pool
- New requests are processed in the new app pool
- Old requests continue to be served by the old app pool
- Once all old requests are processed, the old app pool is terminated
- However, if Overlapped Recycle is disabled, IIS tries to terminate the current app pool while there are still requests being processed. These requests are in deadlock but IIS still tries to terminate the app pool because it is the recycle time. Auto recycling is probably failing. My best guess is that this situation causes the high CPU load.
Note: There could be other reasons of high CPU usage. One of them is a specific scenarios with HttpClient usage. Here are more details: High CPU load in IIS web server caused by HttpClient
Why is the Overlapped Recycle disabled?
If this issue is happening in your server because someone disabled the Overlapped Recycle, you may be hesitant to enabling it back. Here is why someone might have disabled it:
- If you use overlapped recycle, users will lose the session state and variables at the moment they switch to the new app pool
- The application may not be supporting multi-instance scenarios. For example: If the application is locking a file or another resource while working on it, overlapped recycle will make things worse because the new app pool won’t be able to access to the resource since it is locked by the current app pool. This is a rare scenario but it might be the issue depending on the application
However, not using overlapped recycle has more disadvantages than advantages. The biggest disadvantage is that an outage of the site while switching from the current app pool to the new one (Reference 1, Reference 2).
If your application pool is stopped, you may receive 503 error. Here is how to solve this issue: HTTP 503 Service Unavailable (Application pool has been disabled)