Category: IIS

HTTP/2 SETTINGS frame bug and related registry keys

Ned Leave a comment

HTTP/2 protocol improves the performance and security of today’s digital world. It consists of several frames to carry requests between clients and servers. One of these frames is SETTINGS frame which may be used by attackers to increase CPU usage to 100% in IIS and eventually make the server unresponsive (Denial of Service – DoS).

In this post, we will discuss the root cause and the solution for this bug.

What is HTTP/2 SETTINGS frame?

It is part of the HTTP/2 request which contains 6 parameters to manage communication between peers. Here is how IETF HTTP Working Group explains the usage of this frame:

The SETTINGS frame (type=0x4) conveys configuration parameters that affect how endpoints communicate, such as preferences and constraints on peer behavior. The SETTINGS frame is also used to acknowledge the receipt of those parameters.

SETTINGS in Hypertext Transfer Protocol Version 2 (HTTP/2)
HTTP/2 SETTINGS frame
HTTP/2 SETTINGS frame (Source: IETF HTTP Working Group)

SETTINGS frame has the following parameters:

  • SETTINGS_HEADER_TABLE_SIZE: The maximum size of the header compression table used to decode header blocks
  • SETTINGS_ENABLE_PUSH: It can be used to disable server push
  • SETTINGS_MAX_CONCURRENT_STREAMS: The maximum number of concurrent streams that the sender will allow
  • SETTINGS_INITIAL_WINDOW_SIZE: The sender’s initial window size for stream-level flow control
  • SETTINGS_MAX_FRAME_SIZE: The size of the largest frame payload that the sender is willing to receive
  • SETTINGS_MAX_HEADER_LIST_SIZE: The maximum size of header list that the sender is prepared to accept

SETTINGS frame vulnerability

HTTP/2 protocol allows a client to specify any number of SETTINGS frames with any number of SETTINGS parameters. While IIS works on the request, it may cause high CPU load if there are too many frames and parameters to process.

Hopefully, Microsoft took action quickly to address this vulnerability before it is widely leveraged by the attackers.

Other factors such as using instances that are not thread-safe in your code
may cause high CPU load as well. Check out this post for the root cause and solution: High CPU load in IIS web server caused by HttpClient

Solution for SETTINGS frame bug

Microsoft released a security update to fix this bug. The update provides two registry keys to control maximum how many settings can be transferred in a frame and maximum how many settings can be transferred per minute. After installing the update, you should set these registry keys to a desired value based on your environment.

  • Http2MaxSettingsPerFrame (Registry path:
    Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters)
  • Http2MaxSettingsPerMinute (Registry path:
    Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters)

Make sure to restart your server after setting values to these keys. For more information, check out Microsoft Support page.

I recommend setting both of them to 256 but these values are highly dependent on the individual environment. There is currently no official formula to calculate optimum values.

Unfortunately, there is also no performance counters to monitor the count of SETTINGS frames and parameters. Therefore, you may need to dive deep in HttpQueryServiceConfiguration() API or a netsh helper to develop a script for monitoring these values.

HTTP protocol defines errors with status codes. One of them is 503 (Service Unavailable). If you have come across this error, check this post out for step-by-step solution: HTTP 503 Service Unavailable (Application pool has been disabled)

High CPU load in IIS web server caused by HttpClient

Ned Leave a comment

High CPU load is one of the most common issues web servers struggle with. There could be several root causes such as deadlocks, insufficient hardware, high traffic, poor coding. In this post, I will explain the reason and solution for a high CPU load caused by an object which is not thread-safe.

This issue may surface in many ways. One if them is -obviously- high resource utilization. You can monitor the resource usage in Task Manager, Resource Monitor or Performance Monitor.

Before we go further, let’s remember what the thread safety is:

Thread safety is a computer programming concept applicable in the context of multi-threaded programs. A piece of code is thread-safe if it only manipulates shared data structures in a manner that guarantees safe execution by multiple threads at the same time.

Thread safety

High CPU load in Performance Monitor and Event Viewer

The Performance Monitor chart below shows that the CPU load is around 80% (red solid line). It also demonstrates how the error count (brown dashed line) increases in parallel to request total (green dashed line).

High CPU load displayed in Performance Monitor
High CPU load displayed in Performance Monitor

This issue may appear as error or warning messages in Event Viewer that seems unrelated to CPU load at first sight. The message below is recorded in the System container of Event Viewer and it mentions unresponsive application pool which points out high CPU load issue.

Event ID: 5010 (Warning)

A process serving application pool “X” failed to respond to a ping. The process id was “1234”


Root cause of the high CPU load

As I mentioned in the beginning of the post, there could be a variety of reasons why a web server suffers from high CPU load. In this scenario, the reason is that using an instance of a non-thread-safe object multiple times. More specifically, using the same HttpClient instance from different threads increases the CPU usage.

In official documentation for HttpClient, it is recommended to use only one instance of this object:

HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors.

HttpClient Class

However, in practice, it is not a good idea especially for the scenarios where requests step on each other:

If you have requests that are related (or won’t step on eachother) then using the same HttpClient makes a lot of sense.

Best practice usage of HttpClient

If you are targeting the wrong CPU architecture (32-bit or 64-bit) that doesn’t match with your DLL libraries, this may cause an issue as well. Check this post out for a solution: How to find out processor architecture (x86, x64) of dll and exe files?

Solution for high CPU load

In many cases like this one, the application shouldn’t use only one instance of HttpClient. It causes requests to overlap each other and increase the CPU load.

Instead of calling the same instance of HttpClient (or the function that is creating HttpClient object), using a new instance of it should solve the high CPU usage issue.

If you don’t have CPU usage issue but your worker process (w3wp.exe) is crashing periodically, this post will help you to fix it: w3wp.exe crashes every 5 minutes with error code 0xc0000374