The missing Server Hello in TLS handshake (ERR_SSL_PROTOCOL_ERROR)

For TLS communication to be successful, a client machine makes a request to IIS server over TLS by sending Client Hello package first. Then the server is supposed to respond it with Server Hello package. What if the server is not sending Server Hello package?

Here is how TLS handshake (negotiation) works:

If the server is not sending Server Hello package, the communication won’t be established and the customer is likely to see error messages below:

Internet Explorer:

This page can’t be displayed
Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to again. If this error persists, it is possible that this site uses an unsupported protocol or cipher suite such as RC4, which is not considered secure. Please contact your site administrator

Chrome:

This site can’t provide a secure connection
The website sent an invalid response
ERR_SSL_PROTOCOL_ERROR

Solution for missing Server Hello in TLS handshake and ERR_SSL_PROTOCOL_ERROR

While troubleshooting this issue, keep in mind that the TLS protocol errors above might be misleading. However, you should still start the troubleshooting by analyzing TLS negotiation.

Here are the steps to go through if you come across this issue:

• Make sure all TLS versions are enabled in Internet Explorer (This is for testing. You can later disable the unsecure versions once you find out the root cause)
• Check the registry keys below to make sure what you set in Internet Explorer is applied at registry level. If there are working and non-working servers, mirror the working server’s settings. More details SSL/TLS settings: Security Scan (Qualys SSL Labs) shows TLS 1.0 and 1.1 are enabled
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols
• Collect a network trace from the client side. Check if the client and server are agreeing on a cipher suite. If they are not, make sure the client uses the cipher suites the server is trying to use. The Group Policy setting is below. More information on configuring cipher suite list: TLS fatal error code 20. The Windows SChannel error state is 960 (Solved)
  Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order
• If the issue sill exists, look for any network device (proxy, firewall, load balancer etc.) in between that might be blocking TLS traffic

If the TLS protocol looks good:

If you don’t see any issue with TLS protocol and network devices, continue the troubleshooting by focusing on the web server:

• Check the website binding in IIS. Make sure the certificate is valid and the port is set to 443
• Make sure the port 443 is listened in the server (netstat -an -p TCP | find /I "listening"). More details: Port 80 and 443 are not listened in IIS server
• Change the port number to 444 and test website. If it is accessible, it means there is a software blocking or overwriting 443 port
• Disable Windows Firewall (If it works, you can enable it back and set up rules accordingly)
• Look for any third-party applications such as antivirus or endpoint protection software in the server such as Symantec Endpoint Security and Symantec Data Center Security Server Agent (Based on this document, Security Server Agent uses port 443). Uninstall them (Don’t just disable. Uninstall completely. If it works, you can install them back and configure them accordingly)
• Check if there is any Microsoft software that is using port 443. Applications like SQL Server Reporting Services (SSRS) and Windows Admin Center might interfere the port 443. An example: 503 Service Unavailable error related to Windows Admin Center

If nothing works, I would recommend doing tracing at HTTP.SYS level in the web server. However, before going that deep in troubleshooting, look at the big picture and think out of the box. For example:

• Try to access the site from different machines and browsers
• Access it through IP instead of hostname
• Investigate the issue history. When did it start? What changed?