OpenSSL's HollowByte flaw lets tiny payload crash servers

Okta's Red Team has disclosed a critical denial-of-service vulnerability in OpenSSL, named HollowByte, which can be exploited with an 11-byte payload. This flaw allows unauthenticated remote attackers to exhaust server memory, potentially leading to a denial-of-service state. The issue arises as OpenSSL pre-allocates memory based on unvalidated packet claims before any data is received. As a result, the server blocks indefinitely while waiting for the malicious payload.

Older versions of OpenSSL trust the attacker's size declaration, causing a significant memory allocation issue concerning dropped connections, leading to heap fragmentation and requiring process restart for reclamation of memory. Okta tested the attack on unpatched NGINX instances, revealing that an unpatched server could be OOM-killed at 547 MB. The vulnerability affects various software using OpenSSL, including popular web servers and databases. A fix has been implemented in OpenSSL versions 4.0.1 and earlier versions have received backported fixes.