How a capability mismatch broke MySQL caching in Cloudflare Hyperdrive

Cloudflare has announced general availability for MySQL support in Hyperdrive, its database connection pooler for Workers. The release follows a beta period where engineers identified and resolved a subtle caching bug that caused occasional empty query responses. This fix ensures the service meets its 99.99% availability target for MySQL workloads.

What happened

Hyperdrive initially launched with Postgres support, allowing serverless Workers to share warm database connections and cache query results globally. When MySQL support entered beta last year, the team aimed to match the reliability of their Postgres implementation. During this testing phase, high-volume customers reported a consistent but rare error rate where queries returned empty results instead of expected data. This issue occurred in approximately 0.001% of requests, making it difficult to reproduce and trace.

The investigation revealed that the problem was specific to Hyperdrive instances with caching enabled. Engineers traced the issue to how Hyperdrive handles the MySQL client/server protocol, specifically regarding capability negotiation. Unlike Postgres, MySQL uses capability flags to determine how result sets are formatted and terminated. A mismatch between what the Worker’s driver expected and what Hyperdrive cached led to corrupted or unreadable responses when nodes shared a cache.

To reach general availability, the team implemented twenty bug fixes, including corrections for character set encoding and connection reset behaviors. The most critical fix addressed the caching logic to ensure all stored query results use a canonical format. With these changes deployed, the empty response errors disappeared, and the service achieved its reliability goals.

How it works

Hyperdrive uses a split architecture consisting of a Client node and an Endpoint node. The Client runs near the Worker, handling local requests and caching, while the Endpoint sits closer to the origin database, managing the connection pool. Both nodes utilize the local Cloudflare CDN cache. When the Client and Endpoint reside in the same data center, they share this cache to improve performance by serving repeated queries without hitting the database.

The MySQL protocol relies on capability negotiation during the connection phase. One key capability is CLIENT_DEPRECATE_EOF, which changes how result sets are terminated. Modern servers omit an EOF packet after column definitions and replace the final EOF with an OK packet if this capability is negotiated. However, some client libraries, such as mysql2 for Node.js, do not support this flag. Hyperdrive must translate responses between the origin server’s format and the Worker’s expected format.

The bug arose because the caching layer assumed it could store the exact bytes sent to the Worker. If the Endpoint cached a response using Hyperdrive’s internal capabilities, but the Client retrieved it and sent those raw bytes to a Worker expecting a different format, the Worker would misinterpret the data. Conversely, if the Client cached translated bytes, the Endpoint might retrieve them and fail to process them correctly. The fix ensures that only a canonical version of the response, encoded with Hyperdrive’s capabilities, is ever written to the shared cache.

Key details

• Hyperdrive for MySQL is now generally available, having moved out of beta.
• The service targets 99.99% availability for client connections from Workers.
• A rare bug caused 0.001% of cached queries to return empty results or hang.
• The issue stemmed from mismatched CLIENT_DEPRECATE_EOF capability handling.
• Fixing the bug required ensuring canonical byte encoding in the shared CDN cache.
• Additional fixes addressed utf8mb4 encoding and COM_RESET_CONNECTION state resets.

Why it matters

For developers building on serverless platforms, database connection management is a persistent challenge. Establishing new connections for every request adds significant latency and consumes resources. Connection poolers like Hyperdrive mitigate this by keeping connections warm, but they introduce complexity in protocol translation. Understanding how these tools handle wire protocols helps engineers debug obscure issues that arise when mixing different client libraries and database versions.

This incident highlights the risks of assuming protocol uniformity across different database systems. While Postgres and MySQL both use SQL, their underlying network protocols differ significantly. Engineers integrating with multi-protocol infrastructure must be aware that caching layers may need to normalize data formats to prevent desynchronization. Ignoring these nuances can lead to intermittent failures that are hard to reproduce in local development environments.

Furthermore, the reliance on shared caches in distributed systems requires strict consistency guarantees. When multiple components read and write to the same cache, they must agree on the data schema. In this case, the lack of a canonical format meant that the cache became a source of corruption rather than performance. This lesson applies broadly to any system using intermediate caching for protocol translation.

What you can do

• Upgrade to the general available version of Hyperdrive for MySQL to benefit from the stability fixes.
• Verify that your MySQL client library supports utf8mb4 to avoid character encoding issues with emojis or special characters.
• Check if your client driver supports CLIENT_DEPRECATE_EOF to understand how result sets are terminated in your application.
• Monitor your application for intermittent empty query results if you are using a custom or older MySQL driver.
• Use the wrangler CLI to create new Hyperdrive instances for MySQL-compatible databases like PlanetScale.
• Review your connection reset logic to ensure session state does not leak between requests in pooled environments.