Configurable idle keep-alive probe for pooled SQL connections

[vpelikh]

Summary

Adds an application-level keep-alive mechanism to the SQL connection pool so connections that are silently dropped while idle (for example by an intermediate load balancer or the database itself) are detected and replaced with a fresh connection instead of failing the next user query with a Connection timed out error.

The mechanism works in two parts:

1. Probe — when a pooled connection has been idle for at least a configurable interval, the pool issues a lightweight live statement against it before handing it to a caller.
2. Recovery — if the probe fails because the connection is stale, the pool closes it (removing it from the pool) and transparently retries the command on a fresh connection.

Changes

Core (vertx-sql-client)

• PoolOptions.idleKeepAlive / setIdleKeepAlive (with unit, default 0 = disabled). Configures how long a pooled connection can be idle before it is probed.
• spi/protocol/PingCommand — a probe command carrying the lightweight statement used to verify liveness.
• Connection.keepAliveQuery() SPI — the pool reads the probe statement from the connection, keeping the pool database-agnostic.
• Pool (PoolImpl / SqlConnectionPool):
  • probeIfNeeded issues the PingCommand on checkout when the idle threshold is exceeded; on failure it closes the stale connection, which removes it from the pool via the connector onRemove callback (ConnectionPool.evict only removes unused connections, so a leased stale one must be closed instead).
  • execute transparently retries the command on a fresh connection, bounded by MAX_KEEPALIVE_RETRIES.

PostgreSQL (vertx-pg-client)

• PgConnectOptions.keepAliveQuery — the probe statement is moved to configuration (default SELECT 1).
• PgSocketConnection implements keepAliveQuery() and encodes a PingCommand as a simple query on the wire.

Tests

• PgPoolKeepAliveTest:
  • a healthy idle connection is kept alive (same backend PID);
  • a connection dropped at the network level (via ProxyServer, simulating a load-balancer silent close) recovers transparently on a fresh backend.

Why an application-level probe

TCP keep-alive alone is often insufficient: load balancers can drop idle connections without sending a reset, and OS keep-alive intervals are long. An application-level probe actively verifies the connection is usable right before it is used, directly addressing idle stale-connection failures.

Design notes

• Off by default (idleKeepAlive = 0), so no behaviour change for existing users.
• The probe statement is configurable per driver (PgConnectOptions.keepAliveQuery), so each database can provide a correct, portable no-op statement.

Interaction with the vertx-core connection pool

The stale-connection handling is scoped to the SQL client and deliberately does not change the vertx-core connection pool:

• ConnectionPool.evict(...) only removes unused connections (usage == 0). That is an intentional contract of vertx-core, so evicting a stale leased connection there would change public pooling semantics.
• Instead, recovery uses the existing vertx-core mechanism: when the probe fails, the SQL pool closes the stale connection, which fires the connector onRemove() callback. vertx-core's Remove action handles a leased connection correctly — it drops it and, if a waiter is queued, immediately connects a replacement.

This keeps the fix localized to vertx-sql-client and preserves all vertx-core pool guarantees.

[vpelikh]

Hey @vietj @tsegismont. Just pinging you both in case you missed it.

[vietj]

it wasn't set to a milestone, hence out of the radar

[tsegismont]

Thanks for your proposal @vpelikh

So far the problem with stale connections has been solved with maxLifetime option. Are there limitations that caused you some problems?

[vpelikh]

@tsegismont Thanks for the question. Yes, maxLifetime helps, but it has fundamental limitations that mean it cannot replace an application-level keep-alive.

The key point: maxLifetime is a timer for the age of a connection. It doesn't generate any traffic and doesn't check that the connection is still alive. To the database, firewall, or load balancer, a connection just sitting in the pool looks idle.

Problem 1: External idle timeout can fire before maxLifetime.
Every intermediate hop (DB, proxy, LB, NAT) has its own idle timeout. For example, MySQL's wait_timeout or PostgreSQL's idle_session_timeout. If maxLifetime is greater than that timeout, the connection will be killed externally before the pool closes it. The pool won't know and will hand out a dead connection on the next request — exactly the Connection timed out error.

Problem 2: Lowering maxLifetime causes churn.
You could set maxLifetime smaller than the smallest external timeout (as you suggest). But then connections are recreated very frequently: TCP handshake, authentication, SSL, memory allocation on the DB side. With a large pool, that's noticeable overhead.

Problem 3: maxLifetime doesn't check liveness.
Even if the external timeout is greater than maxLifetime, a connection can die for other reasons: DB restart, failover, network partition. maxLifetime is just a timer; it doesn't issue a SELECT 1. The pool will hand out a dead connection, and the user gets an error.

The probe solves all three problems at once: it actively validates the connection right before handing it out and re-establishes it if it's dead. At the same time, maxLifetime can stay large (e.g., 30 minutes), while the probe runs every 5 minutes — so connections live longer, are recreated less often, and dead ones are caught before they reach the user.

This is a well-established pattern used in many database clients and connection pools — for example, Go's database/sql (SetConnMaxLifetime + SetConnMaxIdleTime), Node.js pg (idleTimeoutMillis), DBCP, and others. maxLifetime for "shelf life", keep-alive for "keeping alive". One does not replace the other.