A pre-cancelled token makes the next call fail, after which reusing the FbCommand corrupts the pooled connection (hang) #1281

Summary

Calling an async FbCommand method with a token that is already cancelled sends op_cancel(fb_cancel_raise)
to the server although nothing is running. The server keeps that cancel pending and fails the next, unrelated
call on the attachment with isc_cancelled — typically the implicit transaction start of the next command.

That failure leaves the FbCommand holding an FbTransaction that never started. Executing the same command
again then writes op_allocate_statement plus the first 4 bytes of op_prepare_statement into the connection's
output buffer and throws NullReferenceException. Because the exception is not an IOException, the connection is
returned to the pool as healthy. The next user of that pooled connection flushes the half-written packet, the server
waits for the rest of it, the client waits for responses, and the call hangs forever.

Seen with FirebirdClient 10.3.4 (the code is unchanged on master) against Firebird 2.5, 3, 4 and 5 servers.

Cause

1. ExplicitCancellation.Enter: when cancellationToken.IsCancellationRequested is already true it calls the
   explicit cancel (FbCommand.Cancel() → fb_cancel_raise) before throwing OperationCanceledException. Nothing
   is executing, so the cancel is applied to whatever call the attachment makes next.
2. FbCommand.Prepare / PrepareAsync: _transaction = new FbTransaction(...) is assigned before
   BeginTransaction() runs. If the begin fails, _transaction stays set with a null Transaction, and
   RollbackImplicitTransaction skips it (it only acts when _transaction.Transaction != null). The next execution
   sees _transaction != null, skips the begin, and GdsStatement.SendPrepareToBuffer throws on
   _transaction.Handle after op_prepare_statement is already buffered.
3. FbConnection.Close returns the connection to the pool unless ConnectionBroken is set, which only happens on
   IOException; the half-written buffer and queued deferred packets go back to the pool with it.

Repro

dotnet run probe.cs -- "<connection string>" 200 (.NET 10 file-based app). The hang is timing-dependent (the
cancel must reach the server while it is still busy with the previous packet); it usually appears within a few
dozen iterations. Replacing the reused cmd with a new FbCommand for r1/r2 makes it go away (the stray
cancel still fails one call, but there is no NullReferenceException and no hang).

#:package FirebirdSql.Data.FirebirdClient@10.3.4
using System;
using System.Threading;
using System.Threading.Tasks;
using FirebirdSql.Data.FirebirdClient;

var cs = args[0];
var iterations = int.Parse(args[1]);

static string Run(Action a)
{
	var t = Task.Run(a);
	try
	{
		if (!t.Wait(TimeSpan.FromSeconds(5)))
			return "HANG";
		return "ok";
	}
	catch (AggregateException ex)
	{
		return ex.InnerException!.GetType().Name + ": " + ex.InnerException.Message.Replace('\n', ' ');
	}
}

for (var i = 0; i < iterations; i++)
{
	var cn = new FbConnection(cs);
	cn.Open();
	using (var warm = cn.CreateCommand())
	{
		// DDL keeps the server busy long enough for the cancel to land on the next call
		warm.CommandText = "EXECUTE BLOCK AS BEGIN IF (EXISTS(SELECT 1 FROM rdb$relations WHERE rdb$relation_name = 'ProbeTemp')) THEN EXECUTE STATEMENT 'DROP TABLE \"ProbeTemp\"'; END";
		warm.ExecuteNonQuery();
		warm.CommandText = "EXECUTE BLOCK AS BEGIN IF (NOT EXISTS(SELECT 1 FROM rdb$relations WHERE rdb$relation_name = 'ProbeTemp')) THEN EXECUTE STATEMENT 'CREATE GLOBAL TEMPORARY TABLE \"ProbeTemp\" (ID Int NOT NULL, CONSTRAINT \"PK_ProbeTemp\" PRIMARY KEY (ID)) ON COMMIT PRESERVE ROWS'; END";
		warm.ExecuteNonQuery();
	}

	var cmd = cn.CreateCommand();
	cmd.CommandText = "select 2 from rdb$database";
	using (var cts = new CancellationTokenSource())
	{
		cts.Cancel();
		var r0 = Run(() => cmd.ExecuteNonQueryAsync(cts.Token).GetAwaiter().GetResult()); // sends fb_cancel_raise
		var r1 = Run(() => cmd.ExecuteNonQuery()); // isc_cancelled from the implicit transaction start
		var r2 = Run(() => cmd.ExecuteNonQuery()); // same FbCommand: NullReferenceException, half-written packet
		cmd.Dispose();
		cn.Close();                                // returned to the pool as healthy
		var cn2 = new FbConnection(cs);
		cn2.Open();
		using var cmd2 = cn2.CreateCommand();
		cmd2.CommandText = "select 5 from rdb$database";
		var r3 = Run(() => cmd2.ExecuteNonQuery());
		if (r1 != "ok" || r2 != "ok" || r3 != "ok")
			Console.WriteLine($"iter {i}: precancelled={r0.Split(':')[0]} | reuse1={r1} | reuse2={r2} | next pooled={r3}");
		if (r3 == "HANG")
		{
			Console.WriteLine("hang reproduced");
			Environment.Exit(1);
		}
		cn2.Close();
	}
}
Console.WriteLine("no hang");

Typical output: iter 24: precancelled=OperationCanceledException | reuse1=OperationCanceledException: operation was cancelled | reuse2=NullReferenceException: ... | next pooled=HANG.

A packet capture of the hung connection shows op_cancel, then op_transaction answered with isc_cancelled, then
op_allocate_statement + a lone op_prepare_statement opcode + op_free_statement, and finally the next pooled
user's op_transaction, which the server never answers.

Suggested fixes

• ExplicitCancellation.Enter: if the token is already cancelled, throw without sending fb_cancel_raise — there is
  nothing to cancel, and the cancel otherwise hits a later, unrelated call.
• FbCommand.Prepare / PrepareAsync: assign _transaction only after BeginTransaction succeeds (or reset it when
  the begin throws).
• More generally, an exception while a request is half-serialized into the shared buffer (or while responses are
  outstanding) should reset the buffer and the deferred-packet queue, or mark the connection broken so the pool
  discards it.

Possibly related: #945, #1218, #1270.

Downstream report: linq2db/linq2db#6012

🤖 Generated with Claude Code