Randomly arises strange error on connect NETProvider 8.5.4.0 to Fb 4.0.1 Error writing data to the connection

FirebirdSql.Data.FirebirdClient.FbException (0x80004005): Error writing data to the connection.
Error reading data from the connection. ---> FirebirdSql.Data.Common.IscException: Error writing data to the connection.
Error reading data from the connection. ---> System.IO.IOException: Missing 4 bytes to fill total 4.
в FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytes(Byte[] buffer, Int32 count)
в FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperation()
в FirebirdSql.Data.Client.Managed.AuthBlock.ProcessContAuthResponse(IXdrReader xdr)
в FirebirdSql.Data.Client.Managed.GdsConnection.Identify(String database)
--- Конец трассировки внутреннего стека исключений ---
в FirebirdSql.Data.Client.Managed.GdsConnection.Identify(String database)
в FirebirdSql.Data.Client.ClientFactory.CreateManagedDatabase(ConnectionString options)
в FirebirdSql.Data.Client.ClientFactory.CreateDatabase(ConnectionString options)
в FirebirdSql.Data.FirebirdClient.FbConnectionInternal.Connect()
в FirebirdSql.Data.FirebirdClient.FbConnectionInternal.Connect()
в FirebirdSql.Data.FirebirdClient.FbConnection.Open()

Can you replicate it?

Can you replicate it?

I'll try

The error has gone - antivirus Esset is under suspicion.

Sorry.

OK. In case it returns and you have some way to simulate it, let me know.

@cincuranet

Hey @cincuranet
Did you test it against a remote server, and not using localhost?
The common denominator seems to be when connecting to a remote host, maybe even only on Linux servers.

Yes, in both cases I was running instance on a separate machine (even outside local network).

Hi
I have the same issue only on production server NET 6(not 3.1),NETProvider version:9.1.0.0(.NET 6) /7.5.0(.NET 3.1),Firebird version 3.0.10,Windows server 2016. I've made a test on dev computer with direct connection to production server and no problem,seems that only running from IIS got the issue, is any special configuration to be done in IIS with NET 6+FB?

is any special configuration to be done in IIS with NET 6+FB

None that I'm aware of.

The same error can be reproduce with my sample on #963 I think that somehow both errors are related, because sporadically this error in this ticket is presented

I've tested today this issue with NET 7+NETProvider v9.2.0 alpha 1 and seems to be fixed

Hello!
I may have found the solution to the problem, please take a look.

#if NET5_0_OR_GREATER

		socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, 5);
		socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, 5);
		socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, 5);

#endif

It might be more optimal with different values.

My Error:
FirebirdSql.Data.FirebirdClient.FbException (0x80004005): Error writing data to the connection.
Error reading data from the connection.
---> FirebirdSql.Data.Common.IscException: Error writing data to the connection.
Error reading data from the connection.
---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine..
---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.

@Igor-9309

@Igor-9309 Your problem is unrelated. The 10053 (similar to 10054) error is "normal" error that you should handle.

@cincuranet

@cincuranet Is there an error in my code then?

@Igor-9309

@Igor-9309 If you're not handling that case, then yes.

I have been getting this error as well for a while, but it does seem inconsistent. This is using the latest 10.3.4 version of the provider along with Firebird 5.0.3 (but have been seeing it for a long time with providers 9+ and Firebird 3+).

I managed to sort of code around the issue previously but getting the data from FB in smaller chunks (batches of 500 records, the tables in question have between 50k-90k records) and the problem seemed to go away for a while but then came back again. Managed to get it going consistently wrong on a copy of a clients data, although where it fails on their system compared to the copy running on my system is at different places (fails much earlier on theirs). But after trying a few different things disabling the compression made the problem go away on my system, so it could be related to that? (although yet to try it on the customers system with that configuration)

So worth checking the server Firebird.conf file for WireCompression setting and/or the client connection for the Compression setting (set within the FbConnectionStringBuilder)

If I happen to come across a highly reproducible case with minimal data I will post an update.

FirebirdSql.Data.FirebirdClient.FbException (0x80004005): Error writing data to the connection.
Error reading data from the connection.
 ---> FirebirdSql.Data.Common.IscException: Error writing data to the connection.
Error reading data from the connection.
 ---> System.IO.IOException: Missing 4 bytes to fill total 4.
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(Byte[] buffer, Int32 count, CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadInt32Async(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version11.GdsDatabase.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsStatement.FetchAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at FirebirdSql.Data.Client.Managed.Version10.GdsStatement.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbCommand.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbCommand.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbDataReader.ReadAsync(CancellationToken cancellationToken)
   at LinqToDB.Internal.Linq.QueryRunner.BasicResultEnumerable`1.GetAsyncEnumerable(CancellationToken cancellationToken)+MoveNext()

Can confirm working fine on customers site after turning off wire compression (well so far anyway).

Some additional information, but I can see it is all very strange so doubt it will be easy to figure out what is going on (but at least it we may have narrowed it down slightly?)

I enabled SQL tracing and this is the (slightly truncated) query that is failing with the error

DataConnection: BeforeExecute
-- Firebird.5 Firebird4 (asynchronously)
DECLARE @skip Integer -- Int32
SET     @skip = 77500
DECLARE @take Integer -- Int32
SET     @take = 500

SELECT
	"t1".PLU,
	... /*more fields here*/
FROM
	PLU "t1"
ORDER BY
	"t1".PLU,
OFFSET @skip ROWS FETCH NEXT @take ROWS ONLY 
 
DataConnection: Query Execution Time (AfterExecute) (async): 00:00:00.0409578
 
DataConnection: Total Execution Time (Completed) (async): 00:00:00.3445804. Rows Count: 400.

Interestingly it says Rows Count: 400 - Not 500 as expected...
If I change the skip value to 77499 or 77501 it runs with no error and returns 500 rows as expected (so strange...).

@cincuranet

I was facing this error, and was able to replicate it with the help of AI before I dump the findings of the AI wanted to know, if that is okay to do? My knowledge about packages, and compression aswell as the source of this project is limited, so I cant tell how acurate the findings are.

/cc @cincuranet

Sure. Dump it here, especially if you have a repro.

Okay, thanks.

Written by AI:

Important

With Compression=true, a socket read that returns only bytes that inflate to nothing is taken for the end of the stream, and the connection breaks with "Missing 4 bytes to fill total 4". The server produces such a read by itself whenever a response compresses to a few bytes more than 8192. Below are a program that reproduces it every time against a plain server, and a change to FirebirdNetworkHandlingWrapper.Read / ReadAsync that makes it pass.

The error

FbException: Error writing data to the connection.
Error reading data from the connection.
 ---> IscException: Error writing data to the connection.
Error reading data from the connection.
 ---> IOException: Missing 4 bytes to fill total 4.
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(...)
   ...
   at FirebirdSql.Data.Client.Managed.Version10.GdsStatement.FetchAsync(...)

This is the error of #1021, and the latest comment there fits it: one page of a query fails every time, the pages one row earlier or later do not, and turning compression off makes it go away. We hit it in a test suite: 4 of 6 runs had a failure with compression on, none of 6 with it off.

Environment

Client: FirebirdSql.Data.FirebirdClient 10.3.4. Also reproduced with 11.0.0-alpha1 and with master at b12039a.
Runtime: .NET 10.0.11 on Windows 11 (10.0.26200)
Server: Firebird 4.0.6.3221 SuperServer on the same machine, default TcpRemoteBufferSize (8192), WireCrypt = Disabled

Note

Not tried: other server versions, wire encryption on, a server on another machine, Linux.

Steps to reproduce

Put Program.cs and FbWireCompressionRepro.csproj (both below) in a folder. The .NET 10 SDK is needed.
Run it against a Firebird 3 or later server:
```
dotnet run -- --host localhost --port 3050 --user SYSDBA --password masterkey
```
It creates a database, drops it at the end, and takes about 20 seconds. The database goes into the temp folder of the machine running the program; for a server on another machine, pass --database <file path on the server>.

Part 1 fills a table with 8000 rows (an id plus four pseudo-random numbers as text, the same on every run) and reads them with each fetch size from 320 to 350, compression on, each time on a new connection. Part 2 reads 1000 rows through a relay on 127.0.0.1, see Part 2 below.

To run it with another package version, or with a local build of the provider:

dotnet run -property:FirebirdClientVersion=11.0.0-alpha1 -- --port 3050
dotnet run -property:FirebirdClientProject=<path to FirebirdSql.Data.FirebirdClient.csproj> -- --port 3050

Program.cs

// Reproduces "Error writing data to the connection. Error reading data from the connection." with the inner
// "IOException: Missing 4 bytes to fill total 4." in FirebirdSql.Data.FirebirdClient with wire compression on.
//
// Part 1 needs nothing but the server. It reads 8000 rows of pseudo-random numbers, the same on every run, with every
// fetch size from 320 to 350 rows. The server writes compressed data in pieces of TcpRemoteBufferSize (8192 bytes by
// default). Around 335 of these rows, some fetch responses compress to a few bytes more than 8192, so the server
// sends them as 8192 bytes plus a few, and those few only end the deflate block and the sync flush. The client
// inflates the whole response from the first piece and leaves the few bytes in the socket. The first read of the next
// fetch returns only them, they inflate to nothing, so FirebirdNetworkHandlingWrapper.ReadAsync returns 0, and
// XdrReaderWriter.ReadBytesAsync takes the 0 for the end of the stream.
//
// Part 2 shows the same at a fetch size where part 1 does not fail, through a relay on 127.0.0.1 that forwards all
// bytes unchanged. The relay changes one thing only, and only when told to: it writes the last 4 bytes of every chunk
// it reads from the server 100 ms after the rest. With compression on, those are the last 4 bytes of a sync flush,
// 00 00 FF FF.
//
// dotnet run -- [--host localhost] [--port 3050] [--user SYSDBA] [--password masterkey] [--database <file>]
// --database is the database file to create, as a path on the server; it is dropped at the end. The default is
// FbWireCompressionRepro.fdb in this machine's temp folder, which works for a server on this machine.

using System.Net;
using System.Net.Sockets;
using System.Reflection;
using System.Runtime.InteropServices;
using FirebirdSql.Data.FirebirdClient;

const int TableRows = 8000;
const int FromFetchSize = 320;
const int ToFetchSize = 350;

var options = ParseOptions(args);
var host = options.GetValueOrDefault("host", "localhost");
var port = int.Parse(options.GetValueOrDefault("port", "3050"));
var direct = new FbConnectionStringBuilder
{
    DataSource = host,
    Port = port,
    Database = options.GetValueOrDefault("database", Path.Combine(Path.GetTempPath(), "FbWireCompressionRepro.fdb")),
    UserID = options.GetValueOrDefault("user", "SYSDBA"),
    Password = options.GetValueOrDefault("password", "masterkey"),
    Pooling = false
}.ConnectionString;
var compressed = new FbConnectionStringBuilder(direct) { Compression = true }.ConnectionString;

await FbConnection.CreateDatabaseAsync(direct, overwrite: true);
string serverVersion;
string? remoteBufferSize = null;
await using (var connection = new FbConnection(direct))
{
    await connection.OpenAsync();
    serverVersion = connection.ServerVersion;
    try
    {
        remoteBufferSize = (string?)await new FbCommand(
            "select rdb$config_value from rdb$config where rdb$config_name = 'TcpRemoteBufferSize'", connection).ExecuteScalarAsync();
    }
    catch (FbException)
    {
        // RDB$CONFIG is new in Firebird 4
    }

    // Four numbers from the MINSTD generator per row, so the rows are the same on every run.
    await new FbCommand("create table random_text (id integer not null primary key, txt varchar(40))", connection).ExecuteNonQueryAsync();
    await new FbCommand($"""
        execute block as
        declare i integer = 1;
        declare j integer;
        declare x bigint = 1;
        declare txt varchar(40);
        begin
          while (i <= {TableRows}) do
          begin
            txt = '';
            j = 0;
            while (j < 4) do
            begin
              x = mod(x * 48271, 2147483647);
              txt = txt || x;
              j = j + 1;
            end
            insert into random_text values (:i, :txt);
            i = i + 1;
          end
        end
        """, connection).ExecuteNonQueryAsync();
}

Console.WriteLine($"FirebirdSql.Data.FirebirdClient {typeof(FbConnection).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion}");
Console.WriteLine($"{RuntimeInformation.FrameworkDescription}, {RuntimeInformation.OSDescription}");
Console.WriteLine($"Server {host}:{port}, {serverVersion}, TcpRemoteBufferSize {remoteBufferSize ?? "unknown"}");

Console.WriteLine();
Console.WriteLine($"Part 1, no relay: the {TableRows} rows read once with each fetch size from {FromFetchSize} to {ToFetchSize}, compression on,");
Console.WriteLine("each time on a new connection.");
Console.WriteLine();
var failingFetchSizes = new List<int>();
Exception? firstFailure = null;
for (var fetchSize = FromFetchSize; fetchSize <= ToFetchSize; fetchSize++)
{
    var outcome = await ReadRowsAsync(compressed, fetchSize, TableRows);
    if (outcome.Exception is null)
        continue;
    failingFetchSizes.Add(fetchSize);
    firstFailure ??= outcome.Exception;
}
var fetchSizes = ToFetchSize - FromFetchSize + 1;
if (failingFetchSizes.Count == 0)
{
    Console.WriteLine($"None of the {fetchSizes} fetch sizes failed. Part 2 does not depend on the fetch size.");
}
else
{
    Console.WriteLine($"{failingFetchSizes.Count} of the {fetchSizes} fetch sizes failed; the others read all {TableRows} rows. The failing ones again, three times");
    Console.WriteLine("with compression off and three times with it on:");
    Console.WriteLine();
    Console.WriteLine($"{"fetch size",-12}{"compression off",-17}compression on");
    foreach (var fetchSize in failingFetchSizes)
    {
        var off = new List<Outcome>();
        var on = new List<Outcome>();
        for (var read = 0; read < 3; read++)
        {
            off.Add(await ReadRowsAsync(direct, fetchSize, TableRows));
            on.Add(await ReadRowsAsync(compressed, fetchSize, TableRows));
        }
        Console.WriteLine($"{fetchSize,-12}{Summarize(off),-17}{Summarize(on)}");
    }
}

Console.WriteLine();
Console.WriteLine("Part 2, through the relay: 1000 rows at the default fetch size of 200.");
Console.WriteLine();
Console.WriteLine($"{"compression",-13}{"relay",-44}result");
var relay = new Relay(host, port);
var relayFailures = new List<(string Run, Exception Exception)>();
foreach (var (compression, split) in new[]
{
    (false, Split.Never),
    (false, Split.FromConnect),
    (true, Split.Never),
    (true, Split.FromConnect),
    (true, Split.FromFirstFetch)
})
{
    var throughRelay = new FbConnectionStringBuilder(compression ? compressed : direct)
    {
        DataSource = IPAddress.Loopback.ToString(),
        Port = relay.Port
    }.ConnectionString;
    relay.SplitServerChunks = split == Split.FromConnect;
    var outcome = await ReadRowsAsync(throughRelay, 200, 1000, onExecuted: () => relay.SplitServerChunks = split != Split.Never);
    relay.SplitServerChunks = false;

    var run = $"{(compression ? "on" : "off"),-13}{Describe(split),-44}";
    Console.WriteLine(run + outcome);
    if (outcome.Exception is not null)
        relayFailures.Add((run.TrimEnd(), outcome.Exception));
}

if (firstFailure is not null)
{
    Console.WriteLine();
    Console.WriteLine($"--- part 1, fetch size {failingFetchSizes[0]}");
    Console.WriteLine(firstFailure);
}
foreach (var (run, exception) in relayFailures)
{
    Console.WriteLine();
    Console.WriteLine($"--- part 2: {run}");
    Console.WriteLine(exception);
}

// FirebirdClient leaves the socket of a connection that broke inside a transaction open, so the server keeps that
// attachment. Finalizing the sockets closes them, which lets the database be dropped.
relay.Dispose();
GC.Collect();
GC.WaitForPendingFinalizers();
await DropDatabaseAsync(direct);

// Reads rows 1 to lastId on a new connection; onExecuted runs once the reader is open, before the first fetch.
static async Task<Outcome> ReadRowsAsync(string connectionString, int fetchSize, int lastId, Action? onExecuted = null)
{
    var connection = new FbConnection(connectionString);
    var step = "OpenAsync";
    var rows = 0;
    try
    {
        await connection.OpenAsync();
        step = "ExecuteReaderAsync";
        var command = new FbCommand("select id, txt from random_text where id <= @lastId order by id", connection) { FetchSize = fetchSize };
        command.Parameters.AddWithValue("lastId", lastId);
        var reader = await command.ExecuteReaderAsync();
        onExecuted?.Invoke();
        step = "ReadAsync";
        while (await reader.ReadAsync())
            rows++;
        return new(null, rows, null);
    }
    catch (Exception exception)
    {
        return new(step, rows, exception);
    }
    finally
    {
        try
        {
            await connection.DisposeAsync();
        }
        catch (Exception)
        {
            // closing a connection that already broke is not what this program is about
        }
    }
}

// The server lets go of the attachments of closed connections in its own time, so the drop may need a few tries.
static async Task DropDatabaseAsync(string connectionString)
{
    for (var attempt = 1; ; attempt++)
    {
        try
        {
            await FbConnection.DropDatabaseAsync(connectionString);
            return;
        }
        catch (FbException) when (attempt < 20)
        {
            await Task.Delay(250);
        }
        catch (FbException exception)
        {
            Console.WriteLine($"Could not drop the database: {exception.Message.ReplaceLineEndings(" ")}");
            return;
        }
    }
}

static string Summarize(IEnumerable<Outcome> outcomes) =>
    string.Join(", ", outcomes.GroupBy(outcome => outcome.ToString()).Select(group => $"{group.Key} ({group.Count()})"));

static string Describe(Split split) => split switch
{
    Split.Never => "forwards chunks as they come",
    Split.FromConnect => "last 4 bytes late, from the connect on",
    _ => "last 4 bytes late, from the first fetch on"
};

static Dictionary<string, string> ParseOptions(string[] args)
{
    var options = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
    for (var i = 0; i + 1 < args.Length; i += 2)
        options[args[i].TrimStart('-')] = args[i + 1];
    return options;
}

enum Split { Never, FromConnect, FromFirstFetch }

record Outcome(string? FailedIn, int Rows, Exception? Exception)
{
    public override string ToString() =>
        Exception is null ? "ok"
        : FailedIn == "ReadAsync" ? $"failed after {Rows} rows"
        : $"failed in {FailedIn}";
}

// Forwards bytes between the client and the server unchanged. While SplitServerChunks is on, it writes the last 4
// bytes of every chunk it reads from the server 100 ms after the rest of that chunk, then waits another 100 ms before
// it forwards anything more, so the client gets those 4 bytes on their own.
sealed class Relay : IDisposable
{
    const int LateBytes = 4;
    static readonly TimeSpan Pause = TimeSpan.FromMilliseconds(100);

    readonly TcpListener _listener = new(IPAddress.Loopback, 0);
    readonly string _serverHost;
    readonly int _serverPort;
    volatile bool _splitServerChunks;

    public Relay(string serverHost, int serverPort)
    {
        _serverHost = serverHost;
        _serverPort = serverPort;
        _listener.Start();
        _ = AcceptAsync();
    }

    public int Port => ((IPEndPoint)_listener.LocalEndpoint).Port;

    public bool SplitServerChunks
    {
        get => _splitServerChunks;
        set => _splitServerChunks = value;
    }

    public void Dispose() => _listener.Stop();

    async Task AcceptAsync()
    {
        while (true)
        {
            TcpClient client;
            try
            {
                client = await _listener.AcceptTcpClientAsync();
            }
            catch (Exception exception) when (exception is SocketException or ObjectDisposedException)
            {
                return;
            }
            _ = ForwardAsync(client);
        }
    }

    async Task ForwardAsync(TcpClient client)
    {
        using (client)
        using (var server = new TcpClient())
        {
            client.NoDelay = true;
            server.NoDelay = true;
            await server.ConnectAsync(_serverHost, _serverPort);
            await Task.WhenAny(
                CopyAsync(client.GetStream(), server.GetStream(), fromServer: false),
                CopyAsync(server.GetStream(), client.GetStream(), fromServer: true));
        }
    }

    async Task CopyAsync(NetworkStream from, NetworkStream to, bool fromServer)
    {
        var buffer = new byte[64 * 1024];
        try
        {
            int read;
            while ((read = await from.ReadAsync(buffer)) > 0)
            {
                if (fromServer && _splitServerChunks && read > LateBytes)
                {
                    await to.WriteAsync(buffer.AsMemory(0, read - LateBytes));
                    await Task.Delay(Pause);
                    await to.WriteAsync(buffer.AsMemory(read - LateBytes, LateBytes));
                    await Task.Delay(Pause);
                }
                else
                {
                    await to.WriteAsync(buffer.AsMemory(0, read));
                }
            }
        }
        catch (Exception exception) when (exception is IOException or ObjectDisposedException)
        {
            // one side closed the connection
        }
    }
}

FbWireCompressionRepro.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <!-- Another client version: dotnet run -property:FirebirdClientVersion=11.0.0-alpha1 -->
    <FirebirdClientVersion Condition="'$(FirebirdClientVersion)' == ''">10.3.4</FirebirdClientVersion>
  </PropertyGroup>

  <ItemGroup Condition="'$(FirebirdClientProject)' == ''">
    <PackageReference Include="FirebirdSql.Data.FirebirdClient" Version="$(FirebirdClientVersion)" />
  </ItemGroup>

  <!-- A local build instead: dotnet run -property:FirebirdClientProject=<path to FirebirdSql.Data.FirebirdClient.csproj> -->
  <ItemGroup Condition="'$(FirebirdClientProject)' != ''">
    <ProjectReference Include="$(FirebirdClientProject)" />
  </ItemGroup>

</Project>

Result

On the server here, which listens on port 3052.

Part 1, no relay: 3 of the 31 fetch sizes failed, the other 28 read all 8000 rows. The failing ones, read again three times without and three times with compression:

Fetch size	Compression off	Compression on
334	✅ ok (3 of 3)	❌ failed after 2672 rows (3 of 3)
335	✅ ok (3 of 3)	❌ failed after 7705 rows (3 of 3)
336	✅ ok (3 of 3)	❌ failed after 2352 rows (3 of 3)

Part 2, through the relay, 1000 rows at the default fetch size of 200:

Compression	Relay	Result
off	forwards chunks as they come	✅ ok
off	last 4 bytes late, from the connect on	✅ ok
on	forwards chunks as they come	✅ ok
on	last 4 bytes late, from the connect on	❌ failed in `OpenAsync`
on	last 4 bytes late, from the first fetch on	❌ failed after 200 rows

The first failure of part 1:

FirebirdSql.Data.FirebirdClient.FbException (0x80004005): Error writing data to the connection.
Error reading data from the connection.
 ---> FirebirdSql.Data.Common.IscException: Error writing data to the connection.
Error reading data from the connection.
 ---> System.IO.IOException: Missing 4 bytes to fill total 4.
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(Byte[] buffer, Int32 count, CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadInt32Async(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version11.GdsDatabase.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsStatement.FetchAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at FirebirdSql.Data.Client.Managed.Version10.GdsStatement.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbCommand.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbCommand.FetchAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbDataReader.ReadAsync(CancellationToken cancellationToken)

The fetch failure of part 2 has the same trace. The connect failure happens in GdsDatabase.AttachAsync, on the reply to the server version request (AfterAttachActionsAsync → DatabaseInfoAsync).

Console output

FirebirdSql.Data.FirebirdClient 10.3.4+c6ad75a7f72bb32e1ef6ba65826c0cd554d208bb
.NET 10.0.11, Microsoft Windows 10.0.26200
Server localhost:3052, WI-V4.0.6.3221 Firebird 4.0/tcp (PAA)/P16, TcpRemoteBufferSize 8192

Part 1, no relay: the 8000 rows read once with each fetch size from 320 to 350, compression on,
each time on a new connection.

3 of the 31 fetch sizes failed; the others read all 8000 rows. The failing ones again, three times
with compression off and three times with it on:

fetch size  compression off  compression on
334         ok (3)           failed after 2672 rows (3)
335         ok (3)           failed after 7705 rows (3)
336         ok (3)           failed after 2352 rows (3)

Part 2, through the relay: 1000 rows at the default fetch size of 200.

compression  relay                                       result
off          forwards chunks as they come                ok
off          last 4 bytes late, from the connect on      ok
on           forwards chunks as they come                ok
on           last 4 bytes late, from the connect on      failed in OpenAsync
on           last 4 bytes late, from the first fetch on  failed after 200 rows

Two full runs with 10.3.4 gave the same output, and 11.0.0-alpha1 and master fail at the same fetch sizes. Now and then one of the three reads at a failing fetch size passes, or fails at a later fetch; that is timing, see step 3 below. Which fetch sizes fail depends on how the server compresses these rows: with another zlib they may lie elsewhere, or outside 320 to 350. Part 2 does not depend on that.

What happens

sequenceDiagram
    participant C as FirebirdClient
    participant S as Firebird server
    C->>S: op_fetch, the 23rd fetch of 335 rows
    S-->>C: first send, 8192 compressed bytes
    S-->>C: second send, 6 bytes that only end the deflate block and the sync flush
    Note over C: ReadAsync inflates all 335 rows from the first send<br/>and stops reading, the 6 bytes stay in the socket
    C->>S: op_fetch, the 24th fetch
    Note over C: ReadAsync reads the socket once and gets only the 6 bytes,<br/>which inflate to nothing, so it returns 0
    Note over C: XdrReaderWriter.ReadBytesAsync throws<br/>Missing 4 bytes to fill total 4

The server compresses each response, ends it with a sync flush, and writes the compressed bytes in pieces of TcpRemoteBufferSize, 8192 by default: see REMOTE_deflate and port_buff_size. So a response that compresses to 8198 bytes goes out as two sends, of 8192 and 6 bytes.
Those 6 bytes carry no data. They hold the end of the deflate block and the empty stored block of the sync flush, which ends in 00 00 FF FF. The client inflates the whole response from the first 8192 bytes and has no reason to read on, so the 6 bytes stay in the socket.
The next fetch writes op_fetch and reads the operation code. FirebirdNetworkHandlingWrapper.ReadAsync finds its buffer empty and reads the socket once. It gets the 6 bytes, which are already there, inflates them to nothing, and returns 0. If the second piece arrives together with the first, nothing goes wrong; that is why a read now and then passes.
XdrReaderWriter.ReadBytesAsync takes the 0 for the end of the stream. It marks the connection as failed and throws "Missing 4 bytes to fill total 4".

ReadAsync reads the socket once (Read does the same):

NETProvider/src/FirebirdSql.Data.FirebirdClient/Client/Managed/FirebirdNetworkHandlingWrapper.cs

Lines 90 to 120 in b12039a

    
           public async ValueTask<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default) 
        
           { 
        
           	if (_inputBuffer.Count < count) 
        
           	{ 
        
           		var readBuffer = _readBuffer; 
        
           		int read; 
        
           		try 
        
           		{ 
        
           			read = await _dataProvider.ReadAsync(readBuffer, 0, readBuffer.Length, cancellationToken).ConfigureAwait(false); 
        
           		} 
        
           		catch (IOException) 
        
           		{ 
        
           			IOFailed = true; 
        
           			throw; 
        
           		} 
        
           		if (read != 0) 
        
           		{ 
        
           			if (_decryptor != null) 
        
           			{ 
        
           				_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0); 
        
           			} 
        
           			if (_decompressor != null) 
        
           			{ 
        
           				read = HandleDecompression(readBuffer, read); 
        
           				readBuffer = _compressionBuffer; 
        
           			} 
        
           			WriteToInputBuffer(readBuffer, read); 
        
           		} 
        
           	} 
        
           	var dataLength = ReadFromInputBuffer(buffer, offset, count); 
        
           	return dataLength;

ReadBytesAsync takes 0 for the end of the stream (ReadBytes does the same):

NETProvider/src/FirebirdSql.Data.FirebirdClient/Client/Managed/XdrReaderWriter.cs

Lines 72 to 92 in b12039a

    
           public async ValueTask<byte[]> ReadBytesAsync(byte[] buffer, int count, CancellationToken cancellationToken = default) 
        
           { 
        
           	if (count > 0) 
        
           	{ 
        
           		var toRead = count; 
        
           		var currentlyRead = -1; 
        
           		while (toRead > 0 && currentlyRead != 0) 
        
           		{ 
        
           			toRead -= (currentlyRead = await _dataProvider.ReadAsync(buffer, count - toRead, toRead, cancellationToken).ConfigureAwait(false)); 
        
           		} 
        
           		if (currentlyRead == 0) 
        
           		{ 
        
           			if (_dataProvider is ITracksIOFailure tracksIOFailure) 
        
           			{ 
        
           				tracksIOFailure.IOFailed = true; 
        
           			} 
        
           			throw new IOException($"Missing {toRead} bytes to fill total {count}."); 
        
           		} 
        
           	} 
        
           	return buffer; 
        
           }

To check this, a relay that only forwarded and counted the bytes (not part of the program) recorded the server's sends:

Fetch size	Fetch response	Server's sends	The next fetch failed after
335	23rd	8192 + 6 bytes	7705 rows (23 × 335)
334	8th	8192 + 5 bytes	2672 rows (8 × 334)

In the runs measured, second pieces of 5 to 7 bytes broke the next fetch, and none of 8 bytes or more did. Presumably those still hold bits the client needs for the response itself, so it reads them in time.

In an application this looks random: whether a response ends a few bytes past a multiple of 8192 depends on the data and on everything the connection sent before, because the compressor's window spans responses.

Part 2: any split that leaves only the end of a flush breaks it

Part 2 takes the server's data out of the picture. The relay forwards every byte unchanged and changes only when the bytes arrive: it writes the last 4 bytes of each chunk from the server 100 ms after the rest, as TCP may deliver them. With compression on, those 4 bytes are the 00 00 FF FF that ends a sync flush.

Split once the reader is open: the fetch after the first split response fails.
Split from the connect on: OpenAsync fails. That is the "error on connect" of Randomly arises strange error on connect NETProvider 8.5.4.0 to Fb 4.0.1 Error writing data to the connection #1021's title.
The trace in Randomly arises strange error on connect NETProvider 8.5.4.0 to Fb 4.0.1 Error writing data to the connection #1021's first post fails in AuthBlock.ProcessContAuthResponse, the first read after compression starts in that handshake. That read breaks the same way when the relay sends the first 2 bytes of each chunk late instead of the last 4 (a two-line change, not part of the program). For the first compressed response, those 2 bytes are the zlib header, which inflates to nothing. On this server the first compressed read is the attach response, so Open fails in GdsDatabase.Attach.
With compression off, the same splits do no harm: a read of uncompressed data returns 0 only at the end of the stream.

Suggested fix

Read and ReadAsync read on while the bytes from the socket inflate to nothing, and still stop when the socket returns 0. ReadAsync after the change; Read gets the same:

public async ValueTask<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default)
{
    while (_inputBuffer.Count < count)
    {
        var readBuffer = _readBuffer;
        int read;
        try
        {
            read = await _dataProvider.ReadAsync(readBuffer, 0, readBuffer.Length, cancellationToken).ConfigureAwait(false);
        }
        catch (IOException)
        {
            IOFailed = true;
            throw;
        }
        if (read == 0)
        {
            break;
        }
        if (_decryptor != null)
        {
            _decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
        }
        if (_decompressor != null)
        {
            read = HandleDecompression(readBuffer, read);
            readBuffer = _compressionBuffer;
        }
        WriteToInputBuffer(readBuffer, read);
        // compressed data can inflate to nothing (e.g. only the end of a sync flush), then read on
        if (read != 0)
        {
            break;
        }
    }
    var dataLength = ReadFromInputBuffer(buffer, offset, count);
    return dataLength;
}

With this change on master (b12039a), the program reports no failure in part 1 and only ✅ ok in part 2, in two runs each. That holds for the async API, as in the program, and for the sync one (Open / ExecuteReader / Read). The provider's own test suite was not run.

The full patch

diff --git a/src/FirebirdSql.Data.FirebirdClient/Client/Managed/FirebirdNetworkHandlingWrapper.cs b/src/FirebirdSql.Data.FirebirdClient/Client/Managed/FirebirdNetworkHandlingWrapper.cs
index 17f068a..9a98923 100644
--- a/src/FirebirdSql.Data.FirebirdClient/Client/Managed/FirebirdNetworkHandlingWrapper.cs
+++ b/src/FirebirdSql.Data.FirebirdClient/Client/Managed/FirebirdNetworkHandlingWrapper.cs
@@ -57,7 +57,7 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 
 	public int Read(byte[] buffer, int offset, int count)
 	{
-		if (_inputBuffer.Count < count)
+		while (_inputBuffer.Count < count)
 		{
 			var readBuffer = _readBuffer;
 			int read;
@@ -70,18 +70,24 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 				IOFailed = true;
 				throw;
 			}
+			if (read == 0)
+			{
+				break;
+			}
+			if (_decryptor != null)
+			{
+				_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
+			}
+			if (_decompressor != null)
+			{
+				read = HandleDecompression(readBuffer, read);
+				readBuffer = _compressionBuffer;
+			}
+			WriteToInputBuffer(readBuffer, read);
+			// compressed data can inflate to nothing (e.g. only the end of a sync flush), then read on
 			if (read != 0)
 			{
-				if (_decryptor != null)
-				{
-					_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
-				}
-				if (_decompressor != null)
-				{
-					read = HandleDecompression(readBuffer, read);
-					readBuffer = _compressionBuffer;
-				}
-				WriteToInputBuffer(readBuffer, read);
+				break;
 			}
 		}
 		var dataLength = ReadFromInputBuffer(buffer, offset, count);
@@ -89,7 +95,7 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 	}
 	public async ValueTask<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default)
 	{
-		if (_inputBuffer.Count < count)
+		while (_inputBuffer.Count < count)
 		{
 			var readBuffer = _readBuffer;
 			int read;
@@ -102,18 +108,24 @@ sealed class FirebirdNetworkHandlingWrapper : IDataProvider, ITracksIOFailure
 				IOFailed = true;
 				throw;
 			}
+			if (read == 0)
+			{
+				break;
+			}
+			if (_decryptor != null)
+			{
+				_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
+			}
+			if (_decompressor != null)
+			{
+				read = HandleDecompression(readBuffer, read);
+				readBuffer = _compressionBuffer;
+			}
+			WriteToInputBuffer(readBuffer, read);
+			// compressed data can inflate to nothing (e.g. only the end of a sync flush), then read on
 			if (read != 0)
 			{
-				if (_decryptor != null)
-				{
-					_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
-				}
-				if (_decompressor != null)
-				{
-					read = HandleDecompression(readBuffer, read);
-					readBuffer = _compressionBuffer;
-				}
-				WriteToInputBuffer(readBuffer, read);
+				break;
 			}
 		}
 		var dataLength = ReadFromInputBuffer(buffer, offset, count);

Workaround

Tip

Compression=false avoids it.

Also noticed

After a fetch broke this way, the connection's socket stayed open after DisposeAsync. The server kept the attachment until the socket was finalized, and dropping the database failed with "object ... is in use" until then; that is why the program collects garbage before it drops its database. Seen with 10.3.4. A likely cause, not traced further: DetachAsync throws isc_open_trans before it closes the socket while a transaction is still counted, and DisconnectAsync swallows that.

NETProvider/src/FirebirdSql.Data.FirebirdClient/Client/Managed/Version10/GdsDatabase.cs

Lines 229 to 234 in b12039a

    
           public override async ValueTask DetachAsync(CancellationToken cancellationToken = default) 
        
           { 
        
           	if (TransactionCount > 0) 
        
           	{ 
        
           		throw IscException.ForErrorCodeIntParam(IscCodes.isc_open_trans, TransactionCount); 
        
           	}

NETProvider/src/FirebirdSql.Data.FirebirdClient/FirebirdClient/FbConnectionInternal.cs

Lines 306 to 322 in b12039a

    
           public async Task DisconnectAsync(CancellationToken cancellationToken = default) 
        
           { 
        
           	if (_db != null) 
        
           	{ 
        
           		try 
        
           		{ 
        
           			await _db.DetachAsync(cancellationToken).ConfigureAwait(false); 
        
           		} 
        
           		catch 
        
           		{ } 
        
           		finally 
        
           		{ 
        
           			_db = null; 
        
           			_owningConnection = null; 
        
           			_connectionStringOptions = null; 
        
           		} 
        
           	}

Navigation Menu

Repository navigation

Randomly arises strange error on connect NETProvider 8.5.4.0 to Fb 4.0.1 Error writing data to the connection #1021

14 remaining items

The error

Environment

Steps to reproduce

Result

What happens

Part 2: any split that leaves only the end of a flush breaks it

Suggested fix

Workaround

Also noticed

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	public async ValueTask<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default)
	{
	if (_inputBuffer.Count < count)
	{
	var readBuffer = _readBuffer;
	int read;
	try
	{
	read = await _dataProvider.ReadAsync(readBuffer, 0, readBuffer.Length, cancellationToken).ConfigureAwait(false);
	}
	catch (IOException)
	{
	IOFailed = true;
	throw;
	}
	if (read != 0)
	{
	if (_decryptor != null)
	{
	_decryptor.ProcessBytes(readBuffer, 0, read, readBuffer, 0);
	}
	if (_decompressor != null)
	{
	read = HandleDecompression(readBuffer, read);
	readBuffer = _compressionBuffer;
	}
	WriteToInputBuffer(readBuffer, read);
	}
	}
	var dataLength = ReadFromInputBuffer(buffer, offset, count);
	return dataLength;

	public async ValueTask<byte[]> ReadBytesAsync(byte[] buffer, int count, CancellationToken cancellationToken = default)
	{
	if (count > 0)
	{
	var toRead = count;
	var currentlyRead = -1;
	while (toRead > 0 && currentlyRead != 0)
	{
	toRead -= (currentlyRead = await _dataProvider.ReadAsync(buffer, count - toRead, toRead, cancellationToken).ConfigureAwait(false));
	}
	if (currentlyRead == 0)
	{
	if (_dataProvider is ITracksIOFailure tracksIOFailure)
	{
	tracksIOFailure.IOFailed = true;
	}
	throw new IOException($"Missing {toRead} bytes to fill total {count}.");
	}
	}
	return buffer;
	}

	public override async ValueTask DetachAsync(CancellationToken cancellationToken = default)
	{
	if (TransactionCount > 0)
	{
	throw IscException.ForErrorCodeIntParam(IscCodes.isc_open_trans, TransactionCount);
	}

	public async Task DisconnectAsync(CancellationToken cancellationToken = default)
	{
	if (_db != null)
	{
	try
	{
	await _db.DetachAsync(cancellationToken).ConfigureAwait(false);
	}
	catch
	{ }
	finally
	{
	_db = null;
	_owningConnection = null;
	_connectionStringOptions = null;
	}
	}

Randomly arises strange error on connect NETProvider 8.5.4.0 to Fb 4.0.1 Error writing data to the connection #1021

Description

Activity

cincuranet commented on Jan 17, 2022

andrewkip commented on Jan 17, 2022

andrewkip commented on Jan 20, 2022

cincuranet commented on Jan 20, 2022

14 remaining items

realic commented on Jul 25, 2022

cincuranet commented on Jul 25, 2022

ionutIc commented on Nov 9, 2022

cincuranet commented on Nov 9, 2022

NicFT commented on Dec 21, 2022

mrjohnr commented on Jul 11, 2023

Igor-9309 commented on Nov 11, 2023

cincuranet commented on Nov 14, 2023

Igor-9309 commented on Nov 14, 2023

cincuranet commented on Nov 14, 2023

todaysoftware commented on Nov 21, 2025

todaysoftware commented on Nov 24, 2025

Rand-Random commented on Oct 12, 2026

cincuranet commented on Oct 12, 2026

Rand-Random commented on Oct 13, 2026

The error

Environment

Steps to reproduce

Result

What happens

Part 2: any split that leaves only the end of a flush breaks it

Suggested fix

Workaround

Also noticed

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions