CommunicationInstance: System.IO.IOException: Remote message handler throws an exception. #326

Activity

1. TaviTruman commented on Jul 9, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 9, 2020

   Last edited by TaviTruman

   ContributorAuthor

   More actions

   More data and information for research

   System.IO.IOException
   HResult=0x80131620
   Message=Remote message handler throws an exception.
   Source=Trinity.Core
   StackTrace:
   at Trinity.Storage.RemoteStorage._error_check(TrinityErrorCode err)
   at Trinity.Storage.RemoteStorage._use_synclient(Func`2 func)
   at Trinity.Storage.RemoteStorage.SendMessage(Byte* message, Int32 size, TrinityResponse& response)
   at Trinity.Network.CommunicationModule.SendMessage(IMessagePassingEndpoint endpoint, Byte* buffer, Int32 size, TrinityResponse& response)
   at Trinity.Storage.MessagePassingExtensionMethods.SendMessage[T](IMessagePassingEndpoint storage, Byte* message, Int32 size, TrinityResponse& response)
   at Trinity.Client.TrinityClientModule.MessagePassingExtension.RegisterClient(IMessagePassingEndpoint storage, RegisterClientRequestWriter msg) in G:\IKW-GraphEngine\src\Modules\GraphEngine.Client\Trinity.Client\obj\Debug\netstandard2.0\GeneratedCode\Lib\Protocols.cs:line 279
   at Trinity.Client.ClientMemoryCloud.RegisterClient() in G:\IKW-GraphEngine\src\Modules\GraphEngine.Client\Trinity.Client\ClientSide\ClientMemoryCloud.cs:line 48
   at Trinity.Client.TrinityClient.RegisterClient() in G:\IKW-GraphEngine\src\Modules\GraphEngine.Client\Trinity.Client\ClientSide\TrinityClient.cs:line 93
   at Trinity.Client.TrinityClient.StartPolling() in G:\IKW-GraphEngine\src\Modules\GraphEngine.Client\Trinity.Client\ClientSide\TrinityClient.cs:line 81
   at Trinity.Network.CommunicationProtocolGroup._RaiseStartedEvent()
   at Trinity.Network.CommunicationInstance._RaiseStartedEvents()
   at Trinity.Network.CommunicationInstance.Start()
2. TaviTruman commented on Jul 9, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 9, 2020

   ContributorAuthor

   More actions

   I'm uploading the trinity.log file

   trinity-[07_08_2020_08_40_03_PM].log
3. TaviTruman commented on Jul 9, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 9, 2020

   Last edited by TaviTruman

   ContributorAuthor

   More actions

   So I am able to duplicate the problem on my local SF Cluster and Azure SF Cluster; it looks like we aren't getting a connection to the remote IClientRegistry (memory cloud). So we are getting a connection and we can send data but unable to receive data.
4. TaviTruman commented on Jul 18, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 18, 2020

   Last edited by TaviTruman

   ContributorAuthor

   More actions

   with the SF Load Balancer configured to let TCP traffic flow through to the SF Cluster, traffic seems to flow into the cluster but can't flow out.

   

   

   This is incoming traffic from the Azure LB - via Health Probe

   

   
5. TaviTruman commented on Jul 24, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 24, 2020

   Last edited by TaviTruman

   ContributorAuthor

   More actions

   Okay - I've travel way down the GE rabbit hole now and have an open ticket with Microsoft Service Fabric Support. We are making progress folks. So to actually connect a GE Client to a GE Server running an Azure Service Fabric cluster is a matter of configuring the Azure LB at Level 4 to let TCP traffic pass through; once that done properly the GE GraphEgnineClient API-set is able to partially connect to the GE Service instance running in the SF Cluster. What I have come to understand and to appreciate the brilliance of the Graph Engine networking stack, and that connecting to the server is a multi-step process and that the GE is dogged w.r.t. keep that connection in place.

   Here's all you need to do this point the GE Client to your SF Cluster:

   

   FYI: I've been documenting the ins-n-outs of developing with the GE in the SF and will publish them at my GitHub GraphEngine repository soon.

   This is what I found most recently. Processing on the GE Client-side will make this call into the GE Server

   

   The GE Client is setting up Client Response registration with Server and will get ready to start polling the server before each RPC call as well as lower-level Graph Engine Network infrastructure RPCs into the server; the GE is truly type-safe distributed across memorycloud instances, even in the SF-Cluster. The call, however, fails on the GE Server-side when running in an SF Cluster; otherwise, the stuff just works.

   

   This is bad and as a result, a true or complete connection is never made; the GE Client in the means time is re-trying the Polling and of course that fails too.

   I've got another remote debugging session schedule with Mike Wong from MS SF Support; this guy is great! I think we are getting down the root-cause of this thing and then a fix can be applied.
6. TaviTruman commented on Jul 25, 2020

   

   Tavi Truman (TaviTruman)

   on Jul 25, 2020

   ContributorAuthor

   More actions

   Okay, so when the GE Client connects (TrinityClient.Start()) to my GE Server, outside of the SF Cluster, the RegisterClientHandler is firstly called before CheckInstanceCookie on the server-side; this call sequence is very important because the RegisterClientHandler is the only method that adds the client cookie to m_client_storages:

   `public override void RegisterClientHandler(RegisterClientRequestReader request, RegisterClientResponseWriter response)
       {
           if (m_memorycloud != null)
           {
               response.PartitionCount = m_memorycloud.PartitionCount;
   
               if (ClientRegistry is null)
                   ClientRegistry = m_memorycloud as IClientRegistry;
   
               var cstg = m_client_storages.AddOrUpdate(request.Cookie, _ =>
               {
                   ClientIStorage stg = new ClientIStorage(m_memorycloud) {Pulse = DateTime.Now};
   
                   if (ClientRegistry != null)
                   {
                       int new_id = ClientRegistry.RegisterClient(stg);
                       stg.InstanceId = new_id;
                   }
   
                   return stg;
               }, (_, stg) => stg);
               response.InstanceId = cstg.InstanceId;
           }
       }`
   

   When the GE server is running in the SF Cluster the RegisterClientHandler is not called first, instead, the CheckInstanceCookie is called first:

     `private ClientIStorage CheckInstanceCookie(int cookie, int instanceId)
       {
           if (m_client_storages.TryGetValue(cookie, out var storage) && instanceId == storage.InstanceId) return storage;
           throw new ClientInstanceNotFoundException();
       }`
   

   and the m_client_storages is empty. So even though the GE Client has made the initial connection to the server the order of method calls on the server-side seem to out of order; this sounds like something might be off in the lower-level code like the MessageDispatcher and DispatcherProc code.
7. TaviTruman commented on Aug 6, 2020

   

   Tavi Truman (TaviTruman)

   on Aug 6, 2020

   ContributorAuthor

   More actions

   when this code is called from a GE client over Trinity Native (Default) Sockets and the GE Client is directed to connect to a GE Server running in an SF Cluster:

   

   On the server-side, this code is called via DispatchMessage

   
8. TaviTruman commented on Sep 25, 2020

   

   Tavi Truman (TaviTruman)

   on Sep 25, 2020

   ContributorAuthor

   More actions

   After working with the SF team for a few weeks there is a certain deficiency in the Graph Engine TCP/IP stack; I've been able to narrow it down to code in the TCP layer. I'll come back here to update our continued development and testing we perfect the GE/SF integration, specifically, external GE Client connection to SF/GE Cluster.