Copy:
tao-bugs@list.isis.vanderbilt.edu
OK, answering to my own question to some extent...
I narrowed the problem to Transport_Cache_Manager_T, and its use of Cache_ExtId::index_.
First, how I see it working:
============================
Transport_Cache_Manager_T uses ACE_Hash_Map_Manager to keep a mapping
between Cache_ExtId and Cache_IntId. In case of IIOP (in my case it
really was SSLIOP, but I doubt there is a difference there), Cache_ExtId represents IP-ADDR:PORT/index triple for a connection. IP-ADDR:PORT is
the address that Transport connects to, and index is used to allow
multiple connections to same ip/port address. All three values (address,
port, index) are used to calculate hash when stored to ACE_Hash_Map_Manager
When a new Transport is created, it is registered with cache manager,
and it would create an entry using ip:port:index(0). When another
transport is needed again, Transport_Cache_Manager_T::find_i looks up
for an existing connection, and uses it if it is found and idle.
The problem:
============
Transport_Cache_Manager_T::find_i assumes that indexes of existing
connections are all consecutive numbers starting with 0. It will try to
lookup Transport with index=1 *only* if index=0 entry for the same
IP:port exists, and if it is busy. If IP:port:index=0 entry is
previously purged from the cache, Transport_Cache_Manager_T::find_i will
never try to use index=1 (or any other index in the cache).
This scenario is exactly what happens with BiDirGIOP when client
disappears from the network, and later reconnects( and re-registers
callback with same IP:PORT) value:
- server caches first callback with IP:addr:index=0
- client reconnects/re-registers
- server caches the second callback with IP:addr:index=1
- eventually, server cleans up cache entry with IP:addr:index=0
- but it is never able to access the entry with IP:addr:index=1
I am not too sure on the impact on regular TAO clients, since I didnt
try it, but I would assume that:
- if index=0 entry is busy, second transport is created
- if index=0 entry's transport is closed, index=1
entry is purged from cache, and index=1 entry is no
longer reachable, until index=0 entry for the same IP:PORT is created.
Potential solutions:
====================
- I could fix Transport_Cache_Manager_T::unbind_i so it made sure
that the assumption made in find_i is true: If cache has M elements,
when removing an entry at index=N (where N is in [0,M), all remaining
entries for same IP:addr should have consecutive indexes
in range [0,M-1).
- Alternatively, Transport_Cache_Manager_T can be rewritten
to actually use multi-hashmap. The existing implementation with
hash-map and indexes seems inappropriate and sub-optimal.
Or there is a good reason not to use multi-hash-map, that I am not
aware of...
It seems that this would touch more files in TAO though.
I would like to contribute this patch. I would appreciate if someone
could advise me, which direction should I take.
Thanks, Milan.
Milan Cvetkovic wrote:
TAO VERSION: 2.2.1
ACE VERSION: 6.2.1
HOST MACHINE and OPERATING SYSTEM: Debian wheezy on x86_64
THE $ACE_ROOT/ace/config.h FILE: config-linux.h
THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE:
c++11 = 1
ssl = 1
include ${ACE_ROOT}/include/makeinclude/platform_linux.GNU
AREA/CLASS/EXAMPLE AFFECTED:
BiDirGIOP / Transport_Cache_Manager_T / SSLIOP
DOES THE PROBLEM AFFECT:
EXECUTION: YES
SYNOPSIS: After loss of network connection from a client, server is
no longer able to invoke callback RPCs, even after client reconnected,
and resubmitted its callback IOR.
DESCRIPTION:
I have BiDirGIOP setup over SSLIOP. Client is behind firewall router on 192.168.12.x network. Client incarnates callback object, listening on 192.168.12.113:7770 and port 7771 for ss. Client contacts the server
over the internet, and it sends the IOR to callback object above. Server later uses callback object to send various notifications. This setup
utilizes bidirectional GIOP, over SSLIOP.
Everything works as desired, until client loses connectivity to server.
When client re-registers, server adds the new Transport to Transport
cache manager, however in some scenarios it does not remove the old transport, and keeps using it for callbacks, failing on CORBA::TIMEOUT
My understanding is that Transport_Cache_Manager keeps the hash map
table of all connections. These connections have the same key, being
issued from the same IP:port every time (in the example above, 192.168.12.113:7771). In some cases, the server does not replace the
existing transport entry, but adds it with an increased index, and keeps using index:0 for making callbacks.
I am attaching the portions of TAO logs. Note that second registration
binds with index :1. The stale transport is kept with index :0.
How do I control the content of Transport_Cache_Manager_T. I removed the references to callback objects from server, however the transport is
still cached.
Thanks, Milan.
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)