A nested invocation is one that is made within the context of another Ice operation. For instance, the implementation of an operation in a servant might need to make a nested invocation on some other object, or an AMI callback object might invoke an operation in the course of processing a reply to an asynchronous request. It is also possible for one of these invocations to result in a nested callback to the originating process. The maximum depth of such invocations is determined by the size of the thread pools used by the communicating parties.
On this page:
Applications that use nested invocations must be carefully designed to avoid the potential for deadlock, which can easily occur when invocations take a circular path. For example, this illustration presents a deadlock scenario when using the default thread pool configuration:
Nested invocation deadlock.
In this diagram, the implementation of opA
makes a nested twoway invocation of opB
, but the implementation of opB
causes a deadlock when it tries to make a nested callback. As mentioned in Thread Pools, the communicator's thread pools have a maximum size of one thread unless explicitly configured otherwise. In Server A, the only thread in the server thread pool is busy waiting for its invocation of opB
to complete, and therefore no threads remain to handle the callback from Server B. The client is now blocked because Server A is blocked, and they remain blocked indefinitely unless timeouts are used.
There are several ways to avoid a deadlock in this scenario:
opA
simultaneously. Furthermore, setting the maximum size too large can cause its own set of problems.opB
using a oneway invocation, it would no longer need to wait for a response and therefore opA
could complete, making a thread available to handle the callback from Server B. However, we have made a significant change in the semantics of opA
because now there is no guarantee that opB
has completed before opA
returns, and it is still possible for the oneway invocation of opB
to block.opA
using asynchronous dispatch and invocation.
opA
as an AMD operation and invoking opB
using AMI, Server A can avoid blocking the thread pool's thread while it waits for opB
to complete. This technique, known as asynchronous request chaining, is used extensively in Ice services such as IceGrid and Glacier2 to eliminate the possibility of deadlocks.As another example, consider a client that makes a nested invocation from an AMI callback object using the default thread pool configuration. The (one and only) thread in the client thread pool receives the reply to the asynchronous request and invokes its callback object. If the callback object in turn makes a nested twoway invocation, a deadlock occurs because no more threads are available in the client thread pool to process the reply to the nested invocation. The solutions are similar to some of those presented in the above illustration: increase the maximum size of the client thread pool, use a oneway invocation, or call the nested invocation using AMI.
A number of factors must be considered when evaluating whether an application is properly designed and configured for nested invocations:
Ice.Trace.ThreadPool
property can give you a better understanding of the thread pool behavior in your application.As you can imagine, tracing the call flow of a distributed application to ensure there is no possibility of deadlock can quickly become a complex and tedious process. In general, it is best to avoid circular invocations if at all possible.