Active Connection Management
Active Connection Management (ACM) is a mechanism to close idle connections automatically. It also helps detect connection failure in a timely manner, and can be configured to keep connections alive even when there is no application-level activity.
ACM is enabled by default.
On this page:
ACM Basics
When is a Connection Idle?
A connection is idle when no data is sent or received over this connection. A connection that waits to receive data but does not actually receive any data is idle unless it's sending data at the same time.
A connection with an outstanding invocation or dispatch can be idle and remain idle for a while: idle is all about actual data being sent or received on this connection.
Closing an Unused Connection
When a connection remains idle for the configured ACM timeout (or longer) and there is no outstanding invocation or dispatch, the connection and the application as a whole are functioning properly: all requests have completed and the connection is just unused.
With the default ACM configuration, the ACM of a client connection closes such a connection gracefully to save resources, while the ACM of a server connection does nothing. The server ACM is more conservative because it does not want to close the connection just as a client is sending a oneway request (such a request would be lost). It also counts on the client ACM to close this connection.
Closing an Unhealthy Connection
When a connection remains idle for the configured ACM timeout (or longer) and there is an outstanding invocation, something is wrong with the connection or with the peer: the application is waiting for a reply and nothing indicates the connection and the peer are still alive. In this case, with the default ACM configuration, ACM closes the connection forcefully unless it has an outstanding dispatch. This behavior is the same for client and server connections.
Heartbeat
A heartbeat is a oneway message that ACM sends over a connection to:
- make sure this connection is still writable,
- prevent this connection from remaining idle locally, and
- prevent this connection from remaining idle in the peer
The peer receives and consumes the heartbeat message but does not acknowledge or reply to this message. As a result, sending a heartbeat successfully does not guarantee the peer receives the heartbeat: it just means the heartbeat message was successfully written to the connection.
Configuring ACM
The ACM behavior is controlled by 3 properties: Close, Heartbeat and Timeout.
You use the Close property to configure what to do when a connection remains idle for the configured ACM timeout (or longer). You can for example instruct ACM to do nothing or to close the connection.
You use the Heartbeat property to specify when to send heartbeats. And you use the Timeout property to override the default timeout (60 seconds).
The properties can be set at the Communicator level with the Ice.ACM prefix or overridden for
- the client connections created by a Communicator, with the
Ice.ACM.Clientprefix - the server connections accepted by a Communicator, with the
Ice.ACM.Serverprefix - the server connections accepted by a specific object adapter, with the
adapter.ACMprefix
You can also configure the ACM properties programmatically by calling setACM on a connection object. These settings override all ACM property configurations for that connection.
ACM Timeout Semantics
The Timeout property controls two distinct but related behaviors:
- for how long the connection can remain idle before ACM takes action
- when ACM sends heartbeats
For each connection configured for ACM, ACM checks the connection approximately every (Timeout / 2) seconds. This status check sends a heartbeat if the Heartbeat property calls for one to be sent. This way, when two peers use the same Timeout value, ACM sends a heartbeat well before the connection in the peer is idle for the full Timeout.
Setting Timeout to 0 disables ACM: connections won't be closed by ACM and heartbeats won't be sent.
Recommended Configurations
The ACM properties provide a great deal of flexibility but can also lead to hard-to-diagnose bugs. We recommend you keep this configuration simple and start with one of the following configurations:
| Close | Heartbeat | Timeout | Description | |
|---|---|---|---|---|
| Default Configuration | 3 (client) 2 (server) | 1 | 60 | On the client-side, ACM gracefully closes a connection that has been idle for the ACM timeout (or longer) if this connection has no outstanding invocation or dispatch. On both sides, when there is an outstanding dispatch, ACM sends heartbeats to prevent the connection from remaining idle both locally and in the peer. |
| Keep connection alive | 4 | 3 | 60 | ACM sends heartbeats at regular intervals to prevent the connection from staying idle even when there is no application-level activity. A network failure will be detected in at most Timeout / 2. |
| Disable ACM | 0 | 0 | 0 | ACM does not close any connection. You may want to use this configuration for interop with older versions of Ice without ACM support. |
The ACM configuration of all your applications needs to be in sync, and should be typically the same.
ACM in Depth
Bidirectional Connections
A bidirectional connection allows a server to make callback invocations on a client, offering a simple solution to work around the limitations enforced by network firewalls in which the server would otherwise be prevented from establishing a separate connection back to the client. Given that the client's connection to the server represents the server's only path to that client, this connection must remain open as long as the client needs it.
Consider the following configuration:
# Client Ice.ACM.Close=0 # CloseOff Ice.ACM.Heartbeat=3 # HeartbeatAlways Ice.ACM.Timeout=30 # Server Ice.ACM.Close=4 # CloseOnIdleForceful Ice.ACM.Heartbeat=0 # HeartbeatOff Ice.ACM.Timeout=30
Here the client will always send heartbeats at regular intervals to keep connections alive. If a connection remains idle for ACM timeout (or longer), the server forcefully closes the connection regardless of whether any dispatch or invocations are pending at the time.
You can also reverse the roles: configure the server to send the heartbeats and the client to close forcefully connections that have been idle for ACM timeout or longer.
If your application manually configures bidirectional connections (as opposed to the automatic setup provided by Glacier2 connections, for instance), it's not much extra work to configure the ACM settings individually on the connection object if your requirements for bidirectional connections differ from the communicator-wide settings defined by the ACM properties. As an example, an application may not always need to use a connection for bidirectional purposes. As long as a connection is unidirectional, the application might consider it safe to be closed automatically by ACM. Once it transitions to a bidirectional connection, the program that initiated the connection should modify its ACM configuration similar to the client settings shown above.
Managing Sessions
The notion of a session is a common and very useful solution for managing resources associated with a particular client. Briefly, the idea is that a client creates a session, which usually includes an authentication process, and then allocates some resources. The server associates those resources with the client's session and requires the client to keep the session active, using an expiration timer to reclaim a session and its resources if a client abandons the session without formally terminating it. The strategy represents good defensive programming for a server; without such a solution, ill-behaved clients could continue allocating resources indefinitely.
In versions prior to Ice 3.6, we recommended that the client create a background thread that periodically "pinged" the server using an interval based on the server's session expiration time. The thread is no longer necessary as of Ice 3.6, since the ACM heartbeat functionality serves the same purpose. Note however that there are still a couple of considerations:
- The client needs to determine the server's session expiration time. An application might configure this statically, or the value may only be available at run time by calling an Ice operation. In the latter case, the client would need to transfer this value to a corresponding ACM timeout setting, most likely by calling
setACMon the connection object. Using ACM heartbeats to keep a session alive is convenient for clients but presents a problem for server implementations: How does a server know that the connection is still alive? The solution is for the server to call
setCloseCallbackto get a notification of the connection's closure.
The following configuration settings should get you started:
# Client Ice.ACM.Close=0 # CloseOff Ice.ACM.Heartbeat=3 # HeartbeatAlways Ice.ACM.Timeout=30 # Server Ice.ACM.Close=4 # CloseOnIdleForceful Ice.ACM.Heartbeat=0 # HeartbeatOff Ice.ACM.Timeout=30
These settings assume that the application configures the ACM timeout statically. (For example, perhaps the ACM timeout also serves directly as the session expiration timeout.) The server's setting for ACM connection closure represents a design decision that assists the implementation: When ACM detects that a connection has remained idle for ACM timeout (or longer), it forcefully closes the connection. This process involves calling the callback that the server previously set on the connection; presumably the server interprets this notification as an indication that the session can be destroyed. Without the connection callback, the server would have to implement some other strategy for periodically reaping expired sessions.
In this example, the client has the active role (it sends heartbeats) and the server has the passive role (it expects the client to send heartbeats). It's fine to also reverse these roles. You could for instance configure the client to forcefully close the connection when it remains idle for ACM timeout (or longer) and configure the server to send heartbeats. The server will no longer receive heartbeats but it will still be reliably notified when the connection is closed if it has registered a close callback.
Detecting Peer Problems
ACM is an effective tool for detecting and recovering from catastrophic problems with a peer. Let's suppose that it's safe to enable ACM connection closure in an application for both the client and the server. Furthermore, in this application, the server can take a significant amount of time to dispatch a client invocation. The client is willing to wait as long as it takes for the invocation to complete, however, as a defensive measure, the client also wants the ability to recover in case the server becomes unresponsive. Consider these settings:
# Client Ice.ACM.Close=3 # CloseOnInvocationAndIdle Ice.ACM.Heartbeat=0 # HeartbeatOff Ice.ACM.Timeout=30 # Server Ice.ACM.Close=1 # CloseOnIdle Ice.ACM.Heartbeat=1 # HeartbeatOnDispatch Ice.ACM.Timeout=30
The server configuration causes it to close connections when they remain idle for ACM timeout or longer. Since the client doesn't send heartbeats, this means the client hasn't actively used the connection during the timeout period. The server configuration also sends heartbeats to the client, but only while the server is dispatching invocations. As a result, no matter how long it takes to dispatch the invocations, the Ice run time in the server will continue sending heartbeat messages as an indicator to the client that the server is still "healthy". If the ACM in the client determines that a connection has remained idle for ACM timeout or longer, it either means the connection is no longer being used, or if outgoing invocations are still pending, it means that the server has stopped sending heartbeats for some reason. In the former case, the connection is transparently closed without affecting the client. In the latter case, Ice forcefully closes the connection. Assuming the subsequent automatic retry behavior fails, all pending client invocations on that connection will terminate with an exception.
Don't use ACM timeouts as a mechanism for aborting long-running invocations. Ice provides invocation timeouts for that purpose.
Disabling ACM
An application may be interested in preventing ACM from closing connections. For example, oneway invocations can be silently discarded when a server closes a connection. One solution is to set the following property in the server:
Ice.ACM.Server.Close=0
This setting prevents the server from closing an incoming connection regardless of how long the connection has remained idle.
Another solution is to enable heartbeats in the client so that the connection remains active and never becomes eligible for closure while the client process is operating normally. If the client should terminate unexpectedly, the server will still be able to close the connection within a reasonable amount of time without waiting for a low-level notification from the network layer.