Encoding Version 1.1

The Ice protocol consists of two distinct sets of rules:

  • Rules that describe how data types are marshaled into streams of bytes, known as encoding. For example, according to the Ice encoding rules, a string is always encoded as the string size (using 1 to 4 bytes, depending on the actual string size, in little-endian format), followed by its characters in UTF-8 format.
  • Rules that describe the types of messages used by Ice applications (request, reply, batch request, validate connection, etc.), the format of these messages (headers, a field that carries the length of the message, etc.), and the order in which these messages are exchanged. These rules form the protocol per se.

Ice maintains separate versions for the protocol and encoding, which makes it possible for them to evolve independently. All Ice releases prior to Ice 3.5 supported only one protocol version (1.0) and one encoding version (1.0). Ice 3.5 also supports protocol 1.0 and encoding 1.0, and introduced a new encoding version, 1.1. Likewise, Ice 3.6 and later support encoding versions 1.0 and 1.1.

This page describes the motivation for introducing encoding version 1.1 and presents potential interoperability issues between Ice applications that use different encoding versions (with solutions).

On this page:

Why a New Encoding?

One important new feature introduced in Ice 3.5 is the ability to define optional data members in classes. Such an optional data member may be set or not, and a client and server can exchange class instances even though their Slice definitions for these instances are not completely identical: for example a newer server can read and set optional data members that its older clients do not know about.

Slice
// Client (v1)
class CustomerOrder
{ 
   string orderID;
   Customer* from;
   ItemList items;
   double totalAmount;      
}
 
// Server (v2)
class CustomerOrder
{
   string orderID;
   Customer* from;
   ItemList items;
   double totalAmount;
   optional(1) string specialInstructions;
   optional(2) double salesTaxRate;
}

We did not plan for this feature when we designed the Ice encoding version 1.0 many years ago, and we had no way to encode and decode such optional data members: when marshaling a class instance, the data members are simply encoded one after the other according to each data member's type. Likewise, the recipient decodes the bytes according to the data member types in its Slice definition (which must match the sender's Slice definition). 

With the Ice encoding version 1.1, you can view each class slice as conceptually carrying an extra dictionary<tag, optional data member value> (a very special dictionary since its values can have various types). When an application receives an optional data member it does not know about, the encoding provides enough information to skip the byte(s) of that data member. An application will also successfully unmarshal a class instance when some optional data members are missing in the received message. 

The tag value of each optional data member that was set is encoded in a 1.1 message, but the corresponding data member name and type are not (doing so would add excessive overhead). As a result, you need to be careful to use the optional tag values consistently: if optional tag 1 represents string specialInstructions in your server, it can be present or missing in your client's Slice definition, but it must never represent anything else.

We also took advantage of the introduction of this new encoding revision to add more encoding-dependent features:

  • optional data members in exceptions
  • optional parameters in operations
  • the ability to preserve class and exception slices when unmarshaled and remarshaled by a server that does not know their full definitions
  • a new "compact format" for class instances
  • a more compact encoding for enumerations with a large number of enumerators

Interoperability with Ice 3.4 and earlier releases

When a client using Ice 3.4 (or earlier) sends a request to a server using Ice 3.5 (or later), the client naturally uses the only Ice encoding it knows (version 1.0). The server is able to decode this request and will marshal the response using the same encoding. When using generated code, Ice ensures that each response uses the same encoding as the corresponding request, whether this request is dispatched synchronously or asynchronously.

The reverse is more tricky: when a client using Ice 3.5 (or later) sends a request, it will use by default the latest encoding (version 1.1); if this request reaches a server that does not understand this encoding, the request will fail with an Ice::UnknownLocalException (with its unknown member set to "UnsupportedEncodingException"), sent back by the server.

Ice connections are agnostic with respect to encoding version: requests using different encoding versions can flow on the same Ice connection. When a request fails due to an encoding mismatch, it is the request that fails, not the connection establishment.

The solution is to tell the client to use the 1.0 encoding, as described below.

-e encoding in Proxies and Ice.Default.EncodingVersion

Ice 3.5 introduced a new proxy option, -e encoding, which tells a client invoking on this proxy the encoding version supported by the remote object. The Ice run time in the client must use a compatible encoding version when it marshals parameters as part of an invocation on this proxy. This is all -e encoding does: it does not have any effect until the client invokes an operation (if it ever does). This includes the situation where a client marshals an empty parameter list, for an operation with no in parameters.

Ice 3.4 and prior releases did not have this -e encoding proxy option. Any proxy in a 1.0-encoded message has an implicit -e 1.0 when received by an Ice 3.5 (or later) application. This way, if an newer Ice client sends a 1.0-encoded request to a 3.4 server, any proxy returned by this server (in a 1.0-encoded message) will implicitly carry -e 1.0, and as a result the client does not need to modify the encoding version of this proxy.

When an Ice application manufactures a proxy with stringToProxy or propertyToProxy, and the provided stringified proxy does not specify an encoding, the new proxy uses the encoding specified by the property Ice.Default.EncodingVersion. As of Ice 3.5, the default setting for Ice.Default.EncodingVersion is 1.1.

Likewise, when an Ice application manufactures a proxy by calling add or createProxy on an object adapter, and the application does not include a -e encoding in the object adapter's ProxyOptions property, the newly created proxy uses the value of Ice.Default.EncodingVersion.

Bootstrap with Proxies in Configuration

Most applications rely on one or a few proxies in configuration to get started, for example:

# Direct proxy
Ice.Default.Router=Glacier2/router:ssl -p 4064 -h server-ext.acme.com

or

# Direct proxy
Ice.Default.Locator=IceGrid/Locator:tcp -p 4061 -h server-ext.acme.com
 
# plus well-known proxy (a form of indirect proxy):
App.Proxy=widgetFactory

When the client uses Ice 3.5 (or later) and the server (a Glacier2 router, an IceGrid registry or any other server) uses Ice 3.4 or earlier, you can either add -e 1.0 in the corresponding stringified proxy, or set Ice.Default.Encoding to 1.0:

# Direct proxy to older Glacier2 router
Ice.Default.Router=Glacier2/router -e 1.0:ssl -p 4064 -h server-ext.acme.com

If your IceGrid deployment manages both recent Ice servers and Ice 3.4 or earlier servers, you can also discover at run time the encoding version of a well-known object by calling  Locator::findObjectById:

C++
// Manufactures an indirect proxy, using -e 1.1 by default
auto prxFromConfig = communicator->propertyToProxy("App.Proxy"); 
 
// The locator (implemented by IceGrid) returns the indirect proxy associated with 
// this identity (could contain -e 1.0 or -e 1.1)
auto prxFromIceGrid = communicator->getDefaultLocator()->findObjectById(prxFromConfig->ice_getIdentity());

If this well-known object was registered at run time through the IceGrid::Admin interface or an IceGrid administrative tool, findObjectById simply returns the registered proxy as-is.

Otherwise, this well-known object must have been defined through an IceGrid object descriptor and findObjectById computes the encoding version in the returned proxy as follows: 

  • If you set proxy options for this well-known object, either in the object descriptor or the enclosing object adapter or replica group descriptor, the IceGrid registry uses these proxy options when manufacturing the proxy; if these proxy options do not specify an encoding version (with -e encoding), the new proxy gets the IceGrid registry's default encoding version (Ice.Default.EncodingVersion in the IceGrid registry configuration).
  • If you did not set any proxy options for this well-known object, IceGrid checks the Ice version of the server's descriptor:
    • For Ice 3.4 and earlier servers, IceGrid uses the 1.0 encoding for the proxy returned by findObjectById
    • For Ice 3.5 and later servers, or servers with no Ice version specified, IceGrid uses this server's Ice.Default.EncodingVersion for the proxy returned by findObjectById

Whenever you define a well-known object using an IceGrid object descriptor, the proxy returned by findObjectById is an indirect proxy, such as "widgetFactory -e 1.1 @widgetFactoryAdapter". The first invocation on this proxy resolves the adapter-id; this resolution provides the indirect proxy's encoding version to IceGrid, which filters out any object adapter that does not support the requested encoding version (or better).

If you invoke directly on prxFromConfig, Ice performs under the hood all the steps above (including the adapter-id resolution, as needed), but with one significant difference: Ice cannot change the encoding version in prxFromConfig. If prxFromConfig carries a -e 1.1 and no -e 1.1 object is available, the call fails with an Ice::NoEndpointException. As a result, another way to dynamically discover the encoding version of your target object could be as follows:

C++
// Manufactures an indirect proxy, with by default -e 1.1
auto prxFromConfig = communicator->propertyToProxy("App.Proxy"); 
 
try
{
   prxFromConfig->ice_ping();
} 
catch(const Ice::NoEndpointException&)
{
   // could be an older server: switch to -e 1.0
   prxFromConfig = prxFromConfig->ice_encodingVersion(Ice::Encoding_1_0); 
}

Operations Returning Proxies

When your server returns a proxy, you do not need to worry whether or not the encoding version in the proxy matches the encoding version of the response to be sent to the client.

Let's take for example this simple Slice interface:

Slice
interface WidgetFinder
{
    Widget* find(string criteria);
}

A Widget server using Ice 3.5 or later can always return -e 1.1 proxies. If a client uses Ice 3.4 or an older version of Ice, or marshaled its request using the 1.0 encoding, the Widget proxy will be returned in a 1.0-encoded message. Since the -e proxy option is new in the 1.1 encoding, the -e 1.1 option gets lost as a result: the client receives the proxy within a 1.0-encoded message, and this proxy has an implicit -e 1.0.

Fixed Proxies

A fixed proxy is a special proxy used for bidirectional connections. You create a fixed proxy by calling createProxy on a connection object:

C++
// operation implementation (ignores encoding version)
void 
Server::registerCallback(Ice::Identity ident, const Ice::Current& current)
{
    auto fixedProxy = Ice::uncheckedCast<CallbackPrx>(current.con->createProxy(ident));
}

When the server invokes on this proxy, it encodes the request using the encoding version specified by this proxy. Therefore, when the client uses an older Ice version, it is critical to create this fixed proxy with -e 1.0. The recommended approach is to make the fixed proxy's encoding version match the encoding version of the request from the client, as shown below:

C++
// operation implementation (sets encoding version)
void 
Server::registerCallback(Ice::Identity ident, const Ice::Current& current)
{
    auto fixedProxy = Ice::uncheckedCast<CallbackPrx>(current.con->createProxy(ident));
 
    // adjust the proxy's encoding version:
    fixedProxy = fixedProxy->ice_encodingVersion(current.encoding);
}

Classes with Optional Data Members

Only clients and servers using Ice 3.5 (or later) can exchange class instances with optional data members. You can nevertheless pass such a class instance to an older Ice application: in this case, all the optional data members are simply dropped.
 

With the example below, the old clients use Slice definitions with no optional data member:

Slice
class CustomerOrder
{
   string orderID;
   Customer* from;
   ItemList items;
   double totalAmount;      
}
;
sequence<CustomerOrder> CustomerOrderSeq;
 
interface Customer
{
   CustomerOrderSeq getOrderHistory();
};

The new server and new clients use compatible Slice definitions, but with optional data members:

Slice
 // New Slice used by new clients and servers
class CustomerOrder
{
   string orderID;
   Customer* from;
   ItemList items;
   double totalAmount;
   optional(1) string specialInstructions;
   optional(2) double salesTaxRate;
}
 
sequence<CustomerOrder> CustomerOrderSeq;

interface Customer
{
   CustomerOrderSeq getOrderHistory();
}

With these definitions, the new clients will receive the specialInstructions and salesTaxRate members (when set); responses to older clients are marshaled using the 1.0 encoding and as a result do not include any optional data members.

If desired, you can even use an #ifdef to share the same Ice file between your old and new applications:

Slice
class CustomerOrder
{
   string orderID;
   Customer* from;
   ItemList items;
   double totalAmount;
#if defined(ICE_VERSION) && ICE_VERSION >= 030500
   optional(1) string specialInstructions;
   optional(2) double salesTaxRate;
#endif
};

sequence<CustomerOrder> CustomerOrderSeq;

interface Customer
{
   CustomerOrderSeq getOrderHistory();
};

Dynamic Invocation and Dispatch

Dynamic Invocation and Dispatch is a very powerful tool for building generic servers that efficiently forward requests without unmarshaling and then remarshaling their payloads. The IceStorm service and the Glacier2 router are examples of Ice servers built using Dynamic Invocation and Dispatch.

As of Ice 3.5, the corresponding APIs, ice_invoke on proxies and ice_invoke on Blobject servants, have new and more correct semantics for their inParams and outParams arguments. These values are now Ice encapsulations, with an Ice encapsulation header.

If you use Dynamic Invocation and Dispatch (ice_invoke), you need to check your code carefully when upgrading to Ice 3.5 (or later). The semantics for inParams and outParams have changed.

With Ice 3.5 or later, when you call ice_invoke on proxy, Ice simply forwards the inParams encapsulation you provide as-is. Ice does not check the encoding version specified by this encapsulation, and as a result the encoding version of the proxy (for example -e 1.0 or -e 1.2) has no effect at all on the request or its payload. This way, when a future Ice release introduces the 1.2 encoding, requests with 1.2-encoded parameters can go through servers built with Ice 3.5. These servers would simply forward these requests, without any encoding version check.

While Ice does not check the encoding version of the in parameters encapsulation, it retrieves this information and provides it to a Blobject's ice_invoke method as the encoding member of the Ice::Current object.

Unfortunately we did not get this right in Ice 3.4 and prior releases: the inParams and outParams values in these older Ice versions represent in and out parameters without an Ice encapsulation header. The Ice run time accepts only 1.0-encoded encapsulations when receiving parameters, and creates only 1.0-encoded encapsulations when sending parameters. As a result, a recent IceStorm publisher cannot send messages using the 1.1 encoding to a recent Ice subscriber if these messages go through an Ice 3.4 IceStorm service. Likewise,  a recent Ice client cannot send 1.1-encoded messages to a recent Ice server if these requests go through an Ice 3.4 Glacier2 router.

IceStorm

IceStorm is a highly efficient publish-subscribe service: publishers publish messages to one or more topics (Ice objects hosted by IceStorm), and IceStorm forwards these messages to the subscribers registered with these topics. IceStorm does not perform any filtering of encoding version: if the publisher sends a message using the 1.1 encoding, it forwards this message as-is to all subscribers. The encoding version in the subscriber proxies is not used at all.

The IceStorm service in Ice 3.4 and prior release can only forward 1.0-encoded messages. This is due to a bug in Dynamic Invocation and Dispatch, described in the section above.

If a publisher publishes a message with the 1.0 encoding, all the subscribers to the target topic will receive this 1.0-encoded message; these subscribers could be implemented with any version of Ice.

If a publisher publishes a message with the 1.1 encoding, only subscribers that understand the 1.1 encoding will receive this message properly. Other subscribers, in particular subscribers using older Ice versions, will fail to decode the message and will throw an encoding-mismatch exception (received as an Ice::UnknownLocalException by IceStorm). Whether IceStorm actually receives this exception depends on the type of proxy (oneway or twoway) registered for the subscriber; if IceStorm receives an exception when attempting to forward a message, it automatically unsubscribes this subscriber.

Glacier2

The Glacier2 service helps Ice applications with firewall traversal. A Glacier2 router is inserted between Ice clients outside a firewall and Ice servers on a private network behind this firewall: the Glacier2 router securely routes requests from the clients to the servers, and later routes the responses back to the clients. The Glacier2 router can even forward callbacks from the servers to the clients (and responses to these callbacks), over bidirectional connections with these clients.

A Glacier2 router does not perform any encoding-version processing when forwarding requests and responses, and in particular, it does not check the encoding version in the target proxies. When an Ice client sends a request to a server through a Glacier2 router, the server receives the request sent by the client exactly as encoded by the client; the server later marshals any response using the same encoding. 

There is no particular encoding-version pitfall when using a Glacier2 router: everything is just like when clients and servers communicate directly with each other.

The Glacier2 router in Ice 3.4 and prior releases can only forward 1.0-encoded requests and responses. This is due to a bug in Dynamic Invocation and Dispatch, described in the section above.

Freeze

The Freeze service allows you to define and access persistent Slice dictionaries stored in Berkeley DB databases. Freeze provides two types of persistent dictionaries: Freeze maps (with application-defined keys) and Freeze evictors (which all use Ice::Identity as keys). Both keys and values are encoded using the Ice encoding when written to Berkeley DB. Keys are stored directly (un-encapsulated, meaning without encoding version information), while values are stored within Ice encapsulations (which specify their contents' encoding version).

When Freeze reads a dictionary entry from Berkeley DB, it decodes the key using the Freeze environment's encoding version (set through the Freeze.DbEnv.env-name.EncodingVersion property), and the value using the value's own encoding version. When Freeze writes a dictionary entry, it encodes both the key and value using the Freeze environment's encoding version.

A Freeze environment created with Freeze 3.4 or earlier contains only 1.0-encoded keys and values. It can be read successfully by a Freeze 3.5 (or later) application regardless of the Freeze.DbEnv.env-name.EncodingVersion setting, provided the encoded representation of its keys is the same with encoding 1.0 and 1.1. Fortunately, this is true for the vast majority of keys: strings, ints, longs, and so on, have the same encoding with 1.0 and 1.1. The only exception are keys containing enumerations with 127 or more enumerators: with the 1.0 encoding, any enumerator in such a large enumeration is always encoded with at least 2 bytes, whereas the 1.1 encoding uses a variable-size encoding with enumerators < 127 encoded in a single byte.

If you use such a large enumeration in your keys, you need to set Freeze.DbEnv.env-name.EncodingVersion to 1.0 to allow Freeze to read the keys correctly. There is currently no tool to automatically convert such a Freeze environment to use the 1.1 encoding.

Freeze also provides tools to read the contents of a Berkeley DB database created by Freeze (dumpdb) and to upgrade the structure of Berkeley DB databases created by Freeze (transformdb). transformdb is a powerful tool you want to use each time you rework the definition of a Freeze map or Freeze evictor and need to upgrade the existing contents of your database.

dumpdb and transformdb do not use Freeze.DbEnv.env-name.EncodingVersion. They rely instead on Ice.Default.EncodingVersion to read keys and, in the case of transformdb, to write keys and values.