Writer

Writers are used to publish samples. Three writer types are supported:

  • single-key writer: a single key writer publishes samples for a single data element identified by the key provided on creation of the writer
  • multi-key writer: a multi-key writer publishes samples for multiple data elements identified by a set of keys provided on creation of the writer
  • any-key writer: an any-key writer publishes samples for any data element. No keys are provided when creating the writer.

A writer is created using a topic plus optionally a configuration structure. Each writer keeps the history of the samples written with this writer. Depending on the reader configuration, this history, or part of it, is transmitted to the reader when it connects. The application can be notified when readers connect to or disconnect from a writer by registering listener functions with this writer.

A name can also be assigned to the writer on creation. Readers will use this name for the listener notification and the sample origin.

On this page:

Creation

Single-key writer

A single-key writer enables the publisher to publish samples for a single key. The key is provided when the writer is created. A single-key writer object is created by calling the SingleKeyWriter constructor or by using the makeSingleKeyWriter factory function:

C++
template<typename Key, typename Value, typename UpdateTag=std::string>
class SingleKeyWriter : public Writer<Key, Value, UpdateTag>
{
public:
    SingleKeyWriter(const Topic<Key, Value, UpdateTag>& topic, const Key& key, const string& name = string(), const WriterConfig& config = WriterConfig());
};

template<typename K, typename V, typename UT> SingleKeyWriter<K, V, UT>
makeSingleKeyWriter(const Topic<K, V, UT>& topic, const typename Topic<K, V, UT>::KeyType& key, const string& name = string(), const WriterConfig& config = WriterConfig());

The constructor accepts a topic parameter, a key and an optional name and writer configuration structure. 

The makeSingleKeyWriter function accepts the same parameters. The key, value and update tag template parameters (K, V, UT) don't need to be provided when using this function, they are automatically deduced by the compiler from the topic parameter.

For example, to create a writer to write temperatures for the room "kitchen":

C++
DataStorm::Topic<string, float> temperatures = ...;
SingleKeyWriter<string, float> writer1(temperatures, "kitchen"); // using the constructor
auto writer2 = makeSingleKeyWriter(temperatures, "kitchen"); // using the factory function
auto writer3 = std::make_shared<SingleKeyWriter<string, float>>(temperatures, "kitchen"); // create the writer on the heap using the constructor

When a single-key writer is created, DataStorm notifies connected peers. If these peers have readers matching the writer's key, the readers connect to the writer in order to receive samples from it.

Multi-key writer

A multi-key writer enables the publisher to publish samples for multiple keys. The keys are provided when the writer is created. A multi-key writer object is created by calling the MultiKeyWriter constructor or by using the makeMultiKeyWriter factory function:

C++
template<typename Key, typename Value, typename UpdateTag=std::string>
class MultiKeyWriter : public Writer<Key, Value, UpdateTag>
{
public:
    MultiKeyWriter(const Topic<Key, Value, UpdateTag>& topic, const std::vector<Key>& keys, const string& name = string(), const WriterConfig& config = WriterConfig());
};

template<typename K, typename V, typename UT> MultiKeyWriter<K, V, UT>
makeMultiKeyWriter(const Topic<K, V, UT>& topic, const std::vector<typename Topic<K, V, UT>::KeyType>& keys, const string& name = string(), const WriterConfig& config = WriterConfig());

The constructor accepts a topic parameter, a collection of keys and an optional name and writer configuration structure.

The makeMultiKeyWriter function accepts the same parameters. The key, value and update tag template parameters (K, V, UT) don't need to be provided when using this function, they are automatically deduced by the compiler from the topic parameter.

For example, to create a writer to write temperatures for the room "kitchen" and "living room":

C++
DataStorm::Topic<string, float> temperatures = ...;
MultiKeyWriter<string, float> writer1(temperatures, {"kitchen", "living room"}); // using the constructor
auto writer2 = makeMultiKeyWriter(temperatures, {"kitchen", "living room"}); // using the factory function
auto writer3 make_shared<MultiKeyWriter<string, float>>(temperatures, vector<string> {"kitchen", "living room"}); // create the writer on the heap using the constructor

When a multi-key writer is created, DataStorm notifies connected peers. If these peers have readers matching at least one of the writer's keys, the readers connect to the writer in order to receive samples from it.

Any-key writer

An any-key writer enables the publisher to publish samples for any keys. Contrary to single-key or multi-key writers, no keys are provided when the writer is created. An any-key writer object is created by calling the MultiKeyWriter constructor described above. An empty vector of keys is specified for the keys parameter. A makeAnyKeyWriter factory function is also provided:

C++
template<typename K, typename V, typename UT> MultiKeyWriter<K, V, UT>
makeAnyKeyWriter(const Topic<K, V, UT>& topic, const string& name = string(), const WriterConfig& config = WriterConfig());

The makeMultiKeyWriter function accepts the same parameters as the MultiKeyWriter constructors except for the keys parameter which is missing. The key, value and update tag template parameters (K, V, UT) don't need to be provided, they are automatically deduced by the compiler from the topic parameter.

For example, to create a writer to write temperatures for any room:

C++
DataStorm::Topic<string, float> temperatures = ...;
AnyKeyWriter<string, float> writer1(temperatures); // using the constructor
auto writer2 = makeAnyKeyWriter(temperatures); // using the factory function
auto writer3 make_shared<MultiKeyWriter<string, float>>(temperatures); // create the writer on the heap using the constructor

When an any-key writer is created, DataStorm notifies connected peers. Since an any-key writer might publish samples for any keys, all the readers will connect to the any-key writer to potentially receive samples from it if it publishes samples for matching keys.

Publishing samples

To publish samples, the single-key writer KeyWriter class provides four methods:

  • add(const Value&) publishes an Add sample with the given value
  • update(const Value&)publishes an Update sample with the given value
  • remove(const Value&)publishes a Remove sample with the given value

  • template<typename UpdateValue> std::function<void(const UpdateValue&)> partialUpdate(const UpdateTag&) to obtain a function to publish a PartialUpdate sample

The MultiKeyWriter class for multi-key and any-key writers provides the same methods except that all these methods have an additional key parameter as the first parameter. An application can use these methods as follow to publish samples:

C++
auto single = makeSingleKeyWriter(temperatures, "kitchen");
single.add(10.0f);                       // Add a kitchen temperature with the 10º initial value
single.update(20.5f);                    // Update the kitchen temperature to 20.5º
single.remove();                         // Remove the kitchen temperature

auto any = makeAnyKeyWriter(temperatures);
any.add("kitchen", 10.0f);
any.update("kitchen", 21.5f);
any.update("living room", 16.5f); 
any.remove("living room");

The add and remove methods are useful to provide lifecycle information to subscribers. However, DataStorm doesn't require them to be called to start publishing samples for a given key. It also doesn't impose any restrictions on the order that they are called: you can call remove before calling add for example. The reader will receive a Remove sample before receiving the Add sample.

The partial update method is a template method that accepts the partial update tag and returns a function which can be used to publish partial updates. The application needs to provide the UpdateValue template parameter. The UpdateValue type is the type that will be used to marshal the partial update value over-the-wire. It's important that this type matches the update value type registered with the updater on the topic. If the type doesn't match, readers won't be able to unmarshal the update value. Here's an example that demonstrates how to use the partial update method:

C++
auto single = makeSingleKeyWriter(temperatures, "kitchen");
auto increment = single.partialUpdate<int>("increment");
increment(2); // 2º increment of the temperature

auto any = makeAnyKeyWriter(temperatures);
auto incrementany = any.partialUpdate<int>("increment");
incrementany("living room", 2);
incrementany("kitchen", 2); 

History

A writer queues the samples it publishes in its sample history queue. This queue is used to provide the initial samples to newly connected readers. It can also be accessed by the application using the following writer methods:

  • getLast returns the last published sample
  • getAll returns all the published samples queued in the writer history

The number of samples kept in queue depends on a number of configuration variables. By default a writer only keeps the last published sample in its history.

Configuration

A writer supports several configuration parameters:

C++
class Config
{
    Ice::optional<int> sampleCount;
    Ice::optional<int> sampleLifetime;
    Ice::optional<ClearHistoryPolicy> clearHistory;
};
class WriterConfig : public Config
{
    Ice::optional<int> priority;
};

The parameters are specified as optional values. You can only configure the parameters you need. If a parameter is not set, the existing value will be used.

Configuration parameters for writers can be specified and overriden at several levels:

  • with configuration properties prefixed with DataStorm.Topic (e.g.: DataStorm.Topic.SampleCount).
  • with configuration properties prefixed with DataStorm.Topic.<topic name> (e.g.: DataStorm.Topic.temperatures.ClearHistory).
  • the topic default writer configuration structure set with the setWriterDefaultConfig method of the DataStorm::Topic class.
  • when the writer is created using the last parameter of the writer make factory function presented above.

A parameter set a lower level (configuration provided to the writer make factory function) overrides the one set at a higher level (e.g.: configuration properties).

Sample count

The sample count configuration specifies the number of samples that will be kept in the writer queue. When the writer queue is full, the oldest samples are removed to make room for the newer published samples.

Sample lifetime

The sample lifetime configuration specifies how long a sample will be kept in the history. Samples older than the sample lifetime are removed from the queue.

Clear history

The history can be automatically cleared when a new sample is published and depending on the clear history policy configuration:

C++
enum struct ClearHistoryPolicy
{
    OnAdd,
    OnRemove,
    OnAll,
    OnAllExceptPartialUpdate,
    Never
};

The clear history policy is based on the sample event. It can be cleared for all events or only for Add or Remove. The history can also be cleared for all events except if it's a partial update. This can be useful if you want the writer to keep all the partial updates between full updates.

Priority

You can set a priority for the writer using the priority configuration. A reader can be configured to discard samples based on the priority of the writer so that it only accept samples from the writer with the highest priority.

Listeners

Connected keys

The writer maintains a set of connected keys. This set represents all the keys for which readers are connected. The two listener functions registered with the writer's onConnectedKeys method enable the application to monitor the changes to this set.

C++
enum struct CallbackReason
{
    Initialize,
    Add,
    Remove
};

template<typename Key, typename Value, typename UpdateTag> class Writer
{
    void onConnectedKeys(std::function<void(std::vector<Key>)> init, std::function<void(CallbackReason, Key)> update);
};

The init callback is called with the initial set of connected keys. It's called immediately after the onConnectedKeys method returns. The update callback accepts two parameters: an enumeration to specify if the callback is called following the connection or disconnection of a key and the key.

Connected readers

Readers connected with the writer can be monitored by registering two callback functions with the onConnectedReaders method:

C++
template<typename Key, typename Value, typename UpdateTag> class Writer
{
    void onConnectedReaders(std::function<void(std::vector<std::string>)> init, std::function<void(CallbackReason, std::string)> update);
};

The init callback is called with the initial set of connected readers. It's called immediately after the onConnectedReaders method returns. The update callback accepts two parameters: an enumeration to specify if the callback is called following the connection or disconnection of a reader and the name of the reader.

Reader names are specified when the reader is created. If no name is provided, a default name is computed by DataStorm. This name is guaranteed to be unique within the scope of the DataStorm node.

Coordination

The writer class supports methods to coordinate with readers:

  • hasReaders to check if readers are connected
  • waitReaders to wait for a given number of readers to connect
  • waitForNoReaders to wait for all the readers to disconnect

In addition, the getConnectedKeys method returns the set of keys for which readers are connected to and the getConnectedReaders method returns the names of the connected readers. The wait methods will raise NodeShutdownException if the node is shutdown.