Document replication

For the document [replication](http://en.wikipedia.org/wiki/Replication_(computing\)) feature two types of eXist-db instances are involved: The instance in which a document is added or modified (the producer or 'master' instance) and the instances that shall receive a copy of the document(change): the consumer or 'slave' instances.

The JMS broker receives the document from the producer instance and distributes the data to one or more consumer instances. Theoretically there is no limit on the number of consumer instances, the performance of the producer instance is independent of the number of consumer instances.

During the XML Prague 2013 pre-conference a presentation was given about the replication feature. The slides are available and there is a YouTube recording.

Producer (Master)

The replication extension uses eXist-db triggers to detect when document has been changed. Triggers must defined in collection.xconf files on a per collection basis.

The following XML example shows to connect to an ActiveMQ server on the myserver.local host, the topic name is dynamicTopics/eXistdb:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <triggers>
        <trigger class="org.exist.replication.jms.publish.ReplicationTrigger">
            <!-- 
                Class name of the initial context provider, default value 
                for ActiveMQ
                see javax.naming.Context#INITIAL_CONTEXT_FACTORY
            -->
            <parameter name="java.naming.factory.initial" 
                value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
            <!-- 
                URL of the message broker, default value for ActiveMQ 
                see javax.naming.Context#PROVIDER_URL
            --> 
            <parameter name="java.naming.provider.url" 
                          value="tcp://myserver.local:61616"/>
            <!-- 
                Lookup connection factory
                see javax.naming.InitialContext#lookup(String) 
            -->
            <parameter name="connection-factory" value="ConnectionFactory"/>
            <!--
                Lookup destination (topic)
                see javax.naming.InitialContext#lookup(String) 
            -->
            <parameter name="destination" value="dynamicTopics/eXistdb-replication-example"/>
        </trigger>
    </triggers>
</collection>

Consumer (Slave)

For receiving documents, a document receiver (JMS listener) must be started at each consumer instance. For this the configuration file conf.xml must be extended with the following XML fragment, the database must be restarted for this.

The example connects with the same configuration paramaters to the same server the producer example has been connected with.

<!--
    Start JMS listener for listener of the clustering feature. 
-->
<trigger 
    class="org.exist.replication.jms.subscribe.MessageReceiverStartupTrigger">
    <!-- 
        Class name of the initial context provider, default value for ActiveMQ
        see javax.naming.Context#INITIAL_CONTEXT_FACTORY
    -->
    <parameter name="java.naming.factory.initial" 
        value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
    
    <!-- 
        URL of the message broker, default value for ActiveMQ
        see javax.naming.Context#PROVIDER_URL
    -->
    <parameter name="java.naming.provider.url" 
        value="tcp://myserver.local:61616"/>
    
    <!-- 
        Lookup connection factory
        see javax.naming.InitialContext#lookup(String) 
    -->
    <parameter name="connection-factory" value="ConnectionFactory"/>
    
    <!--
        Lookup destination (topic)
        see javax.naming.InitialContext#lookup(String) 
    -->
    <parameter name="destination" value="dynamicTopics/eXistdb-replication-example"/>
    
    <!-- 
        Set the name used to identify this subscription
        see JMS javax.jms.TopicSession#createDurableSubscriber(Topic,String) 
    -->
    <parameter name="subscriber.name" value="SubscriptionId"/>

    <!-- 
        Set the client identifier for the connection
        see JMS javax.jms.Connection#setClientID(String) 
    -->
    <parameter name="connection.client-id" value="ClientId"/>
    
</trigger>

Quick test

• Setup an ActiveMQ server
• download distribution from http://activemq.apache.org/download.html: The ZIP file for Windows, the TGZ for Unix.
• start server (for testing purposes almost no configuration changes required)
• Setup and configure one or more 'consumer' eXist-db instances
• modify conf.xml as hinted
• create collection /db/replicated
• start servers
• Setup one 'master' eXist-db instance
• start server
• start eXide
• create the collection /db/replicated
• create a collection.xconf' file, store it in /db/replicated`. eXide asks if the file needs to be processed. Please do.
• execute the following query:

let $doc := <doc>just some data</doc>
for $i in (1000 to 3000)
return
    xmldb:store('/db/replicated', concat('mydoc', $i , ".xml"), $doc)

• Verify the 'consumer' instances, each instance should contain 2000 documents in /db/replicated