Skip to content

Document replication

Jump to bottom
Dannes Wessels edited this page Apr 27, 2014 · 35 revisions

For the document [replication](http://en.wikipedia.org/wiki/Replication_(computing\)) feature two types of eXist-db instances are involved: The instance in which a document is added or modified (the producer or 'master' instance) and the instances that shall receive a copy of the document(change): the consumer or 'slave' instances.

The JMS broker receives the document from the producer instance and distributes the data to one or more consumer instances. Theoretically there is no limit on the number of consumer instances, the performance of the producer instance is independent of the number of consumer instances.

During the XML Prague 2013 pre-conference a presentation was given about the replication feature. The slides are available and there is a YouTube recording.

Producer (Master)

The replication extension uses eXist-db triggers to detect when document has been changed. Triggers must defined in collection.xconf files on a per collection basis.

The following XML example shows to connect to an ActiveMQ server on the myserver.local host, the topic name is dynamicTopics/eXistdb:

<collection xmlns="http://exist-db.org/collection-config/1.0">
    <triggers>
        <trigger class="org.exist.replication.jms.publish.ReplicationTrigger">
            <!-- 
                Class name of the initial context provider, default value 
                for ActiveMQ
                see javax.naming.Context#INITIAL_CONTEXT_FACTORY
            -->
            <parameter name="java.naming.factory.initial" 
                value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
            <!-- 
                URL of the message broker, default value for ActiveMQ 
                see javax.naming.Context#PROVIDER_URL
            --> 
            <parameter name="java.naming.provider.url" 
                          value="tcp://myserver.local:61616"/>
            <!-- 
                Lookup connection factory
                see javax.naming.InitialContext#lookup(String) 
            -->
            <parameter name="connection-factory" value="ConnectionFactory"/>
            <!--
                Lookup destination (topic)
                see javax.naming.InitialContext#lookup(String) 
            -->
            <parameter name="destination" value="dynamicTopics/eXistdb-replication-example"/>
        </trigger>
    </triggers>
</collection>

Consumer (Slave)

For receiving documents, a document receiver (JMS listener) must be started at each consumer instance. This can be done via two routes:

  1. Define a StartupTrigger in`conf.xml'
  2. Call an XQuery function

Both are described in the following paragraphs.

conf.xml

For this route the configuration file conf.xml must be extended with the following XML fragment; when finished the database must be restarted to effectuate the change.

The example connects with the same configuration paramaters to the same server the producer example has been connected with.

<!--
    Start JMS listener for listener of the clustering feature. 
-->
<trigger 
    class="org.exist.replication.jms.subscribe.MessageReceiverStartupTrigger">
    <!-- 
        Class name of the initial context provider, default value for ActiveMQ
        see javax.naming.Context#INITIAL_CONTEXT_FACTORY
    -->
    <parameter name="java.naming.factory.initial" 
        value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
    
    <!-- 
        URL of the message broker, default value for ActiveMQ
        see javax.naming.Context#PROVIDER_URL
    -->
    <parameter name="java.naming.provider.url" 
        value="tcp://myserver.local:61616"/>
    
    <!-- 
        Lookup connection factory
        see javax.naming.InitialContext#lookup(String) 
    -->
    <parameter name="connection-factory" value="ConnectionFactory"/>
    
    <!--
        Lookup destination (topic)
        see javax.naming.InitialContext#lookup(String) 
    -->
    <parameter name="destination" value="dynamicTopics/eXistdb-replication-example"/>
    
    <!-- 
        Set the name used to identify this subscription
        see JMS javax.jms.TopicSession#createDurableSubscriber(Topic,String) 
    -->
    <parameter name="subscriber.name" value="SubscriptionId"/>

    <!-- 
        Set the client identifier for the connection
        see JMS javax.jms.Connection#setClientID(String) 
    -->
    <parameter name="connection.client-id" value="ClientId"/>
    
</trigger>

Xquery function

Starting the replication listener can be as easy as calling the replication:register($jmsConfiguration) function:

xquery version "3.0";
import module namespace replication="http://exist-db.org/xquery/replication" 
              at "java:org.exist.replication.xquery.ReplicationModule"; 
let $jmsConfiguration := map {
        "java.naming.factory.initial" 
                := "org.apache.activemq.jndi.ActiveMQInitialContextFactory",
        "java.naming.provider.url" := "tcp://myserver.local:61616",
        "connection-factory" := "ConnectionFactory",
        "destination" := "dynamicTopics/eXistdb-replication-demo",
        "subscriber.name" := "SubscriptionId",
        "connection.client-id" := "ClientId"
    }
              
return
    replication:register($jmsConfiguration)

To have the function started au database startup, please consult (####).

Note that this functions is available starting v0.7 of the extension.

Quick test

  • Setup an ActiveMQ server
  • download distribution from http://activemq.apache.org/download.html: The ZIP file for Windows, the TGZ for Unix.
  • start server (for testing purposes almost no configuration changes required)
  • Setup and configure one or more 'consumer' eXist-db instances
  • modify conf.xml as hinted
  • create collection /db/replicated
  • start servers
  • Setup one 'master' eXist-db instance
  • start server
  • start eXide
  • create the collection /db/replicated
  • create a collection.xconf' file, store it in /db/replicated`. eXide asks if the file needs to be processed. Please do.
  • Test document creation replication
    • execute the following query.
    • verify the 'consumer' instances, each instance should contain 10 documents in /db/replicated 
let $doc := <doc>just some data</doc>
for $i in (1000 to 1010)
return
    xmldb:store('/db/replicated', concat('mydoc', $i , ".xml"), $doc)
  • Test document removal replication
  • execute the following query.
  • verify the 'consumer' instances, each instance should contain 0 documents in /db/replicated 
for $i in (1000 to 1010)
return
    xmldb:remove('/db/replicated/', concat('mydoc', $i , ".xml"))
Clone this wiki locally