OpenNMS Minion

Minion is a next-generation design for distributed OpenNMS components. Initially, it will support SNMP and JMX data collection and in the future other scheduled tasks like polling. It is highly distributable and extremely configurable and uses Apache Karaf, Apache Camel and Apache ActiveMQ as underlaying technologies in the default implementation.

Minion can be used by simply adding a feature URL to a Karaf installation and performing some simple configuration. Karaf scripts that perform this setup are provided for both the OpenNMS server side as well as the Minion client side.

Terms

Dominion

The coordination/configuration controller for Minion clients which runs in Karaf. Currently the Dominion Controller only runs in the Karaf embedded inside OpenNMS.

Minion

The remote client which performs scheduled tasks and reports information back to a Dominion controller.

ActiveMQ

Apache's implementation of JMS (Java Message Service) which is used for communicating between Minions and the Dominion controller.

Requirements

The Dominion controller has the same requirements as OpenNMS since it runs inside OpenNMS's embedded Karaf.

The Minion should run on any platform where Java 8 is available.

Installing from RPMs

There are standalone RPMs available for running Minion clients and OpenNMS with an embedded Dominion controller.

Installing the Minion Client RPM

On the Minion system that will be performing data collections, download the opennms-minion RPM from http://yum.opennms.org/branches/develop/common/opennms/ and install it: rpm -Uvh opennms-minion-*.rpm

Installing the Minion Server Components

First, you should be running the latest version of OpenNMS from the develop branch. RPMs are available at http://yum.opennms.org/branches/develop/common/opennms/.

You'll need to install OpenNMS (at least opennms-core and opennms-webapp-jetty) as well as the Minion webapp from the same location (opennms-webapp-minion).

Configure a remote monitoring location in OpenNMS by following instructions at OpenNMS: Remote Polling.

• Create a polling package in poller-configuration.xml.
• Create a package in collectd-configuration.xml.
• Create a location-def in monitoring-locations.xml for the location you're polling. Make sure it has both the collection-package-name and polling-package-name attributes. TODO: Update this with instructions about using the monitoring locations UI.

Starting the Dominion service inside OpenNMS

1. Open the ActiveMQ port on the OpenNMS server by editing /opt/opennms/etc/opennms-activemq.xml. Change the transport connector configuration to listen on 0.0.0.0 instead of 127.0.0.1:

   <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?useJmx=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
   

2. Connect to the OpenNMS Karaf console with the same admin credentials from the web UI: ssh -p 8101 admin@localhost

3. Run the Dominion configuration script: source http://localhost:8980/minion/opennms-setup.karaf

This will update your local feature repository and then install the sample-receiver-activemq feature to allow OpenNMS to receive data collection samples via ActiveMQ. It will also install the sample-storage-rrd feature which will store the collected data using the currently configured data storage mechanism (RRD, JRobin, or Newts).

Starting the Minion Client

1. Edit the file /etc/sysconfig/minion and put in the URL for the root of your OpenNMS server, the name of your monitoring location, and (optionally) the URL for your ActiveMQ broker.
2. Start the Minion client: sudo systemctl start minion

---

Installing from Source

1. Build OpenNMS develop
2. Build Minion
3. From the Minion source build, copy sampler-repo-webapp/target/minion.war to your OpenNMS jetty-webapps/ directory.

Configuring OpenNMS

1. Configure a remote monitoring location in OpenNMS by following instructions at OpenNMS: Remote Polling.

   • Create a polling package in poller-configuration.xml.
   • Create a package in collectd-configuration.xml.
   • Create a location-def in monitoring-locations.xml for the location you're polling. Make sure it has both the collection-package-name and polling-package-name attributes. TODO: Update this with instructions about using the monitoring locations UI.

2. If you didn't already, copy the minion.war file from the source build above to your $OPENNMS_HOME/jetty-webapps directory.

3. Restart OpenNMS.

Configuring the Dominion Server in OpenNMS

1. Open the ActiveMQ port on the OpenNMS server by editing /opt/opennms/etc/opennms-activemq.xml. Change the transport connector configuration to listen on 0.0.0.0 instead of 127.0.0.1:

   <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?useJmx=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
   

2. Connect to the OpenNMS Karaf console with the same admin credentials from the web UI: ssh -p 8101 admin@localhost

3. Run the Dominion configuration script: source http://localhost:8980/minion/opennms-setup.karaf

If the Dominion Server is behind a firewall you have to open the following ports:

• 8980 The port number for the OpenNMS Webapp. The Minion needs this to download the minion-setup.karaf file.
• 61616 The port for the ActiveMQ JMS. The Minion needs this port to send messages to the Dominion Controller.

Configuring a Minion Client

Ensure that the computer system running the Minion client meets the requirements.

1. Download and unpack Karaf 2.4.0.

2. Run Karaf (bin/karaf)

3. Run the configuration script: source http://opennms-root:8980/minion/minion-setup.karaf instance username password opennms-root location-name

   • instance: possible values are root, activemq, minion, or sampler
   • username: The OpenNMS username allowed to make REST calls
   • password: The password for the REST user
   • opennms-root: The root URL of the OpenNMS server, not including the /opennms part of the URL. For example, if you normally connect to your OpenNMS web UI at http://www.example.com:8980/opennms/, you would use http://www.example.com:8980/ as the OpenNMS root. Note: Do not forget the protocol http or https, otherwise the installation will fail!
   • location-name: The name of the location the Minion is running at. This should match a location in the monitoring-locations.xml file on your OpenNMS server.

Validating Your Minion

All Minions need validation. ;)

If everything is connected correctly, you should be able to browse to the OpenNMS Minion console at http://opennms-host:8980/opennms/minion/index.jsp. Each Minion correctly connected to the Dominion Controller should be listed with the configured location and status Running. In addition each Minion should send data collection samples to the Dominion server.

Troubleshooting

If the Minion does not send samples, follow these instructions:

Ensure the Minion is setup correctly

1. Log in the Karaf Console
2. Set logging to debug: log:set DEBUG
3. Watch logs with log:tail

Ensure that there are no errors in the log file. The errors:

Memory Usage for the Broker (1024 mb) is more than the maximum available for the JVM: 455 mb - resetting to 70% of maximum available: 318 mb
Temporary Store limit is 51200 mb, whilst the temporary data directory: /root/apache-karaf-2.3.5/activemq-data/rackspace/tmp_storage only has 13394 mb of usable space - resetting to maximum available 13394 mb.

can be ignored.

Ensure the logs contain something like this:

org.apache.camel.camel-core - 2.13.2 | Received SampleSet with 184 sample(s)

The number of samples should be greater than 0. If not, watch for these urls:

blueprint-sampler-config.xml: Parsing agents from URL: http://opennms-root:8980/opennms/rest/config/agents/location-name/SNMP.xml
blueprint-sampler-config-snmp.xml: Parsing SNMP XML: http://opennms-root:8980/opennms/rest/config/snmp
blueprint-sampler-config.xml: parseJaxbXml: http://onms-root:8980/opennms/rest/config/location-name/collection

Manually invoke the URLs and see the result in your browser. The result should be a valid non-empty XML result.

In addition the result of the SNMP.xml must contain a sysObjectId, e.g.

<entry>
    <key>sysObjectId</key>
     <value>.1.3.6.1.x.x.x.x.x.x</value>
 </entry>

If everything is set up correctly, you should see the Received SampleSet with 184 sample(s) message every 5 minutes in the karaf log.

Ensure the Dominion is setup correctly

If you do not get resource graphs for your remote node, follow these instructions.

1. Ensure your Minion shows up as running at http://opennms-host:8980/opennms/minion/index.jsp
2. Stop OpenNMS
3. Delete $OPENNMS_HOME/share/rrd/snmp/*
4. Edit $OPENNMS_HOME/etc/service-configuration.xml and comment out service Collectd
5. Ensure that the sample-rrd-storage feature is installed

For this you have to login to the Dominion karaf

$ ssh -p 8101 admin@localhost 

List all installed features:

$ features:list | grep -i sample

You should get a list similar to this:

[installed  ] [15.0.0] sample-api                              opennms-sampler-15.0.0     Sample Collection and Storage :: API
[uninstalled] [15.0.0] minion-controller                       opennms-sampler-15.0.0     Minion :: Controller
[installed  ] [15.0.0] dominion-controller                     opennms-sampler-15.0.0     Dominion :: Controller
[uninstalled] [15.0.0] sampler-config                          opennms-sampler-15.0.0     Sample Collection and Storage :: Configuration
[uninstalled] [15.0.0] sampler-config-snmp                     opennms-sampler-15.0.0     Sample Collection and Storage :: SNMP Configuration
[uninstalled] [15.0.0] sampler-scheduler                       opennms-sampler-15.0.0     Sample Collection and Storage :: Scheduler
[uninstalled] [15.0.0] sampler-snmp                            opennms-sampler-15.0.0     Sample Collection and Storage :: SNMP Collector
[uninstalled] [15.0.0] sample-graphs                           opennms-sampler-15.0.0     Sample Collection and Storage :: Graphing
[installed  ] [15.0.0] sample-storage-rrd                      opennms-sampler-15.0.0     Sample Collection and Storage :: RRD
[uninstalled] [15.0.0] sample-storage-newts                    opennms-sampler-15.0.0     Sample Collection and Storage :: Newts
[uninstalled] [15.0.0] sample-storage-file                     opennms-sampler-15.0.0     Sample Collection and Storage :: File Storage
[uninstalled] [15.0.0] sample-storage-xml                      opennms-sampler-15.0.0     Sample Collection and Storage :: XML Storage
[uninstalled] [15.0.0] sample-dispatch-activemq                opennms-sampler-15.0.0     Sample Collection and Storage :: ActiveMQ Dispatcher
[installed  ] [15.0.0] sample-receiver-activemq                opennms-sampler-15.0.0     Sample Collection and Storage :: ActiveMQ Receiver
[uninstalled] [15.0.0] sampler-with-file-storage               opennms-sampler-15.0.0     Sample Collection and Storage :: Integration :: Sampler with file storage
[uninstalled] [15.0.0] sampler-with-xml-storage                opennms-sampler-15.0.0     Sample Collection and Storage :: Integration :: Sampler with XML storage
[uninstalled] [15.0.0] sampler-with-rrd-storage                opennms-sampler-15.0.0     Sample Collection and Storage :: Integration :: Sampler with RRD storage
[uninstalled] [15.0.0] sampler-with-activemq-export            opennms-sampler-15.0.0     Sample Collection and Storage :: Integration :: Sampler with ActiveMQ export

Ensure that sample-receiver-activemq and sample-storage-rrd is installed. If it is not, install it manually:

$ features:install sample-storage-rrd

After this you should see rrd files at $OPENNMS_HOME/share/rrd/snmp.

Error executing command: Unable to add repositories

If you encounter an error like this:

Error executing command: Unable to add repositories:
    Error resolving artifact org.apache.activemq:activemq-karaf:xml:features:5.10.0: Could not find artifact org.apache.activemq:activemq-karaf:xml:features:5.10.0

You have very likely not set up the org.ops4j.pax.url.mvn.cfg config file correctly.

Check the logs of your Karaf and look for something like java.net.MalformedURLException: no protocol: 192.168.0.2:8980/minion/. This indicates that you have forgotten to set the protocol of the URL (http or https). Please invoke the original installation script with the http|https protocol.