Application Performance Monitoring

We collect metrics about the performance of our Ruby on Rails app, such as throughput, latency or error rates of the various components we use in this app.

An entry point to view those metrics is our Application Performance Overview Dashboard on https://obs-measure.opensuse.org/.

You can login with your GitHub account and should get the Editor role.

Our APM stack consists of:

InfluxDB and influxdb-rails gem

The metrics are stored into InfluxDB with the influxdb-rails gem. The data source is InfluxDB-Rails.

The gem consume the events that Rails instruments by default and stores them directly to InfluxDB using the configuration defined in our application.

How to Send Metrics

# Replace $METRIC_NAME$ and $METRIC_VALUE$ with the according name/value.
# Tags are useful to filter metrics, add them as needed in the hash for `tags`.
InfluxDB::Rails.instrument('$METRIC_NAME$', tags: { a: 1, foo: 'bar' }, values: { value: $METRIC_VALUE$ })

Best Practices

Whenever possible, we extract instrumentation code to the src/api/app/instrumentations directory. This is possible with the use of concerns which are then included in the class which they instrument.

In controllers, rely on filters (all filters are listed here). In ActiveRecord models, rely on callbacks from ActiveRecord::Callbacks. In ActiveModel models, you can also rely on callbacks, but they are from ActiveModel::Callbacks. Using ActiveModel::Callbacks is also possible in other classes by extending the classes with ActiveModel::Callbacks.

Additional Tags

Additionally to the tags and values stored by influxdb-rails we also track this data for every measurement:

Tags:

{
  interface: :webui # [:api, :webui, :job, :clock, :backend]
  beta: false # [true, false]
  anonymous: false # [true, false]
}

OBS Backend instrumentation

We also measure the OBS backend responses with our custom influxdb-rails subscriber.