Micro Benchmarking

Micro-benchmarks are recording observables that measure resource usage of the whole program for a specific time. These measurements are then associated with the subsystem that was observed at that time. Caveat: if the executable under observation runs on a multiprocessor computer where more than one parallel thread executes at the same time, it becomes difficult to associate resource usage to a single function. Even more so, as Haskell’s thread do not map directly to operating system threads. So the expressiveness of our approach is only valid statistically when a large number of observables have been captured.

Counters

The framework provides access to the following O/S counters (defined in ObservableInstance) on Linux:
• monotonic clock
• CPU or total time (/proc//stat)
• memory allocation (/proc//statm)
• network bytes received/sent (/proc//net/netstat)
• disk input/output (/proc//io)

On all platforms, access is provided to the RTS counters.

Implementing micro-benchmarks

Capturing observables in a STM evaluation

In a micro-benchmark we capture operating system counters over an STM evaluation or a function, before and afterwards. Then, we compute the difference between the two and report all three measurements via a Trace to the logging system. Here we refer to the example that can be found in complex example.

STM.bracketObserveIO trace "observeSTM" (stmAction args)

The capturing of STM actions is defined in Cardano.BM.Observer.STM. The function STM.bracketObserveIO has type:

bracketObserveIO :: Configuration → Trace IO a → Severity → Text → STM.STM t → IO t

It accepts a Trace to which it logs, adds a name to the context name and enters this with a SubTrace, and finally the STM action which will be evaluated. Because this evaluation can be retried, we cannot pass to it a Trace to which it could log directly. A variant of this function bracketObserveLogIO also captures log items in its result, which then are threaded through the Trace.

Capturing observables in IO

Capturing observables for a function evaluation in IO, the type of bracketObserveIO (defined in Cardano.BM.Observer.Monadic) is:

bracketObserveIO :: Configuration → Trace IO a → Severity → Text → IO t → IO t

It accepts a Trace to which it logs items, adds a name to the context name and enters this with a SubTrace, and then the IO action which will be evaluated.

example of bracketObserveIO

bracketObserveIO trace "observeDownload" $ do
    license ← openURI "http://www.gnu.org/licenses/gpl.txt"
    case license of
       Right bs → logInfo trace $ pack $ BS8.unpack bs
       Left e → logError trace $ "failed to download; error: " ++ (show e)
    pure ()

Counters are evaluated before the evaluation and afterwards. We trace these as log items ObserveOpen and ObserveClose, as well as the difference with type ObserveDiff.

Configuration of mu-benchmarks

Observed STM actions or IO functions enter a new named context with a SubTrace. Thus, they need a configuration of the behaviour of this SubTrace in the new context. We can define this in the configuration for our example:

CM.setSubTrace c "complex.observeDownload" (Just $ ObservableTrace [NetStats,IOStats])

This enables the capturing of network and I/O stats from the operating system. Other observables are implemented in Cardano.BM.Data.Observable.
Captured observables need to be routed to backends. In our example we configure:

CM.setBackends c "complex.observeIO" (Just [AggregationBK])

to direct observables from named context "complex.observeIO" to the Aggregation backend.