Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Custom Collectors #90

Open
directionless opened this issue Jul 20, 2018 · 11 comments
Open

Support for Custom Collectors #90

directionless opened this issue Jul 20, 2018 · 11 comments

Comments

@directionless
Copy link

It'd be great to have support for custom collectors in the ruby gem. I see them in both python and java.

Any chance of implementation here?

@dmagliola
Copy link
Collaborator

Hi @directionless,
I would like to understand this problem better...
Do you have any examples where you need this functionality?
Thank you

@a0s
Copy link

a0s commented Sep 1, 2019

+1 for Custom Collectors
Examples of usage:

  1. Getting instantly values from ActiveRecord::Base.connection_pool.stat
  2. Getting instantly values from Puma server (through control socket)
  3. Getting instantly values from GC.stat
  4. etc..

@directionless
Copy link
Author

directionless commented Sep 4, 2019

Since submitting this, I've stopped working on the project that needed this. However, the need was real, and I believe should I pick up ruby and prometheus I'll encounter it again.

It generally centers around importing metrics that are generated elsewhere. The python docs have a reasonably clear explanation:

Sometimes it is not possible to directly instrument code, as it is not in your control. This requires you to proxy metrics from other systems.

@a0s has some good examples.

Mine were business/app data, for example:

  • total users User.count
  • active users ActiveRecord::SessionStore::Session.where("updated_at > ?", 15.minutes.ago).count
  • Similar for other important data models
  • Etc.

I usually want these metrics for several reasons. First, it makes for a simple business analytics system. I've already got metrics and graphs, I may as well add in a little bit of BI. Second, when debugging application issues, I usually also want those numbers on a graph. Third, I may want to set alerts. More concretely, I probably know that my application performance is strongly correlated with the number of widgets in the system. So I want to be monitoring that widget count, and comparing it.

For that use case, I also ran into #9 and ended up writing a fork of the gitlab prometheus-client-mmap gem. Submitted upstream at https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/24

@a0s
Copy link

a0s commented Sep 4, 2019

@directionless

BTW,

total users User.count
active users ActiveRecord::SessionStore::Session.where("updated_at > ?", 15.minutes.ago).count

I think there is a better place for this. There is postgres_exporter. You able to add custom metrics with custom SQL scripts. Moreover, making requests like a User.count on a Postgres-side is better for performance.

@lawrencejones
Copy link
Contributor

Hey @directionless!

I totally get your use case, and I think they are valid things to want to export in Prometheus. We export many similar metrics from our services.

Can I ask what is missing to support this, though? I've always produced a Collector class that adheres to the Rack middleware interface, then place this in the scrape middleware chain and have it pull the metrics just before responding to a scrape.

As a result, I'm not sure what we need from the library that is missing?

@a0s
Copy link

a0s commented Sep 4, 2019

@directionless I open-sourced my small library for client_ruby. And add prototype of CustomCollector.

@directionless
Copy link
Author

directionless commented Sep 5, 2019

To reiterate -- I am no longer working on the project that had this use case. So I cannot easily evaluate how well this would replace it. But, comments inline...

I think there is a better place for this. There is postgres_exporter.

While I think that's better for some things, as active record models get more complicated, I'm very leery of separating the reporting logic from it. And, as in your library, that's not going to work for puma stats.

Can I ask what is missing to support this, though? I've always produced a Collector class that adheres to the Rack middleware interface, then place this in the scrape middleware chain and have it pull the metrics just before responding to a scrape.

That sounds interesting, and may cover this, but I don't know that I understand what that would do. Initially it sounds like it would work, is there an example somewhere of how that would work? I can imagine it either exactly meeting the original request for custom collectors. Or feeling like re-implementing a lot of existing features.

There's probably also something caveat about timing. Some metrics are cheap and I want to pull them frequently. Other metrics are expensive, and I want to pull them less frequently. Though perhaps that's best done by sticking some cache+ttl in the metric pull.

@dmagliola
Copy link
Collaborator

dmagliola commented Sep 6, 2019

@lawrencejones How do you feel about having a CustomCollector class that basically implements that Rack interface, and calls a couple of overridable methods (before, after, around, maybe?) so that all you need to do is subclass it, set up your metrics declaration, and override one of those methods to set / increment those metrics in them. (And add to the middleware chain)

It's essentially the same as you said, but it may be more clear / easier to use as an interface?

Either way, I think this is mostly solved by documentation. We possibly don't need that class, just a code snippet in the README on how you'd implement your own custom collector that is easy to understand.

Either way, I don't think this should block the 1.0 release

@lawrencejones
Copy link
Contributor

I'm not sure, thinking about it, a metric that when read fetches its value could be really useful, especially in the fork world.

But it's entirely separate from the 1.0 release- literally nothing should hold that back, there is no reason to delay the 1.0 release as far as I can see. The number of people who adopt Prometheus in ruby every day and use the pre 1.0 grows every day, so the faster this is released the better for the community.

@dmagliola
Copy link
Collaborator

Agreed on that last point. Will get your thoughts on the metric that fetches when read next time I see you :-)

@hiroshi
Copy link

hiroshi commented Nov 17, 2020

I made quick and dirty hack to this kind of purpose for my rails app.

# config/initializers/prometheus.rb
require 'prometheus/client'
Prometheus::Client.registry.gauge(:room_temperature_celsius, docstring: '...', labels: [:room])
# app/controller/prometheus_exporter.rb
require 'prometheus/client'
require 'prometheus/client/formats/text'

class PrometheusExporterController < ApplicationController
  def index
    gauge = Prometheus::Client.registry.get(:room_temperature_celsius)
    gauge.set(21.534, labels: { room: 'kitchen' })
    render plain: Prometheus::Client::Formats::Text.marshal(Prometheus::Client.registry)
  end
end
# config/routes.rb
get '/metrics' => 'prometheus_exporter#index', format: false

And It works for me.

$ curl http://localhost:5000/metrics
# TYPE room_temperature_celsius gauge
# HELP room_temperature_celsius ...
room_temperature_celsius{room="kitchen"} 21.534

dmexe added a commit to bookmate/bm-instrumentations that referenced this issue May 18, 2021
The gem `prometheus-client` doesn't support a custom collectors [issue
90](prometheus/client_ruby#90). This PL add
missing functionality, but only if the management server is used.
dmexe added a commit to bookmate/bm-instrumentations that referenced this issue May 18, 2021
The gem `prometheus-client` doesn't support a custom collectors [issue
90](prometheus/client_ruby#90). This PL add
missing functionality, but only if the management server is used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants