How do I migrate from Harvest 1.6 to 2.0?

[cgrinds]

Harvest 2.0 only supports the time-series databases: Prometheus and InfluxDB. Harvest 1.6 used Graphite, which Harvest 2.0 currently does not support. More details in the FAQ.

[luddite516]

From my recent experience, "forklift upgrade". I am excited by the rewrite of harvest and its new capabilities; however getting here from the "old" Harvest/Graphite has been way more complex then I had anticipated. Its probably not accurate to call it an upgrade, since upgrade implies something which can be done in-place with minimal downtime/learning curve.

Understand that I am not knocking the new architecture. I do think that we have to evolve. But I would like to mention some of the challenges I am dealing with.

The first surprise: it was trivial in Graphite to specify the retention period of metrics. Its apparently not recommended to keep metrics in the Prometheus datastore for longer than a few weeks. We like to keep capacity metrics for a year or two to use them for forecasting/planning and for trend comparison when troubleshooting performance issues.
The second surprise was that Prometheus does not do any downsampling of the data (thinning out data points over time to make long-range queries perform better). Graphite did this out of the box.
Third surprise: say what you want about the inefficiency and old-fashionedness of Graphite storing metrics in a pile of text files, but you have to admit they were very easy to understand, manage, and troubleshoot. Prometheus has been a very steep learning curve for me.
Fourth: I am being forced to essentially rebuild all of our Grafana dashboards (about 30 of them) to migrate from the Graphite metric syntax to Prometheus. Fringe benefit of this: it has been a crash-course in learning how Prometheus queries work.

To solve the above problems, I chose to install Thanos to take care of long-term storage and downsampling of data. It seems to do the job well, but this has significantly increased the complexity of the system.

[Flo-Fly]

Hi Luddite

i'm a step behind you...i'm facing a problem with thanos querier down sampling, when i'm querying through grafana i did not find the way to have auto down sampling used, i mean i must use raw data or 5mn or 1h. did you meet this issue? thanks

[werenzo]

Fourth: I am being forced to essentially rebuild all of our Grafana dashboards (about 30 of them) to migrate from the Graphite metric syntax to Prometheus. Fringe benefit of this: it has been a crash-course in learning how Prometheus queries work.

Hello @luddite516, all

I'm gathering informations because we currently use 1.6 with default dashboards. Do you think I'll forcibly rebuild dashboard or harvest 2.0 with Prometheus comes with same default dashboards? (Cluster, Volume and so on)

Thank you

Lorenzo

[luddite516]

Yes I did have that problem. I partially solved it by setting the option '--query.auto-downsampling' in the Thanos Query commandline. I still have the problem that if I set the timescale to a long time (greater than 90 days), the graph does not show all of the data. Not sure why.

…

On Wed, Apr 27, 2022 at 11:54 AM Flo-Fly ***@***.***> wrote: Hi Luddite i'm a step behind you...i'm facing a problem with thanos querier down sampling, when i'm querying through grafana i did not find the way to have auto down sampling used, i mean i must use raw data or 5mn or 1h. did you meet this issue? thanks — Reply to this email directly, view it on GitHub <#659 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKO2LMS62IU3JG5FVJOJVVLVHFPMVANCNFSM5HX4S4XQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Flo-Fly]

what is your retention for raw data? i also enabled '--query.auto-downsampling' but did not see any enhancement. my explanation is that when the query comes from grafana by default raw data is queried unless you specify another sampling. things that was not happening with graphite. i keep you posted if i find something thanks

[werenzo]

Hello,

harvest 1.6 had the option to gather capacity metrics from Oncommand Unified Manager. Do 2.0 have this kind of interoperability?

Thanks

Lorenzo

[cgrinds]

hi @werenzo Harvest 2 does not collect data from OnCommand Unified Manager