Updates to g32influx script #107
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add argparse for passing in targets to move out of testing phase. Swap to a nestable progress bar, and add documentation.
|
Thanks for the PR, Adam! Getting that kind of speed improvement would be great. I'm looking forward to reviewing this, but probably won't be able to look much at it until early next week, after our SMuRF/DAQ meeting at Princeton. |
BrianJKoopman
force-pushed
the
g32influx-script
branch
from
63a9be0 to
a64b8b0
Compare
BrianJKoopman
added a commit
that referenced
this pull request
Jul 17, 2020
This incorporates several changes from #107, but with some modifications. Most notably it includes format_timeline(), which comes with a dramatic performance increase, both by publishing all fields that share a timeline simultaneously and by utilizing InfluxDB's line protocol, which is faster than the previously used json based protocol. We maintain the use of a local sqlite DB instead of switching to using InfluxDB to track published .g3 files. We also remove the use of an additional md5sum tag on data uploaded with this script. Many thanks to @ahincks for the work this was based on.
BrianJKoopman
added a commit
that referenced
this pull request
Jul 17, 2020
This incorporates several changes from #107, but with some modifications. Most notably it includes format_timeline(), which comes with a dramatic performance increase, both by publishing all fields that share a timeline simultaneously and by utilizing InfluxDB's line protocol, which is faster than the previously used json based protocol. We maintain the use of a local sqlite DB instead of switching to using InfluxDB to track published .g3 files. We also remove the use of an additional md5sum tag on data uploaded with this script. Many thanks to @ahincks for the work this was based on.
BrianJKoopman
added a commit
that referenced
this pull request
Jul 17, 2020
* Start work on g32influx script * Handle new filename format * Incorporate select changes from #107 This incorporates several changes from #107, but with some modifications. Most notably it includes format_timeline(), which comes with a dramatic performance increase, both by publishing all fields that share a timeline simultaneously and by utilizing InfluxDB's line protocol, which is faster than the previously used json based protocol. We maintain the use of a local sqlite DB instead of switching to using InfluxDB to track published .g3 files. We also remove the use of an additional md5sum tag on data uploaded with this script. Many thanks to @ahincks for the work this was based on. * gitignore: Ignore any local databases * docs: Add documentation for g32influx
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As I work on assessing the scalability of
influxdbas an "official" storage format, I've made some modifications to this script.Speed is greatly improved (on my laptop, ~5 seconds per file instead of ~1 minute):
influxdbDB.Rather than using a standalone
sqliteDB to record which files have been written, there is now a measurement in theinfluxdbDB,.g32influx.log, that records when files have been completed (and also records if they have been started, as well as if an error has occurred. This can be queried to determine whether a file needs to be written or not (which can be overridden with-f).In addition to the
feedtag, anmd5sumtag is added to each HK point so that the provenance of data can be traced.Although I'm creating this pull request here so that you can see the changes, it could reflect a simple branch of this code away from
ocsas I continue my own investigation. So, if you have concerns about anything I've done here and don't think this belongs inocs, no problem.