-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support limited non-alphanumeric metadata keys #3354
Support limited non-alphanumeric metadata keys #3354
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks OK to me, but...do we guard against metadata keys containing dots, colons, and any other characters that we have special semantics for?...and do we have tests that ensure that we disallow the things we're supposed to disallow? And, I found a typo and I have a code suggestion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good; just two "small questions".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
PBENCH-1111 Internally defined and managed Pbench Server metadata keys are lowercase alphanumeric strings; however the dataset.metalog namespace is extracted directly from the tarball metadata.log file and some benchmarks (especially fio and uperf generate tool and iteration keys that aren't accessible because they fail the normal key path validation checks. This PR loosens validation checks in some paths so `GET` APIs that retrieve metadata will be able to find these keys, for example uperf iterations like `dataset.metalog.iterations/1-tcp_stream-64B-1i`. (NOTE: I first noticed this problem when I added general metadata filtering, and made special allowances in the `filter` keyword processing, but at the time didn't want to expand the scope to deal with more general handling. I was reminded when I added the key aggregator, and finally I'm fixing it.)
(There could always be more, but I think this covers the basics.)
Some more test cases, but also now recursively validating JSON keys to avoid that loophole.
Confirm that multiple unacceptable JSON keys are collected across various levels of the JSON tree.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
PBENCH-1111
Internally defined and managed Pbench Server metadata keys are lowercase alphanumeric strings; however the dataset.metalog namespace is extracted directly from the tarball metadata.log file and some benchmarks (especially fio and uperf generate tool and iteration keys that aren't accessible because they fail the normal key path validation checks.
This PR loosens validation checks in some paths so
GET
APIs that retrieve metadata will be able to find these keys, for example uperf iterations likedataset.metalog.iterations/1-tcp_stream-64B-1i
.(NOTE: I first noticed this problem when I added general metadata filtering, and made special allowances in the
filter
keyword processing, but at the time didn't want to expand the scope to deal with more general handling. I was reminded when I added the key aggregator, and finally I'm fixing it.)