Skip to content

Conversation

@pmai
Copy link
Contributor

@pmai pmai commented Jun 11, 2025

No description provided.

@pmai pmai self-assigned this Jun 11, 2025
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
@pmai pmai force-pushed the feature/automatic-checks branch from aa3e037 to a42942d Compare June 11, 2025 18:17
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
@pmai pmai force-pushed the feature/automatic-checks branch from 49cc25d to 4ca4d18 Compare June 20, 2025 22:47
)

if expected_version is None:
logging.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider trying to extract the version from the trace file name and only fall back to OSI_FALLBACK_VERSION if that extraction fails. This approach would make it easier to use the tool from the CLI (at a later stage) without requiring the user to explicitly specify the version. Of course, providing the version explicitly via an CLI argument could still remain a valid option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be the idea, however I wanted to delay that code to the point in time when we integrate MCAP support, as we can then do it generically, rather than just special coding for the single-trace files today.

level=IssueSeverity.ERROR,
rule_uid=exp_version_rule_uid,
)
check_message_against_rules(message, rule_uid_map, result)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we agreed to use the .yml files internally and largely hide them from the user, I see potential in allowing their use for custom rule definitions. This could enable users to define additional .yml files to check not only for standard conformity but also for custom requirements - such as a minimum number of detections, the presence of specific fields, and so on. These files could also be exchanged between collaborating companies prior to exchanging trace files.

We could reuse a lot of the present code and might want to restructure it now already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I agree that this is the idea - the change should be trivial already (just the option), but I'd postpone this to a new PR, as this is a simple orthogonal change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given I was reorganizing the cli interface for better interactive use already - due to other requirements, I implemented this change now as well, as it was easier to do this way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

I tested the CLI with a custom trace file and the default config and naively did not specify the trace files type. This causes an exception (KeyError) as config.get_config_param("osiType") is None an thus this line fails:

trace = OSITrace(
        config.get_config_param("InputFile"), config.get_config_param("osiType")
    )

As you noted here that you'd prefer to postpone the "guessing" of the trace file type, I suggest we add a simple check for it during argument parsing in args_entrypoint() for now.

    parser.add_argument(
        "--output_config",
        type=pathlib.Path,
        help="Path to save the configuration after running the checks.",
    )
+   args = parser.parse_args()

+   if args.input_file and not args.osiType:
+       parser.error("argument --input_file requires --osiType to also be specified")

-   return  parser.parse_args()        
+   return args

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check actually needs to be a check on the config, as the source of osiType, like inputFile, can be either a loaded config or argument handling or both.

That said none of the checkers in the framework currently do not do any validation on the config, blindly assuming that e.g. inputFile will be set. So there is probably room to debate how the bundles should treat missing required config settings, so that these are aligned in their treatment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In any case there is a fallback to SensorView in the current code, which is just not consistently used - hence will rework that to be more apparent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is probably room to debate how the bundles should treat missing required config settings, so that these are aligned in their treatment.

Agreed. I just wanted to highlight that is easy to cause general exceptions arbitrarily. This should be avoided, as it can leave users unaware of what went wrong in terms of configuration without access to the source code.

rulename = rule_name_from_rule(rule)
rule_uid = result.register_rule(
checker_bundle_name=constants.BUNDLE_NAME,
checker_id=osirules_constants.CHECKER_ID,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states:

A check should address exactly one rule, unless there is a strong reason not to. In exceptional cases, a check may address multiple rules, or multiple checks may be required to check a particular rule.

While this technically allows multiple rules, it also spams the log:

2025-06-24 14:00:13,496 - There are 175 rules registered to the check osirules_osi. A check should address exactly one rule, unless there is a strong reason not to. See the following document for more information: https://github.com/asam-ev/qc-framework/blob/main/doc/manual/checker_library.md#check-characteristics

There is also this issue: asam-ev/qc-framework#163

Should we then not only create rules dynamically but also checks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some discussion on the qc framework side whether this rule "1 check per 1 rule" should not be loosened more, in which case the warning would be removed.

That said we might want to have some more fine-granular checkers than just one, however it is unclear that we really want to have one per rule, as that is likely to grow the output file substantially, and makes the overview potentially unwieldy - since we have potentially better ways to e.g. remove a single rule from consideration, this might also agitate against having strictly 1:1 mappings here. Maybe group by field, for example.

Given this uncertainty I would stay with the one checker for now, as we can easily split up later on (either dynamically generating the checkers/checkerids, and/or grouping them usefully), which would require some discussion that we should have going forward.

logging.info("Executing osirules automatic checks")

for message in trace:
if not message.HasField("version"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is already included in deserialization_osi should it not occur in only one place? This also causes the issues to be reported twice

image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the idea would be to likely remove that check going forward from the deserialization checker bundle - either because we fold the whole deserialization checker bundle into the osirules bundle, or we specialize the deserialization checker bundle to focus on pure deserialization issues.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No further inputs. Decision from the 08.07.2025 OSI multi-trace meeting: ready to be merged

pmai added 5 commits June 25, 2025 12:52
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Note that override of config parameters requires further changes
to baselib to support this.

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
This is the preferred way, as it ensures that the right virtual
environment is active, see other qc repos.

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Update to dependent changes in qc-baselib-py

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
@pmai pmai force-pushed the feature/automatic-checks branch from 88fbf0a to 68d7e14 Compare June 29, 2025 09:30
expected_type = OSITrace.map_message_type(expected_type_name)

trace = OSITrace(
config.get_config_param("InputFile"), config.get_config_param("osiType")
Copy link

@msandrk-avl msandrk-avl Jul 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to allow leaving out osiType parameter from the config.xml or not? Line 364 seems to suggest that we do allow the this and in that case we default to SensorView. If so, this line should be changed to trace = OSITrace(config.get_config_param("InputFile"), expected_type_name) (use variable defined above).

Currently, leaving out the osiType parameter in the input config file would result in a KeyError: None.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that was a carryover mistake, now fixed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW: Not that the SensorView default currently is the best idea - we probably want to default to auto-detection in the future, with an error fallback when auto detect does not work - but that should be handled in a separate PR that introduces multi-trace file handling.

Copy link
Contributor

@jdsika jdsika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add "the warning"

pmai added 6 commits July 8, 2025 14:51
Also for the moment add information on message index and time to the
generated issue descriptions, as we do not yet have support for better
location reporting in the qc-baselib-py.

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
This is the only check_if rule combination currently used in the OSI
rules, hence this completes the set of checked rules necessary for
current OSI built-in rules.

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
This currently uses file locations with the row indicating the message
index, as the framework does not yet have support for more fitting
location reporting for binary and time-based formats.

Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
Signed-off-by: Pierre R. Mai <pmai@pmsf.de>
@pmai pmai force-pushed the feature/automatic-checks branch from cdb6973 to 688d0bf Compare July 8, 2025 12:52
@pmai pmai merged commit cc2acbb into main Jul 8, 2025
16 checks passed
@pmai pmai mentioned this pull request Jul 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants