Skip to content

Address "Updated expected results" C++ compiler compatibility issues #299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
May 10, 2023

Conversation

lcartey
Copy link
Collaborator

@lcartey lcartey commented Apr 28, 2023

Description

This PR addresses issues highlighted by our compiler compatibility matrix testing for C++ compilers triaged a "Update expected results". This covers a bucket of issues where the expected results on a specific compiler differ from our expected results, but not in a semantically meaningful way.

These changes are grouped into three categories:

  • Improving our C++ standard library stubs - in a number of cases, the difference between our unit tests and real compilers was due to differences in the way we modelled our C++ stubs. Rather than just accepting the differences I've modified the standard library stubs, so the unit tests now better reflect real compilers.
  • Accept empty results for impossible to test queries - a couple of queries require compiling multiple compilation units with different compilers. That is not possible within our framework, so I've effectively "disabled" those by committing empty expected result files.
  • Accept different expected result files where standard library macros have different parameter names.

Change request type

  • Release or process automation (GitHub workflows, internal scripts)
  • Internal documentation
  • External documentation
  • Query files (.ql, .qll, .qls or unit tests)
  • External scripts (analysis report or other code shipped as part of a release)

Rules with added or modified queries

  • No rules added
  • Queries have been added for the following rules:
    • rule number here
  • Queries have been modified for the following rules:
    • A14-7-2 - qcc, gcc, clang
    • A15-1-5 - qcc, gcc, clang
    • A18-0-3 - qcc, gcc, clang
    • A26-5-1 - qcc, gcc, clang
    • ERR59-CPP - qcc, gcc, clang
    • M18-2-1 - qcc, gcc
    • M18-7-1 - qcc, gcc, clang
    • M27-0-1 - qcc, gcc, clang
    • MSC50-CPP - qcc, gcc, clang

Release change checklist

A change note (development_handbook.md#change-notes) is required for any pull request which modifies:

  • The structure or layout of the release artifacts.
  • The evaluation performance (memory, execution time) of an existing query.
  • The results of an existing query in any circumstance.

If you are only adding new rule queries, a change note is not required.

Author: Is a change note required?

  • Yes
  • No

🚨🚨🚨
Reviewer: Confirm that format of shared queries (not the .qll file, the
.ql file that imports it) is valid by running them within VS Code.

  • Confirmed

Reviewer: Confirm that either a change note is not required or the change note is required and has been added.

  • Confirmed

Query development review checklist

For PRs that add new queries or modify existing queries, the following checklist should be completed by both the author and reviewer:

Author

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

Reviewer

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

lcartey and others added 11 commits April 27, 2023 19:26
Move "rand" definition to the global namespace and import into std
with "using", to better reflect "real" compilers.
Our compiler testing only uses a single compiler and cannot replicate
execution boundaries, so we simply ignore this test by expecting no
results.
Move global locale definitions to locale.h, and import them into std
with "using" to better reflect real compilers.
Our compiler testing only uses a single compiler and cannot replicate
execution boundaries, so we simply ignore this test by expecting no
results.
Macro parameter names vary by compiler.
Better reflect how real compilers implement csignal.
Modify cstdio to more closely match real compilers.
The source location is different between compiler testing and qltest,
leading to spurious differences. This simplifies the query, reporting
only the base name of the file but still reporting the link.
@mbaluda mbaluda self-requested a review May 9, 2023 21:57
Copy link
Contributor

@mbaluda mbaluda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All issues fixed 🚢

@github github deleted a comment from jsinglet May 10, 2023
@github github deleted a comment from jsinglet May 10, 2023
@github github deleted a comment from jsinglet May 10, 2023
@jsinglet
Copy link
Contributor

🤖 Beep Boop! qcc/cpp/aarch64le Matrix Testing for this PR has been completed. See below for the results!


PACKAGE              : Templates
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : TemplateSpecializationNotDeclaredInTheSameFile
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A14-7-2
TEST_PASS            : True

PACKAGE              : TrustBoundaries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : ExceptionsThrownAcrossExecutionBoundaries
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A15-1-5
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : LocaleMacrosUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A18-0-3
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : LocaleFunctionsUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A18-0-3
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : LocaleTypeLConvUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A18-0-3
TEST_PASS            : True

PACKAGE              : BannedFunctions
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : MacroOffsetofUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M18-2-1
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : CsignalFunctionsUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M18-7-1
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : CsignalTypesUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M18-7-1
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : CstdioFunctionsUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M27-0-1
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : CstdioMacrosUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M27-0-1
TEST_PASS            : True

PACKAGE              : BannedLibraries
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : CstdioTypesUsed
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : M27-0-1
TEST_PASS            : True

PACKAGE              : TrustBoundaries
SUITE                : CERT-C++
COMPILE_PASS         : True
QUERY                : DoNotThrowAnExceptionAcrossExecutionBoundaries
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : ERR59-CPP
TEST_PASS            : True

PACKAGE              : BannedFunctions
SUITE                : CERT-C++
COMPILE_PASS         : True
QUERY                : DoNotUseRandForGeneratingPseudorandomNumbers
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : MSC50-CPP
TEST_PASS            : True

PACKAGE              : BannedFunctions
SUITE                : AUTOSAR
COMPILE_PASS         : True
QUERY                : PseudorandomNumbersGeneratedUsingRand
COMPILE_ERROR_OUTPUT : 
TEST_DIFFERENCE      : 
RULE                 : A26-5-1
TEST_PASS            : True


@lcartey lcartey enabled auto-merge May 10, 2023 15:13
@lcartey lcartey added this pull request to the merge queue May 10, 2023
Merged via the queue into main with commit 9de7232 May 10, 2023
@lcartey lcartey deleted the cpp-compiler-compat-update-results branch May 10, 2023 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

3 participants