Improve the test coverage of the C API #447

Gnimuc · 2025-01-13T09:50:49Z

Description

as per #440 (comment)

Fixes # (issue)

Type of change

Please tick all options which are relevant.

Bug fix
New feature
Requires documentation updates

Testing

Please describe the test(s) that you added and ran to verify your changes.

Checklist

I have read the contribution guide recently

github-actions

clang-tidy made some suggestions

unittests/CppInterOp/InterpreterTest.cpp

github-actions

clang-tidy made some suggestions

unittests/CppInterOp/FunctionReflectionTest.cpp

unittests/CppInterOp/InterpreterTest.cpp

github-actions

clang-tidy made some suggestions

lib/Interpreter/CXCppInterOp.cpp

unittests/CppInterOp/FunctionReflectionTest.cpp

unittests/CppInterOp/ScopeReflectionTest.cpp

unittests/CppInterOp/TypeReflectionTest.cpp

github-actions

clang-tidy made some suggestions

unittests/CppInterOp/InterpreterTest.cpp

codecov · 2025-01-13T12:06:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.31%. Comparing base (f0f59f2) to head (35cbc9c).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #447      +/-   ##
==========================================
+ Coverage   72.71%   75.31%   +2.59%     
==========================================
  Files           9        9              
  Lines        3618     3617       -1     
==========================================
+ Hits         2631     2724      +93     
+ Misses        987      893      -94

Files with missing lines	Coverage Δ
lib/Interpreter/CXCppInterOp.cpp	`47.27% <100.00%> (+25.18%)`	⬆️
lib/Interpreter/CppInterOp.cpp	`83.03% <100.00%> (+0.02%)`	⬆️

... and 1 file with indirect coverage changes

Files with missing lines	Coverage Δ
lib/Interpreter/CXCppInterOp.cpp	`47.27% <100.00%> (+25.18%)`	⬆️
lib/Interpreter/CppInterOp.cpp	`83.03% <100.00%> (+0.02%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

vgvassilev · 2025-01-13T19:02:09Z

unittests/CppInterOp/FunctionReflectionTest.cpp

+
+  // C API
+  auto* I = clang_createInterpreterFromRawPtr(Cpp::GetInterpreter());
+  EXPECT_TRUE(clang_hasDefaultConstructor(make_scope(Decls[0], I)));


Do you think we should put the tests for the c layer in a separate file?

I don't think that's practical at the moment because of the low API coverage. We need to use a lot C++ API to set up the environment and do the test.

github-actions

clang-tidy made some suggestions

unittests/CppInterOp/FunctionReflectionTest.cpp

Gnimuc · 2025-01-18T07:42:32Z

This only improved the coverage from 70% to 74% because as I said before nearly half of the code are from libclang.

github-actions

clang-tidy made some suggestions

unittests/CppInterOp/ScopeReflectionTest.cpp

Gnimuc · 2025-03-09T05:43:45Z

@mcbarton @vgvassilev

I think this is ready to merge.

vgvassilev · 2025-03-09T06:07:16Z

@mcbarton @vgvassilev

I think this is ready to merge.

Seems so. Please address the clang-tidy suggestions and we should be good to go.

Gnimuc · 2025-03-09T13:06:28Z

done!

mcbarton · 2025-03-14T17:42:56Z

@Gnimuc can you rebase? Now that we have automated wasm tests we need to just double check that the new tests pass for an Emscripten build. Once the ci passes after rebase I will approve and merge.

Gnimuc · 2025-03-15T06:14:53Z

@mcbarton any hints on how to investigate those Emscripten build errors?

mcbarton · 2025-03-17T10:44:51Z

@Gnimuc My suggestion is to wait for this PR to go in #523 and rebase. This PR will fix the current crashing tests, so hopefully it will fix your tests too (or at least allow to have more useful error messages)

mcbarton · 2025-03-18T20:56:33Z

@anutosh491 I had a quick go at trying to fix the errors without success. Do you have any suggestions about what @Gnimuc can try to debug/fix the failing/crashing tests?

anutosh491 · 2025-03-19T03:46:54Z

Hmmm, can look into it in a while

Gnimuc · 2025-03-19T10:21:02Z

@anutosh491 does Emscripten build need any special setup when creating the interpreter instance? The current interpreter C API is not equivalent to the C++ one(it's just a thin wrapper over clang::Interpreter with no additional setups), so probably we should just skip those C API tests on Emscripten?

anutosh491 · 2025-03-19T10:41:09Z

so probably we should just skip those C API tests on Emscripten?

Yeah absolutely we can do that for now. Its not suprising atleast to me to see things work natively and not with emscripten right now. I would not stress on getting it to pass through this PR itself. Let's skip it for now.

@mcbarton as you've been monitoring the emscripten tests my suggestion here is to let this one go and we create a todo-list through an issue to keep track and keep addressing one skipped test at a time. I see only two reasons for tests to fail right now

i) some tests are not catered for the emscripten build (for eg the functions for detecting resource dir use system calls that won't (shouldn't) work with emscripten ... so we just update it to returning an empty string cause that's expected)
ii) errors/missing implementation in clang-repl-lite

We have all tools to address i if not ii which will be done slowly.

anutosh491 · 2025-03-24T07:06:34Z

Hey @Gnimuc

I see two errors here. I would only expect the 1st one be addressed cause that's doable.

The first one says this as shown here

TypeError: resolved is not a function

What this basically means is its trying to call an undefined symbol at runtime. Doing some more debugging shows me this is the exact symbol, I had added some PRs back (reasoning on why it is missing should be mentioned there)
Check #425

So if you export this like I did, the first one would be resolved (and possibly we can close my PR then)

Coming to the 2nd one. I see this being added

  clang_Interpreter_addSearchPath(CXI, "dummy", false, false);
  clang_Interpreter_addIncludePath(CXI, "dummy");

Search paths for native cases vs for emscripten cases where a virtual file system is being put to use should work differently. So I would not stress on getting this to pass just yet. Could you please skip this ? I think this should be ready then.

P:S Not sure how important those 2 lines are. Probably just commenting them out or removing should be enough too ?

Gnimuc · 2025-03-24T09:57:26Z

@anutosh491 Thanks for looking into this. I cherry-picked your commit and did a rebase. 🤞🏻

clang_Interpreter_addSearchPath(CXI, "dummy", false, false);
clang_Interpreter_addIncludePath(CXI, "dummy");

That's my ugly trick for improving the coverage.😅 Let’s remove them until we figure out a way to do a proper test.

Gnimuc · 2025-03-24T10:09:34Z

Oops! Should not just cherry-pick that commit. I manually applied the changes.

anutosh491 · 2025-03-24T10:20:11Z

Let’s remove them until we figure out a way to do a proper test.

Perfect. Nothing wrong with that I think

anutosh491

I think the emscripten tests pass now.

If these are all the tests that were meant to be addressed through this PR (someone can let me know if this is all) then this should be ready I think !

mcbarton · 2025-03-24T10:50:02Z

lib/Interpreter/CMakeLists.txt

    PRIVATE "SHELL: -s WASM_BIGINT"
    PRIVATE "SHELL: -s SIDE_MODULE=1"
    PRIVATE "SHELL: ${SYMBOLS_LIST}"
+    PUBLIC "SHELL: -Wl,--export=__clang_Interpreter_SetValueNoAlloc"


These should be added to the exports.ld file we already have. e.g. here https://github.com/Gnimuc/CppInterOp/blob/c-api-test/lib/Interpreter/exports.ld

This can be done too through a subsequent PR.

Gnimuc · 2025-03-24T11:21:06Z

Thanks!

Gnimuc changed the title ~~Improve the test coverage of the C API -- Take 1~~ Improve the test coverage of the C API Jan 13, 2025

github-actions bot reviewed Jan 13, 2025

View reviewed changes

unittests/CppInterOp/InterpreterTest.cpp Outdated Show resolved Hide resolved

unittests/CppInterOp/InterpreterTest.cpp Outdated Show resolved Hide resolved

Gnimuc marked this pull request as ready for review January 13, 2025 10:16

github-actions bot reviewed Jan 13, 2025

View reviewed changes

Gnimuc force-pushed the c-api-test branch from 365681c to c898693 Compare January 13, 2025 11:04

github-actions bot reviewed Jan 13, 2025

View reviewed changes

Gnimuc force-pushed the c-api-test branch 3 times, most recently from 6f81874 to 11fea7d Compare January 13, 2025 11:57

github-actions bot reviewed Jan 13, 2025

View reviewed changes

unittests/CppInterOp/InterpreterTest.cpp Outdated Show resolved Hide resolved

vgvassilev reviewed Jan 13, 2025

View reviewed changes

Gnimuc force-pushed the c-api-test branch 2 times, most recently from 2a1eca2 to 0420a80 Compare January 18, 2025 06:03

github-actions bot reviewed Jan 18, 2025

View reviewed changes

unittests/CppInterOp/FunctionReflectionTest.cpp Outdated Show resolved Hide resolved

Gnimuc requested a review from vgvassilev January 18, 2025 11:29

Gnimuc force-pushed the c-api-test branch from 0c1f212 to b7c0766 Compare January 19, 2025 07:09

github-actions bot reviewed Jan 19, 2025

View reviewed changes

unittests/CppInterOp/ScopeReflectionTest.cpp Show resolved Hide resolved

Gnimuc force-pushed the c-api-test branch from b7c0766 to c8f1c63 Compare March 9, 2025 02:38

Gnimuc force-pushed the c-api-test branch from da16e01 to f2b5129 Compare March 12, 2025 06:07

vgvassilev force-pushed the c-api-test branch from f2b5129 to 7fc6113 Compare March 14, 2025 17:50

Gnimuc force-pushed the c-api-test branch from 7fc6113 to f489086 Compare March 17, 2025 10:35

mcbarton force-pushed the c-api-test branch from f489086 to 967d1fd Compare March 17, 2025 15:33

Gnimuc force-pushed the c-api-test branch from 967d1fd to 2b3f2d0 Compare March 19, 2025 02:01

Improve the test coverage of the C API

1689434

Gnimuc force-pushed the c-api-test branch from 2b3f2d0 to cdf3db6 Compare March 24, 2025 09:52

Export runtime symbols

35cbc9c

Gnimuc force-pushed the c-api-test branch from cdf3db6 to 35cbc9c Compare March 24, 2025 10:08

anutosh491 approved these changes Mar 24, 2025

View reviewed changes

mcbarton reviewed Mar 24, 2025

View reviewed changes

vgvassilev merged commit 6dc5278 into compiler-research:main Mar 24, 2025
73 checks passed

anutosh491 mentioned this pull request Mar 24, 2025

Export __clang_Interpreter_SetValueNoAlloc to allow automatic printf in xeus-cpp-lite #425

Closed

4 tasks

Gnimuc mentioned this pull request Mar 24, 2025

Use exports.ld to export symbols #537

Merged

4 tasks

Gnimuc deleted the c-api-test branch March 24, 2025 11:19

Improve the test coverage of the C API #447

Improve the test coverage of the C API #447

Uh oh!

Conversation

Gnimuc commented Jan 13, 2025

Description

Type of change

Testing

Checklist

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vgvassilev Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

Gnimuc Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gnimuc commented Jan 18, 2025

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gnimuc commented Mar 9, 2025

Uh oh!

vgvassilev commented Mar 9, 2025

Uh oh!

Gnimuc commented Mar 9, 2025

Uh oh!

mcbarton commented Mar 14, 2025

Uh oh!

Gnimuc commented Mar 15, 2025

Uh oh!

mcbarton commented Mar 17, 2025

Uh oh!

mcbarton commented Mar 18, 2025

Uh oh!

anutosh491 commented Mar 19, 2025

Uh oh!

Gnimuc commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anutosh491 commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 13, 2025 •

edited

Loading

Gnimuc commented Mar 19, 2025 •

edited

Loading

anutosh491 commented Mar 19, 2025 •

edited

Loading