Improve testing DevX #7362
Comments
robert-zaremba
added
T: Dev UX
Do you mean that you'd prefer not to run commands testing via |
What do you mean by that? The
That's easy, it's one-line patch: alessio@phoenix:~/work/cosmos-sdk$ git diff
diff --git a/Makefile b/Makefile
index aa03819bc..fe418355b 100644
--- a/Makefile
+++ b/Makefile
@@ -74,7 +74,7 @@ ifeq (,$(findstring nostrip,$(COSMOS_BUILD_OPTIONS)))
BUILD_FLAGS += -trimpath
endif
-all: tools build lint test
+all: tools build simd lint test-all
# The below include contains the tools and runsim targets.
include contrib/devtools/Makefile
|
That is already a thing: |
No, unit tests should be fast and well encapsulated. Unit test which runs a blockchain and a rest client is not a unit test. There are some tests marked as unit which run 30+ seconds. |
|
The
Yes, I know that. I must confess that the whole team knew that all along. We thought that running a single |
Have a look at the Proposal section of OP ;) .
|
|
How do you suggest splitting unit, integration and e2e tests? |
It might depend on the developer's latitude and longitude (can't tell on that), but... |
|
@alessio - you don't need to copy paste your cached test suite. Try to modify something in |
@marbar3778 thanks for asking. I suggest to use |
If I modify something in
This sounds interesting as well as a bit convoluted. Could you prepare a draft PR so that the teams can be in the position to assess the impact of the proposed change please? |
|
@alessio it seams that I have difficulty to explain you the motivation for wisely splitting unit tests. I know how
All this things are written in the proposal. |
Please don't bother about this. It si already very, very fast. The whole point of the Go compiler is to compile fast. Let's avoid engaging in flights of fancy and solve problems that are not problems at all, shall we?
Yes, unit tests should be fast, you're right and I agree to that. Yet this Now, I like using |
|
@alessio - you are showing cached test results. Not relevant. I already explained that few times above.
Yes, but now I'm busy with other tasks. I have described the idea above. Again, this is a meta task. let's don't spam here. If you want to contribute or share some advice about one of the proposals, then let's do it in a sub-issue. Thanks. |
|
Using EDIT: As in: test suites can run in parallel, yet test cases of a suite cannot. |
|
Ah, that's bad. I'm usually using AFAIK gocheck handles parallel tests and has nice support skipping tests out of the box. |
|
Hmm, I think gocheck has same support as testify. I think you can mimic parallel tests in testify using |
|
Switching to |
|
Here's the relevant issue stretchr/testify#187. Suites could be parallelized if a deep copy was done of the suite for every test. Seems feasible to me. |
It is, though we would need to introduce some hack-ish, custom code to handle parallelism. So unless test execution times become untenable, I'd wait until upstream introduces this as a feature. |
I'm assuming this would need to come from upstream. And we could gently put pressure or introduce a PR if it's an issue for us. |
|
I believe this is now obsolete? |
|
Test separation is technically not done (there are many of unit tests which are integration tests rather than unit tests). |
|
Want to capture that / a list of places to look out for in a new issue? |
|
@ValarDragon -- you mean to list "bad tests"? I was talking about it in various events. Instead of having a right test distribution (70% unit tests, 20 % integration, 10% functional: Most of our tests are integration - in every module. There is tonnes of copy-paste around, lacking a proper setup nor fixtures repository. At Regen we were experimenting with better setup to enable more managable unit-tests and mocking capabilities. Not sure what to put in a new issue. I think this issue is kind of an epic to have better testing patterns in the SDK. |
Summary
As a developer I want an easy way to run and manage tests. Moreover, I want to have a clear separation of unit tests, integration tests and functional tests. Finally, unit tests must be very quick to run and don't involve lot of dependencies.
NOTE
Problem Definition
Current setup is far from good (in terms of DevX):
make test-unittakes loooot of time, and it's more than unit testsProposal
make) Enable test caches #7314testing.T.Skipinstead of build flags whenever possible to skip some tests.For Admin Use
The text was updated successfully, but these errors were encountered: