Build directory structure for single tests #1052
Comments
|
Part of the solution is to use --test-root and --output-root pointing to the same directory. Then, all the output related to this test is in one directory, including the sharedlibroot. Cleaning a test is more complicated since there is often more than one case directory and therefore, more than one case.setup. Maybe we need a special clean option for tests? |
|
@gold2718 I'm not sure I understand your suggestion or maybe it's not consistent with the way I'd like to work. I do not want my bld/run to be in the same location as my case. I think that's the problem. What I'd like is for all the build stuff to be under bld and for all the run stuff to be under run and for my case to be separate. If that's no longer possible, then I will adjust. But that seems to be how it works for create_newcase. In fact, I just tried if for create_newcase and I get entry id="CIME_OUTPUT_ROOT" value="/glade/scratch/tcraig"> while create_test --test-root . gives me something more like entry id="CIME_OUTPUT_ROOT" value="/glade/scratch/tcraig"> So I think my request just comes down to setting the SHAREDLIBROOT to the same value as create_newcase does by default for a single test. |
|
Hmm. If I do --test-root <place_for_case_stuff> --output-root <place_for_bld_and_run> |
|
I played with it earlier and I thought not, and I still think not. If I do ./create_test SMS_EN.T31_g37_rx1.A.yellowstone_intel -t xyy --test-root . --output-root /glade/scratch/tcraig/xyz --no-build then I get entry id="CIME_OUTPUT_ROOT" value="/glade/scratch/tcraig/xyz" so while SHAREDLIBROOT in in CIME_OUTPUT_ROOT, it's not the same as what create_newcase does which is to put it under EXEROOT. Also, when I use --output-root, I am responsible for setting a path, while if the scripts do it for me, then it's a whole bunch easier and I get exactly what I want with the appropriate CASENAME in the path. I continue to believe that for single tests, it's easier and clearer for most of the community if the SHAREDLIBROOT defaults as I suggest without having to mess with a bunch of extra command line options. In fact, I might even argue the default for a single test should also be --test-root . just like it is for create_newcase. I guess it depends if you think create_newcase and create_test are fundamentally different tools or whether they are largely the same tool but do slightly different things. I am more in the latter camp and think create_newcase is largely for production runs where a particular case name is desired while create_test is for everything else including development and it should behave a lot like create_newcase for most users. |
|
./create_newcase in my opinion was never meant *just* for production runs.
The way I use it, and I think others use it (I'd like some comments if this
is not the case) is if I do code development, I always do it with a case -
not with a test. Once the code development is done - then I start executing
tests. I do think that create_test and create_newcase serve different
purposes and were designed with that aspect in mind.
…On Thu, Jan 26, 2017 at 8:53 PM, apcraig ***@***.***> wrote:
I played with it earlier and I thought not, and I still think not. If I do
./create_test SMS_EN.T31_g37_rx1.A.yellowstone_intel -t xyy --test-root .
--output-root /glade/scratch/tcraig/xyz --no-build
then I get
entry id="CIME_OUTPUT_ROOT" value="/glade/scratch/tcraig/xyz"
entry id="EXEROOT" value="$CIME_OUTPUT_ROOT/$CASE/bld
entry id="SHAREDLIBROOT" value="/glade/scratch/tcraig/
xyz/sharedlibroot.xyy
so while SHAREDLIBROOT in in CIME_OUTPUT_ROOT, it's not the same as what
create_newcase does which is to put it under EXEROOT. Also, when I use
--output-root, I am responsible for setting a path, while if the scripts do
it for me, then it's a whole bunch easier and I get exactly what I want
with the appropriate CASENAME in the path.
I continue to believe that for single tests, it's easier and clearer for
most of the community if the SHAREDLIBROOT defaults as I suggest without
having to mess with a bunch of extra command line options. In fact, I might
even argue the default for a single test should also be --test-root . just
like it is for create_newcase. I guess it depends if you think
create_newcase and create_test are fundamentally different tools or whether
they are largely the same tool but do slightly different things. I am more
in the latter camp and think create_newcase is largely for production runs
where a particular case name is desired while create_test is for everything
else including development and it should behave a lot like create_newcase
for most users.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1052 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AHlxE1y7FcRhMMq5R9xd1sFKEqW2TOKzks5rWWo1gaJpZM4LvN22>
.
|
|
@apcraig , I'm not sure what you mean by 'community' but I have interacted with a large number of CESM users (many met via the CESM tutorials) and none of them ever use create_test. create_test was invented simply to allow us to test CESM, all science users use create_newcase. |
|
Those are reasonable comments. I think it's just faster and easier to leverage the ability to test and develop in parallel. Sometimes I use an ERS to develop, sometimes an SMS, sometimes something else. I can quickly create a configuration leveraging _D, _E, _N, _P and other flags that come with tests and which are now not entirely are supported in create_newcase. Using tests just provides additional capabilities and allows more rapid progress in some cases. I can create cases faster for what I want to test, whether it's particular pe counts or other useful flags. Creating a case and manually doing all that stuff will take extra time. What I'm proposing has no impact on either create_newcase or create_test when running test suites. I think what I'm proposing is unlikely to have any negative impacts on other users. |
gold2718
added
ty: enhancement
type: enhancement request
and removed
ty: enhancement
labels
|
I think one thing preventing what you want is that an individual test doesn't know if its part of a suite or just one test. What I think you need is a --sharedlib-root option to explicitly set the value in the xml file. I've always thought create_test functionality is a superset of create_newcase. It still just calls create_newcase in the end right? |
|
@rljacob I can't comment whether create_test is still a superset of create_newcase, I've just started using the new scripts and am still trying to figure out how they work, but I think that's still largely the case. Also, it seems to me the arguments in create_test used to create a test suite vs a single test are quite different. That might be enough to trigger the behavior. But maybe I'm wrong about that. |
|
Sometimes a test is not just one test. If a test has two build phases (e.g., ERP), the builds for gptl, mct, & pio are shared which is a significant time saving. So the disadvantages that I see to this request are possibly significant engineer time to implement it and slower test build time. Can you concisely state the advantages to aid this discussion? |
|
When an ERP test is created, is it generating two case directories? If so, then I see the point. If it is building multiple executables or rebuilding during the run in a single case, it seems like it wouldn't matter where the location of the SHAREDLIBROOT is defined. And remember, as proposed, this would only impact users that are generating a single test. If they end up building the share code twice for a single test they are going to reuse multiple times, that doesn't seem like a big deal. Test suites would always be leveraging the common SHAREDLIBROOT which I think works well. And the test suite mode would be invoked anytime create_test is asked to generate more than one test at a time. If the choice is to use ERP, get some clean builds, pay a little extra time to build the share code twice, and have clean directories OR save a little time by having a common share build OR take a bunch more time by having to use create_newcase to do all my testing and developoment, I choose the first. I think I've concisely stated by points above. Mainly, I think moving the SHAREDLIBROOT into the bld directory for a single test makes the system simpler and clearer. It makes it easier to delete the bld directory in cases where the clean/cleanall doesn't do a complete job. I wasted some time initially trying to figure out where the share code was being built and then trying to understand why when I deleted the bld directory, I wasn't getting a clean rebuild. The thing I didn't understand was that there was this SHAREDLIBROOT directory and that it was totallly separate from the rest of the case bld. Now that I know it exists, I won't run into the same problems. But other might. I'm just advocating for this because I think it makes the system simpler and clearer for users that are creating tests one by one. The other point is that I could argue that you should only reuse the SHAREDLIBROOT for test suites where you know the source code isn't changing. If I setup multiple tests one at a time on different days as code is evolving, I may get different outcomes if I happen to use the same testid or different testids. It's almost like every invocation of create_test should have it's own SHAREDLIBROOT. Testid usually handles that. Putting the SHAREDLIBROOT inside the individual bld directories for each individual test also takes care of that concern. I think what I'm doing here is proposing a use case that may not have been considered in the initial design, and I'm trying to provide input about what I think would make it better. It's a use mode that I use all the time and it's quite efficient. Maybe others should be using it more often :). Anyway, It's just a proposal that I think make usage simpler and clearer, and whether it's implemented or not, I'll continue to be able to work. |
|
I also use create_test with single tests quite a lot - largely because it's an easy way to get the right testmods in place, but also because it's just more convenient to copy & paste a test name than to create the right create_newcase command. So I see some of @apcraig 's points - especially defaulting to It seems like the real question here is whether It sounds like part of the problem is that As to this:
You definitely don't want to use the same test id for different invocations of create_test. That was the motivation for adding the unique string to the end of the default test id. We had discussed checking a user-defined testid to make sure there isn't already a sharedlibroot with that testid, and decided it wasn't worth the trouble - but I could imagine revisiting that discussion if you or others feel it's important. |
|
"this was fixed" |
When building a single test with create_test, the build generates a separate directory called something like "sharedlibroot.$testid" that is outside the CASE, bld, or run directories. That directory is useful when generating test suites as the same "share" build is reused for multiple tests, but may not be ideal for just a single test. When create_newcase is used, the sharedlibroot directory is not created and all the build happens under the bld directory.
I would like to propose that when using create_test for a single test, that it behave, in it's build, like create_newcase and NOT create the sharedlibroot by default. The sharedlibroot would be created when a test suite was being run or maybe with an optional flag on the create_test command line if others felt that was useful.
The problem is when people use create_test like create_newcase (which I often do) to setup cases to do development and run single tests, where code often gets modified during the course of development, tests get rebuilt often, and such. It's a different paradigm than running regression test suites. As such, currently that sharedlibroot is not well known and the clean or cleanall doesn't seem to address it at all, so it makes the system more complicated to use for much of the community. The sharelibroot is really something that is most useful for regression testing of test suites, and I think the default case/test behavior should not include setting up that directory.
This issue might also be related to the clean issue, #1045
The text was updated successfully, but these errors were encountered: