Combined GemStone/Squeak/Pharo tests - revisited

[dalehenrich]

In Issue #29, I mentioned that I would eventually need this feature and at the time I figured that I could handle this requirement in GsDevKit_home and @fniephaus postulated that the following could be used to run the tests from travis:

script:
  - $SMALLTALK_CI_HOME/run.sh
  - $SMALLTALK_CI_HOME/run.sh -s Pharo-5.0
  - $SMALLTALK_CI_HOME/run.sh -s Squeak-5.0

However as I get into this in earnest, I am finding that things are not quite that simple.

To start with, here are the GsDevKit_home scripts that I'm using to create a Pharo client with the GemStone-GCI project loaded (NOTE: I am starting to use the smalltalk.ston file for more than just travis!!!):

    createClient -t pharo gciClient -v Pharo-5.0 -s $GS_HOME/sys/local/pharo/gciClient.ston # create the client with GemStone-GCI loaded
    startClient gciClient  # bring up the gciClient image ...

The file $GS_HOME/sys/local/pharo/gciClient.ston is basically the .smalltalk.ston file for GemStone-GCI, however, note that the smalltalk.ston file is not located in the GemStone-GCI git checkout.

I actually have the above creation scripts "working", but I have to use the .smalltalk.ston file from the GemStone-GCI project to do the load ... and of course the tests are run as well because I have to use SmalltalkCI class>>runCIFor: -- it would be much more convenient if the creation protocol were a bit more diverse with the following additions to the class-side api:

• runCIFor:projectDirectory: - run the CI file from one path and use the specified proejct directory
• loadCIFor:projectDirectory: - execute the #loading directives only (I.e. just load the code and don't run tests)
• testCIFor: - execute only the #testing directive and assume that code is already loaded

I expect that the GemStone-GCI .travis.yml file will eventually look like the following and allow me to test different combinations of clients and GemStone versions:

language: smalltalk
sudo: false

os:
  - linux
  - osx

env:
  - GSCI_CLIENT=Pharo-5.0
  - GSCI_CLIENT=Pharo-4.0
  - GSCI_CLIENT=Pharo-3.0

smalltalk:
  - GemStone-3.2.13
  - GemStone-3.3.0

cache:
  directories:
    - $SMALLTALK_CI_CACHE

This will also take some monkey business in the gemstone/run.sh implementation.

I also have some proposals for extensions to the smalltalk.ston fields, but that will come in a separate message

[fniephaus]

SGTM and great that there already is a second use-case for smalltalkCI config files :)
I have one suggestion at the moment: You are using envs as a variation point which is ok, but I'd like to avoid the use of too many env vars if possible. One idea that I haven't worked on yet is that one could also specify a STON config for a build just like it's for example possible to specify gem files in Ruby on Travis. So your example could look something like this:

language: smalltalk
sudo: false

os:
  - linux
  - osx

smalltalk_config:
  - smalltalk_pharo-5.0.ston
  - smalltalk_pharo-4.0.ston
  - smalltalk_pharo-3.0.ston

smalltalk:
  - GemStone-3.2.13
  - GemStone-3.3.0

cache:
  directories:
    - $SMALLTALK_CI_CACHE

This would mean that the client selection is encoded in the STON config which I think would make sense. What do you think? I guess I could start working on it if this would help here.

[dalehenrich]

Okay, I think I see what you are doing with the multiple config files and I agree that your proposal would be a useful feature, but I'm not sure that it addresses my use case ...

The point of using the GSCI_CLIENT env var is two fold:

1. I don't want to introduce the overhead of having to maintain multiple, duplicate config files when I am only varying a single parameter.

2. I am thinking that the pattern for -gs-* options is that each gemstone option will be settable via:
   - a GSCI_*environment variable
   - --gs-* option
   - .smalltalk.ston slot

   The .smalltalk.ston has the default value, which can be overridden by a command line option or environment variable. And the command line option will override the value of the environment variable. Finally this scheme would be apply equally to command line and travis environments.

So I am sensitive to over reliance on environment vars, but I extend that to where I want to minimize the proliferation of --gs-* options as well, but when they are needed, I think that they should follow this pattern ... with all of that said, I would say that having multiple config files would be a feature that I would definitely use ...

[dalehenrich]

Continuing with the discussion of clients and smalltalkCI configuration file extensions...
I've already described how I intend to create and start clients for GsDevKit_home:

createClient -t pharo gciClient -v Pharo-5.0 -s $GS_HOME/sys/local/pharo/gciClient.ston # create the client with GemStone-GCI loaded
startClient gciClient  # bring up the gciClient image ...

However, if I want to be able to run a travis test involving the client, I need to create and start a stone; then create and start a client; run the stone tests and finally run the client tests all while producing coherent tests results for travis (otherwise known as producing one or more sunit test results files that are properly displayed by trvis/smalltalk.rb --- a problem for which I do not have a solution, regardless of how the tests get started and run).

We've already taught SmalltalkCI how to create and start a stone, so I need to add additional attributes to config file to be able to create and start the client. Specifically, I need to supply the various arguments needed by both the createClient and startClient commands.

My proposal is to add a SCIGemStoneClientConfigurationSpec class that is used to specify the necessary attributes for creating and starting the client and given the currently known attributes it would look like the following:

SCIGemStoneClientConfigurationSpec {
     #name : 'gciClient',
     #clientVersion : 'Pharo-5.0',
     #smalltalkCIConfig : '$GS_HOME/sys/local/pharo/gciClient.ston'
  }

In the current use case only the #clientVersion needs to be variable and would have a GSCI_* environment variable and a --gs-* command line option.

At the same time, I've hit some orthogonal requirements for being able to specify some configuration options for the stone itself, namely: gemConfPath and timeZone.

In this case I propose to add a SCIGemStoneSystemConfigurationSpec class where stone-specific attributes may be specified and it would look like the following:

SCIGemStoneSystemConfigurationSpec {
  #stoneName : 'travis',
  #gemConfPath : 'tests/gemstone/gem.conf',
  #timeZone : 'Europe/Prague' }

In the current use case all three of the attributes: #stoneName, #gemConfPath, and #timeZone; would have a GSCI_* and --gs-* variants.

Since clients are technically independent of the stones, the two configuration instances would be peers and a full .smalltalk.ston would look like the following:

SmalltalkCISpec {
  #configuring: [
      SCIGemStoneClientConfigurationSpec {
        #name : 'gciClient',
        #clientVersion : 'Pharo-5.0',
        #smalltalkCIConfig : '$GS_HOME/sys/local/pharo/gciClient.ston' },
      SCIGemStoneSystemConfigurationSpec {
        #name : 'gciClientTest',
        #gemConfPath : 'tests/gemstone/gem.conf',
        #timeZone : 'Europe/Prague' }
    ],
  #loading : ['...'],
  #testing : {},
}

If there is the likelihood that other platforms would need system configuration attributes specified, then perhaps an SCISystemConfigurationSpec class could be created for common attributes (perhaps #name?)

Whew ... I know this is a lot, but as I have said, I have a couple of use cases that I am currently working on, so this is not based on conjecture, but real requirements ....

So what do you think ... I will be starting work on this, because the projects needing this are at the top of my list (at the moment), but I want to hear what you think and we should definitely discuss alternatives --- shoot, I need to validate my assumptions about what is needed as well :)

[fniephaus]

Have to think about this tomorrow. I will follow up asap

[dalehenrich]

I know its a lot to digest ... so thanks!

[fniephaus]

Once again, I realized that I have very limited knowledge about GemStone, but I trust your judgement.
Here are my thoughts (I hope they make sense):
I do understand that you need additional config options via the smalltalkCI config file and it sounds reasonable to call this #configuring and to use an abstract class for this. Since we already have SCIAbstractLoadSpec, maybe call it SCIAbstractConfiguration or SCIAbstractConfig?
I can't think of any Squeak or Pharo-related use cases, but I don't see how your approach would interfere in any way and if there are use cases, we already have the infrastructure :)

I also understand you want to reference a config for the client in the system config (via #smalltalkCIConfig). I'm curious how smalltalkCI configs for system and client would differ. We already have the #platforms slot. I wonder if it is possible to do something like this:

SmalltalkCISpec {
  #loading : [
    SCIMetacelloLoadSpec {
      #baseline : 'MySystemProject',
      ...
      #platforms : [ #gemstone ]
    },
    SCIMetacelloLoadSpec {
      #baseline : 'MyClientProject',
      ...
      #platforms : [ #gemstoneClient ]
    },
  ],
  ...
}

This way you only have one config file.

In addition, I'm not quite sold on the idea to be able to configure something in the smalltalkCI config file and then be able to override it using cmd line parameters and env vars. I'm also not sure how you want to use the options within the config file. Will you start a client from within the system GemStone image or how do you want to access these options within the bash script?

Lots of questions, but I hope this is helpful. :)

[dalehenrich]

On 4/1/16 1:27 PM, Fabio Niephaus wrote:

Once again, I realized that I have very limited knowledge about
GemStone, but I trust your judgement.
Here are my thoughts (I hope they make sense):
I do understand that you need additional config options via the
smalltalkCI config file and it sounds reasonable to call this
|#configuring| and to use an abstract class for this. Since we already
have |SCIAbstractLoadSpec|, maybe call it |SCIAbstractConfiguration|
or |SCIAbstractConfig|?

Unless told otherwise I'll go with SCIAbstractConfig --- it can always
be changed.

I can't think of any Squeak or Pharo-related use cases, but I don't
see how your approach would interfere in any way and if there /are/
use cases, we already have the infrastructure :)

Well I suppose that database connections would need to be configured ---
say a seaside application that is designed to connect to multiple data
bases --- which does happen to lineup with with my use case, except that
the data base is GemStone :) In fact for the Tugrik project I will
likely be testing a client against both GemStone and Mongo, but it will
be in the context of the GsDevKi_home environment and with these
features, I will be able to arrange to connect to Mongo just as easily
as GemStone:)

I also understand you want to reference a config for the /client/ in
the /system/ config (via |#smalltalkCIConfig|). I'm curious how
smalltalkCI configs for /system/ and /client/ would differ. We already
have the |#platforms| slot. I wonder if it is possible to do something
like this:

SmalltalkCISpec {
#loading: [
SCIMetacelloLoadSpec {
#baseline: 'MySystemProject',
...
#platforms: [ #gemstone ]
},
SCIMetacelloLoadSpec {
#baseline: 'MyClientProject',
...
#platforms: [ #gemstoneClient ]
},
],
...
}

This way you only have one config file.

Since the #platforms attribute already exists for the MetacelloLoadSpec,
I didn't mention that I would be leveraging this feature as well and I
do expect to have the #loading and #testing specs along with the
configuration specs ... I need to be able to have configuration files
that are NOT in the root directory of a project ... For example For
Obex I may choose to load a different set of groups than those used for
travis testing and to avoid conflicting with the standard obex
.smalltalk.ston file, I'll want to make a separate copy that I own and
stash it somewhere in my project ...

In addition, I'm not quite sold on the idea to be able to configure
something in the smalltalkCI config file and then be able to override
it using cmd line parameters /and/ env vars.

Well the env vars are mainly for creating test matrices for travis (you
can't create a test matrix in travis without using env vars) and the
command line options are mainly for using smalltalkCI on a local machine
(it doesn't make sense to require a user to define env vars before
running a bash script). Since env vars can exist on a local machine and
one may set run.sh options in a .travis.yml, there needs to be a
precedence order defined.... and of course both are necessary to avoid
requiring users to manage mostly duplicated sets of config files...

Keep in mind that this is a rule of thumb. For example I don't think
that it would make sense to create a travis test matrix for the stone
name used by smalltalkCI, but when running locally it is necessary to be
able to override the default stone name, because two stones with that
same name may not run on the same machine ... But sometimes it is just
easier to follow the rule of thumb than to explain to a user why you've
disallowed her perfectly good use case, because at the time you didn't
think it was necessary:) So I will be inclined to follow the rule of
thumb pretty mindlessly

I'm also not sure how you want to use the options within the config file.

Well, to start with there are a very large number of configuration
options for GemStone[1] and there are two basic config files that
GemStone uses: system.conf[2] and gem.conf[3].

Over the long haul, I expect that it will be useful to specify both the
system.conf and gem.conf files in the config file which would allow a
user to customize any number of GemStone configuration options, but in
the specific use case a user is running out of temporary object space
when running tests on travis using smalltalkCI (actually the Saaside
project[5]). In builderCI, I wired the temp obj cache size to 100M, but
with SmalltalkCI and GsDevKit_home, the default temp obj cache size is
50M, so we have a user that needs to increase the temp obj cache size
--- for travis builds. Rather than add a GemStone option per GemStone
configuration option (there might be hundreds) I am opting to allow the
user to specify the gem.conf file to be used when running travis and the
user can set it to whatever value she would like for the temp obje cache
--- literally killing hundreds of birds with one stone.

The second use case for the stone is the default TimeZone for the system
... GemSTone uses PST by default and of course actual developers will
want to run their tests using the appropriate time zone.

In both of these cases, the developer cannot change the settings from
within the vm itself ... the temp obj cache needs to specified before
starting the gem and the time zone needs to be set by the System User ...

[1]
https://downloads.gemtalksystems.com/docs/GemStone64/3.3.x/GS64-SysAdminGuide-3.3/GS64-SysAdminGuide-3.3.htm?https://downloads.gemtalksystems.com/docs/GemStone64/3.3.x/GS64-SysAdminGuide-3.3/A-ConfigOptions.htm#pgfId-355643
[2]
https://downloads.gemtalksystems.com/docs/GemStone64/3.3.x/GS64-SysAdminGuide-3.3/1-Server.htm#pgfId-83703
[3]
https://downloads.gemtalksystems.com/docs/GemStone64/3.3.x/GS64-SysAdminGuide-3.3/GS64-SysAdminGuide-3.3.htm?https://downloads.gemtalksystems.com/docs/GemStone64/3.3.x/GS64-SysAdminGuide-3.3/2-Clients.htm#pgfId-82579
[4]
https://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-ProgGuide-3.2/GS64-ProgGuide-3.2.htm?https://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-ProgGuide-3.2/14-Performance.htm#pgfId-350732
[5] https://github.com/SeasideSt/Seaside

Will you start a client from within the system GemStone image or how
do you want to access these options within the bash script?

Well this is the crux of the problem. Right now I start a tODE client to
run the server side tests ...

For a Tugrik client, a tODE client is not used, so I must arrange to
build and run tests in two coordinated processes --- smalltalkCI without
these additional client/server options is only designed to manage a
single image at a time and if I want to run both client and server
tests, I must be able to run two images at the same time in a
coordinated fashion ... at some level the actual details of how this
is done (from within a "system GemStome image" or a bash script) is
immaterial --- I need to be able to specify that instead of running
EITHER a #gemstone platform OR a #pharo platform ... I must be able to
specify both and I need to coordinate between the two ..

So the client must be run within the GsDevKit_home environment itself
otherwise how would the client know the name of the stone or which
version of the GemStone libraries to load or what user name to use to
login or ....

To be hosnest before SmalltalkCI came along and you invented the
cross-platform config file, I was at a loss as to how to solve these
problems ... I think that by adding GemStone specific configurations
classes for the server and the client is an elegant solution to a very
difficult and complicated problem ...

Dale

[fniephaus]

SCIAbstractConfig sounds good :)

For MongoDB, you might want to have a look at this as well.

Sure, there is nothing wrong with using multiple STON configs, I just thought that for the system/client setup it might be handy to be able to also use just one file.

I had no idea how many options there are for GemStone. I must admit that I never even had a look at the GemStone/S 64 Bit Programmer's Guide. But now I think it might be better to use a GemStone-specific config as well. One could provide this config via --gs-config system.conf or for Travis via the corresponding GSCI_CONFIG env var and then smalltalkCI sets $GEMSTONE_EXE_CONF accordingly? Why would you want to reference this file in the STON config if it operates on bash-level, or is this config file loaded in a GemStone image? Additionally, if a user wants to change the config locally, he can edit the config file locally or just provide a different one. I assume this is what you wanted to achieve with #gemConfPath, but I just don't understand why it needs to be in the STON.

I'm also not quite sure yet how we could synchronize two builds (system and client). We could initiate two separate builds and tell the system build to keep the image open until the client build tells it to close the session. Multiple JUnit XML files within the build directory should already be picked up by the prettifier, so just make sure system and client tests generate a JUnit file each.
Instead of using your suggested #smalltalkCIConfig for a client, we could also somehow implement before and after events for smalltalkCI. Then you could for example say that after a system build, you want to trigger a client build. Or we could support something like this:

./run.sh -s GemStone system.ston client.ston

Not sure if this would be helpful though.

I'm glad that we're apparently able to solve a complex problem with this approach. I didn't invent anything myself here, I'd say that this was and continues to be a team effort. And as always, communication is the key ;) Having said that, I'm happy and flexible enough to jump on a quick video call if you think it'd be better to discuss this without writing yet another novel 😜

Fabio

[dalehenrich]

@fniephaus, I'm pretty comfortable with the basic solution that I have been proposing:

• gs-specific env vars
• gs-specific command line arguments
• gs-specific entries in system config object in smalltalk.ston

In the standard case, the .smalltalk.ston file is sufficient ...

When one runs smalltalkCI locally for either debugging a travis failure or using it as part of ones own local CI runs, the command line arguments are available for overriding any of .smalltalk.ston entries --- if necessary.

If one wants to vary entries in the .smalltalk.ston file for a travis test matrix the env vars can be used ...

For GsDevKit_home, I am adopting the .smalltalk.ston files for use in configuring the creation of stones and custom clients.

The important point here is:

The exact same information that is needed for setting up and running a travis build is needed for setting up a stone/client system for GsDefKit_home ...

I am trying to keep the GemStone-specific changes within the scope of smalltalkCI and travis and I think that my proposal does that ...

[fniephaus]

The above all sounds good to me, except that I haven't fully understood yet how you intend to trigger the client build using the second STON file defined in the first STON file. But I guess I will find out soon, since you are already working on it :)

[dalehenrich]

Good! I haven't quite figured out how myself ... part of it has to do with figuring out how to get the results displayed (sounds like I might have to arrange to use non-default xml log file names, or perhaps call print_results myself?) and part of it has to do with figuring out when I need to run the client tests --- as you point out: How will the gemstone.sh script know to run a client test?. I have several ideas in this area, but I haven't quite got to that point yet :)

[dalehenrich]

Here's where I'm tracking my progress towards figuring out the client/server testing split

[fniephaus]

Maybe running a duo build already is such a special use case, so that we don't need to support three or more synchronous builds. By that I mean, maybe we can do both in one run of smalltalkCI which eliminates the need to overwrite script in the travis.yml and we also don't have to worry about things like print_results.
The more I think about it, the less I believe we need to support n passes of smalltalkCI per build. Usually you would want different Travis CI builds for that.

[dalehenrich]

Not sure what your point is here ... I have a relatively common use case (for GemStone) where I want to run both client and server tests in the same build using a single smalltalk.ston config file --- and I expect to solve the problem within the context of the gemstone/run.sh build driver ... outside of GemStone I agree that n passes does not make sense - although you already make two passes for the SmalltalkCI travis tests.

I'm just not sure what you are fretting about :)

[fniephaus]

I was just thinking out loud 😄 and I guess my point was that I think it'd be enough to just add support for system/client duo tests to smalltalkCI and nothing more (e.g. my suggested after and before). Let keep it simple :) And of course, you can run smalltalkCI multiple times if you want to

[dalehenrich]

Okay, I wan't paying that much attention to before/after discussion, since it didn't seem to be applicable ... GemStone is more like a regular smalltalk image than a database service: the client server scenarios I am interested are cases where the project needs to be loaded into both the client and the server and tests run on both the client and server ... I think I finally have all the pics figured out ... but we'll see how far I get :)

[dalehenrich]

fixed with PR #123