diff --git a/davinci-grid.md b/davinci-grid.md index c94f200..d47f4d5 100644 --- a/davinci-grid.md +++ b/davinci-grid.md @@ -123,7 +123,7 @@ To look at the `root` file produced by the job start a new terminal, and type: ```bash -$ lb-run DaVinci v41r2 $SHELL +$ lb-run DaVinci/v41r2 $SHELL $ root -l path/to/the/job/output ``` diff --git a/davinci.md b/davinci.md index fc147c8..0f80844 100644 --- a/davinci.md +++ b/davinci.md @@ -58,7 +58,7 @@ One of the most important ones is *DaVinci*, which provides lots of *Algorithms* You can run DaVinci using the following command: ```bash -lb-run DaVinci v41r2 gaudirun.py +lb-run DaVinci/v41r2 gaudirun.py ``` This will run the `gaudirun.py` command using version v41r2 of DaVinci. @@ -89,7 +89,7 @@ During this run, DaVinci didn't do anything: We didn't specify any algorithms to Usually, you will write an option file (e.g. `options.py`) and specify it as an argument to `gaudirun.py`: ```bash -lb-run DaVinci v41r2 gaudirun.py options.py +lb-run DaVinci/v41r2 gaudirun.py options.py ``` An `option.py` is just a regular Python script that specifies how to set things up in the software. @@ -97,7 +97,7 @@ Many of the following lessons will teach you how to do something with DaVinci by You can use the above command to test it. You can also specify several option files like this: ```bash -lb-run DaVinci v41r2 gaudirun.py options1.py options2.py +lb-run DaVinci/v41r2 gaudirun.py options1.py options2.py ``` They will then both be used to set up DaVinci. @@ -108,7 +108,7 @@ lb-run --list DaVinci Do you want to start a shell that already contains the LHCb environment, so you don't have to use `lb-run`? Execute ```bash -lb-run DaVinci v41r2 $SHELL +lb-run DaVinci/v41r2 $SHELL ``` A simple `gaudirun.py` should work as well now. Typing `exit` will close the shell and leave the LHCb environment behind. @@ -119,7 +119,5 @@ Typing `exit` will close the shell and leave the LHCb environment behind. > `lb-run` is the new way of doing things and has some nice benefits over `SetupProject`. > For most purposes, `SetupProject DaVinci v41r2` is equivalent to > ```bash -> lb-run DaVinci v41r2 $SHELL +> lb-run DaVinci/v41r2 $SHELL > ``` - - diff --git a/interactive-dst.md b/interactive-dst.md index 0a15eec..a25b54e 100644 --- a/interactive-dst.md +++ b/interactive-dst.md @@ -10,7 +10,7 @@ minutes: 10 > * Print all nodes in a DST > * Explore the contents of the TES > * Inspect a track -> * Inspect a stripping location +> * Inspect a stripping location Data is stored in files called DSTs, which are processed by DaVinci to make nTuples. However you can also explore @@ -51,7 +51,7 @@ Place this into a file called `first.py` and run the following command in a new terminal: ```bash -$ lb-run DaVinci v41r2 ipython -i first.py 00035742_00000002_1.allstreams.dst +$ lb-run DaVinci/v41r2 ipython -i first.py 00035742_00000002_1.allstreams.dst ``` This will open the DST and print out some of the TES locations @@ -178,8 +178,8 @@ print_decay = appMgr.toolsvc().create( print_decay.printTree(cands[0]) ``` -With our candidates in hand, it would be nice to be able to retrieve and -compute the variables we need for an analysis. On to [LoKi +With our candidates in hand, it would be nice to be able to retrieve and +compute the variables we need for an analysis. On to [LoKi functors](loki-functors.html)! > ## Fast DST browsing {.callout} @@ -192,10 +192,10 @@ functors](loki-functors.html)! > For example, to explore the `DST` we could have simply done: > > ``` -> lb-run Bender bender 00035742_00000002_1.allstreams.dst +> lb-run Bender/latest bender 00035742_00000002_1.allstreams.dst > ``` > -> This leaves us in a prompt in which we can proceed as discussed in this +> This leaves us in a prompt in which we can proceed as discussed in this > lesson, with the advantage that some functions are already provided > for us, such as `seekStripDecision` (which replaces our `advance`) or > `ls` and `get`, which allow to list and get TES locations. @@ -205,7 +205,7 @@ functors](loki-functors.html)! > `Bender` also provides a useful command `dst-dump`, which is a quick way of > figuring out what objects are present on a `DST` and where. Try out: > ``` -> lb-run Bender dst-dump -f -n 100 00035742_00000002_1.allstreams.dst +> lb-run Bender/latest dst-dump -f -n 100 00035742_00000002_1.allstreams.dst > ``` > The `-f` option tells `Bender` to try and "unpack" the locations such as > `/Event/AllStreams/pPhys/Particles` that we mentioned above, while `-n 100` diff --git a/lhcb-dev.md b/lhcb-dev.md index bb4f371..69fab84 100644 --- a/lhcb-dev.md +++ b/lhcb-dev.md @@ -8,15 +8,15 @@ minutes: 10 > ## Learning Objectives {.objectives} > * Learn the basics of how to work with and modify LHCb software packages -This lesson introduces you to two commands: +This lesson introduces you the commands: - `lb-dev` for setting up a new development environment - - `getpack` for downloading LHCb software packages + - `git lb-use` and `git lb-checkout` for downloading LHCb software packages If you want to make changes to a software package, you will need to set up a development environment. `lb-dev` is your friend here: ```bash -lb-dev --name DaVinciDev DaVinci v41r2 +lb-dev --name DaVinciDev DaVinci/v41r2 ``` The output should look similar to this: @@ -43,6 +43,14 @@ You can customize the configuration by editing the files 'build.conf' and 'CMakeLists.txt' (see http://cern.ch/gaudi/CMake for details). ``` +> ## `lb-dev` created local projects are Git repositories {.callout} +> When `lb-dev` creates the local project directory and create the initial files +> there, it also calls `git init` and commits to the local Git repository the +> first version of the files (try with `git log` in there). +> +> You can then use git to keep track of your development, and share your code +> with others (for example with a [new project in gitlab.cern.ch](https://gitlab.cern.ch/projects/new)). + Follow those instructions and once you've compiled your software run ```bash @@ -52,7 +60,7 @@ Follow those instructions and once you've compiled your software run inside the directory. This will (similar to `lb-run`) give you a new bash session with the right environment variables. Your new development environment won't be very useful without any software to modify and build. -So why not check out out one of the existing LHCb packages, which are stored in the LHCb SVN repository? +So why not check out one of the existing LHCb packages, which are stored in the [LHCb Git repositories](https://gitlab.cern.ch/lhcb)? Let's assume you have already used `lb-dev` to set up your development environment and you are currently inside it. In order to obtain the source code of the package you want to work on, use the [Git4LHCb](https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb) scripts. @@ -68,6 +76,20 @@ make configure Under the hood, `git lb-use` will add the [`Stripping`](https://gitlab.cern.ch/lhcb/Stripping) repository as a remote in git. `git lb-checkout` will then perform a partial checkout of the master branch of the Stripping repository, only adding the files under [`Phys/StrippingSelections`](https://gitlab.cern.ch/lhcb/Stripping/tree/master/Phys/StrippingSelections). +> ## Which project to use in `git lb-use`? {.callout} +> The project name to pass to `git lb-use` depends on the directories you want +> to check out and work on, and not on the project name you passed to `lb-dev`. +> Moreover you can call `git lb-use` several times for different remote +> projects in the same local project: +> +> ```bash +> lb-dev --name DaVinciDev DaVinci/v40r2 +> cd DaVinciDev +> git lb-use Analysis +> git lb-use Stripping +> git lb-use DaVinci +> ``` + You can now modify the `StrippingSelections` package and run `make purge && make` to build it with your changes. You can test your changes with the `./run` script. It works similar to `lb-run`, without the need to specify a package and version: @@ -75,26 +97,17 @@ It works similar to `lb-run`, without the need to specify a package and version: ./run gaudirun.py options.py ``` -> ## What if git asks for my password 1000 times? {.callout} -> `git` might ask you for your password several times. -> To avoid this, you can create a kerberos token with -> ``` -> kinit $USER@CERN.CH -> ``` -> You will have to enter your password once, and further password prompts will be skipped. - -If you just want to take a look at a source file, without checking it out, you can comfortably access the repository through two different web UIs. - - * [Trac](https://svnweb.cern.ch/trac/lhcb/) - * [SVNweb](http://svnweb.cern.ch/world/wsvn/lhcb) +> ## What if `git` asks for my password? {.callout} +> Make sure you followed the [prerequisites](https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb#Prerequisites) +> section in the [Git4LHCb TWiki page](https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb). -(which one to use? It is just a matter of taste, pick the one that looks nicest) +If you just want to take a look at a source file, without checking it out, you can comfortably access the repository through +the [Gitlab web interface](https://gitlab.cern.ch/lhcb). -To get an idea of how a certain component of the LHCb software works, you can access its doxygen documentation. -There is a page for each project, lists of projects can be found here: - - * [https://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/latest_doxygen/index.html](https://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/latest_doxygen/index.html). - * [http://lhcb-comp.web.cern.ch/lhcb-comp/](http://lhcb-comp.web.cern.ch/lhcb-comp/) +To get an idea of how a certain component of the LHCb software works, you can access the doxygen documentation. +One set of doxygen web pages is generated for several related projects, and is linked in all the +projects web sites, like https://cern.ch/LHCb-release-area/DOC/davinci/. +See also the the [LHCb Computing web page](http://cern.ch/lhcb-comp/) for a list of projects. Another useful tool available on lxplus machines is `Lbglimpse`. It allows you to search for a given string in the source code of LHCb software. ```bash @@ -102,27 +115,33 @@ Lbglimpse "PVRefitter" DaVinci v41r2 ``` It works with every LHCb project and released version. -If you have made changes that are supposed to be integrated into the official LHCb repositories, you can use `svn commit`. -But read the instructions in the SVN usage guidelines first. - - * [SVNUsageGuidelines](https://twiki.cern.ch/twiki/bin/view/LHCb/SVNUsageGuidelines) +If you have made changes that are supposed to be integrated into the official LHCb repositories, you can use `git lb-push`. +But read the instructions in the [TWiki page](https://twiki.cern.ch/twiki/bin/view/LHCb/Git4LHCb#Using_Git_for_LHCb_development) first. -Be advised that you should always communicate with the package maintainers before committing changes! -The release managers have to find the packages to include in a new release. Therefore make sure -to document your changes in the release notes which are found in `doc/release.notes`. -After that put your commit into the tag collector. +Depending on the project, you may be required to document your changes in the +release notes which are found in `doc/release.notes`. - * [Tag collector FAQ](https://twiki.cern.ch/twiki/bin/view/LHCb/FAQ/TagCollectorFAQ) - * [LHCb Tag Collector](https://lhcb-tag-collector.web.cern.ch/lhcb-tag-collector/index.html) +After the call to `git lb-push`, go to the project repository web page to +create a new merge request, for example https://gitlab.cern.ch/lhcb/Stripping/merge_requests/new. -(The login button is in the top right corner.) +> ## Quick link to create a merge request {.callout} +> When pushing to a branch in a project in Gitlab you will see a message like: +> +> ``` +> remote: +> remote: Create merge request for my-branch: +> remote: https://gitlab.cern.ch/lhcb/Stripping/merge_requests/new?merge_request%5Bsource_branch%5D=my-branch +> remote: +> ``` +> +> You can use the URL in the message to quickly create a merge request for the +> changes you just pushed. -It is advisable to test new developments on the so-called nightly builds. -They take everything which is committed in the head versions of all packages -and try to build the given project. -You can checkout this version by doing: +It is advisable to test new developments on the so-called [nightly builds](https://lhcb-nightlies.cern.ch). +They take build all the projects with all pending merge requests. +You can use a nightly build version of a project with: ```bash -lb-dev DaVinci HEAD --nightly lhcb-head +lb-dev --nightly lhcb-head DaVinci/HEAD ``` A more detailed description of the command is found here: @@ -133,10 +152,8 @@ Therefore the nightly tests are performed. They first try to build the full soft If that is successful, they run some reference jobs and compare the output of the new build with a reference file. The results of the nightly builds can be found here. -* [Nightly tests](https://buildlhcb.cern.ch/nightlies/) +* [Nightly builds summaries](https://lhcb-nightlies.cern.ch) If the aim of the commit was to change the ouput, e.g. because you increased the -track reconstruction efficiency by a factor of two, inform the release manager of the package -such that he can update the reference file. - - +track reconstruction efficiency by a factor of two, mention it in the merge request +description, such that the manager of the affected project can update the reference file. diff --git a/minimal-dv-job.md b/minimal-dv-job.md index ac0aab4..b3e5e61 100644 --- a/minimal-dv-job.md +++ b/minimal-dv-job.md @@ -5,15 +5,15 @@ subtitle: Running a minimal DaVinci job locally minutes: 10 --- -[Looping event-by-event](interactive-dst.html) over a file and [inspecting -interesting quantities with LoKi functors](loki-functors.html) is great for -exploration: to checking that the file contains the candidates you need, that +[Looping event-by-event](interactive-dst.html) over a file and [inspecting +interesting quantities with LoKi functors](loki-functors.html) is great for +exploration: to checking that the file contains the candidates you need, that the topology makes sense, and so on. -It's impractical for most cases, though, where you want _all_ the candidates -your trigger/stripping line produced, which could be tens of millions of +It's impractical for most cases, though, where you want _all_ the candidates +your trigger/stripping line produced, which could be tens of millions of decays. -In these cases we use DaVinci, the application for analysing high-level -information such as tracks and vertices, which we'll look at in this lesson to +In these cases we use DaVinci, the application for analysing high-level +information such as tracks and vertices, which we'll look at in this lesson to produce a ROOT ntuple. > ## Learning Objectives {.objectives} @@ -22,23 +22,23 @@ produce a ROOT ntuple. > * Inspect the ntuple output > * Set up the job to run in Ganga -With some stripped data located, it's useful to store the information on the +With some stripped data located, it's useful to store the information on the selected particles inside an ntuple. -This allows for quick, local analysis with [ROOT](https://root.cern.ch/), -rather than always searching through a DST that contains lots of things we're +This allows for quick, local analysis with [ROOT](https://root.cern.ch/), +rather than always searching through a DST that contains lots of things we're not interested in. -As well as being the application that runs the stripping, -[DaVinci](http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci) -allows you to access events stored in DSTs and copy the information to ROOT +As well as being the application that runs the stripping, +[DaVinci](http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci) +allows you to access events stored in DSTs and copy the information to ROOT ntuples. -You tell DaVinci what you want it to do through _options files_, written in +You tell DaVinci what you want it to do through _options files_, written in Python. -There are lots of things you can do with DaVinci options files, as there's lots -of information available on the DST, but for now we'll just work on getting the +There are lots of things you can do with DaVinci options files, as there's lots +of information available on the DST, but for now we'll just work on getting the bare essentials up and running. -Our main tool will be the `DecayTreeTuple` object, which we'll create inside a +Our main tool will be the `DecayTreeTuple` object, which we'll create inside a file we will call `ntuple_options.py`: ```python @@ -55,38 +55,38 @@ dtt.Inputs = ['/Event/{0}/Phys/{1}/Particles'.format(stream, line)] dtt.Decay = '[D*(2010)+ -> (D0 -> K- pi+) pi+]CC' ``` -This imports the `DecayTreeTuple` class, and then creates an object called +This imports the `DecayTreeTuple` class, and then creates an object called `dtt` representing our ntuple. -Once DaVinci has run, the resulting ntuple will be saved in a folder within the +Once DaVinci has run, the resulting ntuple will be saved in a folder within the output ROOT file called `TupleDstToD0pi_D0ToKpi`. -The `Inputs` attribute specifies where `DecayTreeTuple` should look for -particles, and here we want it to look at the output of the stripping line +The `Inputs` attribute specifies where `DecayTreeTuple` should look for +particles, and here we want it to look at the output of the stripping line we're interested in. -As stripping lines can save many decays to a DST, the `Decay` attribute +As stripping lines can save many decays to a DST, the `Decay` attribute specifies what decay we would like to have in our ntuple. -If there are no particles at the `Input` location, or the `Decay` string +If there are no particles at the `Input` location, or the `Decay` string doesn't match any particles at that location, the ntuple will not be filled. > ## Decay descriptors {.callout} > -> There is a special syntax for the `Decay` attribute string, commonly called +> There is a special syntax for the `Decay` attribute string, commonly called > 'decay descriptors', that allow a lot of flexibility with what you accept. -> For example, `D0 -> K- X+` will match any D0 decay that contains one +> For example, `D0 -> K- X+` will match any D0 decay that contains one > negatively charged kaon and one positively charged track of any species. -> More information the decay descriptor syntax can be found on the [LoKi decay -> finders TWiki +> More information the decay descriptor syntax can be found on the [LoKi decay +> finders TWiki > page](https://twiki.cern.ch/twiki/bin/view/LHCb/FAQ/LoKiNewDecayFinders). Now we need to tell DaVinci how to behave. -The `DaVinci` class allows you to tell DaVinci how many events to run over, -what type of data is being used, what algorithms to run over the events, and so +The `DaVinci` class allows you to tell DaVinci how many events to run over, +what type of data is being used, what algorithms to run over the events, and so on. -There are [many configuration -attributes](http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/releases/v41r2/doxygen/py/dc/d2f/class_da_vinci_1_1_configuration_1_1_da_vinci.html#ac788f6a80f5f61d47056debe7b86ca71) -defined on the `DaVinci` object, but we will only set the ones that are +There are [many configuration +attributes](http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/releases/v41r2/doxygen/py/dc/d2f/class_da_vinci_1_1_configuration_1_1_da_vinci.html#ac788f6a80f5f61d47056debe7b86ca71) +defined on the `DaVinci` object, but we will only set the ones that are necessary for us. ```python @@ -106,40 +106,40 @@ DaVinci().CondDBtag = 'sim-20130522-1-vc-md100' DaVinci().DDDBtag = 'dddb-20130929-1' ``` -Nicely, a lot of the attributes of the `DaVinci` object are self-explanatory: -`InputType` should be `'DST'` when giving DaVinci DST files; `PrintFreq` -defines how often DaVinci should print its status; `DataType` is the year of -data-taking the data corresponds to, which we get from looking at the -bookkeeping path used to get the input DST; `Simulation` should be `True` when -using Monte Carlo data, `Lumi` defines whether to store information on the -integrated luminosity the input data corresponds to; and `EvtMax` defines how +Nicely, a lot of the attributes of the `DaVinci` object are self-explanatory: +`InputType` should be `'DST'` when giving DaVinci DST files; `PrintFreq` +defines how often DaVinci should print its status; `DataType` is the year of +data-taking the data corresponds to, which we get from looking at the +bookkeeping path used to get the input DST; `Simulation` should be `True` when +using Monte Carlo data, `Lumi` defines whether to store information on the +integrated luminosity the input data corresponds to; and `EvtMax` defines how many events to run over, where a value of `-1` means "all events". -The `CondDBtag` and `DDDBtag` attributes specify the exact detector conditions +The `CondDBtag` and `DDDBtag` attributes specify the exact detector conditions that the Monte Carlo was generated with. -Specifying these tags is important, as without them you can end up with the +Specifying these tags is important, as without them you can end up with the wrong magnet polarity value in your ntuple, amongst other Bad Things. -You can find the values for these tags in the [bookkeeping -file](data/MC_2012_27163003_Beam4000GeV2012MagDownNu2.5Pythia8_Sim08e_Digi13_Trig0x409f0045_Reco14a_Stripping20NoPrescalingFlagged_ALLSTREAMS.DST.py) +You can find the values for these tags in the [bookkeeping +file](data/MC_2012_27163003_Beam4000GeV2012MagDownNu2.5Pythia8_Sim08e_Digi13_Trig0x409f0045_Reco14a_Stripping20NoPrescalingFlagged_ALLSTREAMS.DST.py) we downloaded earlier. > ## Database tags {.callout} > -> Generally, the `CondDB` and `DDDB` tags are different for each dataset you +> Generally, the `CondDB` and `DDDB` tags are different for each dataset you > want to use, but will be the same for all DSTs within a given dataset. -> When using simulated data, *always* find out what the database tags are for +> When using simulated data, *always* find out what the database tags are for > your dataset! -> For real collision data, you shouldn't specify these tags, as the default -> tags are the latest and greatest, so just remove those lines from the options +> For real collision data, you shouldn't specify these tags, as the default +> tags are the latest and greatest, so just remove those lines from the options > file. -In order to run an algorithm that we have previously created, we need to add it +In order to run an algorithm that we have previously created, we need to add it to the `UserAlgorithms` list. -The `TupleFile` attribute defines the name of the ROOT output file that DaVinci +The `TupleFile` attribute defines the name of the ROOT output file that DaVinci will store any algorithm output in, which should be our ntuple. All that's left to do is to say what data we would like to run over. -As we already have a data file [downloaded locally](files-from-grid.html), we +As we already have a data file [downloaded locally](files-from-grid.html), we define that as our input data. ```python @@ -151,19 +151,19 @@ IOHelper().inputFiles([ ], clear=True) ``` -This says to use the `.dst` file that is in the same directory as the options +This says to use the `.dst` file that is in the same directory as the options file, and to clear any previous input files that might have been defined. That's it! We're ready to run DaVinci. -In the same folder as your options file `ntuple_options.py` and your DST file +In the same folder as your options file `ntuple_options.py` and your DST file ending in `.dst`, there's just a single command you need run on `lxplus`. ```shell -$ lb-run DaVinci v41r2 gaudirun.py ntuple_options.py +$ lb-run DaVinci/v41r2 gaudirun.py ntuple_options.py ``` -The full options file we've created, `ntuple_options.py`, is [available +The full options file we've created, `ntuple_options.py`, is [available here](./code/minimal-dv/ntuple_options.py). A slightly modified version that uses remote files (using an XML catalog as [described here](files-from-grid.html)) is [available @@ -173,12 +173,12 @@ here](./code/minimal-dv/ntuple_options_xmlcatalog.py) > > A microDST (or µDST) is a smaller version of a DST. > Some stripping lines go to µDSTs, and some go to DSTs. -> There are two things that need changing in our options file in order to have +> There are two things that need changing in our options file in order to have > it work when it is used with a stripping line that goes to a µDST: > > 1. The `DecayTreeTuple.Inputs` attribute should start at the word > `Phys`; and -> 2. The `RootInTES` attribute on the `DaVinci` object has to be set to +> 2. The `RootInTES` attribute on the `DaVinci` object has to be set to > `/Event/$STREAM` > > In context, the changes look like