Skip to content
This repository has been archived by the owner on Jan 30, 2024. It is now read-only.

2.2.0 #219

Merged
merged 68 commits into from
Oct 10, 2019
Merged

2.2.0 #219

merged 68 commits into from
Oct 10, 2019

Conversation

PalNilsson
Copy link
Collaborator

  • Added MANIFEST file - Pilot 2 is now registered with pypi which means it can be pip-installed without referencing github
    • pip install panda-pilot
  • Data component upgrade
    • Refactored and unified ES StagingClients
    • Automatically prefer LAN protocol (read_lan/write_lan) for stage-in/stage-out file if source/destination RSE is local for given PQ (defined in inputddms=astorages['read_lan'])
    • Base movers workflow upgraded
    • Introduced require_input_protocols mode to look up and manually form input replicas for specific copytool (activated for the objectstore mover, ES workflow)
    • Refactored and simplified the objectstore copytool
    • Implemented fail-over transfer for ES stage-out
  • Preparing for containerized middleware commands [minor update]
  • Added debug messages for potential problem with relying on SC_CLK_TCK
  • Added LFN in diagnostics message for checksum errors. Corrected mislabelled checksum types (MD5SUM reported instead of ADLER32). Requested by R. Walker
  • Cleaned up stage-in/out error messages containing irrelevant Traceback info (should now be concise)
  • Following an update in the auto-setup script, the pilot is now using RUCIO_LOCAL_SITE_ID instead of the deprecated DQ2_LOCAL_SITE_ID for localsite in Rucio traces
  • Simplification of pilot arguments: now using resource name from queuedata instead of relying on pilot option -r (which can now be removed from wrapper)
  • Instead of a traceback, now reporting the real error returned from rucio download or upload. However, the current version of rucio does not propagate errors well so the message will always be "None of the requested files have been downloaded". D. Cameron is working on fixing this so a future version of Rucio will report the real error
  • Changed minimum allowed local space from 5 GB to 2 GB (as verified during payload running); the higher limit affected event index jobs run at OU. Requested by H. Severini
  • Pilot is now always setting ATHENA_CORE_NUMBER (previously only set for event service jobs)
  • Updated memory leak calculation to be consistent with new prmon field names (changed PSS+Swap to pss+swap)
  • Added new error code 1352, “Failed to stat proc file for CPU consumption calculation” which is set when the pilot cannot access /prod/pid/stat. Requested by P. Svirin
  • Corrected the local/remoteSite sent with the traces - previously if the pilot overwrote the requested ddmendpoint (ie if the requested ddmendpoint was not allowed), then the trace was not updated as well. Now it is.

Code contributions from D. Cameron, A. Anisenkov, W. Guan, F. Barreiro, P. Nilsson

anisyonk and others added 30 commits August 3, 2019 16:50
@anisyonk
- base movers workflow upgraded
8e565e1 
 - refactor and unify ES StagingClients
 - automatically prefer `LAN` protocol for stage-in/stage-out transfers if source/destination RSE is local for given PQ (defined in `inputddms=astorages['read_lan']`)
 - introduce `require_input_protocols` mode to look up and manually form input replicas for specific copytool (activated for the objecstore mover)
 - refactor and simplify `objecstore` copytool

To BE TESTED
@anisyonk
implemented fail-over transfers for ES stage-out (`failover_storage_a…
86ea7fa 
…ctivity = ['es_failover', 'pw']`)
@PalNilsson
New version 2.1.26
cfc3a94
@PalNilsson
Corrected repo name, pilot->panda-pilot
3ce9386
@PalNilsson
Testing container use in lsm stage-in/out
e259677
@PalNilsson
do_use_container() now supports containers in stage-in/out
ece7d70
@PalNilsson
Added some protection against unset job object in container code
d05410c
@PalNilsson
Job object is now optional in singularity_wrapper(). Middleware image…
7b781b3 
… will be used if job object is not set
@PalNilsson
Set usercontainer=False in lsm
e4625b2
@PalNilsson
Merge pull request #185 from anisyonk/dataapi_movers_esclient_refactor
c350175 
Data API updates, unify ES StagingClients
@PalNilsson
Merge remote-tracking branch 'upstream/next' into next
5e1a7e6
@PalNilsson
Updated build number after merge with AA PR
0203236
@PalNilsson
Now confirming that the worker node has a proper SC_CLK_TCK (problems…
593a9d6 
… seen on MPPMU)
@PalNilsson
Inclreased sleep from 0.1 to 1 s in thread counting function
2f6a8d3
@anisyonk
- fix copytools failover logic for stage-in (enforce replica look up…
4a316f7 
… for allowed schemas)

 - add root:// protocol into supported list for the gfal sitemover
@anisyonk
code style fix: resolve complexity error
767c717
@PalNilsson
Merge pull request #210 from anisyonk/copytools_failover_fix
50dedfe 
Copytools failover fix
@PalNilsson
Merge remote-tracking branch 'upstream/next' into next
12bf682
@PalNilsson
Updated build number after merge with AA PR 2
1231039
@PalNilsson
Correction in ExtendedConfig to resolve hasattr() difference between …
0e15fb4 
…python versions
@PalNilsson
Correction in ExtendedConfig to resolve DictProxyType difference betw…
0990399 
…een python versions
@anisyonk
fix typo in StageInESClient
11298ad
@PalNilsson
Merge pull request #211 from anisyonk/typo_fix
6443818 
fix typo in StageInESClient
@PalNilsson
Merge remote-tracking branch 'upstream/next' into next
395763d
@PalNilsson
Merge with AA PR; fix for type in ES code
3fb50f6
@PalNilsson
Corrected ad32->adler32 which led to MD5SUM error being reported inst…
6f02707 
…ead of ADLER32 error (reported by Rod)
@PalNilsson
Cleanup
193df85
@davidgcameron
add check for output file after upload
03c5aa1
@PalNilsson
Cleanup
4034323
@PalNilsson
Additional cleanup
c132631
PalNilsson and others added 29 commits September 24, 2019 16:31
@PalNilsson
Updated build number, 2.1.26 -> 2.2.0
0b63f3f
@PalNilsson
Removed debug messages
de12931
@PalNilsson
Added fix for objectstore copytool + added timeout decorator to rucio…
c9dd73c 
…, default 10800, 3h
@PalNilsson
Now using file size based time-outs for rucio api transfers on the pi…
d577e2c 
…lot side as an extra layer of protection for cases where the time-out mechanism on the rucio api side does not work - pilot waits an additional 10 s to let rucio abort first
@PalNilsson
Now using turl = fspec.turl or fspec.surl as recommended by Alexey
8b17d7c
@davidgcameron
fix stage in/out code and propagate first error message
3280a30
@davidgcameron
fix test after queue was renamed
a46332a
@PalNilsson
Merge pull request #214 from davidgcameron/rucioerror
d3cdd86 
fix stage in/out code and propagate first error message
@PalNilsson
Merge with David's PR
75a2398
@PalNilsson
Updated build number after merge with David's code
7aaf2f4
@PalNilsson
Corrected usages of format_diagnostics() (wrong argument) after merge…
6347b84 
… with David's code
@PalNilsson
Changed free_space_limit from 5 GB to 2 GB as requested by H. Severini
360e346
@PalNilsson
Added ATHENA_CORE_NUMBER to add_athena_proc_number()
acdc8c3
@PalNilsson
Added ATHENA_CORE_NUMBER to add_athena_proc_number()
c9c6e88
@davidgcameron
always try to extract detail of error message
2f2011e
@davidgcameron
get error information from rucio traces if available
cc62dd2
@PalNilsson
Merge pull request #216 from davidgcameron/ruciotrace
bc00581 
Propagate error info from rucio traces
@PalNilsson
Corrected PSS+Swap -> pss+swap in leak calculation. Merge with David'…
07995b0 
…s latest PR
@PalNilsson
Corrected diagnostics
7e0ff22
@PalNilsson
Improved formatting of error diagnostics in format_diagnostics()
828ffa9
@PalNilsson
Added handling for STATFILEPROBLEM, new error code 1352
3e8f999
@PalNilsson
Avoiding using timeout function since it leads to SIGTERMs. To be inv…
008a7d8 
…estigated
@PalNilsson
Corrected spelling error
80c279e
@PalNilsson
Removed timeout usages
4e12971
@PalNilsson
Now updating localsite and remotesite before trace is sent (necessary…
b5a98ae 
… in case the requested ddmpoint was not allowed and thus overwritten - trace report was not updated for this; also, the localsite was only set to the RSE value and not the env variable RUCIO_LOCAL_SITE_ID)
@PalNilsson
Temporarily removed the traces_copy_out from the rucio download clien…
0386148 
…t calls since the rucio client was rolled back to the previous version
@PalNilsson
Updated comment
3508141
@PalNilsson
Clarified logic in add_athena_proc_number()
a70cf96
@PalNilsson
Merge pull request #217 from PalNilsson/next
e9d7bb4 
2.2.0 (25)
@PalNilsson PalNilsson merged commit 092335b into master Oct 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@PalNilsson @anisyonk @davidgcameron