Canfar run v1.4 #677

martinkilbinger · 2024-03-06T14:29:00Z

Summary

Changes to get shapepipe run on the canfar science portal. Used to create the SP v1.4 catalogue.

The PRs
martinkilbinger#3 ... martinkilbinger#14
were merged before to https://github.com/martinkilbinger/shapepipe-1.

Reviewer Checklist

Reviewers should tick the following boxes before approving and merging the PR.

…added init_canfar.sh script

…gument

…pipe-1 into science_portal_run

Conflicts: Dockerfile environment.yml scripts/sh/init_canfar.sh

… into docker_image

martinkilbinger · 2024-03-06T15:10:26Z

docs/source/post_processing.md


-1. Retrieve `ShapePipe` result files


Obsolete VM system on canfar.

martinkilbinger · 2024-03-06T15:11:10Z

docs/source/vos_retrieve.md

@@ -0,0 +1,51 @@
+## Retrieve files from VOspace


Isolated vos HOWTO into separate doc file.

martinkilbinger · 2024-03-06T15:12:13Z

environment.yml

@@ -4,30 +4,30 @@ channels:
 dependencies:


Due to library inconsistencies I removed most of the set versions. We can try to find an updated set that works on our different platforms.

martinkilbinger · 2024-03-06T15:14:30Z

install_shapepipe

@@ -25,20 +25,6 @@ libpng_ver="1.6.37"
 mpi4py_ver="3.1.3"
 openblas_ver="0.3.18"

-# SExtractor Package


astromatic packages now installed via conda.

martinkilbinger · 2024-03-06T15:19:47Z

install_shapepipe

@@ -193,7 +117,7 @@ check_conda() {
  CONDA_VERSION_MAJOR=$(cut -d'.' -f1 <<<$CONDA_VERSION)
  CONDA_VERSION_MINOR=$(cut -d'.' -f2 <<<$CONDA_VERSION)
  CONDA_VERSION_PATCH=$(cut -d'.' -f3 <<<$CONDA_VERSION)
-  CONDA_SH=/etc/profile.d/conda.sh
+  CONDA_SH=/opt/conda/etc/profile.d/conda.sh


The correct path on canfar; can we do this in a general way?

martinkilbinger · 2024-03-06T15:20:16Z

nsswitch.conf

@@ -0,0 +1,62 @@
+#


From Sebastien Fabbro.

… develop

martinkilbinger · 2024-03-06T17:53:07Z

shapepipe/modules/psfex_interp_runner.py

@@ -66,7 +66,16 @@ def psfex_interp_runner(
            module_config_sec,
            'ME_DOT_PSF_DIR',
        )
-        dot_psf_dir = get_last_dir(run_dirs['run_log'], module)
+        module_name = module.split(":")[-1]


Added flexibility to use "last:dir" and "all:dir" as search strings.

martinkilbinger · 2024-03-06T17:54:51Z

shapepipe/modules/setools_package/setools.py

-            self._data = cat_file.get_data()
+            try:
+                self._data = cat_file.get_data()
+            except:


catch error, e.g. if file is corrupt or has zero size.

martinkilbinger · 2024-03-06T17:55:32Z

shapepipe/modules/vignetmaker_package/vignetmaker.py

-                    image_dir + '/' + image_pattern + '-'
-                    + exp_name + '-' + str(ccd) + '.fits'
-                )
+                # Look for input image


Bug fix: Need to go through list of input directories instead of just one.

martinkilbinger · 2024-03-06T17:56:19Z

shapepipe/modules/vignetmaker_runner.py

-                last_dir = get_last_dir(run_dirs['run_log'], module)
-                image_dir.append(last_dir)
+                module_name = module.split(":")[-1]
+                if "last" in module:


added flexibility to input run(s)

martinkilbinger · 2024-03-06T17:56:56Z

shapepipe/pipeline/args.py

@@ -128,5 +128,10 @@ def create_arg_parser():
        help='configuration file name',
    )

+    optional.add_argument(
+        '-e',


new option, to run shapepipe on exclusively one single image ID.

martinkilbinger · 2024-03-06T17:58:25Z

shapepipe/pipeline/file_handler.py

+                id_to_test = f"-{self._exclusive.replace('.', '-')}"
+                if number == id_to_test:
+                    if self._verbose:
+                        print(f"-- Using exclusive number {self._exclusive} ({id_to_test})")


Test print, should probably be removed.

martinkilbinger · 2024-03-06T17:59:42Z

shapepipe/pipeline/file_handler.py

@@ -1120,6 +1135,9 @@ def _format_process_list(
            ])
            process_list.append(process_items)

+        if len(process_list) == 0:


Catching an error that sometimes occurs, e.g. when something is wrong with input files or directories.
If not, caused unexpected behaviour later on. Check and error message could still be improved.

…cally

Cleanup canfar

sfarrens

Hey @martinkilbinger I have some questions about the Dockerfile (see opened threads). In order to move things along a bit faster, it would probably be easier to implement these changes in #600, but I would like to understand everything first.

sfarrens · 2024-03-27T10:45:22Z

Dockerfile

+COPY ./environment.yml ./
+COPY install_shapepipe README.rst setup.py setup.cfg ./


@martinkilbinger is there any particular reason you are copying things to / rather than e.g. /home?

sfarrens · 2024-03-27T10:45:56Z

Dockerfile

+COPY shapepipe ./shapepipe
+COPY scripts ./scripts


Again, is there any reason you only copy these directories and now the whole repo?

sfarrens · 2024-03-27T10:46:59Z

Dockerfile

+COPY shapepipe ./shapepipe
+COPY scripts ./scripts
+
+RUN source activate shapepipe


The following line will generate an Image that will launch containers with ShapePipe pre-activated.

RUN echo "conda activate shapepipe" >> ~/.bashrc

sfarrens · 2024-03-27T10:47:33Z

Dockerfile

+
+COPY ./environment.yml ./
+COPY install_shapepipe README.rst setup.py setup.cfg ./
+RUN touch ./README.md


What is this for?

sfarrens · 2024-03-27T10:48:19Z

Dockerfile

+COPY install_shapepipe README.rst setup.py setup.cfg ./
+RUN touch ./README.md
+
+RUN conda update -n base -c defaults conda -c defaults


Was this needed to get things working?

sfarrens · 2024-03-27T10:49:06Z

Dockerfile

+ARG CC=gcc-9
+ARG CXX=g++-9
+
+# gcc < 10 is required to compile ww


I guess you checked this?

sfarrens and others added 30 commits November 16, 2022 09:36

added Dockerfile and GH action

6934d27

added make dependency

82bc72a

notebook

5b44743

docker testing

29c609e

Merge remote-tracking branch 'upstream/develop' into docker_image

da18970

Dockerfile tests

5770367

Added cent-os to CI tests

3166e3c

Added pyproject

373722e

canfar pyproject update

b025b6a

cfis vos configs: missing keywords, DR5 added

ba135f2

Testing dockerfile with only conda env installed

07cafb4

Fixing numpy and astropy versions

92a999c

installation on canfar working: removed most versions from yml file; …

2b6600d

…added init_canfar.sh script

Dockerfile for image 0:7, 0:8

b6e5c97

Installation of sextractor and psfex with conda

bc66565

script to call curl with canfar contained updated, adding NCORE as ar…

0b3aac6

…gument

added init scripts for canfar; job_sp added n_smp for further jobs

69317cb

script to call curl for canfar container: added resources

d35bd69

testing input numbers

d93a60e

removed debug msg

fb83a27

numpy error fixed with version upgrade

e33972b

Merge branch 'science_portal_run' of github.com:martinkilbinger/shape…

edbec76

…pipe-1 into science_portal_run

Getting ready for exclusive-one-tile processing

9aa16be

Added command line option to specify exclusive ID for processing

e5d0335

Merge remote-tracking branch 'origin/exclusive' into science_portal_run

49d8880

curl script exclusive ID

d02347e

Dockerfile conda -> source activate

12e6b63

Merge remote-tracking branch 'origin/docker_image' into docker_image

7d20699

Conflicts: Dockerfile environment.yml scripts/sh/init_canfar.sh

Merge branch 'docker_image' of github.com:martinkilbinger/shapepipe-1…

1d3df1a

… into docker_image

comment added

b05f2a5

martinkilbinger commented Mar 6, 2024

View reviewed changes

docs/source/post_processing.md

1. Retrieve `ShapePipe` result files

Copy link

Contributor Author

martinkilbinger Mar 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obsolete VM system on canfar.

martinkilbinger commented Mar 6, 2024

View reviewed changes

nsswitch.conf

@@ -0,0 +1,62 @@

#

Copy link

Contributor Author

martinkilbinger Mar 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Sebastien Fabbro.

martinkilbinger added 4 commits March 6, 2024 16:21

Merge branch 'develop' of github.com:martinkilbinger/shapepipe-1 into…

339f13f

… develop

repaired plot spectro notebook

bddc491

added doc string

4dffd47

Merge branch 'develop' of github.com:martinkilbinger/shapepipe-1 into…

41cfd97

… develop

martinkilbinger commented Mar 6, 2024

View reviewed changes

python example runner: added back mccd and pysap

5675c06

martinkilbinger commented Mar 6, 2024

View reviewed changes

corrected typo in doc string

14569b2

martinkilbinger commented Mar 6, 2024

View reviewed changes

martinkilbinger requested a review from sfarrens March 6, 2024 18:02

martinkilbinger marked this pull request as draft March 6, 2024 18:03

martinkilbinger added 8 commits March 13, 2024 07:01

updated canfar pipeline doc

b406438

fixed combined mask input dir

fe66831

added parameters to final cat

282d76b

summary

9ac6bb4

Removed rarely used ngmix njob/ngal options; splits computed automati…

25ef4e2

…cally

canfar init + run job: removed prev Ms and Mc dirs for job 256

75fb975

merge sep cats: error if ngmix #HDUs < 6

376bbf1

Merge pull request #15 from martinkilbinger/cleanup_canfar

06564a1

Cleanup canfar

sfarrens reviewed Mar 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Canfar run v1.4 #677

Canfar run v1.4 #677

martinkilbinger commented Mar 6, 2024 •

edited

Loading

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

martinkilbinger Mar 6, 2024

sfarrens left a comment

sfarrens Mar 27, 2024

sfarrens Mar 27, 2024

sfarrens Mar 27, 2024

sfarrens Mar 27, 2024

sfarrens Mar 27, 2024

sfarrens Mar 27, 2024

		COPY ./environment.yml ./
		COPY install_shapepipe README.rst setup.py setup.cfg ./

Canfar run v1.4 #677

Are you sure you want to change the base?

Canfar run v1.4 #677

Conversation

martinkilbinger commented Mar 6, 2024 • edited Loading

Summary

Reviewer Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfarrens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinkilbinger commented Mar 6, 2024 •

edited

Loading