Example PBS Script #1260
Comments
|
Hi, |
|
I believe PBS_ARRAYID is indexed from zero, although this will not matter - we ensure to create exactly one scheduler either way. |
|
@apatlpo you can install dask using pip or conda. If you have a local python installation (such as would be created with anaconda) then you should be able to follow the instructions here without help from your system administrator: http://distributed.readthedocs.io/en/latest/install.html |
|
One obvious possibility is to have the (conda) python distribution in a directory accessible to all machines. Also, code in knit exists to build an environment to a description or wrap an existing one and place it in a particular location. (pip install against a system python will work too, using the |
|
I believe the conda install went successfully. After a very quick playing around before dinner time, it's not working yet but the output is: If you have any suggestions I'll give it a try, thanks |
|
it looks like it's treating each of the individual lines in the dask-worker command as a separate commend. Looking at my text from above it looks like I forgot a trailing slash after |
|
The code is not crashing after few minutes with the trailing slash but there is not output. Note also that in the preceding error code, the variables PBS_ARRAYID, PBS_NUM_PPN were empty. |
|
Can you also try printing $PBS_ARRAY_INDEX ? import os
from dask.distributed import Client
Client(scheduler_file=os.path.join(os.path.expanduser('~'), 'scheduler.json'))(note that you echo |
|
Hi everybody, and thank you Matthew for this script. I was just planning to make such a tool script for our HPC cluster at CNES. I will start from yours and try to make it work on our supercomputer. I hope I will be able to look at it this week. But here is a few things I can tell you after a first look:
Looking at @apatlpo solution, here is what I can say:
Okay so on a second thought, the idea of using job arrays seemed promising, but I believe the flaw I underline makes it not usable. In order to be able to launch several process, so several workers, in only one PBS job, we will need to use something like pbsdsh command, which allows to launch a given command on every resource asked to PBS. I have done this for launching Spark clusters using PBS, and this is probably well suited also for Dask. I will try to improve your proposal with this solution, and share the result here. |
|
@guillaumeeb : I thought for an array, you can specify "I need X tasks, of which Y must run concurrently", although I don't know the syntax for this. |
|
@martindurant: You may be right, I will check that tomorrow and tell you, This would be a really nice and clean solution. |
|
Another thought, what about the DRMAA proposed contributions on the Setup Network page, like https://github.com/dask/dask-drmaa? It seems a very elegant solution to be able to spawn a Dask cluster on PBS directly from a Python API. As anybody played with this? |
|
In my experience users ask far more often for PBS scripts than for help with DRMAA. I'm inclined to meet them with what they already know and use. That being said, adding DRMAA to the documentation seems like a fine idea to me. |
|
I am also far more used to PBS scripts :). I don't even know if DRMAA tools can work with our PBS Scheduler version. Another thing I need to check. Starting with the script sounds much more simple to me, and still a very good and usable option. |
|
As anticipated by Guillaume, the script does seem to work, here is the relevant part of the output log: Note that I have modified the number script and am now using upon Guillaume suggestion: Let me know if you want me to try something else. |
|
So I have discussed with a colleague, and it seems there is no such thing as proposed by @martindurant : "I need X tasks, of which Y must run concurrently". The job array philosophy is: "I need to run X tasks, just schedule them as best as you can". But my coworker was very interested by the idea of having a Dask cluster that fills the Gaps available in the PBS resources. So for him the option of a Dask cluster in an array is a really promising one: you ask for 100 resources, but your cluster can begin with only the 10 that are currently available, and grows as new resources become available. So I believe we must keep the two options, the job array when there is no minimal size required, and we want to start as soon as possible, and the unique job if we know we need only a fixed amount of resources and we prefer to wait for them. @apatlpo the PBS directive for a job array is: This should lead to something like this: |
|
Hey, seems to work for me: There is a problem with the number of cores though, I should have 8x2=16 cores, but I get 192=8x24, 24 cores being the total available cores of each of my node. Moreover, how can I had a memory limitation on my workers? |
|
The code seems to go further with the code you suggested but I sill get errors: I get in the output log files for the first process: and for other processes: @guillaumeeb do you have any suggestions for what I could try? |
|
@apatlpo You seem to have a networking problem, that is weird. Apparently, dask only find the loopback address (127.0.0.1) to bind on. So every worker, and your client, tries to connect to its localhost, which is probably not the host where the scheduler is started. |
Yes, see the help text. I recommend using |
|
OK, learning about the difference between job arrays that can fill in gaps and larger jobs is useful. While I agree that there are many useful situations for job arrays I'm inclined to set our default published PBS script to the simpler "wait until you have N machines please". I think this means that we should change to |
|
Cores limitation should be handled by the So what is the right solution here for |
|
Printing out the dask-worker options as well for reference: |
|
(note that this post was updated, the reason being that I was not waiting long enough before running the Client command in the python shell) Ok, I was unable to run with 'PBS -l nodes=...' so I stuck with the array option for now the PBS script looks now like: and I got: and in the log for the first process: and for one member of the array: |
|
It looks like the What does |
|
I had to update my preceding post, sorry about that. |
|
Yeah, it's not seeing anything beneath |
|
@apatlpo Be carefull, You are submitting 28 jobs, each using 14x2=28 cores, so at the end you will use 784 cores! Apparently, only 4 jobs were launched at the time you connected with your client: the scheduler and 3 workers. The good options would be: So you have a Dask cluster, but as @mrocklin said, the |
The |
|
Oh, I didn't notice this error. But it looks like the scheduler and workers are now binding on a correct IP and are able to talk to each others. |
|
See #1367 for an mpi4py based solution |
|
@mrocklin So do you want the |
|
I still think that PBS/SLURM examples would be useful. Having options can be very helpful. Deployment often has unexpected constraints. |
|
I completely agree. |
|
Hi everyone, I think I will get back at this soon because some people are beginning to show interest on dask distributed at CNES. But @subhasisb perhaps you could please clarify some points first:
What is your thought about that?
|
|
There was some significant progress on deploying Dask on HPC systems over at pangeo-data/pangeo#2 . |
|
+1 for an LSF example template script. I'd be happy to help with this. |
|
Any efforts would be welcome. If you can put something together @lzamparo I'd be happy to offer thoughts. I don't have an LSF system handy. |
|
@mrocklin Made an attempt setting up a scheduler & some workers using LSF's job dependency syntax (see my gist). However, I'm not sure that I'm connecting the workers to the scheduler correctly. While Any tips on debugging / error logging in distributed? Update: closing the interactive session revealed some hopefully useful traceback info, which I appended to the gist's session output. |
|
scheduler.json is written by the scheduler when it first starts up and has no workers. I would not expect the workers field to be populated even if things ran successfully. A simple way to determine if workers have connected would be to check the logs for either side. You might want to check that the network address published by the scheduler in the scheduler.json file is visible to the workers and client. If not then you might want to select a different network interface with These are the only things that come to mind at the moment. |
The scheduler's log suggests that the workers can see it and try to register. The filesystem is shared among all compute nodes, no worries there. |
This example worker log suggests two problems:
|
|
Yes, see It did connect though, which shows that they found each other. Is your client on a machine that is able to see the scheduler? Perhaps everything connected except for that? |
|
@mrocklin yes, all of the client machine, scheduler, and worker nodes are on the same network, so visibility should not be an issue. |
|
Also, I looked at the docs for |
|
Hi everybody, Found time to work again on this script this morning. Thanks to some examples found in https://github.com/opendatacube github project and the work of a coworker ont it. I used a conda environment like what is done by @mrocklin in pangeo-data/pangeo#2. Here is what I've got: This seems to work from the client perspective: I tried to use different network interfaces because I have problem when loading Dask web UI: this takes a lot of time and most of the time it never works. Don't know why yet, if somebody has a hint... |
|
Hi everyone, A quick word to share with you work that has been done on Dask (and also Spark) on PBS at @CNES. PBS scripts to launch Dask are available in this repo: https://github.com/guillaumeeb/big-data-frameworks-on-pbs. Cheers. |
|
@guillaumeeb, fantastic! Thanks for sharing your progress. Would you and the CNES folks be interested in collaborating on a project benchmarking the performance of dask vs spark on common analysis workflows? |
|
Hi @rabernat @guillaumeeb, at Ifremer we would be interested in looking at such benchmarking ! |
|
+1 on Dask/Spark benchmarking. This is a frequently asked question. |
|
@rabernat (and @gmaze @mrocklin), don't know if you got my email? We would be glad at CNES to collaborate on such a project. We should just define together how we could contribute, on which use cases, and on which place to discuss about it (e.g. Pangeo? Another Dask or Distributed issue?) Any idea yet? |
|
Hi Guillaume! Yes, we got it. So sorry that we have been slow to reply. I
have been hopping around with lots of travel and conferences, but your
email is definitely on my mind. I will reply to that email and share my
thoughts. Then we can follow up with some specific github issues where
necessary.
…On Sat, Jan 20, 2018 at 8:14 AM, Guillaume EB ***@***.***> wrote:
@rabernat <https://github.com/rabernat> (and @gmaze
<https://github.com/gmaze> @mrocklin <https://github.com/mrocklin>),
don't know if you got my email? We would be glad at CNES to collaborate on
such a project. We should just define together how we could contribute, on
which use cases, and on which place to discuss about it (e.g. Pangeo?
Another Dask or Distributed issue?)
Any idea yet?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1260 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABJFJiqJvV1ccQvsHO7hOwSIWqxG29xyks5tMebFgaJpZM4OZdk6>
.
|
|
@lzamparo Did you figure out your issue? I was experiencing the sample problem. I noticed that at the client I should put the scheduler address directly instead of the scheduler file and it worked. |
|
@alvarouc Thanks for following up. I never did fix this for a bunch of reasons. Good to hear you found a workaround. I presume you still interrogate the |
|
This has been resolved by the dask-jobqueue project. |
People using Dask on traditional job schedulers often depend on PBS scripts. It would be useful to include a plain example in the documentation that users to download, modify, and run.
What we do now
Currently we point users to the setup network docs, and in particular the section about using job schedulers with a shared network file system. The instructions there suggest that users submit two jobs, one for the scheduler and one for the workers:
However this is flawed because the scheduler or workers may start and run independently from each other. It would be better to place them into a single job, where one special node is told to be the
dask-schedulerprocess and all other nodes are told to be thedask-workerprocesses. Additionally we would like to offer some guidance on tuning the number of CPUs and pointing workers to use local high-speed scratch disk if available.PBS script options
Many docs on PBS scripts exist online, but each seems to be made by an IT group at a separate super-computer. It is difficult to tease out what is general to all systems and what is specific to a single supercomputer or job scheduler. After reading from a number of pages I've cobbled together the following example.
https://wiki.hpcc.msu.edu/display/hpccdocs/Advanced+Scripting+Using+PBS+Environment+Variables
http://www.pbsworks.com/documentation/support/PBSProUserGuide10.4.pdf
Questions
ncpusandppn?-t 1-8andnodes=8?Does this actually work? I suspect not. I don't have a convenient testing system for this and would appreciate coverage by a few different groups.
The text was updated successfully, but these errors were encountered: