Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demo day 2022-10-27 #282

Closed
mrocklin opened this issue Oct 17, 2022 · 30 comments
Closed

Demo day 2022-10-27 #282

mrocklin opened this issue Oct 17, 2022 · 30 comments

Comments

@mrocklin
Copy link
Member

mrocklin commented Oct 17, 2022

We'll be organizing a demo day (see #280) Thursday next week at 8am US Pacific, 5pm Central European Time.

Context

I'd like to solicit 5-10 minute demos that show off ongoing or lesser known work. I hope to have 3-5 of these during the meeting. Meetings will be recorded and advertised on social. Hopefully this helps to educate folks on some of the great work people are up to.

If you're interested please respond to this issue with a brief (a couple sentences) description. If you have colleagues who you think should be interested please let them know.

Agenda

  1. "Matplotlib is popular": @mrocklin
  2. "Task Queuing / Constant Memory Scheduling" : @fjetter or @gjoseph92
  3. "Kubernetes stuff" : @jacobtomlinson
  4. "Apache Beam on Dask": @alxmrs
  5. "Prometheus" : @ntabris or @dchudz
  6. ... (room for one more!)

If there is time at the end I'll show some simple work with the arXiv dataset, which was fun

Meeting Invite

Matthew Rocklin is inviting you to a scheduled Zoom meeting.

Topic: Dask Demo Day
Time: Oct 27, 2022 10:00 AM Central Time (US and Canada)

Join Zoom Meeting
https://us06web.zoom.us/j/85409976995?pwd=L1ZrbG00OVkzZ2RvV0R3bTdFQlRDdz09

Meeting ID: 854 0997 6995
Passcode: 130413
One tap mobile
+13462487799,,85409976995# US (Houston)
+16694449171,,85409976995# US

Dial by your location
+1 346 248 7799 US (Houston)
+1 669 444 9171 US
+1 669 900 6833 US (San Jose)
+1 719 359 4580 US
+1 253 215 8782 US (Tacoma)
+1 564 217 2000 US
+1 646 931 3860 US
+1 929 436 2866 US (New York)
+1 301 715 8592 US (Washington DC)
+1 309 205 3325 US
+1 312 626 6799 US (Chicago)
+1 360 209 5623 US
+1 386 347 5053 US
Meeting ID: 854 0997 6995
Find your local number: https://us06web.zoom.us/u/kcshPwvXdS

@dchudz
Copy link

dchudz commented Oct 17, 2022

Folks from Coiled could demo some fairly basic (boring?) Coiled functionality, e.g.:

  • a setup CLI we have that gets a user's AWS or GCP account prepared for Coiled to work properly
  • a "cluster details" page showing some of the infrastructure, and the history of cluster-related events (launching instances, etc).

For something that's not yet in the Coiled product, maybe we could demo some stuff we've been playing around with collecting metrics from the Prometheus endpoint and plotting them in Graphana.

@jrbourbeau
Copy link
Member

@dchudz I suspect there will be folks who haven't used Coiled before. Something like a "Getting started with Coiled in 5 minutes" demo could be informative for folks to see. We could walk through (1) using the coiled CLI to connect to a cloud provider account, (2) spin up a Coiled cluster (using local environment syncing), (3) walk through several of the niceties one gets with Coiled (e.g. logs, package details, analytics). Thoughts?

@jacobtomlinson, you've usually always got interesting things to show, is there something you're working on that you'd like to demo?

@jrbourbeau
Copy link
Member

Whoops, seeing @jacobtomlinson proposed demo-ing the Dask Kubernetes Operator over in #280 (comment)

@fjetter
Copy link
Member

fjetter commented Oct 17, 2022

Task queuing / Constant memory scheduling

See also dask/distributed#7128

A new configuration parameter / feature toggle was introduced that significantly impacts scheduling behavior and is typically expected to deliver constant memory scheduling and therefore reduce spilling, increase stability and in some cases even drastically improves runtime.

During the demo we can briefly discuss tradeoffs between show how to enable this and discuss a couple of example workloads where this matters

cc @gjoseph92

@dchudz
Copy link

dchudz commented Oct 17, 2022

"Getting started with Coiled in 5 minutes" demo ... Thoughts?

Sounds great. I'd be happy to either do it, or invite someone else from our Platform team to give the demo.

@jacobtomlinson
Copy link
Member

I'm in no rush if there are other topics that folks also want to demo 🙂

@mrocklin
Copy link
Member Author

Folks should list things. If there is a surplus then I'll arbitrarily select a set and push others to next month.

@jacobtomlinson
Copy link
Member

The Dask Kubernetes Operator would be a nice thing to show.

@mrocklin
Copy link
Member Author

We're still low on user-focused demos. Does anyone know anyone who would be good here?

@alxmrs
Copy link

alxmrs commented Oct 19, 2022

I'd be happy to demo apache/beam#22421 when it's ready (fairly soon).

@TomAugspurger
Copy link
Member

I'd be happy to do something around "Scalable geospatial data analysis with Dask". That'd be user-focused (albeit on a specific user community).

@gjoseph92
Copy link

@TomAugspurger "scalable geospatial analysis with dask" might play nicely with constant-memory scheduling dask/distributed#7128 if you can update the PC images to the latest version.

@mrocklin
Copy link
Member Author

@alxmrs are you good to go for this Thursday? WIP work is still quite welcome.

@mrocklin
Copy link
Member Author

I've placed a tentative agenda and Zoom invite in the opening comment. I'll tweet this out later today.

@TomAugspurger I think that you're still the guardian of the calendar. Thoughts on how to make this show up more easily in folks' calendars?

@alxmrs
Copy link

alxmrs commented Oct 24, 2022

Yes! I should be good to go -- I'm happy to present WIP work.

@TomAugspurger
Copy link
Member

There should be an event for Thursday the 27th at 10:00 US/Central. I see it on the embedded calendar at https://docs.dask.org/en/latest/support.html. Did it show up for others who are already subscribed.

I have conflicts during that time so I won't be able to attend. I can update the agenda on the calendar as it solidifies though.

@gjoseph92
Copy link

I also won't be able to attend that time. Maybe someone else can talk about queuing?

@jrbourbeau
Copy link
Member

Did it show up for others who are already subscribed

I see the event 👍

@mrocklin
Copy link
Member Author

I also won't be able to attend that time. Maybe someone else can talk about queuing?

@fjetter ?

@mrocklin
Copy link
Member Author

This has been advertised on Twitter: https://twitter.com/dask_dev/status/1584935939213705216

@ljstrnadiii
Copy link

Exciting! I am hoping to quietly sit in on this.

@TomAugspurger
Copy link
Member

Just to note, since I'm listed as presenting, I unfortunately have another commitment that overlaps with the demo day time. I'm looking forward to the recording and will join in on the next one! I'll edit the original post to remove myself.

@mrocklin
Copy link
Member Author

Alas! Oh well.

See you next month?

@mmccarty
Copy link
Member

Sorry I missed it. Had a conflict. Is there a recording?

@gtmaskall
Copy link

I'm always up for some scalable geoscience, so will have to look out for your demo @TomAugspurger .

@mrocklin
Copy link
Member Author

Finished processing, but I should trim things off of the beginning and end. Hopefully we'll get the recording out tomorrow.

@mrocklin
Copy link
Member Author

@ntabris
Copy link

ntabris commented Oct 28, 2022

https://youtu.be/VlTgcLqb1DQ

YouTube tells me the video is private (so I can't see it).

@mrocklin
Copy link
Member Author

Sorry! It should be open now.

@mrocklin
Copy link
Member Author

Thanks all who participating for making this a success. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests