Skip to content

Meeting Minutes

Jump to bottom
Rohini Joshi edited this page Nov 28, 2024 · 11 revisions

Karabo Dev meetings

28/11

Present: Lukas (LG), Michel (MP), Rohini (RJ)

Minutes

Roundtable updates

  • Lukas: some comments on Andreas' PR about TelescopeClass, but changes should make it into this month's release
  • Lukas (on behalf of Delberin): Discussion about some issues with wsclean and everybeam, resolution can be patch old versions or build new versions of both? Have gone for the second option which is hopefully backwards compatible. RASCIL has some issues, probably a few weeks to get through the last bits and to get to python 3.10
  • Michel: Updated PI24 run scripts have been merged, relevant docs done and demo WIP, should be wrapped in PI24
  • Lukas: Note Daint decommissioning EoY, esp if we need to do more dataset generation, where will it happen? Eiger or Alps?
    • Rohini: Maybe ingesting visibility data is an option, or using Karabo dataset generation as a way to validate Alps delivery
  • Exploring MSv4 data ingestion wrt SRCNet (it may not be as hierarchical as MSv2)

21/11

Present: Lukas (LG), Andreas (AW), Michel (MP), Rohini (RJ)

Minutes

Roundtable updates

  • Andreas: working on Telescope Class, discussions ongoing in PR.
  • Michel: Working on wrap-up from Karabo runs for SRCNet, small PR incoming. Workflow streamlining to generate data, ingest metadata, have it be discoverable. Discovery possible via SCS (simple cone search) or a TAP client + ADQL.
  • Lukas: PR review, supporting others. Delberin working on upgraded python support, assuming happy path should be in soon, perhaps not for next release.
  • Monthly release should include new AA layouts, Telescope class PR, maybe data ingestion scripts
  • Rohini: Not a lot of obvious Karabo work in SRCNet for PI25. Some data ingestion could be continued. We could work on adding noise to Karabo data, but need science user input.
    • Lukas: Adding MSv4 could be an option. But might be too soon, unsure how metadata will be handled.
    • Michel: Could do noise addition, if it can get prioritised. Or Karabo run + data ingestion to test the SRCNodes.
    • Rohini: Could be added to test campaign preparation ticket (Not too much to update on the paper atm)

07/11

Present: Lukas (LG), Simon (SF), Andreas (AW), Michel (MP), Delberin (DA), Rohini (RJ)

Minutes

Roundtable updates

  • Andreas: changes to the telescope class (due to discrepancy between OSKAR/RASCIL usage). For ex, baselines returned should now be the same irrespective of backend used. waiting on CI pipelines to clear. and review of course
  • Simon: First pass of paper complete, to be discussed with Rohit. Publication location not decided yet. Contributors welcome!
  • Lukas: CI is failing since yesterday due to some breaking change elsewhere from some dependency. Availability is low in the coming 2 weeks so need someone to look into this. (starting point can be going back to last successful run in github actions and the diff of dependency resolution since then)
  • Michel - data ingestion SRCNet side ongoing (related PR open, waiting for review)
  • Delberin - discussing with Simon, currently working on wsclean compatibility
  • Rohini - work in progress to define the SRCNet work plan, including for Karabo

31/10

Present: Lukas (LG), Simon (SF), Andreas (AW), Michel (MP), Rohini (RJ)

Agenda

  • Roundtable update
  • Followup discussion after OSKAR/RASCIL meeting

Minutes

Roundtable update

  1. MP finishing up Karabo run, run went well, just took a while. Visibility data (1.5 TB for this run) doubles in size after cleaning the image, due to wsclean implementation, needs further investigation. Okay for now, but a problem as datasets get larger. Not able to create full image cubes, issue created (https://github.com/i4Ds/Karabo-Pipeline/issues/624). SF: Dask could be an option to consider.
  2. Simon and Rohit continuing work on the paper.
  3. Delberin and Simon in touch about compilation issues.
  4. AW worked on telescope discrepancy issue wrt OSKAR and RASCIL. Working on this, still a shift of 0.1 degree present, sometimes with a telescope center shift. MP: source telescope configurations might be different so this could be a source of discrepancy.

Followup discussion after OSKAR/RASCIL meeting

Recap of the few important points

  • originally OSKAR thought for SKA-LOW sims, development progress is limited, no concrete long term roadmap but things are implemented/fixed based on need. Recommendation to not use vis file format, just use MS. (vis to dump to disk but that's about it)
  • RASCIL is python based (thus slow), started as a PoC, but then evolved accidentally to be used for sims. RASCIL-main retirement happening soon and migration of core functionality to SDP packages. no new simulation functionality planned.

Discussion

24/10

Present: Lukas (LG), Simon (SF), Andreas (AW)

Agenda

  • Roundtable update
  • Date for followup discussion after OSKAR/RASCIL meeting?

Minutes

  1. Delberin is still stuck with the compiling error related to the everybeam package.
  2. How should we handle user requests? Would be cool if we could respond to them. But we still suffer from a lack of human resources.
  3. How to go on after OSKAR/RASCIL meeting was not discussed. Does the outcome of the meeting affect future Karabo development? Maybe we should bring it up in next Dev meeting.
  4. Working on the Karabo paper ongoing. Simon finished another chapter.

Actions

  1. Simon - will have a look at the error and discuss ideas with Delberin.
  2. No immediate actions
  3. Schedule update for next Karabo Dev meeting. AW will ask MP for presentation.
  4. No immediate actions.

17/10

Present: Michel (MP), Lukas (LG), Pascal (PH), Andreas (AW), Delberin (DA), Rohini (RJ)

Minutes

  • Lukas - meeting with OSKAR/RASCIL devs
  • Andreas - Jennifer chat this week, HI science would require HPC infra so might be a good idea to deploy to Alps
  • Michel - Followup discussion after OSKAR/RASCIL meeting needed. AA layout integration complete (all SKA LOW and MID layouts), to be merged.
  • Delberin: python version upgrade nearly complete, 2 packages outstanding Async update
  • Simon and Rohit making progress on the paper 🎉

Actions

  • RJ: Followup meeting from OSKAR/RASCIL discussion for Karabo roadmap impact

10/10

Present: Michel, Lukas, Pascal, Simon

Minutes

  • Michel PR #620 merged
  • Michel suspects RASCIL interferometry doesn’t work anymore (#626). Weird issue, needs to be looked at.
  • Simon worked on paper
  • Should we automatically convert .vis to MS when required? Might take a lot of time & space.
  • We should show user feedback for long running tasks.

Actions

SF: Create issue for automatic conversions of visibility formats (#627) SF: User Feedback for long running tasks. (#)

03/10

Meeting was cancelled due to a lot of absences and no need for a meeting among the people who were available.

26/09 ‘All-hands’ dev meeting

Present: Michel, Lukas, Simon, Vincenzo, Andreas, Delberin, Pascal, Rohini

Agenda/minutes

Time Item Discussion Actions
14.00 Roundtable updates    
14.15 MilestonesClose out milestones? (other than icebox)Make milestone for v1.0 - What should go into this? https://github.com/i4Ds/Karabo-Pipeline/milestones We should close out 0.23 and 0.24 milestones. Check issues dont get deleted when they are removed for a milestone.V1.0MP: how strict would we be with breaking changes with v1.0 and beyondSF: Want to send a message that this software is usableLG: should define the public api, even if breaking changes come in v1.0+ with an API review for public APIsOther things for v1.0 AW: should running on CSCS infra (alps) be linked to this?RJ: link to python version?Probably no to bothSF: focusing more on PR angle when thinking about which issues should be resolved for v1.0SF: would be good to get a test science user to run Karabo, go over documentation and give feedbackJan 2025 might be okay (SF: probably not later than this) RJ: accept pending invite!API review (public/private) (issue 609)Find test science user to run Karabo, go over documentation and give feedbackRJ: Create v1.0 milestoneAll: add issues to this milestone to be reviewed in future dev meeting
14.30 LabelsWhat do we mean by prio-low/med/highDoes it imply things getting into the next release?Can we use them to identify good-first-issues? SF: Originally prio-high were meant to be the to-do list, prio-med is next steps, and prio low is similar to iceboxLG: need some ownership/responsiblity for labels and relabeling as neededLG: add a work-in-progress label to indicate work that has been picked upMP: rename prios to must/should/could haveLG: some way to group larger topics spanning multiple issues/releases(milestones could work but not ideal) Remove proj-pinochio and proj-sdc3 labelsrename prios to must/should/could have
14.45 Dev meeting formatSuggestions here welcome!Roundtable updates, identify items for longer discussion as a meet after Going around based on issues wasn’t the best approach in the pastwork to make Karabo more accessible, visible, etc that doesn’t entail PRs to the repoMinutes on the wiki, note actions? Separate user feedback/use case meetings as we need them? MP: Look at new issues made in the last week and look at labels  

User/Stakeholder meetings

Sync with OSKAR, RASCIL teams

Oct 15th

People

Celeste Lu, Danielle Fenech, Mark Ashdown, Fred Dulwich, Michel Plüss, Pascal Herzog, Lukas Gehrig, Rohini Joshi, Jeremy Coles

Agenda, Minutes

  • Introductions
  • Intro to Karabo and what we have done to expand on the backends, how/why we wrap it to provide a package

Using slides: https://confluence.skatelescope.org/pages/viewpage.action?pageId=280401927&preview=/280401927/284182692/Karabo.pdf

  • OSKAR, RASCIL roadmaps, plans
    • Dependency management plans ?
    • Tell us a bit more about RASCIL retirement but core functionality retained in different repository structure, how/where and the plans for this?

Fred: Current users? Rohini/Lukas: Some SDC use cases, and users at ETH, and EPFL mainly SRCNet at the moment

Fred: note there are existing datasets out there from existing instruments. Also, OSKAR originally built for SKA-Low, simulating dish arrays with gaussian beams wasn't the original intention :)

Mark: Development work process is distributed? Fred: Collaborative effort, yes

Celeste: RASCIL python models mostly moved out, RASCIL misuse also prevalent. RASCIL-main is what's left after moving the imaging tooling, data models, python models etc. This will be archived ~March, Gitlab for understanding the latest status wrt RASCIL

Don't touch RASCIL, also don't touch OSKAR 😁

Semantic versioning? Discussion ongoing, push to move to version 1 and be more intentional about versioning

Celeste: Other pipelines being looked at in DP - processing CoP Mark: Analysis pipelines not simulation pipelines https://confluence.skatelescope.org/display/SE/Processing+Community+of+Practice

  • What is the current usage of OSKAR/RASCIL/Casa tools in the Construction ARTs, or DP ART?

  • Appetite/need for simulated data generation pipeline for SDP? Other feature work for the upcoming PI? Communication and discussion welcome here, SDP side more useful to have smaller (than AA*) but precise simulations

  • Q&A

    • MPI support for multi-node clusters? Dask hasn't worked quite well for RASCIL. OSKAR originally had MPI but this was messy and no planned support for this going forward.
    • Difference between the visibility data formats? Don't touch .vis files 😁 they have a very specific function and intended use, ie dump to disk and load into OSKAR or python when you don't want to use MS interaction tools like casacore. For interoperability, use MS files.

Note some useful links:

Clone this wiki locally