Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ICESat-2 data read-in functionality #222

Merged
merged 79 commits into from
Nov 23, 2021
Merged

Add ICESat-2 data read-in functionality #222

merged 79 commits into from
Nov 23, 2021

Commits on Sep 30, 2021

  1. add read class module and basic outline

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    ff7f807 View commit details
    Browse the repository at this point in the history

  2. start input check

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    38a33d2 View commit details
    Browse the repository at this point in the history

  3. add more infrastructure to read class and needed intake dependencies

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    a478825 View commit details
    Browse the repository at this point in the history

  4. write file/dir and path matching functions. Still in progress for ini… 

    …t pattern_as_path input needs
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    7acde07 View commit details
    Browse the repository at this point in the history

  5. outline basic tests file for read module validation functions

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    78ef804 View commit details
    Browse the repository at this point in the history

  6. add function to create glob path from pattern

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    cf42fe9 View commit details
    Browse the repository at this point in the history

  7. iterate over directories and confirm files with specified pattern exi… 

    …st in source
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    581aee3 View commit details
    Browse the repository at this point in the history

  8. add basic build catalog function to read data object

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    5edcce1 View commit details
    Browse the repository at this point in the history

  9. start turning notebook into example; add docstring to read.build_catalog

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    df4fef3 View commit details
    Browse the repository at this point in the history

  10. begin debugging save-load and var_path_params issues

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    4af16dc View commit details
    Browse the repository at this point in the history

  11. add rough example notebook for data read in

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    f4f95a2 View commit details
    Browse the repository at this point in the history

  12. debug var_path_params entry to build_catalog

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    667f817 View commit details
    Browse the repository at this point in the history

  13. separate intake catalog functions to another module

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    552fd3f View commit details
    Browse the repository at this point in the history

  14. add ability to get list of variables from a file and create the varia… 

    …bles object on the read object
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    fcae6d6 View commit details
    Browse the repository at this point in the history

  15. commit ongoing changes so can merge dev updates

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    eee1f2e View commit details
    Browse the repository at this point in the history

  16. update variable names for clarity and in line with common usage (#211) 

    * change 'dataset' to 'product' in core and test code
* update 'source' input parameter to 'path' in variables module (part of a not-yet-implemented feature).
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    73140fc View commit details
    Browse the repository at this point in the history

  17. write load function; clean up is2cat module and notebook

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    1b1e978 View commit details
    Browse the repository at this point in the history

  18. add dict to list parsing and flag to provide non-tiered var paths

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    d049bc8 View commit details
    Browse the repository at this point in the history

  19. fix LocalCatEntry building with args

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    fec72b1 View commit details
    Browse the repository at this point in the history

  20. distinguish between data paths and metadata paths and read in groups … 

    …that have data paths
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    8f31ce3 View commit details
    Browse the repository at this point in the history

  21. draft gt to beam function

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    0b71fa4 View commit details
    Browse the repository at this point in the history

  22. fix h5py read in spec to remove warning

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    c69f2dc View commit details
    Browse the repository at this point in the history

  23. add ability to provide some engine args

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    2c0f055 View commit details
    Browse the repository at this point in the history

  24. change var name to be clearer

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    e319ee3 View commit details
    Browse the repository at this point in the history

  25. add items to required list

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    4892236 View commit details
    Browse the repository at this point in the history

  26. basic infrastructure for reading in a single dataset. Work ongoing

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    e84fd74 View commit details
    Browse the repository at this point in the history

  27. debug new function

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    90bca3f View commit details
    Browse the repository at this point in the history

  28. preliminary single dataset read-in working

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    ed654dd View commit details
    Browse the repository at this point in the history

  29. ongoing dev work and example writing

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    59e71d7 View commit details
    Browse the repository at this point in the history

  30. add some more default variables for read in

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    0ea6463 View commit details
    Browse the repository at this point in the history

  31. basic reader functionality complete and working

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    717d9b1 View commit details
    Browse the repository at this point in the history

  32. clean up and add quickstart to example notebook

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    867fdb2 View commit details
    Browse the repository at this point in the history

  33. clear up beam/spot nomenclature in variable naming

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    5b469cc View commit details
    Browse the repository at this point in the history

  34. refactor load into multiple functions

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    5eba9fb View commit details
    Browse the repository at this point in the history

  35. add variables example

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    0af55b7 View commit details
    Browse the repository at this point in the history

  36. complete variables example notebook draft

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    aa0a6b4 View commit details
    Browse the repository at this point in the history

  37. trim variables module details out of subsetting example

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    4fdd4a6 View commit details
    Browse the repository at this point in the history

  38. add more explanatory text to example notebook

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    4c704f3 View commit details
    Browse the repository at this point in the history

  39. update examples from 2020 Hackweek tutorials

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    b4a12b3 View commit details
    Browse the repository at this point in the history

  40. add note about using a hosted version of the data products

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    7b90290 View commit details
    Browse the repository at this point in the history

  41. auto-detect input data type and remove files without specified produc… 

    …t in name
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    dec9527 View commit details
    Browse the repository at this point in the history

  42. add some tests for non-class functions in Read module

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    5abb033 View commit details
    Browse the repository at this point in the history

  43. add handling for failed xarray merge and return list of datasets instead

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    fc26d76 View commit details
    Browse the repository at this point in the history

  44. add automated check that user specified product matches that in file

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    0a8eebb View commit details
    Browse the repository at this point in the history

  45. clean up notebook and improve figures

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    5ece50d View commit details
    Browse the repository at this point in the history

  46. generalize path in notebook

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    bd55562 View commit details
    Browse the repository at this point in the history

  47. tests that require an active NSIDC Earthdata session (#209) 

    * update and run tests that require an active NSIDC Earthdata session
* add flat data resource note
* NOTE: using the fixture to successfully start a session can be considered the test
* transition options to json for cleaner test file
    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    a3f5536 View commit details
    Browse the repository at this point in the history

  48. updated product_summary_info function to latest fields returned by CM… 

    …R (product_id --> title) (#218)
    @jpswinski @JessicaS11
    jpswinski authored and JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    214eb5e View commit details
    Browse the repository at this point in the history

  49. add read module to docs

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    e87ef92 View commit details
    Browse the repository at this point in the history

  50. update requirements 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    4b0fa8a View commit details
    Browse the repository at this point in the history

  51. move warnings import to top level and use is instead of == 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    77cb363 View commit details
    Browse the repository at this point in the history

  52. improve messages to user 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    b8f3ec4 View commit details
    Browse the repository at this point in the history

  53. specify string for gran_idx 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    3139f90 View commit details
    Browse the repository at this point in the history

  54. reformat GitHub commits

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    eea627d View commit details
    Browse the repository at this point in the history

  55. remove changes caused by dev rebase

    @JessicaS11
    JessicaS11 committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    3b85632 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2021

  1. Merge branch 'development' into dataobj

    @JessicaS11
    JessicaS11 authored Oct 15, 2021
    Configuration menu
    Copy the full SHA
    b0fdb15 View commit details
    Browse the repository at this point in the history

  2. force data types for certain coordinates/variables

    @JessicaS11
    JessicaS11 committed Oct 15, 2021
    Configuration menu
    Copy the full SHA
    634a596 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2021

  1. Merge branch 'development' into dataobj

    @JessicaS11
    JessicaS11 authored Oct 18, 2021
    Configuration menu
    Copy the full SHA
    e119165 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2021

  1. initial PR review updates - clear outputs on notebooks, fix errors, u… 

    …pdate docstrings
    @JessicaS11
    JessicaS11 committed Oct 26, 2021
    Configuration menu
    Copy the full SHA
    6fa6d60 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dataobj' of github.com:icesat2py/icepyx into dataobj

    @JessicaS11
    JessicaS11 committed Oct 26, 2021
    Configuration menu
    Copy the full SHA
    eb36780 View commit details
    Browse the repository at this point in the history

  3. fix typos

    @JessicaS11
    JessicaS11 committed Oct 26, 2021
    Configuration menu
    Copy the full SHA
    3ba8220 View commit details
    Browse the repository at this point in the history

  4. use glob_path instead of glob 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Oct 26, 2021
    Configuration menu
    Copy the full SHA
    1b06422 View commit details
    Browse the repository at this point in the history

  5. fix or silence xarray deprecation warnings

    @JessicaS11
    JessicaS11 committed Oct 26, 2021
    Configuration menu
    Copy the full SHA
    db253da View commit details
    Browse the repository at this point in the history

  6. make imports alphebetical 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Oct 26, 2021
    Configuration menu
    Copy the full SHA
    c96b1dd View commit details
    Browse the repository at this point in the history

  7. specify unsigned int for some dimensions 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Oct 26, 2021
    Configuration menu
    Copy the full SHA
    377ca1d View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2021

  1. re-cast dtypes after merge; turn gt into a variable

    @JessicaS11
    JessicaS11 committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    9ce555c View commit details
    Browse the repository at this point in the history

  2. add explicit parameter name 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Oct 27, 2021
    Configuration menu
    Copy the full SHA
    68b9fa3 View commit details
    Browse the repository at this point in the history

  3. some code and notebook cleanup

    @JessicaS11
    JessicaS11 committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    45cb1d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2021

  1. Merge branch 'development' into dataobj

    @JessicaS11
    JessicaS11 authored Oct 28, 2021
    Configuration menu
    Copy the full SHA
    c2a9a48 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2021

  1. Merge branch 'development' into dataobj

    @JessicaS11
    JessicaS11 authored Nov 15, 2021
    Configuration menu
    Copy the full SHA
    e800ba3 View commit details
    Browse the repository at this point in the history

  2. fix example notebook error

    @JessicaS11
    JessicaS11 committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    714d065 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2021

  1. Apply suggestions from code review 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    a4ec72d View commit details
    Browse the repository at this point in the history

  2. Update examples/ICESat-2_Data_Read-in_Example.ipynb 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    2eaba79 View commit details
    Browse the repository at this point in the history

  3. Update examples/ICESat-2_Data_Read-in_Example.ipynb 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    7f71117 View commit details
    Browse the repository at this point in the history

  4. Update examples/ICESat-2_Data_Read-in_Example.ipynb 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    ad2968d View commit details
    Browse the repository at this point in the history

  5. Update examples/ICESat-2_Data_Read-in_Example.ipynb 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    6672910 View commit details
    Browse the repository at this point in the history

  6. Update examples/ICESat-2_Data_Read-in_Example.ipynb 

    Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
    @JessicaS11 @weiji14
    JessicaS11 and weiji14 authored Nov 23, 2021
    Configuration menu
    Copy the full SHA
    2ac4e2e View commit details
    Browse the repository at this point in the history

  7. fix last few typos and issues

    @JessicaS11
    JessicaS11 committed Nov 23, 2021
    Configuration menu
    Copy the full SHA
    745e24e View commit details
    Browse the repository at this point in the history

  8. add sorting to list comparisons

    @JessicaS11
    JessicaS11 committed Nov 23, 2021
    Configuration menu
    Copy the full SHA
    b57d247 View commit details
    Browse the repository at this point in the history