Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide byte-range reading of remote datasets #1267

Merged
merged 24 commits into from
Mar 19, 2019
Merged

Provide byte-range reading of remote datasets #1267

merged 24 commits into from
Mar 19, 2019

Commits on Jan 2, 2019

  1. Provide byte-range reading of remote datasets

    re: issue #1251
    
    Assume that you have the URL to a remote dataset
    which is a normal netcdf-3 or netcdf-4 file.
    
    This PR allows the netcdf-c to read that dataset's
    contents as a netcdf file using HTTP byte ranges
    if the remote server supports byte-range access.
    
    Originally, this PR was set up to access Amazon S3 objects,
    but it can also access other remote datasets such as those
    provided by a Thredds server via the HTTPServer access protocol.
    It may also work for other kinds of servers.
    
    Note that this is not intended as a true production
    capability because, as is known, this kind of access to
    can be quite slow. In addition, the byte-range IO drivers
    do not currently do any sort of optimization or caching.
    
    An additional goal here is to gain some experience with
    the Amazon S3 REST protocol.
    
    This architecture and its use documented in
    the file docs/byterange.dox.
    
    There are currently two test cases:
    
    1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
       for a remote netcdf-3 file and a remote netcdf-4 file.
    2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
       datasets.
    
    This PR also incorporates significantly changed model inference code
    (see the superceded PR #1259).
    
    1. It centralizes the code that infers the dispatcher.
    2. It adds support for byte-range URLs
    
    Other changes:
    
    1. NC_HDF5_finalize was not being properly called by nc_finalize().
    2. Fix minor bug in ncgen3.l
    3. fix memory leak in nc4info.c
    4. add code to walk the .daprc triples and to replace protocol=
       fragment tag with a more general mode= tag.
    
    Final Note:
    Th inference code is still way too complicated. We need to move
    to the validfile() model used by netcdf Java, where each
    dispatcher is asked if it can process the file. This decentralizes
    the inference code. This will be done after all the major new
    dispatchers (PIO, Zarr, etc) have been implemented.
    DennisHeimbigner committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    bf2746b View commit details
    Browse the repository at this point in the history
  2. Ignore lgtm alerts

    DennisHeimbigner committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    b2e24ea View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    84c2bc0 View commit details
    Browse the repository at this point in the history
  4. It turns out the the type H5FD_class_t was changed

    between HDF5 versions 1.8 and 1.10.
    So modify H5FDhttp.c to be conditional on the
    HDF5 major+minor version from H5public.h
    DennisHeimbigner committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    a7fa2d8 View commit details
    Browse the repository at this point in the history
  5. Fix typo

    DennisHeimbigner committed Jan 2, 2019
    Configuration menu
    Copy the full SHA
    6e76972 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fe3ba09 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2019

  1. typos3

    DennisHeimbigner committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    dc491f9 View commit details
    Browse the repository at this point in the history
  2. typeo4

    DennisHeimbigner committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    8cd4502 View commit details
    Browse the repository at this point in the history
  3. typo5

    DennisHeimbigner committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    ac2e6f9 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2019

  1. Configuration menu
    Copy the full SHA
    b27c7d8 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2019

  1. Configuration menu
    Copy the full SHA
    9df45bc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c49f425 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2019

  1. Configuration menu
    Copy the full SHA
    3662c4f View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2019

  1. ckp

    DennisHeimbigner committed Feb 1, 2019
    Configuration menu
    Copy the full SHA
    65e9a46 View commit details
    Browse the repository at this point in the history
  2. Made some mistakes in handling the PNETCDF case.

    So, fixed the following:
    1. Forgot to check for NC_FORMATX_PNETCDF case
       in one of the switches in NC_infermodel.
    2. Accidentally turned on both the NC_64BIT_OFFSET
       and the NC_64BIT_DATA mode flags.
    DennisHeimbigner committed Feb 1, 2019
    Configuration menu
    Copy the full SHA
    e4cb5c8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6723a38 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2019

  1. Configuration menu
    Copy the full SHA
    1fde39c View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2019

  1. Configuration menu
    Copy the full SHA
    a1f080d View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2019

  1. Configuration menu
    Copy the full SHA
    f19cac7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    24510f2 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2019

  1. master merge

    DennisHeimbigner committed Feb 24, 2019
    Configuration menu
    Copy the full SHA
    45a8a26 View commit details
    Browse the repository at this point in the history
  2. conflict resolution

    DennisHeimbigner committed Feb 24, 2019
    Configuration menu
    Copy the full SHA
    959c213 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0c59e13 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2019

  1. Configuration menu
    Copy the full SHA
    e2b31ff View commit details
    Browse the repository at this point in the history