Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor some algorithms to improve efficiency and/or cleanup #3078

Closed
wants to merge 57 commits into from
Closed

Refactor some algorithms to improve efficiency and/or cleanup #3078

wants to merge 57 commits into from

Conversation

DennisHeimbigner
Copy link
Collaborator

Major Changes

  • Convert the loaded_plugin list to use insertion sort plus binary search. This is a straight speed for space tradeoff.
  • Cleanup some dead code.
  • Replace some dicey macros in ncjson.
  • Refactor libdispatch/dutil.c functions to have their own header: ncutil.h.
  • Refactor the byte swapping code.
  • Move to consistently type HDF5 filter ids as int to correspond with H5Z_filter_t. This change excludes ids in the netcdf.h API.
  • Make Zarr format V3 actually use the "bytes" endian handling codec.
  • Fix a weird problem with NC_mktmp.

DennisHeimbigner and others added 30 commits December 23, 2024 15:10
@DennisHeimbigner
I am posting this PR as a draft so that Manuel Reis and others
150d130 
can work with it. I will revise this PR description later.

### Notes:
1. It does not yet run on github actions without errors.
2. It appears to have no memory leaks
3. It has problems with the testing infrastructure that I am working on.
4. CMake does not work for testing v3_nczarr_test because of CMake Policy
   [CMP002](https://cmake.org/cmake/help/latest/policy/CMP0002.html).
5. Ubuntu + Automake appears to work including testing v3_nczarr_test.
6. make distcheck does not work because of difficulties with VPATH.
@DennisHeimbigner
cleanup actions
9646782
@DennisHeimbigner
v3infer
a91ffb0
@DennisHeimbigner
infer1
cc5bb07
@DennisHeimbigner
trace1
31e27a6
@DennisHeimbigner
trace5
fdc2481
@DennisHeimbigner
hyrax1
210219d
@DennisHeimbigner
unpush
0a4c59b
@DennisHeimbigner
resync with v3meta.tmp
85a5a97
@DennisHeimbigner
Remove warnings from selected files
d4b96a9 
Add various casts etc to cleanup warnings from
selected modules.

The modules affected are as follows:
* libdispatch
* libncpoco
* libnczarr
* libsrc4
* nczarr_test

Additionally, clean up some Zarr V3 dependencies
in Makefile.am and configure.ac.

Note: This PR is a delta on v3plug.dmh, so cannot be applied
until that previous PR is applied.
@DennisHeimbigner
more cast fixes
8197a40
@DennisHeimbigner
example1
4e1ac75
@DennisHeimbigner
fix example/C filter tests
d23adb7
@DennisHeimbigner
Fix some bugs in zformat3.c
8da0d36
@DennisHeimbigner
Fix blosc compressor to accept uncompressable input
11e4f0d
@DennisHeimbigner
Apply patch to fix blosc compression
2cd4e30 
H/T Manual Reis
Also turn off some debugging output.
@DennisHeimbigner
Fix cygwin action
69ef73e
@DennisHeimbigner
Merge branch 'v3plug.dmh' of https://github.com/DennisHeimbigner/netc…
56db98b 
…df-c into v3plug.dmh
@DennisHeimbigner
Merge branch 'master' into v3plug.dmh
ea212e1
@DennisHeimbigner
Merge branch 'master' into clean1.dmh
bb60f19
@DennisHeimbigner
Merge branch 'v3plug.dmh' into clean1.dmh
f499bd9
@DennisHeimbigner
update docs
624b3ab
@DennisHeimbigner
Merge branch 'v3plug.dmh' of https://github.com/DennisHeimbigner/netc…
625bf74 
…df-c into v3plug.dmh
@DennisHeimbigner
Add Manuel's commits 348ca1a and 48b283b
44d5fc5
@DennisHeimbigner
Merge branch 'master' into v3plug.dmh
aae33e6
@DennisHeimbigner
Merge branch 'v3plug.dmh' of https://github.com/DennisHeimbigner/netc…
33c8477 
…df-c into v3plug.dmh
@DennisHeimbigner
fix static builds
348b6ab
@DennisHeimbigner
Merge branch 'v3plug.dmh' of https://github.com/DennisHeimbigner/netc…
e9fad85 
…df-c into v3plug.dmh
@DennisHeimbigner
Merge branch 'v3plug.dmh' into clean1.dmh
400134c
@DennisHeimbigner
Merge branch 'clean1.dmh' of https://github.com/DennisHeimbigner/netc…
987da3b 
…df-c into clean1.dmh
@WardF
Copy link
Member

WardF commented Jan 19, 2025

Hi Dennis, I'll take a look at this but right now I'm focusing on the other giant PR, the V3 one, so that we can get the consolidated metadata work in as well. I've run into an issue on that one as well (more on that on Tuesday), where the libnczarr tests are failing on my local Windows machine. Once we get that sorted I'll take a look at this!

@DennisHeimbigner
Copy link
Collaborator Author

Sigh! I really have to to quit doing these big PRs. It is not fair to you.
What happens is that I am working on something -- adding Zarr V3 for example -- and I
see something that should be fixed. Rather than spinning off a separate PR, I just end up
doing it in the same PR.

@WardF
Copy link
Member

WardF commented Jan 21, 2025

If we could focus on the first v3 PR that would be great, and I see additional failures have crept in here as opposed to the others. Let me see if I can figure out what's going on with that 'out of space' error happening.

WardF and others added 24 commits January 21, 2025 12:00
@WardF
Flag git ownership as safe for cygwin issue.
b9f665b
@WardF
Specify shell.
ba1d08c
@WardF
Restore pathing to something accepted (hopefully) by Github Actions.
66221d2
@WardF
Add disk space check post-failure, to see if the 'out of disk space' …
2869dff 
…error message is congruent with what we observe in the filesystem, at this point.
@WardF
Merge branch 'v3plug.dmh' of github.com:DennisHeimbigner/netcdf-c int…
93b35ff 
…o v3plug.dmh
@WardF
Correct syntax error.
779bbbf
@DennisHeimbigner
ckp
00932be
@WardF
Added fenceposts to only add leading '/' to nczarr tests when working…
6875ae3 
… in a filesystem which requires it. This may fail on cygwin, but I don't have an easy way to test that, so we shall see what the Github Actions reveal.
@DennisHeimbigner
Add -N flag to zmapio to suppress outputting the size of objects
9d7871b
@DennisHeimbigner
Add -N flag to zmapio to suppress outputting the size of objects
fcebd93
@DennisHeimbigner
Disable Zarr Over HTTP Support
1547d4b 
re: Unidata#3068 (comment)

At Manuel Reis request, This commit disables the current Zarr-Over-HTTP (ZOH)
support for URLS.
@DennisHeimbigner
debug1
206663f
@DennisHeimbigner
Merge branch 'clean1.dmh' of https://github.com/DennisHeimbigner/netc…
f9a25f1 
…df-c into clean1.dmh
@DennisHeimbigner
update
dc21726
@DennisHeimbigner
update
9b886a7
@DennisHeimbigner
Fix duplicate macro
ff78398
@DennisHeimbigner
Merge branch 'v3plug.dmh' into clean1.dmh
ca08544
@DennisHeimbigner
Merge branch 'v3plug.dmh' into insert1.dmh
3015059
@DennisHeimbigner
Merge branch 'v3plug.dmh' into zoh.tmp
79e9297
@DennisHeimbigner
make
146a75e
@DennisHeimbigner
Merge branch 'clean1.dmh' into insert1.dmh
c905c83
@DennisHeimbigner
Merge branch 'zoh.tmp' into insert1.dmh
eab796e
@DennisHeimbigner
Add zoh suppression
23ded3c
@DennisHeimbigner
Fix debug
0e430fb
@DennisHeimbigner
Copy link
Collaborator Author

Closing to replace with a set of smaller, more manageable commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WardF WardF Awaiting requested review from WardF WardF will be requested when the pull request is marked ready for review WardF is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DennisHeimbigner @WardF