Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't save Cube as NetCDF #23

Closed
felixcremer opened this issue Dec 10, 2020 · 3 comments
Closed

Can't save Cube as NetCDF #23

felixcremer opened this issue Dec 10, 2020 · 3 comments

Comments

@felixcremer
Copy link
Member

When I try to save an example datacube which has been generated from an ENVI file, but here loaded as a zarr file, I can't save it as a netcdf.

julia> envi = YAXArrays.Cube("/home/crem_fe/Daten/tmp/smallenvi.zarr/")
YAXArray with the following dimensions
Y                   Axis with 10 Elements from -98.6335185299432 to -98.63190156243179
X                   Axis with 10 Elements from 20.6311005446814 to 20.629483577169985
Band                Axis with 271 elements: Band_1 Band_2 .. Band_270 Band_271 
Total size: 105.86 KB

julia> savecube(envi, "/home/crem_fe/Daten/tmp/smallenvi_zarr2.nc", backend=:netcdf, overwrite=true)
ERROR: ArgumentError: array must be non-empty
Stacktrace:
 [1] popfirst! at ./array.jl:1255 [inlined]
 [2] jl2nc(::DataType) at /home/crem_fe/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:91
 [3] getNCType(::DataType) at /home/crem_fe/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:101
 [4] NcVar(::String, ::Array{NcDim,1}; atts::Dict{String,Any}, t::DataType, compress::Int64, chunksize::Tuple{Int64}) at /home/crem_fe/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:256
 [5] nccreate(::String, ::String, ::String, ::Vararg{Any,N} where N; atts::Dict{String,Any}, gatts::Dict{Any,Any}, compress::Int64, t::DataType, mode::UInt16, chunksize::Tuple{Int64}) at /home/crem_fe/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:1209
 [6] #add_var#75 at /home/crem_fe/.julia/packages/YAXArrayBase/OgYsV/src/datasets/netcdf.jl:50 [inlined]
 [7] add_var at /home/crem_fe/.julia/packages/YAXArrayBase/OgYsV/src/datasets/netcdf.jl:49 [inlined]
 [8] #add_var#2 at /home/crem_fe/.julia/packages/YAXArrayBase/OgYsV/src/datasets/datasetinterface.jl:46 [inlined]
 [9] add_var at /home/crem_fe/.julia/packages/YAXArrayBase/OgYsV/src/datasets/datasetinterface.jl:46 [inlined]
 [10] arrayfromaxis(::YAXArrayBase.NetCDFDataset, ::CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},:Band,Array{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},1}}, ::Int64) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DatasetAPI/Datasets.jl:315
 [11] (::YAXArrays.Datasets.var"#60#69"{YAXArrayBase.NetCDFDataset})(::CategoricalAxis{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},:Band,Array{Zarr.MaxLengthStrings.MaxLengthString{8,UInt8},1}}, ::Int64) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DatasetAPI/Datasets.jl:250
 [12] foreach(::Function, ::Array{CubeAxis,1}, ::Tuple{Int64,Int64,Int64}) at ./abstractarray.jl:2010
 [13] createdataset(::Type{T} where T, ::Array{CubeAxis,1}; path::String, persist::Nothing, T::Type{T} where T, chunksize::Tuple{Int64,Int64,Int64}, chunkoffset::Tuple{Int64,Int64,Int64}, overwrite::Bool, properties::Dict{String,Any}, datasetaxis::String, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DatasetAPI/Datasets.jl:249
 [14] generateOutCube(::Type{YAXArrayBase.NetCDFDataset}, ::Type{T} where T, ::YAXArrays.DAT.OutputCube, ::Array{Int64,1}, ::Tuple{Int64,Int64,Int64}; kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:path, :overwrite),Tuple{String,Bool}}}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:513
 [15] generateOutCube(::YAXArrays.DAT.OutputCube, ::Base.RefValue{Bool}, ::Float64, ::Array{Int64,1}, ::Tuple{Int64,Int64,Int64}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:534
 [16] (::YAXArrays.DAT.var"#106#107"{YAXArrays.DAT.DATConfig{1,1},Tuple{}})(::YAXArrays.DAT.OutputCube) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:529
 [17] foreach(::YAXArrays.DAT.var"#106#107"{YAXArrays.DAT.DATConfig{1,1},Tuple{}}, ::Tuple{YAXArrays.DAT.OutputCube}) at ./abstractarray.jl:2009
 [18] generateOutCubes(::YAXArrays.DAT.DATConfig{1,1}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:527
 [19] mapCube(::YAXArrays.DAT.var"#cop#260", ::Tuple{YAXArray{Union{Missing, Float32},3,ZArray{Union{Missing, Float32},3,Zarr.BloscCompressor,DirectoryStore},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Array{Int64,1}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:275
 [20] #mapCube#34 at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/DAT.jl:177 [inlined]
 [21] savecube(::YAXArray{Union{Missing, Float32},3,ZArray{Union{Missing, Float32},3,Zarr.BloscCompressor,DirectoryStore},Array{CubeAxis,1}}, ::String; chunksize::Dict{Any,Any}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:overwrite,),Tuple{Bool}}}) at /home/crem_fe/.julia/packages/YAXArrays/LTFsC/src/DAT/CubeIO.jl:38
 [22] top-level scope at REPL[33]:1

I can save the envi file directly into netcdf, and I can also save the so generated netcdf back into a netcdf file, therefore, I don't really understand, what is happening when I first convert the data into the Zarr format.

@meggart
Copy link
Member

meggart commented Mar 9, 2021

I think this was related to the axis data type being a non-default string type and was fixed already?

@felixcremer
Copy link
Member Author

This is related to it, but in YAXArrays 0.1.2 and NetCDF 0.11.3 this is not yet fixed.
I think, we discussed this but haven't come to a conclusion, where the string conversion should happen.
I can mitigate this by converting every categorical axes to a String like this.

for ax in caxes(cube)
   if ax isa CategoricalAxis
      newax = CategoricalAxis(YAXArrays.Cubes.Axes.axname(ax), String.(ax.values))
      renameaxis!(cube, YAXArrays.Cubes.Axes.axname(ax)=>newax)
   end   
end

@felixcremer
Copy link
Member Author

I am closing this in favor of JuliaDataCubes/YAXArrayBase.jl#3 because this is something that needs to be fixed either in YAXArraysBase or NetCDF.jl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants