Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More metadata for FileDFG and restore as saved #1038

Merged
merged 5 commits into from
Aug 18, 2023
Merged

Conversation

Affie
Copy link
Member

@Affie Affie commented Aug 18, 2023

No description provided.

Comment on lines +117 to +122
function getBlobEntryFirst(var::Variable, key::Regex)
firstIdx = findfirst(x->contains(string(x.label), key), var.blobEntries)
if isnothing(firstIdx)
throw(KeyError("$key"))
end
return var.blobEntries[firstIdx]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
function getBlobEntryFirst(var::Variable, key::Regex)
firstIdx = findfirst(x->contains(string(x.label), key), var.blobEntries)
if isnothing(firstIdx)
throw(KeyError("$key"))
end
return var.blobEntries[firstIdx]
function getBlobEntryFirst(var::Variable, key::Regex)
firstIdx = findfirst(x -> contains(string(x.label), key), var.blobEntries)
if isnothing(firstIdx)
throw(KeyError("$key"))
end
return var.blobEntries[firstIdx]

end
return var.blobEntries[firstIdx]
end

getBlobEntry(var::AbstractDFGVariable, key::AbstractString) = getBlobEntry(var,Regex(key))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
getBlobEntry(var::AbstractDFGVariable, key::AbstractString) = getBlobEntry(var,Regex(key))
getBlobEntry(var::AbstractDFGVariable, key::AbstractString) = getBlobEntry(var, Regex(key))

Comment on lines 160 to 161
varFiles = sort(readdir(varFolder; sort=false); lt=natural_lt)
factorFiles = sort(readdir(factorFolder; sort=false); lt=natural_lt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
varFiles = sort(readdir(varFolder; sort=false); lt=natural_lt)
factorFiles = sort(readdir(factorFolder; sort=false); lt=natural_lt)
varFiles = sort(readdir(varFolder; sort = false); lt = natural_lt)
factorFiles = sort(readdir(factorFolder; sort = false); lt = natural_lt)

Comment on lines +163 to 180
if isa(dfgLoadInto, GraphsDFG) && GraphsDFGs._variablestype(dfgLoadInto) == Variable
variables = @showprogress 1 "loading variables" map(varFiles) do varFile
jstr = read("$varFolder/$varFile", String)
return JSON3.read(jstr, PackedVariable)
end
else
variables = DFGVariable[]
@showprogress 1 "loading variables" for varFile in varFiles
jstr = read("$varFolder/$varFile", String)
try
packedData = JSON3.read(jstr, PackedVariable)
push!(variables, unpackVariable(packedData))
catch ex
@error("JSON3 is having trouble reading $varFolder/$varFile into a PackedVariable")
@show jstr
throw(ex)
end
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
if isa(dfgLoadInto, GraphsDFG) && GraphsDFGs._variablestype(dfgLoadInto) == Variable
variables = @showprogress 1 "loading variables" map(varFiles) do varFile
jstr = read("$varFolder/$varFile", String)
return JSON3.read(jstr, PackedVariable)
end
else
variables = DFGVariable[]
@showprogress 1 "loading variables" for varFile in varFiles
jstr = read("$varFolder/$varFile", String)
try
packedData = JSON3.read(jstr, PackedVariable)
push!(variables, unpackVariable(packedData))
catch ex
@error("JSON3 is having trouble reading $varFolder/$varFile into a PackedVariable")
@show jstr
throw(ex)
end
end
if isa(dfgLoadInto, GraphsDFG) && GraphsDFGs._variablestype(dfgLoadInto) == Variable
variables = @showprogress 1 "loading variables" map(varFiles) do varFile
jstr = read("$varFolder/$varFile", String)
return JSON3.read(jstr, PackedVariable)
end
else
variables = DFGVariable[]
@showprogress 1 "loading variables" for varFile in varFiles
jstr = read("$varFolder/$varFile", String)
try
packedData = JSON3.read(jstr, PackedVariable)
push!(variables, unpackVariable(packedData))
catch ex
@error("JSON3 is having trouble reading $varFolder/$varFile into a PackedVariable")
@show jstr
throw(ex)
end

end
end
@info "Loaded $(length(variables)) variables - $(map(v->v.label, variables))"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
end

Comment on lines 63 to 70
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))

#FIXME Deprecate remove in DFG v0.24
setdiff!(commonfields, [:blobStores])
blobStores = Dict{Symbol, AbstractBlobStore}()
!isnothing(packed.blobStores) && merge!(blobStores, packed.blobStores)

props = (k => getproperty(packed, k) for k in commonfields)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))
#FIXME Deprecate remove in DFG v0.24
setdiff!(commonfields, [:blobStores])
blobStores = Dict{Symbol, AbstractBlobStore}()
!isnothing(packed.blobStores) && merge!(blobStores, packed.blobStores)
props = (k => getproperty(packed, k) for k in commonfields)
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))
#FIXME Deprecate remove in DFG v0.24
setdiff!(commonfields, [:blobStores])
blobStores = Dict{Symbol, AbstractBlobStore}()
!isnothing(packed.blobStores) && merge!(blobStores, packed.blobStores)

Comment on lines +72 to +76
VT = isnothing(packed.typePackedVariable) || !packed.typePackedVariable ? DFGVariable : Variable
FT = isnothing(packed.typePackedFactor) || !packed.typePackedFactor ? DFGFactor : PackedFactor
# VT = isnothing(packed.typePackedVariable) || packed.typePackedVariable ? Variable : DFGVariable
# FT = isnothing(packed.typePackedFactor) || packed.typePackedFactor ? PackedFactor : DFGFactor
GraphsDFG{typeof(packed.solverParams), VT, FT}(;blobStores, props...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
VT = isnothing(packed.typePackedVariable) || !packed.typePackedVariable ? DFGVariable : Variable
FT = isnothing(packed.typePackedFactor) || !packed.typePackedFactor ? DFGFactor : PackedFactor
# VT = isnothing(packed.typePackedVariable) || packed.typePackedVariable ? Variable : DFGVariable
# FT = isnothing(packed.typePackedFactor) || packed.typePackedFactor ? PackedFactor : DFGFactor
GraphsDFG{typeof(packed.solverParams), VT, FT}(;blobStores, props...)
props = (k => getproperty(packed, k) for k in commonfields)
VT = if isnothing(packed.typePackedVariable) || !packed.typePackedVariable
DFGVariable
else
Variable
end
FT = if isnothing(packed.typePackedFactor) || !packed.typePackedFactor
DFGFactor
else
PackedFactor
end
# VT = isnothing(packed.typePackedVariable) || packed.typePackedVariable ? Variable : DFGVariable
# FT = isnothing(packed.typePackedFactor) || packed.typePackedFactor ? PackedFactor : DFGFactor
return GraphsDFG{typeof(packed.solverParams), VT, FT}(; blobStores, props...)

FT = isnothing(packed.typePackedFactor) || !packed.typePackedFactor ? DFGFactor : PackedFactor
# VT = isnothing(packed.typePackedVariable) || packed.typePackedVariable ? Variable : DFGVariable
# FT = isnothing(packed.typePackedFactor) || packed.typePackedFactor ? PackedFactor : DFGFactor
GraphsDFG{typeof(packed.solverParams), VT, FT}(;blobStores, props...)
end

function unpackDFGMetadata!(dfg::GraphsDFG, packed::PackedGraphsDFG)
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))
commonfields = intersect(fieldnames(GraphsDFG), fieldnames(PackedGraphsDFG))

Comment on lines +82 to +84
#FIXME Deprecate remove Nothing union in DFG v0.24
setdiff!(commonfields, [:blobStores])
!isnothing(packed.blobStores) && merge!(dfg.blobStores, packed.blobStores)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
#FIXME Deprecate remove Nothing union in DFG v0.24
setdiff!(commonfields, [:blobStores])
!isnothing(packed.blobStores) && merge!(dfg.blobStores, packed.blobStores)
#FIXME Deprecate remove Nothing union in DFG v0.24
setdiff!(commonfields, [:blobStores])
!isnothing(packed.blobStores) && merge!(dfg.blobStores, packed.blobStores)

Comment on lines +181 to +182
function packVariable(v::Variable; includePPEs::Bool=true, includeSolveData::Bool=true, includeDataEntries::Bool=true)
return v
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
function packVariable(v::Variable; includePPEs::Bool=true, includeSolveData::Bool=true, includeDataEntries::Bool=true)
return v
function packVariable(
v::Variable;
includePPEs::Bool = true,
includeSolveData::Bool = true,
includeDataEntries::Bool = true,
)
return v

@codecov
Copy link

codecov bot commented Aug 18, 2023

Codecov Report

Merging #1038 (e57eac3) into master (17f63da) will decrease coverage by 0.15%.
Report is 4 commits behind head on master.
The diff coverage is 47.45%.

@@            Coverage Diff             @@
##           master    #1038      +/-   ##
==========================================
- Coverage   78.09%   77.94%   -0.15%     
==========================================
  Files          26       26              
  Lines        2524     2181     -343     
==========================================
- Hits         1971     1700     -271     
+ Misses        553      481      -72     
Files Changed Coverage Δ
src/DataBlobs/services/BlobEntry.jl 50.98% <0.00%> (-4.63%) ⬇️
src/DistributedFactorGraphs.jl 100.00% <ø> (ø)
src/GraphsDFG/entities/GraphsDFG.jl 100.00% <ø> (ø)
src/GraphsDFG/services/GraphsDFG.jl 71.23% <0.00%> (-1.52%) ⬇️
src/services/AbstractDFG.jl 83.29% <ø> (-0.20%) ⬇️
src/services/DFGFactor.jl 73.07% <0.00%> (-7.57%) ⬇️
src/services/DFGVariable.jl 85.06% <0.00%> (-0.27%) ⬇️
src/services/Serialization.jl 70.62% <20.00%> (-3.24%) ⬇️
src/GraphsDFG/services/GraphsDFGSerialization.jl 72.72% <57.14%> (-6.00%) ⬇️
src/FileDFG/services/FileDFG.jl 77.69% <67.85%> (-0.23%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Affie Affie merged commit c0f1719 into master Aug 18, 2023
3 of 6 checks passed
@Affie Affie added the FileDFG label Aug 18, 2023
@Affie Affie added this to the v0.23.2 milestone Aug 18, 2023
@Affie Affie deleted the 23Q3/enh/packed_filedfg branch October 30, 2024 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants