Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pretty printing for DataLoader #122

Merged
merged 2 commits into from
Oct 2, 2022
Merged

Pretty printing for DataLoader #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9d55bfd
pretty printing for DataLoader
mcabbott Sep 23, 2022
9043acc
tidy, tests
mcabbott Sep 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions src/eachobs.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,3 +255,41 @@ end
e.parallel && throw(ArgumentError("Transducer fold protocol not supported on parallel data loads"))
_dataloader_foldl1(rf, val, e, ObsView(e.data))
end

# Base uses this function for composable array printing, e.g. adjoint(view(::Matrix)))
function Base.showarg(io::IO, e::DataLoader, toplevel)
print(io, "DataLoader(")
Base.showarg(io, e.data, false)
Comment on lines +259 to +262
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention with overloading this (not just show) is that things like CuIterator(DataLoader(adjoint(::Matrix{... could then work without packages knowing about each other.

Copy link
Contributor Author

@mcabbott mcabbott Sep 23, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice if that could also apply to something like Iterators.flatten(Iterators.repeated(d, n)), or Iterators.take(Iterators.cycle(d), n * length(d))... but these seem obscure & have very complicated types.

Maybe we should make repeat(d::DataLoader, epochs::Int) just work?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but I thought there would be a default implementation for repeat that calls the iterator repeatedly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there should be. I thought it was only arrays, but turns out to repeat strings too:

julia> methods(repeat)
# 6 methods for generic function "repeat" from Base:
 [1] repeat(A::AbstractArray; inner, outer)
     @ abstractarraymath.jl:392
 [2] repeat(A::AbstractArray, counts...)
     @ abstractarraymath.jl:355
 [3] repeat(c::Char, r::Integer)
     @ strings/string.jl:349
 [4] repeat(c::AbstractChar, r::Integer)
     @ strings/string.jl:348
 [5] repeat(s::Union{SubString{String}, String}, r::Integer)
     @ strings/substring.jl:251
 [6] repeat(s::AbstractString, r::Integer)
     @ strings/basic.jl:715

I guess it's eager for all of those, so perhaps unclear whether repeat((x for x in 1:3), 3) should make an iterator or collect.

e.buffer == false || print(io, ", buffer=", e.buffer)
e.parallel == false || print(io, ", parallel=", e.parallel)
e.shuffle == false || print(io, ", shuffle=", e.shuffle)
e.batchsize == 1 || print(io, ", batchsize=", e.batchsize)
e.partial == true || print(io, ", partial=", e.partial)
e.collate == Val(nothing) || print(io, ", collate=", e.collate)
e.rng == Random.GLOBAL_RNG || print(io, ", rng=", e.rng)
print(io, ")")
end

Base.show(io::IO, e::DataLoader) = Base.showarg(io, e, false)

function Base.show(io::IO, m::MIME"text/plain", e::DataLoader)
if Base.haslength(e)
print(io, length(e), "-element ")
else
print(io, "Unknown-length ")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this can happen, can DataLoader be used with iterators which don't support indexing? Are there any which allow indexing but don't have a length?

julia> DataLoader(1:10)
10-element DataLoader(::UnitRange{Int64})
  with first element:
  1-element UnitRange{Int64}

julia> DataLoader(x for x in 1:10)
10-element DataLoader(::Base.Generator{UnitRange{Int64}, typeof(identity)})
  with first element:Error showing value of type DataLoader{Base.Generator{UnitRange{Int64}, typeof(identity)}, Random._GLOBAL_RNG, Val{nothing}}:
ERROR: MethodError: no method matching getindex(::Base.Generator{UnitRange{Int64}, typeof(identity)}, ::UnitRange{Int64})
Stacktrace:
  [1] getobs(data::Base.Generator{UnitRange{Int64}, typeof(identity)}, idx::UnitRange{Int64})
    @ MLUtils ~/.julia/dev/MLUtils/src/observation.jl:49

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think DataLoader expects random access indexing (for shuffling, batching, etc.). Even an infinite data container must support this through getobs (falls back to getindex) despite being slow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random access indexing is required as Kyle said

end
Base.showarg(io, e, false)
print(io, "\n with first element:")
print(io, "\n ", _expanded_summary(first(e)))
end

_expanded_summary(x) = summary(x)
function _expanded_summary(xs::Tuple)
parts = [_expanded_summary(x) for x in xs]
"(" * join(parts, ", ") * ",)"
end
function _expanded_summary(xs::NamedTuple)
parts = ["$k = "*_expanded_summary(x) for (k,x) in zip(keys(xs), xs)]
"(; " * join(parts, ", ") * ")"
end
Comment on lines +291 to +294
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is perhaps slightly odd, but the idea is to show what's inside something like this:

julia> nt = (x=rand(10,100), y=rand(Bool,10,100));

julia> summary(nt)  # no sizes
"NamedTuple{(:x, :y), Tuple{Matrix{Float64}, Matrix{Bool}}}"

julia> repr(nt)  # no sizes, too long
"(x = [0.4301236060985135 0.30528931378945046 0.9865249486891879 0.880700604424396 0.0411513866531249 0.5940861560957025 0.8857114440668031 0.07167460028948913 0.011754567396461968 0.1843492781834919 0.6304545382381233 0.4430950147392062 0.8014564359131793 0.8161412755930899 0.47800508950976983 0.11072415162810345 0.6459516433095668 0.6872" ⋯ 20611 bytes ⋯ " 1 1 0 1 0 0 0 1 1 0 0 1 1 1 0 0 0 0 0 1 1 1 1 0 0 0 1 1 0 1 0 0 1 0 1 0 1 0 0 1 0 1 1 0 0 1 0 0 0 0 1 1 1 1 1 1 0 1 1 0 1 0 0 0 1 1 1 1; 1 1 0 0 0 0 1 1 1 0 1 0 1 0 0 1 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 1 0 1 1 0 0 1 0 0 0 1 1 0 0 1 0 0 1 1 1 0 1 1 0 0 1 1 0 1 1 1 0 1 1 1])"

julia> MLUtils._expanded_summary(nt)
"(; x = 10×100 Matrix{Float64}, y = 10×100 Matrix{Bool})"

Maybe ideally Base.summary would do something more useful here.


21 changes: 21 additions & 0 deletions test/dataloader.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,4 +214,25 @@
dloader = DataLoader(1:1000; batchsize = 2, shuffle = true)
@test copy(Map(x -> x[1]), Vector{Int}, dloader) != collect(1:2:1000)
end

@testset "printing" begin
X2 = reshape(Float32[1:10;], (2, 5))
Y2 = [1:5;]

d = DataLoader((X2, Y2), batchsize=3)

@test contains(repr(d), "DataLoader(::Tuple{Matrix")
@test contains(repr(d), "batchsize=3")

@test contains(repr(MIME"text/plain"(), d), "2-element DataLoader")
@test contains(repr(MIME"text/plain"(), d), "2×3 Matrix{Float32}, 3-element Vector")

d2 = DataLoader((x = X2, y = Y2), batchsize=2, partial=false)

@test contains(repr(d2), "DataLoader(::NamedTuple")
@test contains(repr(d2), "partial=false")

@test contains(repr(MIME"text/plain"(), d2), "2-element DataLoader(::NamedTuple")
@test contains(repr(MIME"text/plain"(), d2), "x = 2×2 Matrix{Float32}, y = 2-element Vector")
end
end