Skip to content

Commit

Permalink
some tweaks to clean up the diff
Browse files Browse the repository at this point in the history
  • Loading branch information
ssfrr committed Nov 24, 2017
1 parent 6769f25 commit 861de39
Show file tree
Hide file tree
Showing 2 changed files with 131 additions and 84 deletions.
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ file format.
Sometimes you want to read or write files that are larger than your available
memory, or might be an unknown or infinite length (e.g. reading an audio or
video stream from a socket). In these cases it might not make sense to process
the whole file at once, but instead process it a chunk at a time. For these situations FileIO provides the `loadstreaming` and `savestreaming` functions, which return an object that you can `read` or `write`, rather than the file data itself.
the whole file at once, but instead process it a chunk at a time. For these
situations FileIO provides the `loadstreaming` and `savestreaming` functions,
which return an object that you can `read` or `write`, rather than the file data
itself.

This would look something like:

Expand Down Expand Up @@ -69,7 +72,8 @@ do loadstreaming("bigfile.wav") audio
end
```

Note that in these cases you may want to use `read!` with a pre-allocated buffer for maximum efficiency.
Note that in these cases you may want to use `read!` with a pre-allocated buffer
for maximum efficiency.

## Adding new formats

Expand Down Expand Up @@ -166,7 +170,13 @@ automatically even if the code inside the `do` scope throws an error.)
Conversely, `load(::Stream)` and `save(::Stream)` should not close the
input stream.

`loadstreaming` and `savestreaming` use the same query mechanism, but return a decoded stream that users can `read` or `write`. You should also implement a `close` method on your reader or writer type. Just like with `load` and `save`, if the user provided a filename, your `close` method should be responsible for closing any streams you opened in order to read or write the file. If you are given a `Stream`, your `close` method should only do the clean up for your reader or writer type, not close the stream.
`loadstreaming` and `savestreaming` use the same query mechanism, but return a
decoded stream that users can `read` or `write`. You should also implement a
`close` method on your reader or writer type. Just like with `load` and `save`,
if the user provided a filename, your `close` method should be responsible for
closing any streams you opened in order to read or write the file. If you are
given a `Stream`, your `close` method should only do the clean up for your
reader or writer type, not close the stream.

```julia
struct WAVReader
Expand Down
199 changes: 118 additions & 81 deletions src/loadsave.jl
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,52 @@ add_loader
"`add_saver(fmt, :Package)` triggers `using Package` before saving format `fmt`"
add_saver

"""
- `load(filename)` loads the contents of a formatted file, trying to infer
the format from `filename` and/or magic bytes in the file.
- `load(strm)` loads from an `IOStream` or similar object. In this case,
the magic bytes are essential.
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
- `load(f; options...)` passes keyword arguments on to the loader.
"""
load

"""
Some packages may implement a streaming API, where the contents of the file can
be read in chunks and processed, rather than all at once. Reading from these
higher-level streams should return a formatted object, like an image or chunk of
video or audio.
- `loadstreaming(filename)` loads the contents of a formatted file, trying to infer
the format from `filename` and/or magic bytes in the file. It returns a streaming
type that can be read from in chunks, rather than loading the whole contents all
at once
- `loadstreaming(strm)` loads the stream from an `IOStream` or similar object. In this case,
the magic bytes are essential.
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
- `load(f; options...)` passes keyword arguments on to the loader.
"""
loadstreaming

"""
- `save(filename, data...)` saves the contents of a formatted file,
trying to infer the format from `filename`.
- `save(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
- `save(f, data...; options...)` passes keyword arguments on to the saver.
"""
save

"""
Some packages may implement a streaming API, where the contents of the file can
be written in chunks, rather than all at once. These higher-level streams should
accept formatted objects, like an image or chunk of video or audio.
- `savestreaming(filename, data...)` saves the contents of a formatted file,
trying to infer the format from `filename`.
- `savestreaming(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
- `savestreaming(f, data...; options...)` passes keyword arguments on to the saver.
"""
savestreaming

for fn in (:load, :loadstreaming, :save, :savestreaming)
@eval $fn(s::@compat(Union{AbstractString,IO}), args...; options...) =
Expand All @@ -63,12 +109,14 @@ function savestreaming{sym}(df::Type{DataFormat{sym}}, s::IO, data...; options..
checked_import(libraries[1])
eval(Main, :($savestreaming($Stream($(DataFormat{sym}), $s),
$data...; $options...)))
end

function save{sym}(df::Type{DataFormat{sym}}, s::IO, data...; options...)
libraries = applicable_savers(df)
checked_import(libraries[1])
eval(Main, :($save($Stream($(DataFormat{sym}), $s),
$data...; $options...)))
end

function savestreaming{sym}(df::Type{DataFormat{sym}}, f::AbstractString, data...; options...)
libraries = applicable_savers(df)
Expand All @@ -82,104 +130,93 @@ for fn in (:loadstreaming, :savestreaming)
@eval function $fn(f::Function, args...; kwargs...)
str = $fn(args...; kwargs...)
try
ret = f(str)
f(str)
finally
close(str)
end

ret
end
end

# Fallbacks
for fn in (:load, :loadstreaming)
@eval function $fn{F}(q::Formatted{F}, args...; options...)
if unknown(q)
isfile(filename(q)) || open(filename(q)) # force systemerror
throw(UnknownFormat(q))
end
libraries = applicable_loaders(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.$fn), Library)
throw(LoaderError(string(library), "$fn not defined"))
end
return eval(Main, :($(Library.$fn)($q, $args...; $options...)))
catch e
push!(failures, (e, q))

# TODO: this definitely should be refactored to reduce duplication
function load{F}(q::Formatted{F}, args...; options...)
if unknown(q)
isfile(filename(q)) || open(filename(q)) # force systemerror
throw(UnknownFormat(q))
end
libraries = applicable_loaders(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.load), Library)
throw(LoaderError(string(library), "load not defined"))
end
return eval(Main, :($(Library.load)($q, $args...; $options...)))
catch e
push!(failures, (e, q))
end
handle_exceptions(failures, "loading \"$(filename(q))\"")
end
handle_exceptions(failures, "loading \"$(filename(q))\"")
end
for fn in (:save, :savestreaming)
@eval function $fn{F}(q::Formatted{F}, data...; options...)
unknown(q) && throw(UnknownFormat(q))
libraries = applicable_savers(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.$fn), Library)
throw(WriterError(string(library), "$fn not defined"))
end
return eval(Main, :($(Library.$fn)($q, $data...; $options...)))
catch e
push!(failures, (e, q))

function loadstreaming{F}(q::Formatted{F}, args...; options...)
if unknown(q)
isfile(filename(q)) || open(filename(q)) # force systemerror
throw(UnknownFormat(q))
end
libraries = applicable_loaders(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.loadstreaming), Library)
throw(LoaderError(string(library), "loadstreaming not defined"))
end
return eval(Main, :($(Library.loadstreaming)($q, $args...; $options...)))
catch e
push!(failures, (e, q))
end
handle_exceptions(failures, "saving \"$(filename(q))\"")
end
handle_exceptions(failures, "opening \"$(filename(q))\" for streamed loading")
end

"""
- `load(filename)` loads the contents of a formatted file, trying to infer
the format from `filename` and/or magic bytes in the file.
- `load(strm)` loads from an `IOStream` or similar object. In this case,
the magic bytes are essential.
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
- `load(f; options...)` passes keyword arguments on to the loader.
"""
load

"""
Some packages may implement a streaming API, where the contents of the file can
be read in chunks and processed, rather than all at once. Reading from these
higher-level streams should return a formatted object, like an image or chunk of
video or audio.
- `loadstreaming(filename)` loads the contents of a formatted file, trying to infer
the format from `filename` and/or magic bytes in the file. It returns a streaming
type that can be read from in chunks, rather than loading the whole contents all
at once
- `loadstreaming(strm)` loads the stream from an `IOStream` or similar object. In this case,
the magic bytes are essential.
- `load(File(format"PNG",filename))` specifies the format directly, and bypasses inference.
- `load(f; options...)` passes keyword arguments on to the loader.
"""
loadstreaming

"""
- `save(filename, data...)` saves the contents of a formatted file,
trying to infer the format from `filename`.
- `save(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
- `save(f, data...; options...)` passes keyword arguments on to the saver.
"""
save

"""
Some packages may implement a streaming API, where the contents of the file can
be written in chunks, rather than all at once. These higher-level streams should
accept formatted objects, like an image or chunk of video or audio.
function save{F}(q::Formatted{F}, data...; options...)
unknown(q) && throw(UnknownFormat(q))
libraries = applicable_savers(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.save), Library)
throw(WriterError(string(library), "save not defined"))
end
return eval(Main, :($(Library.save)($q, $data...; $options...)))
catch e
push!(failures, (e, q))
end
end
handle_exceptions(failures, "saving \"$(filename(q))\"")
end

- `savestreaming(filename, data...)` saves the contents of a formatted file,
trying to infer the format from `filename`.
- `savestreaming(Stream(format"PNG",io), data...)` specifies the format directly, and bypasses inference.
- `savestreaming(f, data...; options...)` passes keyword arguments on to the saver.
"""
savestreaming
function savestreaming{F}(q::Formatted{F}, data...; options...)
unknown(q) && throw(UnknownFormat(q))
libraries = applicable_savers(q)
failures = Any[]
for library in libraries
try
Library = checked_import(library)
if !has_method_from(methods(Library.savestreaming), Library)
throw(WriterError(string(library), "savestreaming not defined"))
end
return eval(Main, :($(Library.savestreaming)($q, $data...; $options...)))
catch e
push!(failures, (e, q))
end
end
handle_exceptions(failures, "opening \"$(filename(q))\" for streamed saving")
end

function has_method_from(mt, Library)
for m in mt
Expand Down

0 comments on commit 861de39

Please sign in to comment.