Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readdlm problem when reading many files #3884

Closed
cmmp opened this issue Jul 30, 2013 · 1 comment
Closed

readdlm problem when reading many files #3884

cmmp opened this issue Jul 30, 2013 · 1 comment

Comments

@cmmp
Copy link

cmmp commented Jul 30, 2013

I'm getting a strange error when using readdlm over many files.

Consider a simple write code:

for i in 1:300
    writedlm("$i.dat", rand(10))
end

and the read code:

for i in 1:300
    x = readdlm("$i.dat")
end

The write code runs fine, but when trying to run the read code I get:

ERROR: could not open file 244.dat
 in open at io.jl:291
 in open at io.jl:301
 in readdlm_auto at datafmt.jl:46
 in readdlm at datafmt.jl:41
 in readdlm at datafmt.jl:38
 in anonymous at no file:2
 in include_from_node1 at loading.jl:92
 in process_options at client.jl:274
 in _start at client.jl:352
at /Users/cassio/Desktop/test/read.jl:3

However, if I try to open it in the REPL there's no problem:

Cassios-iMac:test cassio$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.2.0-prerelease+2890
 _/ |\__'_|_|_|\__'_|  |  Commit master/9c392b7* 2013-07-30 14:51:08 UTC
|__/                   |  x86_64-apple-darwin12.4.0

julia> x = readdlm("244.dat")
10x1 Float64 Array:
 0.485284
 0.249354
 0.305629
 0.412677
 0.740868
 0.574576
 0.23548
 0.0276011
 0.963765
 0.297823

I'm running OS X 10.8.4 with julia: master/9c392b7*

Any ideas?

Thanks,
Cássio

@tanmaykm
Copy link
Member

I think it's because of the file handle opened for mmap is not closed and you are running out of file handles. This would happen in tight loops where gc does not get an opportunity to run.

I shall push a fix shortly.

JeffBezanson added a commit that referenced this issue Jul 31, 2013
close file handle after mmap. fixes #3884
KristofferC pushed a commit that referenced this issue May 2, 2024
Stdlib: Pkg
URL: https://github.com/JuliaLang/Pkg.jl.git
Stdlib branch: master
Julia branch: master
Old commit: 8f772ffa7
New commit: a0851bcfd
Julia version: 1.12.0-DEV
Pkg version: 1.12.0
Bump invoked by: @KristofferC
Powered by:
[BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl)

Diff:
JuliaLang/Pkg.jl@8f772ff...a0851bc

```
$ git log --oneline 8f772ffa7..a0851bcfd
a0851bcfd Relax overly-specific manifest tests (#3884)
299b77093 guard against isdir EACESS in completions (#3877)
b9c39a7be Merge pull request #3883 from JuliaLang/kc/resolve_tree_hash
8ab562f72 instantiate the test environment if it is in the current workspace when testing
1d961c1c9 get the correct manifest file for a workspace
e468facb5 allow `resolve` to download packages that have been deleted from disk but has a manifest entry
20ceec9b8 do not precompile full env on `using` callback from REPL (#3876)
195e17e3f Replace UnstableIO with IOContext{IO} (#3735)
00aa38cb3 add missing `**5.**` (#3870)
ac5672dad Revert "Move "Creating Packages" to Julia docs" (#3818)
```

Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
lazarusA pushed a commit to lazarusA/julia that referenced this issue Jul 12, 2024
…#54316)

Stdlib: Pkg
URL: https://github.com/JuliaLang/Pkg.jl.git
Stdlib branch: master
Julia branch: master
Old commit: 8f772ffa7
New commit: a0851bcfd
Julia version: 1.12.0-DEV
Pkg version: 1.12.0
Bump invoked by: @KristofferC
Powered by:
[BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl)

Diff:
JuliaLang/Pkg.jl@8f772ff...a0851bc

```
$ git log --oneline 8f772ffa7..a0851bcfd
a0851bcfd Relax overly-specific manifest tests (JuliaLang#3884)
299b77093 guard against isdir EACESS in completions (JuliaLang#3877)
b9c39a7be Merge pull request JuliaLang#3883 from JuliaLang/kc/resolve_tree_hash
8ab562f72 instantiate the test environment if it is in the current workspace when testing
1d961c1c9 get the correct manifest file for a workspace
e468facb5 allow `resolve` to download packages that have been deleted from disk but has a manifest entry
20ceec9b8 do not precompile full env on `using` callback from REPL (JuliaLang#3876)
195e17e3f Replace UnstableIO with IOContext{IO} (JuliaLang#3735)
00aa38cb3 add missing `**5.**` (JuliaLang#3870)
ac5672dad Revert "Move "Creating Packages" to Julia docs" (JuliaLang#3818)
```

Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants