Pkg.add("ParallelAccelerator") results in dirty state #17610

tkelman · 2016-07-25T15:59:41Z

Due to one of the files having executable permissions in git, but something wrong happening with the libgit2 clone or checkout. The executable permissions are removed in the working copy, and strangely that change is staged? The package being dirty is very bad, as it means Pkg won't upgrade or remove it unless you go in and manually reset the change.

~/julianightly> rm -rf pkgtmp; JULIA_PKGDIR=$PWD/pkgtemp bin/julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-pre+5654 (2016-07-24 09:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit aa2e898 (1 day old master)
|__/                   |  x86_64-unknown-linux-gnu

julia> Pkg.init()
INFO: Initializing package repository /global/homes/t/tkelman/julianightly/pkgtemp/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("Images", v"0.5.5", v"0.5.5+")
INFO: Cloning cache of ColorTypes from https://github.com/JuliaGraphics/ColorTypes.jl.git
INFO: Cloning cache of ColorVectorSpace from https://github.com/JuliaGraphics/ColorVectorSpace.jl.git
INFO: Cloning cache of Colors from https://github.com/JuliaGraphics/Colors.jl.git
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of FileIO from https://github.com/JuliaIO/FileIO.jl.git
INFO: Cloning cache of FixedPointNumbers from https://github.com/JeffBezanson/FixedPointNumbers.jl.git
INFO: Cloning cache of Graphics from https://github.com/JuliaGraphics/Graphics.jl.git
INFO: Cloning cache of Images from https://github.com/timholy/Images.jl.git
INFO: Cloning cache of Reexport from https://github.com/simonster/Reexport.jl.git
INFO: Cloning cache of SIUnits from https://github.com/Keno/SIUnits.jl.git
INFO: Cloning cache of TexExtensions from https://github.com/Keno/TexExtensions.jl.git
INFO: Cloning cache of Zlib from https://github.com/dcjones/Zlib.jl.git
INFO: Installing ColorTypes v0.2.5
INFO: Installing ColorVectorSpace v0.1.5
INFO: Installing Colors v0.6.6
INFO: Installing Compat v0.8.6
INFO: Installing FileIO v0.1.0
INFO: Installing FixedPointNumbers v0.1.4
INFO: Installing Graphics v0.1.3
INFO: Installing Images v0.5.5
INFO: Installing Reexport v0.0.3
INFO: Installing SIUnits v0.0.6
INFO: Installing TexExtensions v0.0.3
INFO: Installing Zlib v0.1.12
INFO: Package database updated

julia> Pkg.status()
1 required packages:
 - Images                        0.5.5              8840a952 (dirty)
11 additional packages:
 - ColorTypes                    0.2.5
 - ColorVectorSpace              0.1.5
 - Colors                        0.6.6
 - Compat                        0.8.6
 - FileIO                        0.1.0
 - FixedPointNumbers             0.1.4
 - Graphics                      0.1.3
 - Reexport                      0.0.3
 - SIUnits                       0.0.6
 - TexExtensions                 0.0.3
 - Zlib                          0.1.12

julia> exit()
~/julianightly> cd pkgtemp/v0.5/Images/
~/julianightly/pkgtemp/v0.5/Images> git status
Not currently on any branch.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   test/map.jl

~/julianightly/pkgtemp/v0.5/Images> ls -al test
total 8
drwxr-x--- 2 tkelman tkelman  4096 Jul 25 08:54 .
drwxr-x--- 6 tkelman tkelman   512 Jul 25 08:54 ..
-rw-r----- 1 tkelman tkelman    85 Jul 25 08:54 REQUIRE
-rwxr-x--- 1 tkelman tkelman 25276 Jul 25 08:54 algorithms.jl
-rw-r----- 1 tkelman tkelman 18441 Jul 25 08:54 core.jl
-rw-r----- 1 tkelman tkelman  1337 Jul 25 08:54 corner.jl
-rw-r----- 1 tkelman tkelman   463 Jul 25 08:54 distances.jl
-rw-r----- 1 tkelman tkelman 18633 Jul 25 08:54 edge.jl
-rw-r----- 1 tkelman tkelman 13339 Jul 25 08:54 map.jl
-rw-r----- 1 tkelman tkelman  3362 Jul 25 08:54 overlays.jl
-rw-r----- 1 tkelman tkelman   729 Jul 25 08:54 parallel.jl
-rw-r----- 1 tkelman tkelman   631 Jul 25 08:54 restrict.jl
-rw-r----- 1 tkelman tkelman   433 Jul 25 08:54 runtests.jl
-rw-r----- 1 tkelman tkelman  1657 Jul 25 08:54 writemime.jl

The text was updated successfully, but these errors were encountered:

timholy · 2016-07-25T16:38:24Z

I've noticed the executable bit keeps getting set on some files. Does this show up in GitHub PRs and I'm just not noticing? (Very likely.) Or is it from someone pushing straight to master?

Of relevance for using this as a test case, I just unset the executable flag in all Images.jl files (sorry). I'm tempted to use git config core.fileMode false, but http://stackoverflow.com/questions/1580596/how-do-i-make-git-ignore-file-mode-chmod-changes warns that it's not great practice. In this case I can't really see any problem, though.

tkelman · 2016-07-25T16:41:57Z

It does show up in PR's, ref https://github.com/timholy/Images.jl/pull/516/files#diff-af28e1ea7bb1f554f483a8671aab187b - so if you make a new tag this might not be the best test case any more, but we could do Pkg.add("Images", v"0.5.5", v"0.5.5+") for testing. Or commit something in a smaller package like Example.jl for testing purposes.

timholy · 2016-07-25T16:46:42Z

I am on the verge of tagging a new Images, but as you say the previous version should work as a test case.

tkelman · 2016-07-28T06:43:14Z

@wildart I suspect this is a libgit2 bug with executable permissions.

wildart · 2016-07-28T15:36:29Z

I'm not sure, but I use default settings for checked out files:

julia/base/libgit2/types.jl

Lines 109 to 110 in c389aa3

    
           dir_mode::Cuint = Cuint(0), # Cuint(0o755), 
        
           file_mode::Cuint = Cuint(0), #Cuint(0o644),

tkelman · 2016-07-28T16:03:53Z

How do we track down where in either libgit2 at the C level or in our bindings this is going wrong? Having packages not be updatable or removable (without manual fixes first) if they have any executable files in them would be pretty bad.

wildart · 2016-07-28T16:59:40Z

test\map.jl has executable bit set in v0.5.5, but unset on master.

~~I wonder if package caching of introduces this error. Beause repo first cached and then checked out from this cache. Maybe during this transition executable bit is lost.~~

tkelman · 2016-07-28T17:11:08Z

yeah we can put in place fixes for packages where we know this is happening if they don't actually need the executable bits, but for miscellaneous scripts etc it may help to have the ability to do so without things going wrong. not a bad theory that it might be cache (bare clone) related.

wildart · 2016-07-28T17:15:16Z

Cache is not a problem. Apparently, something funny is going when checking out particular commit. Because another file test\algorithms.jl has execution bit set as well, and it's correctly checked out, but not test\map.jl.

timholy · 2016-07-28T18:59:45Z

As I've made additional commits I've unset executable bits in Images, so it might be me doing the unsetting. I am sure I am not doing the setting, but I am not the only one contributing to that repo.

wildart · 2016-07-28T19:11:51Z

Well, you already committed map.jl file with execution bit set for v0.5.5. Whoever gets this version should have file with execution bit set.

timholy · 2016-07-28T19:20:13Z

Maybe I pulled someone else's branch and then pushed? If it would help to debug, I can try to figure it out. Which commit turned it on? (The commit linked by @tkelman above was me turning it off.)

wildart · 2016-07-28T19:23:09Z

It's probably the good thing that this happened, cause the bug was found. No need for further investigation, I can reproduce it.

tkelman · 2016-07-28T19:55:40Z

thank pkgeval - very strange versions being tested or failing to install wound up due to requirements conflicts happening because of this bug

tkelman · 2016-08-05T15:52:58Z

@wildart where in libgit2 do file permissions get handled?

wildart · 2016-08-05T19:16:12Z

Still investigating, it's not trivial It looks like upstream problem.

tkelman · 2016-08-17T15:52:44Z

Any update here? Not absolutely release blocking but would be ideal to fix before final if we can.

tkelman · 2017-03-05T01:31:02Z

The original test case doesn't cause this at the moment, but ParallelAccelerator does:

~/Julia/julia-0.6$ rm -rf ../jlpkgtmp && \
JULIA_PKGDIR=$HOME/Julia/jlpkgtmp ./julia -e 'versioninfo();
Pkg.init();
Pkg.add("ParallelAccelerator", v"0.2.2", v"0.2.2+");
Pkg.status()' && \
cd ../jlpkgtmp/v0.6/ParallelAccelerator && \
git diff && \
git status && \
ls -al test
Julia Version 0.6.0-pre.alpha.48
Commit 08f00b5* (2017-03-04 23:55 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)
INFO: Initializing package repository /home/tkelman/Julia/jlpkgtmp/v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
INFO: Cloning cache of Compat from https://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of CompilerTools from https://github.com/IntelLabs/CompilerTools.jl.git
INFO: Cloning cache of DataStructures from https://github.com/JuliaCollections/DataStructures.jl.git
INFO: Cloning cache of DocOpt from https://github.com/docopt/DocOpt.jl.git
INFO: Cloning cache of ParallelAccelerator from https://github.com/IntelLabs/ParallelAccelerator.jl.git
INFO: Installing Compat v0.19.0
INFO: Installing CompilerTools v0.2.1
INFO: Installing DataStructures v0.5.3
INFO: Installing DocOpt v0.2.1
INFO: Installing ParallelAccelerator v0.2.2
INFO: Building ParallelAccelerator
ParallelAccelerator: build.jl begin.
ParallelAccelerator: Building j2c-array shared library
System installed BLAS found
Using g++ to build ParallelAccelerator array runtime.
ParallelAccelerator: build.jl done.
INFO: Package database updated
1 required packages:
 - ParallelAccelerator           0.2.2              6da2e374 (dirty)
4 additional packages:
 - Compat                        0.19.0
 - CompilerTools                 0.2.1
 - DataStructures                0.5.3
 - DocOpt                        0.2.1
Not currently on any branch.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   test/test_kmeans.jl
        modified:   test/test_lr.jl

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        deps/a.out

total 132
drwxr-xr-x 2 tkelman tkelman 4096 Mar  4 20:29 .
drwxrwxr-x 8 tkelman tkelman 4096 Mar  4 20:29 ..
-rw-r--r-- 1 tkelman tkelman 2219 Mar  4 20:29 abs.jl
-rw-r--r-- 1 tkelman tkelman 1833 Mar  4 20:29 aug_assign.jl
-rw-r--r-- 1 tkelman tkelman 2602 Mar  4 20:29 BitArray.jl
-rw-r--r-- 1 tkelman tkelman 2688 Mar  4 20:29 broadcast.jl
-rw-r--r-- 1 tkelman tkelman 2307 Mar  4 20:29 cat.jl
-rw-r--r-- 1 tkelman tkelman 2651 Mar  4 20:29 complex.jl
-rw-r--r-- 1 tkelman tkelman 1884 Mar  4 20:29 const_promote.jl
-rw-r--r-- 1 tkelman tkelman  486 Mar  4 20:29 gemv_test.jl
-rw-r--r-- 1 tkelman tkelman 1910 Mar  4 20:29 hcat.jl
-rw-r--r-- 1 tkelman tkelman 3107 Mar  4 20:29 lib.jl
-rw-r--r-- 1 tkelman tkelman 3893 Mar  4 20:29 mapreduce.jl
-rw-r--r-- 1 tkelman tkelman 4671 Mar  4 20:29 misc.jl
-rw-r--r-- 1 tkelman tkelman 2478 Mar  4 20:29 parfor.jl
-rw-r--r-- 1 tkelman tkelman 1799 Mar  4 20:29 print.jl
-rw-r--r-- 1 tkelman tkelman 2464 Mar  4 20:29 rand.jl
-rw-r--r-- 1 tkelman tkelman 2088 Mar  4 20:29 range.jl
-rw-r--r-- 1 tkelman tkelman 2451 Mar  4 20:29 ranges.jl
-rw-r--r-- 1 tkelman tkelman   34 Mar  4 20:29 REQUIRE
-rw-r--r-- 1 tkelman tkelman 3489 Mar  4 20:29 runtests.jl
-rw-r--r-- 1 tkelman tkelman 2225 Mar  4 20:29 seq.jl
-rw-r--r-- 1 tkelman tkelman 2290 Mar  4 20:29 strings.jl
-rw-r--r-- 1 tkelman tkelman  448 Mar  4 20:29 test_at_mul_b.jl
-rw-r--r-- 1 tkelman tkelman 2395 Mar  4 20:29 test_kmeans.jl
-rw-r--r-- 1 tkelman tkelman  406 Mar  4 20:29 test_lda_evaluate.jl
-rw-r--r-- 1 tkelman tkelman 2143 Mar  4 20:29 test_lr.jl
-rw-r--r-- 1 tkelman tkelman  433 Mar  4 20:29 test_sum_maximum.jl
-rw-r--r-- 1 tkelman tkelman  690 Mar  4 20:29 test_update_w.jl
-rw-r--r-- 1 tkelman tkelman  438 Mar  4 20:29 transpose_test.jl
-rw-r--r-- 1 tkelman tkelman 1901 Mar  4 20:29 vcat.jl
-rw-r--r-- 1 tkelman tkelman  428 Mar  4 20:29 vecnorm_test.jl

This is a really nasty bug that causes packages to be dirty as soon as they're installed, so they never get updated again until the user notices the problem.

tkelman · 2017-03-08T03:54:28Z

fixed by #20916

tkelman · 2017-03-08T04:26:27Z

Apparently this one was happening on Windows, interestingly enough.

tkelman added bug Indicates an unexpected problem or unintended behavior packages Package management and loading libgit2 The libgit2 library or the LibGit2 stdlib module labels Jul 25, 2016

tkelman added this to the 0.5.x milestone Jul 28, 2016

wildart self-assigned this Jul 28, 2016

tkelman mentioned this issue Aug 1, 2016

Checklist of steps towards 0.5 release [candidates] #17418

Closed

16 tasks

StefanKarpinski added help wanted Indicates that a maintainer wants help on an issue or pull request and removed help wanted Indicates that a maintainer wants help on an issue or pull request labels Oct 27, 2016

This was referenced Dec 19, 2016

Add +x to deps/cmake_check.sh JuliaLang/MbedTLS.jl#91

Merged

MbedTLS package is "dirty" JuliaLang/MbedTLS.jl#92

Closed

tkelman assigned simonbyrne and unassigned wildart Mar 5, 2017

tkelman changed the title ~~Pkg.add("Images") results in dirty state~~ Pkg.add("ParallelAccelerator") results in dirty state Mar 5, 2017

simonbyrne mentioned this issue Mar 6, 2017

do checkout before changing HEAD #20916

Merged

tkelman closed this as completed Mar 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pkg.add("ParallelAccelerator") results in dirty state #17610

Pkg.add("ParallelAccelerator") results in dirty state #17610

tkelman commented Jul 25, 2016 •

edited

Loading

timholy commented Jul 25, 2016

tkelman commented Jul 25, 2016 •

edited

Loading

timholy commented Jul 25, 2016

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016 •

edited

Loading

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016 •

edited

Loading

timholy commented Jul 28, 2016

wildart commented Jul 28, 2016

timholy commented Jul 28, 2016

wildart commented Jul 28, 2016 •

edited

Loading

tkelman commented Jul 28, 2016

tkelman commented Aug 5, 2016

wildart commented Aug 5, 2016

tkelman commented Aug 17, 2016

tkelman commented Mar 5, 2017 •

edited

Loading

tkelman commented Mar 8, 2017

tkelman commented Mar 8, 2017

Pkg.add("ParallelAccelerator") results in dirty state #17610

Pkg.add("ParallelAccelerator") results in dirty state #17610

Comments

tkelman commented Jul 25, 2016 • edited Loading

timholy commented Jul 25, 2016

tkelman commented Jul 25, 2016 • edited Loading

timholy commented Jul 25, 2016

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016 • edited Loading

tkelman commented Jul 28, 2016

wildart commented Jul 28, 2016 • edited Loading

timholy commented Jul 28, 2016

wildart commented Jul 28, 2016

timholy commented Jul 28, 2016

wildart commented Jul 28, 2016 • edited Loading

tkelman commented Jul 28, 2016

tkelman commented Aug 5, 2016

wildart commented Aug 5, 2016

tkelman commented Aug 17, 2016

tkelman commented Mar 5, 2017 • edited Loading

tkelman commented Mar 8, 2017

tkelman commented Mar 8, 2017

tkelman commented Jul 25, 2016 •

edited

Loading

tkelman commented Jul 25, 2016 •

edited

Loading

wildart commented Jul 28, 2016 •

edited

Loading

wildart commented Jul 28, 2016 •

edited

Loading

wildart commented Jul 28, 2016 •

edited

Loading

tkelman commented Mar 5, 2017 •

edited

Loading