Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write_tag_metadata does not add files on Windows #28

Closed
tkelman opened this issue Feb 9, 2016 · 22 comments · Fixed by #80
Closed

write_tag_metadata does not add files on Windows #28

tkelman opened this issue Feb 9, 2016 · 22 comments · Fixed by #80
Labels

Comments

@tkelman
Copy link
Contributor

tkelman commented Feb 9, 2016

This might be a bug in base's LibGit2.add!, not sure. Will have to come back and look into it in more detail later. I end up getting "No METADATA changes to commit" at the end of a Pkg.tag because no sha1 or requires files have been added (they've been created, but not added to the metadata git repo).

@wildart
Copy link
Member

wildart commented Apr 1, 2016

Message "No METADATA changes to commit" displayed when you try to register or tag repo which is dirty. This is expected behavior.

@tkelman
Copy link
Contributor Author

tkelman commented Apr 1, 2016

Pkg.tag is expected to git add the new files it creates. It doesn't on Windows, last I checked.

@tkelman
Copy link
Contributor Author

tkelman commented Apr 1, 2016

when you try to register or tag repo which is dirty. This is expected behavior.

That really needs a more descriptive error message for that condition then.

@wildart
Copy link
Member

wildart commented Apr 1, 2016

Pkg.tag is expected to git add the new files it creates.

To be clear, tag does not add anything to the tagged package, it adds files to METADATA by calling write_tag_metadata. So if repo is dirty all created files are added but not committed. Are you sure that it didn't add anything to METADATA repo? It is highly unlikely.

@tkelman
Copy link
Contributor Author

tkelman commented Apr 1, 2016

Are you sure that it didn't add anything to METADATA repo? It is highly unlikely.

Yes, that was the bug.

@wildart
Copy link
Member

wildart commented Apr 1, 2016

Message "No METADATA changes to commit" displayed when you try to register or tag repo which is dirty.

Sorry, I made a mistake this message displayed when METADATA is dirty while you try to register or tag any package. So, probably, you had something uncommitted before tagging.

This is an exceptional situation, usually METADATA mustn't have any uncommitted changes.

@tkelman
Copy link
Contributor Author

tkelman commented Apr 1, 2016

julia> pwd()
"C:\\cygwin64\\home\\Tony\\.julia\\v0.5\\METADATA"

shell> git status
On branch metadata-v2
Your branch is up-to-date with 'origin/metadata-v2'.
nothing to commit, working directory clean

julia> PkgDev.tag("PkgDev")
INFO: Tagging PkgDev v0.1.1
INFO: No METADATA changes to commit

shell> git status
On branch metadata-v2
Your branch is up-to-date with 'origin/metadata-v2'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        PkgDev/versions/0.1.1/

nothing added to commit but untracked files present (use "git add" to track)

shell> git log -1
commit e9f6c9a07effedb166d49f2eddfc2b464f37cdfd
Merge: d4d5998 32b116f
Author: Stefan Karpinski <stefan@karpinski.org>
Date:   Thu Mar 31 19:11:55 2016 -0400

    Merge pull request #4891 from spencerlyon2/metadata-v2

    Tag PlotlyJS v0.1.4

@wildart
Copy link
Member

wildart commented Apr 1, 2016

Cannot reproduce on linux box. And it's LibGit2.add! indeed. What path gives PkgDev.dir("PkgDev") on your machine?

@wildart
Copy link
Member

wildart commented Apr 1, 2016

Aha, I managed to get the same problem. I cannot stage changes anymore.

@tkelman
Copy link
Contributor Author

tkelman commented Apr 1, 2016

Might have a file-system race condition somewhere, which is likely to be slower and more apparent on Windows.

@tkelman
Copy link
Contributor Author

tkelman commented Aug 3, 2016

This is still a bug on latest master of everything. A very nasty one if LibGit2.add! has never worked on Windows, and needs to be fixed in Base.

  | | |_| | | | (_| |  |  Version 0.5.0-rc0+146 (2016-08-03 00:47 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 37e6397* (0 days old master)
|__/                   |  x86_64-w64-mingw32

julia> using PkgDev

julia> PkgDev.generate("TestGenPkg", "MIT")
INFO: Initializing TestGenPkg repo: C:\cygwin64\home\Tony\.julia\v0.5\TestGenPkg
INFO: Origin: https://github.com/tkelman/TestGenPkg.jl.git
INFO: Generating LICENSE.md
INFO: Generating README.md
INFO: Generating src/TestGenPkg.jl
INFO: Generating test/runtests.jl
INFO: Generating REQUIRE
INFO: Generating .gitignore
INFO: Generating .travis.yml
INFO: Generating appveyor.yml
INFO: Committing TestGenPkg generated files

julia> PkgDev.register("TestGenPkg")
INFO: Registering TestGenPkg at https://github.com/tkelman/TestGenPkg.jl.git
INFO: No METADATA changes to commit

julia> cd(Pkg.dir("METADATA"))

shell> git status
On branch metadata-v2
Your branch is up-to-date with 'origin/metadata-v2'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        TestGenPkg/

nothing added to commit but untracked files present (use "git add" to track)

julia> readdir("TestGenPkg")
1-element Array{String,1}:
 "url"

julia> readstring("TestGenPkg/url")
"https://github.com/tkelman/TestGenPkg.jl.git\n"

@tkelman
Copy link
Contributor Author

tkelman commented Aug 3, 2016

At least PkgDev.generate seems to do the right thing, is it adding/committing files differently?

julia> cd(Pkg.dir("TestGenPkg"))

shell> git status
On branch master
nothing to commit, working directory clean

shell> git log
commit 8e4bb2d4d874c602d11b51b6ec925650fa642d50
Author: Tony Kelman <tony@kelman.net>
Date:   Wed Aug 3 08:21:05 2016 +0100

    TestGenPkg.jl generated files.

        license:  MIT
        authors:  Tony Kelman
        years:    2016
        user:     tkelman

    Julia Version 0.5.0-rc0+146 [37e6397*]

commit 3de52e9d2272f4b263affabeb0044fd70af0eecf
Author: Tony Kelman <tony@kelman.net>
Date:   Wed Aug 3 08:21:04 2016 +0100

    initial empty commit

shell> git show
commit 8e4bb2d4d874c602d11b51b6ec925650fa642d50
Author: Tony Kelman <tony@kelman.net>
Date:   Wed Aug 3 08:21:05 2016 +0100

    TestGenPkg.jl generated files.

        license:  MIT
        authors:  Tony Kelman
        years:    2016
        user:     tkelman

    Julia Version 0.5.0-rc0+146 [37e6397*]

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..8c960ec
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,3 @@
+*.jl.cov
+*.jl.*.cov
+*.jl.mem
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 0000000..d9927ff
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,19 @@
+# Documentation: http://docs.travis-ci.com/user/languages/julia/
+language: julia
+os:
+  - linux
+  - osx
+julia:
+  - release
+  - nightly
+notifications:
+  email: false
+# uncomment the following lines to override the default test script
+#script:
+#  - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
+#  - julia -e 'Pkg.clone(pwd()); Pkg.build("TestGenPkg"); Pkg.test("TestGenPkg"; coverage=true)'
+after_success:
+  # push coverage results to Coveralls
+  - julia -e 'cd(Pkg.dir("TestGenPkg")); Pkg.add("Coverage"); using Coverage; Coveralls.submit(Coveralls.process_folder())'
+  # push coverage results to Codecov
+  - julia -e 'cd(Pkg.dir("TestGenPkg")); Pkg.add("Coverage"); using Coverage; Codecov.submit(Codecov.process_folder())'
diff --git a/LICENSE.md b/LICENSE.md
new file mode 100644
index 0000000..73f42e7
--- /dev/null
+++ b/LICENSE.md
@@ -0,0 +1,22 @@
+The TestGenPkg.jl package is licensed under the MIT "Expat" License:
+
+> Copyright (c) 2016: Tony Kelman.
+>
+> Permission is hereby granted, free of charge, to any person obtaining a copy
+> of this software and associated documentation files (the "Software"), to deal
+> in the Software without restriction, including without limitation the rights
+> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+> copies of the Software, and to permit persons to whom the Software is
+> furnished to do so, subject to the following conditions:
+>
+> The above copyright notice and this permission notice shall be included in all
+> copies or substantial portions of the Software.
+>
+> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+> SOFTWARE.
+>
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..6434699
--- /dev/null
+++ b/README.md
@@ -0,0 +1,7 @@
+# TestGenPkg
+
+[![Build Status](https://travis-ci.org/tkelman/TestGenPkg.jl.svg?branch=master)](https://travis-ci.org/tkelman/TestGenPkg.jl)
+
+[![Coverage Status](https://coveralls.io/repos/tkelman/TestGenPkg.jl/badge.svg?branch=master&service=github)](https://coveralls.io/github/tkelman/TestGenPkg.jl?branch=master)
+
+[![codecov.io](http://codecov.io/github/tkelman/TestGenPkg.jl/coverage.svg?branch=master)](http://codecov.io/github/tkelman/TestGenPkg.jl?branch=master)
diff --git a/REQUIRE b/REQUIRE
new file mode 100644
index 0000000..d5d6467
--- /dev/null
+++ b/REQUIRE
@@ -0,0 +1 @@
+julia 0.4
diff --git a/appveyor.yml b/appveyor.yml
new file mode 100644
index 0000000..ace01f4
--- /dev/null
+++ b/appveyor.yml
@@ -0,0 +1,34 @@
+environment:
+  matrix:
+  - JULIAVERSION: "julialang/bin/winnt/x86/0.4/julia-0.4-latest-win32.exe"
+  - JULIAVERSION: "julialang/bin/winnt/x64/0.4/julia-0.4-latest-win64.exe"
+  - JULIAVERSION: "julianightlies/bin/winnt/x86/julia-latest-win32.exe"
+  - JULIAVERSION: "julianightlies/bin/winnt/x64/julia-latest-win64.exe"
+
+branches:
+  only:
+    - master
+    - /release-.*/
+
+notifications:
+  - provider: Email
+    on_build_success: false
+    on_build_failure: false
+    on_build_status_changed: false
+
+install:
+# Download most recent Julia Windows binary
+  - ps: (new-object net.webclient).DownloadFile(
+        $("http://s3.amazonaws.com/"+$env:JULIAVERSION),
+        "C:\projects\julia-binary.exe")
+# Run installer silently, output to C:\projects\julia
+  - C:\projects\julia-binary.exe /S /D=C:\projects\julia
+
+build_script:
+# Need to convert from shallow to complete for Pkg.clone to work
+  - IF EXIST .git\shallow (git fetch --unshallow)
+  - C:\projects\julia\bin\julia -e "versioninfo();
+      Pkg.clone(pwd(), \"TestGenPkg\"); Pkg.build(\"TestGenPkg\")"
+
+test_script:
+  - C:\projects\julia\bin\julia -e "Pkg.test(\"TestGenPkg\")"
diff --git a/src/TestGenPkg.jl b/src/TestGenPkg.jl
new file mode 100644
index 0000000..36fb519
--- /dev/null
+++ b/src/TestGenPkg.jl
@@ -0,0 +1,5 @@
+module TestGenPkg
+
+# package code goes here
+
+end # module
diff --git a/test/runtests.jl b/test/runtests.jl
new file mode 100644
index 0000000..b1eb816
--- /dev/null
+++ b/test/runtests.jl
@@ -0,0 +1,5 @@
+using TestGenPkg
+using Base.Test
+
+# write your own tests here
+@test 1 == 2

@tkelman
Copy link
Contributor Author

tkelman commented Aug 21, 2016

@wildart could you please respond to direct requests for assistance?

@tkelman tkelman added the bug label Sep 19, 2016
@pfitzseb
Copy link
Member

pfitzseb commented Sep 27, 2016

Just wanted to report that I've been running into this as well when trying to Pkg.tag() (on Windows, 0.5 release).

It seems like LibGit2.add! doesn't handle absolute file paths, which might be the source of this bug:

julia> mkdir("git_test")

julia> cd("git_test")

julia> LibGit2.init(pwd())
Base.LibGit2.GitRepo(Ptr{Void} @0x00000000088990d0)

julia> r = LibGit2.GitRepo(pwd())
Base.LibGit2.GitRepo(Ptr{Void} @0x00000000088993d0)

julia> (f, _) = mktemp(pwd());

julia> LibGit2.add!(r, f)

shell> git status
On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        jl_85BE.tmp

nothing added to commit but untracked files present (use "git add" to track)

julia> LibGit2.add!(r, splitdir(f)[2])

shell> git status
On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   jl_85BE.tmp

I won't pretend to understand what's going on in PkgDev or LibGit2 enough to figure out why this happens or even if that's a red herring, just throwing it out there.

@tkelman
Copy link
Contributor Author

tkelman commented Sep 29, 2016

That is a very useful piece of information, thanks @pfitzseb!

@tkelman
Copy link
Contributor Author

tkelman commented Sep 29, 2016

Hm. Throwing in some @show annotations, LibGit2.add! isn't getting called on any absolute paths here as far as I can tell. But the relative paths are Windows-style, with backslashes. Those might be getting escaped too many times inside the libgit2 code, or handled poorly by the C library, but it looks like if I replace backslashes with forward slashes in sha1file and reqsfile, then things work. Should have tried that sooner.

@pfitzseb
Copy link
Member

pfitzseb commented Sep 29, 2016

Hm, I kinda would've expected cding into d to work, but this

function write_tag_metadata(repo::GitRepo, pkg::AbstractString, ver::VersionNumber, commit::AbstractString, force::Bool=false)
    pkgdir = PkgDev.dir(pkg)
    content = with(GitRepo, pkgdir) do pkg_repo
        LibGit2.cat(pkg_repo, LibGit2.GitBlob, "$commit:REQUIRE")
    end
    reqs = content !== nothing ? Reqs.read(split(content, '\n', keep=false)) : Reqs.Line[]
    cd(Pkg.dir("METADATA")) do
        d = joinpath(pkg,"versions",string(ver))
        mkpath(d)
        cd(d) do
            sha1file = "sha1"
            if !force && ispath(sha1file)
                current = readchomp(sha1file)
                current == commit ||
                    throw(PkgError("$pkg v$ver is already registered as $current, bailing"))
            end
            open(io->println(io,commit), sha1file, "w")
            LibGit2.add!(repo, sha1file)
            reqsfile = "requires"
            if isempty(reqs)
                ispath(reqsfile) && LibGit2.remove!(repo, reqsfile)
            else
                Reqs.write(reqsfile,reqs)
                LibGit2.add!(repo, reqsfile)
            end
        end
    end
    return nothing
end

doesn't for me.

And in the example I posted above it doesn't seem to matter if the path is specified unix or windows style -- no files are added either way.
Confusing...

Nevermind, I'm an idiot: The path must of course be relative to the git dir's cwd. Unix style path seperators work then. The libgit2 docs even say as much:
"The pathspec is a list of file names or shell glob patterns that will matched against files in the repository's working directory."

@tkelman
Copy link
Contributor Author

tkelman commented Sep 29, 2016

we should probably do normalization here for 0.5.0's sake even if Base.LibGit2 gets fixed for this

@pfitzseb
Copy link
Member

Should be sufficient to replace joinpath(pkg,"versions",string(ver)) with join([pkg,"versions",string(ver)], '/'), right?

@tkelman
Copy link
Contributor Author

tkelman commented Sep 29, 2016

That isn't exactly what I tested, but should be a more direct way to get the same result. should have a comment to these issues explaining why it isn't calling joinpath

@tkelman
Copy link
Contributor Author

tkelman commented Sep 29, 2016

and the PkgDev.tag tests aren't detailed enough if they were missing this on appveyor

@pfitzseb
Copy link
Member

Should probably add a @test LibGit2.isdirty(LibGit2.GitRepo(joinpath(pkgdir,"METADATA"))) == false here. I'll prepare a PR for that soonish.

pfitzseb added a commit to pfitzseb/PkgDev.jl that referenced this issue Sep 30, 2016
pfitzseb added a commit to pfitzseb/PkgDev.jl that referenced this issue Oct 3, 2016
tkelman pushed a commit that referenced this issue Oct 12, 2016
* change git paths to always be / separated
fixes #28.

* fix tests?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants