Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add appveyor.yml for Windows CI. Closes GH-76 #77

Merged
merged 33 commits into from
Apr 30, 2017
Merged

Add appveyor.yml for Windows CI. Closes GH-76 #77

merged 33 commits into from
Apr 30, 2017

Conversation

satoshinm
Copy link
Owner

#76

Current status: linking craft.exe fails with undefined references in curl even though curl is found by cmake:

- Found CURL: C:/Tools/curl/lib/libcurl.a (found version "7.54.0") 
[ 25%] Linking C executable craft.exe
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0xae): undefined reference to `_imp__curl_easy_init'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x11a): undefined reference to `_imp__curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x189): undefined reference to `_imp__curl_easy_perform'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1a4): undefined reference to `_imp__curl_easy_getinfo'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1ad): undefined reference to `_imp__curl_easy_cleanup'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x27): undefined reference to `_imp__curl_global_init'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x768): undefined reference to `_imp__curl_global_cleanup'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\craft.dir\build.make:530: recipe for target 'craft.exe' failed
mingw32-make[2]: *** [craft.exe] Error 1
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/craft.dir/all' failed
mingw32-make[1]: *** [CMakeFiles/craft.dir/all] Error 2
Makefile:128: recipe for target 'all' failed
mingw32-make: *** [all] Error 2
Command exited with code 2

@satoshinm
Copy link
Owner Author

satoshinm commented Apr 27, 2017

https://curl.haxx.se/docs/faq.html#Link_errors_when_building_libcur

When building an application that uses the static libcurl library, you must add -DCURL_STATICLIB to your CFLAGS. Otherwise the linker will look for dynamic import symbols. If you're using Visual Studio, you need to instead add CURL_STATICLIB in the "Preprocessor Definitions" section.

If you get linker error like "unknown symbol __imp__curl_easy_init ..." you have linked against the wrong (static) library. If you want to use the libcurl.dll and import lib, you don't need any extra CFLAGS, but use one of the import libraries below. These are the libraries produced by the various lib/Makefile.* files:

 Target: static lib. import lib for libcurl*.dll.
 -----------------------------------------------------------
 MingW: libcurl.a libcurldll.a
 MSVC (release): libcurl.lib libcurl_imp.lib
 MSVC (debug): libcurld.lib libcurld_imp.lib
 Borland: libcurl.lib libcurl_imp.lib

@satoshinm satoshinm changed the title [WIP] Add appveyor.yml for Windows CI. Closes GH-76 Add appveyor.yml for Windows CI. Closes GH-76 Apr 27, 2017
@satoshinm satoshinm changed the title Add appveyor.yml for Windows CI. Closes GH-76 [WIP] Add appveyor.yml for Windows CI. Closes GH-76 Apr 27, 2017
@satoshinm
Copy link
Owner Author

Definining CURL_STATICLIB only changes what fails to link:

[ 25%] Linking C executable craft.exe
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0xad): undefined reference to `curl_easy_init'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x12b): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x143): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x15c): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x174): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x188): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x190): undefined reference to `curl_easy_perform'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1aa): undefined reference to `curl_easy_getinfo'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1b2): undefined reference to `curl_easy_cleanup'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x26): undefined reference to `curl_global_init'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x766): undefined reference to `curl_global_cleanup'
collect2.exe: error: ld returned 1 exit status
CMakeFiles\craft.dir\build.make:530: recipe for target 'craft.exe' failed
mingw32-make[2]: *** [craft.exe] Error 1
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/craft.dir/all' failed
mingw32-make[1]: *** [CMakeFiles/craft.dir/all] Error 2
Makefile:128: recipe for target 'all' failed
mingw32-make: *** [all] Error 2
Command exited with code 2

@satoshinm
Copy link
Owner Author

satoshinm commented Apr 30, 2017

Testing on a Windows 10 VM with the same steps, it can't find the curl library (not in C:\Program Files\CURL\lib). Looking closely at the AppVeyor logs, its picking up some other version seemingly preinstalled on AppVeyor:

-- Found CURL: C:/Tools/curl/lib/libcurl.a (found version "7.49.1") 

AppVeyor is supposed to include curl 7.53.1? (maybe only newer images?) appveyor/ci#1431

@satoshinm
Copy link
Owner Author

Stabbing in the dark, but made some progress: removed the need to download and unpack the curl distribution, instead using AppVeyor's preinstalled C:\Tools\curl. Nonetheless, still same link error.

Without -DCURL_STATICLIB: https://ci.appveyor.com/project/satoshinm/netcraft/build/1.0.58

-- Found CURL: C:/Tools/curl/lib/libcurl.a (found version "7.53.1") 

[ 25%] Linking C executable craft.exe
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0xae): undefined reference to `_imp__curl_easy_init'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x11a): undefined reference to `_imp__curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x189): undefined reference to `_imp__curl_easy_perform'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1a4): undefined reference to `_imp__curl_easy_getinfo'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1ad): undefined reference to `_imp__curl_easy_cleanup'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x27): undefined reference to `_imp__curl_global_init'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x775): undefined reference to `_imp__curl_global_cleanup'
collect2.exe: error: ld returned 1 exit status

With -DCURL_STATICLIB: https://ci.appveyor.com/project/satoshinm/netcraft/build/1.0.61

-- Found CURL: C:/Tools/curl/lib/libcurl.a (found version "7.53.1") 

[ 25%] Linking C executable craft.exe
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0xad): undefined reference to `curl_easy_init'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x12b): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x143): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x15c): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x174): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x188): undefined reference to `curl_easy_setopt'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x190): undefined reference to `curl_easy_perform'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1aa): undefined reference to `curl_easy_getinfo'
CMakeFiles\craft.dir/objects.a(auth.c.obj):auth.c:(.text+0x1b2): undefined reference to `curl_easy_cleanup'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x26): undefined reference to `curl_global_init'
CMakeFiles\craft.dir/objects.a(main.c.obj):main.c:(.text.startup+0x766): undefined reference to `curl_global_cleanup'

Library linkage is missing somewhere.

@satoshinm
Copy link
Owner Author

satoshinm commented Apr 30, 2017

The headers in C:\Tools\curl\include are fine, but unsure about the libraries.. they are .a files, but the compiler output shows .obj files instead of .o, suggesting the expected library input would be .lib instead of .a? Windows-style extensions? Is there a .lib file for curl, or how can the .a file be consumed? This is all that's on AppVeyor's image in C:\Tools\curl:

 Volume in drive C is Windows
 Volume Serial Number is D4AB-4044
 Directory of C:\Tools\curl
03/23/2017  02:34 AM    <DIR>          .
03/23/2017  02:34 AM    <DIR>          ..
02/24/2017  07:50 AM    <DIR>          bin
02/24/2017  07:50 AM                66 BUILD-HOMEPAGE.url
02/24/2017  07:50 AM               299 BUILD-README.txt
02/24/2017  07:50 AM           185,520 CHANGES.txt
08/30/2014  10:41 PM             1,928 COPYING-libssh2.txt
02/26/2017  10:00 AM             1,179 COPYING-nghttp2.txt
01/15/2017  05:29 PM             5,302 COPYING-zlib.txt
02/21/2017  07:09 AM             1,110 COPYING.txt
02/24/2017  07:50 AM    <DIR>          docs
02/24/2017  07:50 AM    <DIR>          include
02/24/2017  07:50 AM    <DIR>          lib
02/16/2017  11:58 AM             6,251 LICENSE-openssl.txt
11/07/2016  09:18 AM            19,214 mk-ca-bundle.pl
11/07/2016  09:18 AM             1,659 README.txt
02/24/2017  07:49 AM             1,421 RELEASE-NOTES.txt
              11 File(s)        223,949 bytes
 Directory of C:\projects\netcraft
File Not Found
 Directory of C:\Tools\curl\bin
02/24/2017  07:50 AM    <DIR>          .
02/24/2017  07:50 AM    <DIR>          ..
02/24/2017  07:50 AM           261,889 curl-ca-bundle.crt
02/24/2017  07:50 AM         3,533,824 curl.exe
02/24/2017  07:50 AM           756,736 libcurl.dll
               3 File(s)      4,552,449 bytes
 Directory of C:\projects\netcraft
File Not Found
 Directory of C:\Tools\curl\lib
02/24/2017  07:50 AM    <DIR>          .
02/24/2017  07:50 AM    <DIR>          ..
02/24/2017  07:50 AM           733,392 libcurl.a
02/24/2017  07:50 AM            37,568 libcurldll.a
               2 File(s)        770,960 bytes
 Directory of C:\projects\netcraft
File Not Found
 Directory of C:\Tools\curl\include
02/24/2017  07:50 AM    <DIR>          .
02/24/2017  07:50 AM    <DIR>          ..
02/24/2017  07:50 AM    <DIR>          curl
               0 File(s)              0 bytes
               3 Dir(s)  32,208,166,912 bytes free

Found this: http://stackoverflow.com/questions/12107885/libcurl-lib-where-can-i-get-it#12492652 - refers to http://curl.haxx.se/download/libcurl-7.19.3-win32-ssl-msvc.zip as containing libcurl.lib. But this link is 404. Latest downloads: https://curl.haxx.se/download.html - where is win32-ssl-msvc?

https://curl.haxx.se/mail/lib-2010-08/0094.html refers to https://curl.haxx.se/latest.cgi?curl=win32-ssl-devel-msvc but it says "The recent-version-off-a-mirror system has no info about your requested package "win32-ssl-devel-msvc"! :-( This is most likely because there is no up-to-date release for "win32-ssl-devel-msvc".".

How about without ssl? win32-devel-msvc same error: https://curl.haxx.se/latest.cgi?curl=win32-devel-msvc

There are various Windows downloads here, some labeled SSL, maybe they renamed the files: https://curl.haxx.se/download.html#Win32. Win32 - Generic, Win32 zip 7.54.0 binary SSL SSH looks promising, but 404's: http://www.paehl.com/open_source/?CURL_7.54.0. There is a 7z: https://bintray.com/artifact/download/vszakats/generic/curl-7.54.0-win32-mingw.7z - this one downloads, requires 7zip to extract, but it contains the same lib/libcurl.a and lib/libcurldll.a files.

Is the problem msvc vs mingw? Can't find a libcurl.lib (msvc) anywhere, should try to get to build with libcurl.a (mingw). msvc (Microsoft Visual Studio 2013) support was added by merging this pull request: fogleman/Craft#139 "Visual Studio 2013 has C99 support which allows Craft to be built successfully using the MSVC compiler." - just adding some conditionals and fixes. Why is cmake -G "MinGW Makefiles" trying to use msvc? (or is it? idk enough about Windows to be sure)

@satoshinm
Copy link
Owner Author

%PATH%:

C:\MinGW\bin
C:\Tools\curl\bin
C:\Perl\site\bin
C:\Perl\bin
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\Program Files\7-Zip
C:\Program Files\Microsoft\Web Platform Installer\
C:\Tools\GitVersion
C:\Tools\PsTools
C:\Program Files\Git LFS
C:\Program Files (x86)\Subversion\bin
C:\Program Files\Microsoft SQL Server\120\Tools\Binn\
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\
C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\
C:\Program Files\Microsoft SQL Server\120\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\
C:\Tools\WebDriver
C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.4\
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PrivateAssemblies\
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI\wbin
C:\Ruby193\bin
C:\Tools\NUnit\bin
C:\Tools\xUnit
C:\Tools\MSpec
C:\Tools\Coverity\bin
C:\Program Files (x86)\CMake\bin
C:\go\bin
C:\Program Files\Java\jdk1.8.0\bin
C:\Python27
C:\Program Files\nodejs
C:\Program Files (x86)\iojs
C:\Program Files\iojs
C:\Users\appveyor\AppData\Roaming\npm
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\
C:\Program Files (x86)\MSBuild\14.0\Bin
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120
C:\Tools\NuGet
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow
C:\Program Files\Microsoft DNX\Dnvm
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\
C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\
C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\
C:\Program Files\Microsoft SQL Server\130\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\
C:\Program Files (x86)\Apache\Maven\bin
C:\Python27\Scripts
C:\Tools\NUnit3
C:\Program Files\Mercurial\
C:\Program Files (x86)\Yarn\bin
C:\Program Files\LLVM\bin
C:\Program Files\dotnet\
C:\Program Files\erl8.3\bin
C:\Tools\curl\bin
C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code
C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager
C:\Program Files\Amazon\AWSCLI\
C:\Program Files\Git\cmd
C:\Program Files (x86)\nodejs\
C:\ProgramData\chocolatey\bin
C:\Users\appveyor\AppData\Local\Yarn\.bin
C:\Users\appveyor\AppData\Roaming\npm

@satoshinm
Copy link
Owner Author

satoshinm commented Apr 30, 2017

What compiler were the libraries in C:\tools\curl\lib built with, mingw? http://stackoverflow.com/questions/42663089/c-adding-static-library-to-cmake-project-results-in-undefined-references demonstrates an error caused by trying to link Visual Studio libraries into a mingw application. Try downloading the mingw libraries? I am going to try building locally.


I can replicate the failure locally as follows: download and extract https://bintray.com/artifact/download/vszakats/generic/curl-7.54.0-win32-mingw.7z to C:\Program Files\CURL (note requires using 7zip-compatible tool, not included in Windows), cmake -G "MinGW Makefiles", then mingw32-make. Can now investigate locally instead of cluttering up AppVeyor.


Whoa.. if I add add_definitions(-DCURL_STATICLIB) on my local Windows 10 VM, I get a bunch of different linker errors, about not finding OpenSSL symbols. Haven't seen that on AppVeyor.


This issue occurs with fogleman/Craft, not specific to NetCraft. There is an issue on it: fogleman/Craft#144 which says to download and build curl from source (same cmake -G "MinGW Makefiles" and mingw32-make in the curl directory, then Craft).

@satoshinm
Copy link
Owner Author

satoshinm commented Apr 30, 2017

Attempt at building curl on my clean Windows 10 VM locally:

cmake -G "MinGW Makefiles"
-- The C compiler identification is GNU 5.3.0
-- Check for working C compiler: C:/MinGW/bin/gcc.exe
-- Check for working C compiler: C:/MinGW/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning at CMakeLists.txt:49 (message):
  the curl cmake build system is poorly maintained.  Be aware


-- curl version=[7.54.0]
CMake Warning at CMakeLists.txt:224 (message):
  Found no *nroff program


CMake Error at C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find Perl (missing: PERL_EXECUTABLE)
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:377 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.8/Modules/FindPerl.cmake:74 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:228 (find_package)

Seriously, I need to install Perl? but, curl does build fine on AppVeyor (they have Perl).

@satoshinm
Copy link
Owner Author

Got past the curl trouble by not using curl, but another challenge arises (looks like its compiling all the glfw examples? probably don't need these):

[ 25%] Building C object deps/glfw/examples/CMakeFiles/boing.dir/boing.c.obj
[ 26%] Building RC object deps/glfw/examples/CMakeFiles/boing.dir/glfw.rc.obj
C:\MinGW\bin\windres.exe: invalid option -- s
Usage: C:\MinGW\bin\windres.exe [option(s)] [input-file] [output-file]
 The options are:
  -i --input=<file>            Name input file
  -o --output=<file>           Name output file
  -J --input-format=<format>   Specify input format
  -O --output-format=<format>  Specify output format
  -F --target=<target>         Specify COFF target
     --preprocessor=<program>  Program to use to preprocess rc file
     --preprocessor-arg=<arg>  Additional preprocessor argument
  -I --include-dir=<dir>       Include directory when preprocessing rc file
  -D --define <sym>[=<val>]    Define SYM when preprocessing rc file
  -U --undefine <sym>          Undefine SYM when preprocessing rc file
  -v --verbose                 Verbose - tells you what it's doing
  -c --codepage=<codepage>     Specify default codepage
  -l --language=<val>          Set language when reading rc file
     --use-temp-file           Use a temporary file instead of popen to read
                               the preprocessor output
     --no-use-temp-file        Use popen (default)
  -r                           Ignored for compatibility with rc
  @<file>                      Read options from <file>
  -h --help                    Print this help message
  -V --version                 Print version information
FORMAT is one of rc, res, or coff, and is deduced from the file name
extension if not specified.  A single file name is an input file.
No input-file is stdin, default rc.  No output-file is stdout, default rc.
C:\MinGW\bin\windres.exe: supported targets: pe-i386 pei-i386 elf32-i386 elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex
deps\glfw\examples\CMakeFiles\boing.dir\build.make:86: recipe for target 'deps/glfw/examples/CMakeFiles/boing.dir/glfw.rc.obj' failed
mingw32-make[2]: *** [deps/glfw/examples/CMakeFiles/boing.dir/glfw.rc.obj] Error 1
CMakeFiles\Makefile2:245: recipe for target 'deps/glfw/examples/CMakeFiles/boing.dir/all' failed
mingw32-make[1]: *** [deps/glfw/examples/CMakeFiles/boing.dir/all] Error 2
Makefile:128: recipe for target 'all' failed
mingw32-make: *** [all] Error 2

@satoshinm satoshinm changed the title [WIP] Add appveyor.yml for Windows CI. Closes GH-76 Add appveyor.yml for Windows CI. Closes GH-76 Apr 30, 2017
@satoshinm
Copy link
Owner Author

satoshinm commented Apr 30, 2017

Finally got Windows builds on CI working reasonably well (caveat: no curl), only took until (AppVeyor build) version 1.0.100!

screen shot 2017-04-29 at 9 38 48 pm

@satoshinm satoshinm merged commit 877698c into master Apr 30, 2017
@satoshinm satoshinm deleted the appveyor branch April 30, 2017 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant