Support to make NVIDIA's Caffe fork (for NVIDIA DIGITS) work in Windows #6
Comments
|
@lunzueta I am always happy to help when I can. Here is what I can suggest from the error messages above:
I added a preprocessor definition in the root CMakeLists.txt for this. Make the same change.
This one I don't remember ever seeing this.
You need a define: #define mkdir _mkdir
Same here. You need a define for these two. See msvc.hpp. HTH |
|
Thanks for the hints @willyd. After applying your suggestions, the only compilation error I get now (apart from some linking errors that come in the end which I guess I still can't solve until I fix this), is that related to 'sys/ioctl.h'. I guess you didn't see this before because it seems to be something specific to NVIDIA's fork (in file caffe\src\caffe\parallel.cpp). Any idea about what to do with this? I'll search about what to do for porting this to Windows with the minimal possible changes, but this would be my first time with this file, so any suggestion from somebody more experienced is welcome. |
|
@lunzueta I am not so experienced in porting code from Linux to Windows myself either. So your guess is as good as mine. Google is your friend. |
|
I got a bit stuck with this now. In the parallel.cpp file there are inclusions of two header files which are specific to Linux: sys/ioctl.h and sys/mman.h. If you just skip them and compile the parallel.cpp file (to see which functions rely on those inclusions) you don't get any error but just the following warnings: From this info, I don't see clearly what to do next, but I guess something has to be done instead of just skipping those inclusions. Don't know... Anyway, from this point on, if I compile the whole caffe.vcxproj project, I don't get any error either. The caffe-nv.lib file (caffe.lib in your Windows port, @willyd) is generated in Caffe-prefix\src\Caffe-build\lib\Release folder (caffe-nv-d.lib in Caffe-prefix\src\Caffe-build\lib\Debug folder), but those files aren't automatically copied to the install/lib folder, as it happens with caffe.lib and caffe-d.lib in your port. What should be changed to make the builder copy automatically those files to install/lib? Something related to CMake files? |
|
You need to build the INSTALL project. Equivalent of make install on Linux. |
|
Thanks @willyd, that worked for the copy. Anyway, I see that more changes have to be done to make this work. I've done another fork of NVIDIA's fork to have a version control of the changes, and to be able to share this with others: https://github.com/lunzueta/caffe Anyone interested in collaborating on this, let me know, so that I can give access. |
|
@willyd, I have one question about how to adapt your super builder to make it work with my fork. In the file Caffe.cmake I did the following change: Instead of this: I put this: I'm not sure what GIT_TAG is for, because I still didn't find a good explanation about it on the Internet. In my repository there's not visualstudio branch, so I just removed that. Would that be ok? |
|
@lunzueta GIT_TAG stands for anything you would do: git checkout ${GIT_TAG}so a branch, a commit sha-1 or a tag. You can also remove GIT_TAG altogether so that the default branch (usually master) is checked out. |
|
Ok, thanks. Understood. In my case, for now, it's simpler just to remove it. |
|
At this moment, I'm able to compile NVCaffe from my fork (https://github.com/lunzueta/caffe) in Release mode, except the projects "runtest" and "test.testbin", but I guess I can start testing DIGITS with the compiled executables and libraries. I'll report about these tests when possible. Anyway, I'd like to be able to compile all projects in Release and also in Debug to finish properly this port. The errors I get are the following: runtest and test.testbin projects (Release): pycaffe (Debug): The linking error of pycaffe in Debug is strange because python27_d.lib doesn't appear in the project linking properties as a dependence, but python27.lib. In @willyd's equivalent project this project doesn't have this issue in Debug. Any suggestions? |
You need to
std::max is a template function. It seems MSVC cannot deduce the type it should use. You have to specify it manually like so |
|
Thanks @willyd, the change about python worked. Now pycaffe compiles correctly in Debug too. Regarding the template issue, I'm not sure how to apply what you tell me. The only differences I see in test_gradient_check_util.hpp in Nvidia's fork with respect to yours are the following two:
In Nvidia's fork:
In Nvidia's fork: The rest of the file is exactly the same in both sides. What do you think would be the best to solve this? |
These are my changes not Nvidia's. Weirdly enough github's markdown removes the Something is confusing the compiler type deduction. There are multiple overloads of Dtype scale = std::max(
std::max(fabs(computed_gradient),
fabs(estimated_gradient)), Dtype(1.)); |
|
Thanks @willyd. This last change worked. Now I can compile all the NVCaffe projects in Release and Debug in VS2013 and x64. I guess this is a good moment to start testing Digits with this and see if everything is ok :) |
|
@willyd, while testing the generated NVCaffe executable for training the MNIST data, I've seen that I get this failure, which I don't get with your fork (your fork does this correctly): As you can see it doesn't find the known layer types properly. It's not straightforward to figure out where this behavior can come from. While you were doing your Windows port, did you find this kind of behavior at some point (and fixed it)? |
|
@lunzueta You need to integrate all my changes into your fork if you want the build to work. Because of the way VS works I need to use dumpbin to parse the caffe.lib and generate a header file to force the inclusion of some symbols the linker would get rid of otherwise. Do you have any header called caffe/forcelink.hpp in your build tree? |
|
@willyd It's a bit messy to apply your changes in Nvidia's fork, because of the differences between the original Caffe and Nvidia's fork. Initially, I tried to do this through Git (trying to merge Nvidia's fork with yours somehow), but in the end I think that it's better to do this manually, file by file (using Notepad++ and the compare plugin, any alternative ideas are welcome). Probably I missed something on the way, and I'm checking it again (manually). I didn't know this dumpbin that you mention. Maybe I should use it too or adapt something? How should that be? I don't have any header called caffe/forcelink.hpp but I can't see it either in your fork (only one comment inside src\caffe\CMakeLists.txt). Should I have that somewhere? |
|
@lunzueta I use the dumpbin utility to generate a list of # ---[ Visual Studio specifics
# configure headers for dynamic or static linking on windows
unset(CAFFE_DLLEXPORT)
unset(CAFFE_DLLIMPORT)
unset(CAFFE_INCLUDE_FORCELINK)
if(MSVC)
if(BUILD_SHARED_LIBS)
set(CAFFE_DLLEXPORT __declspec(dllexport))
set(CAFFE_DLLIMPORT __declspec(dllimport))
else()
get_filename_component(forcelink_header_name ${forcelink_header} NAME)
set(CAFFE_INCLUDE_FORCELINK "# include \"caffe/${forcelink_header_name}\"")
# add a post build command to generate a header file that will
# force the linker to include the statically registered layers
# in a final executable.
include(${PROJECT_SOURCE_DIR}/cmake/GenerateForceLinkSymbolsHeader.cmake)
add_generate_force_link_symbols_header_post_build_command(caffe ${forcelink_header} caffe_force_link)
# install the generated file
install(FILES ${forcelink_header} DESTINATION include/caffe)
endif()
endif()You could also forego this process altogether and use the Visual Studio feature called "Use Library Dependency Inputs" if you prefer this. But this conflicts with the CMake build system that will regenerate the projects overwriting any change you make to them via the IDE. I think this is what @happynear uses in his fork, but he doesn't use CMake.
As the file is generated by CMake it should be in the build tree and not in the source tree. It should be in |
|
@willyd Many thanks for the explanation. I also like more the CMake way. Now I see that I have an important difference in that forcelink.hpp file. In your version I see that there are many lines like the following (I only include a few): And in my forcelink.hpp I don't have any of these. I'll review what I missed, but do you have an explanation of what happened? |
|
@willyd @lukeyeager Finally, I've been able to run Nvidia Digits on Windows 10 :-) I could do this directly by using the current Windows version of Caffe (the recently created 'windows' branch on BVLC's Caffe repo, which is based on Microsoft's Caffe's fork), the latest version of Nvidia Digits pulled from Github and that's it! I used PyCharm to setup the dependencies and so on. Apart from this I also had to install Graphviz for Windows and set its bin folder in the 'path' environment variable, so that the nets could be visualized in Digits. Then, I was able to generate a dataset, train correctly a classification model (I tried with LeNet), and then test some images correctly too. More tests could be done, but I guess we could say this compatibility issue, in principle, is solved. |
|
@lunzueta DIGITS can be run in windows long ago since this pull request NVIDIA/DIGITS#199 . |
|
@happynear I thought that having NVIDIA's Caffe fork was a requirement, but I've seen that it isn't. Using the latest Windows version of Caffe (already integrated in BVLC's Caffe as a branch, based on Microsoft's Caffe fork), it's more simple to compile and use it, than before. Hopefully, since this Windows version of Caffe appeared it will be easier to get an official Windows version of Digits. |
|
@lunzueta, there is no difference between willyd's or my caffe-windows repo and the official one. I remember that I just copied the caffe.exe to a specified folder, then everything worked well. |
|
@happynear Ok, I see. Looking at comments in some threads here and there, for me (and for more people as I could see in those comments) it wasn't clear enough what should be done to make Nvidia Digits work on Windows, especially because of the use of Nvidia's Caffe fork, instead of BVLC's. Here (NVIDIA/DIGITS#47) I was requested to prepare a brief guide of what I did to make it work on Windows. I wrote a brief guide, but still some doubts arose. Maybe you can also comment there to clarify the doubts. Thanks for your help. |
|
@lunzueta , I have answered the question now. It seems that you haven't came across the folder and version issue? |
|
@happynear I've just tested to install this in another PC, to see if I forgot to include something in the guide explained in the other thread, but I don't think I missed anything apart from what I already explained there. I have it working in both PCs. I don't know what you mean with the folder and version issue. Let's continue the discussion in the other thread :-P |
|
This discussion was very helpful, |
@lukeyeager, I include you here because of your answer in this issue of DIGITS (NVIDIA/DIGITS#47). I'm not sure where on GitHub would be the best place to include this (here, @willyd's Caffe Windows port, NVIDIA's Caffe fork, DIGITS, or somewhere else), but I guess this could be a good place.
Based on @willyd's great Windows port and builder of Caffe (the most recent of the "unofficial Windows ports" cited in Caffe's Wiki: https://github.com/BVLC/caffe/wiki/Installation), I've done a first (unsuccessful) attempt to make NVIDIA's Caffe fork (https://github.com/NVIDIA/caffe) work in Windows too. I guess this is a necessary step to make NVIDIA DIGITS (https://github.com/NVIDIA/DIGITS) work in Windows, as I've seen that it has some extra files not included in this Caffe Windows port.
For that, in the Caffe.cmake file of Caffe-builder I replaced the repository from which Caffe is downloaded with that of NVIDIA's fork. Then, after compiling in VS2013-64-bit all the projects generated by Caffe-builder with NVIDIA's Caffe fork, the only one that failed was that of Caffe (obviously). So, I started checking what the compiler was reporting and based on that I did the following changes:
After these changes I could solve some of the initial errors but I still get some that I don't know how to solve:
I'm not so experienced in porting so complex projects from Linux to Windows so that's why I got stuck, but my impression is that somebody more experienced could help to accomplish this goal. Can anybody help on this, please?
The text was updated successfully, but these errors were encountered: