How to get Dataverse running on a Windows computer for development

[zhul-pku]

No description provided.

[pdurbin]

As I said on the "Do you want to develop Dataverse on Windows or can you speak for someone who does?" thread I started at https://groups.google.com/d/msg/dataverse-community/Hs9j5rIxqPI/-q54751aAgAJ for now the answer is to use vagrant up on Windows. In 24340e9 I updated the dev guide to mention the use of Vagrant on Windows. The big problem is that if you clone the git repo on Windows, you end up with Windows line endings rather than Unix line endings such that the you have to use dos2unix on the installer and other files or you'll see crazy errors like /usr/bin/perl^M: bad interpreter as I mentioned in f4f0feb when updating the dev guide.

@akio-sone contributed some code for Windows users at https://github.com/akio-sone/windows-tools-for-dataverse_4_or_later but it looks like it hasn't been updated in a while.

I'd like to know if which of the following statements best applied to Windows developers:

• I want to install all the required software (Glassfish Postgres, Solr, etc.) directly on my Windows development computer.
• I want to install Vagrant and VirtualBox on my Windows development computer and have all the required required software (Glassfish Postgres, Solr, etc.) contained within a Vagrant virtual machine.
• I want to install Docker on my Windows development computer and have all the required required software (Glassfish Postgres, Solr, etc.) contained within a Docker image.

If a different statement applies better, that's fine. Please just let us know.

[zhul-pku]

Personally, the second statement best describes my needs.
(I want to install Vagrant and VirtualBox on my Windows development computer and have all the required required software (Glassfish Postgres, Solr, etc.) contained within a Vagrant virtual machine).

In this way, I can keep windows environment for other applications and daily work which I am very familiar with, and at the same time has a virtual linux developing and deploy environment.

For the first statement, it could be a lot of work to modify the codes to fit into windows environment. For the third statement, it could be a little difficult and it seems not necessary now.

[pdurbin]

@zhul-pku thanks. Yes I agree it would be a lot of work to get the installation process to work directly on Windows. By this I mean that the installation process assumes a Linux/Unix environment to run the Perl script which calls a number of Bash scripts.

It's fine, of course, that you find Docker difficult. I passed this along at #3937 (comment) . To be honest, while I've used Docker a bit here and there, I'm much more familiar with Vagrant. But don't let me bias anyone who comes along and answers my question above! 😄

Finally, Mike Mormando had this to say at https://groups.google.com/d/msg/dataverse-community/Hs9j5rIxqPI/OZzNrQpvAgAJ

I'd like very much to work on Windows, a lot of people are quite religious about it, I'm not one of them, but all I own are Windows machines, I do both Java and C#, and I play a game from time to time.

A lot of the developers I know would be lost without Visual Studio. I know it can be had on the Mac now, but it isn't really the same from what the few I know that have tried it have told me.
I'll look up Vagrant.

Thanks
Mike

[pdurbin]

I bumped into @zhul-pku on Monday and it sounds like she's sticking with Windows, as opposed to switching to Mac. She's going to think about which issue she'd like to tackle while she's in town.

[dlmurphy]

@pdurbin and I spent some time today trying to use Docker with Minishift on my Windows 7 laptop today. After lengthy wrestling with at least 4 different errors, I'm comfortable saying that this process is more complicated and fiddly than it's worth for me, though others' mileage may vary. I would love to get a Dataverse dev environment running on my PC but so far I've had very little luck with Vagrant or Docker.

[oscardssmith]

what about the bash subsystem? Does that help?

[pdurbin]

Are there any developers on Windows who would like to try a new Docker-based dev environment? I just created pull request #4168 and I would be happy for people to try it out!

[pdurbin]

I have a Windows laptop on loan from our help desk and in 361de42 I push a new "Windows" page under the dev guide with my findings. In short, vagrant up worked fine when you use this VERY IMPORTANT git trick:

git config --global core.autocrlf input

Also, I tried to follow https://docs.microsoft.com/en-us/windows/wsl/install-win10 to install the new Windows Subsystem for Linux thing but it didn't work. As I wrote in the commit, the "Get" button for Unbuntu was greyed out.

I did all this testing on Windows 10 and I'm dismayed by how when you reboot the thing is will inform you that it failed to install security updates. I don't know. At least Vagrant worked. Phew.

I can make a pull request based on that commit if there is interest. Let me know. Mostly I thought I'd take a quick look at Windows since SpinachCon is on Friday and I'm planning on using #4505 to incorporate feedback.

Any Windows users listening should please let us know what your expectations are for hacking on a Java app from Windows. I'm way out out my depth. At home I use Linux and at work I use a Mac.

[pdurbin]

At standup today the consensus seemed to be that I should go ahead and make a pull request so I just did: #4531

I'll keep this issue in development in case I have time during SpinachCon to mess around with minishift/openshift on Windows. One of the BU students I'm mentoring is using Windows and I think he's had success with minishift.

[pdurbin]

I'm putting this issue into Code Review in https://waffle.io/IQSS/dataverse because I believed I've delivered some incremental progress in pull request #4531 and as far as I understand, the original reporter switched to Mac.

In preparing for SpinachCon 2018 (#4505) I ran through the dev environment instructions on a fresh install of Mac and Windows. I'm feeling pretty good about the Mac and Linux writeup (pull request #4502) but Windows support is still trailing way behind. In pull request #4531 I'm suggesting Vagrant and also offering Minishift as an option. I also ask for more feedback from Windows developers on what they want or expect. It's a whole different world to me.

[pdurbin]

what about the bash subsystem? Does that help?

@oscardssmith I couldn't get it working and mention my struggles in the pull request.

[pdurbin]

I just listened to https://www.allthingsgit.com/episodes/dont_ignore_the_fundamentals.html where @ethomson argues pretty convincingly that rather than asking Windows developers to configure git config --global core.autocrlf input on each of their computers as I recommend at https://github.com/IQSS/dataverse/blob/f52384afda381a2939a8eb96b5c860283726b623/doc/sphinx-guides/source/developers/windows.rst#configure-git-to-use-unix-line-endings a better practice is to add a .gitattributes file in the root of the repo with * text=auto as the content. http://edwardthomson.com/blog/git_for_windows_line_endings.html says

"With this set, Windows users will have text files converted from Windows style line endings (\r\n) to Unix style line endings (\n) when they’re added to the repository."

The blog post doesn't mention what happens when you git clone but it's probably safe to assume that files will come down with Unix line endings. Again, the main problem we're trying to avoid is having our installer (which is in Perl) not run in Vagrant with an error like this:

/tmp/vagrant-shell: ./install: /usr/bin/perl^M: bad interpreter: No such file or directory

I haven't tested a .gitattributes file and it probably isn't worth pulling pull request #4531 out of QA at this point, but I thought I'd mention it as a possible future improvement. I like the idea of having the repo contain the config about line endings rather that each Windows developer making a config change on their computer.

[ethomson]

👋 One thing that I didn't mention in that particular blog post is that .gitattributes gives you more fine-grained control over how your files are handled. For example, if you wanted most of your files to have native line endings (* text=auto) you could also override this on a per-file basis to have a particular set of line endings. eg:

* text=auto
install eol=lf

will ensure that the install file always has Unix-style line endings when checked out.