Windows Support #58

mattmoor · 2017-09-19T22:58:22Z

I've run into issues with dependencies pulled in by skydoc in several repos I collaborate on, you can see a sample error here.

This being clean on Windows is blocking Windows support on at least google/subpar and bazelbuild/rules_python. To be clear: we can comment this stuff out to uncover other issues, but we cannot enable Windows CI or declare support without this working.

@davidstanke @davidzchen @dslomov FYI.

The text was updated successfully, but these errors were encountered:

alexeagle · 2018-03-26T18:38:58Z

This is also a blocker for Angular rules, my failure looks like


INFO: From Compiling external/protobuf/src/google/protobuf/generated_message_reflection.cc [for host]:
--
  | external/protobuf/src/google/protobuf/generated_message_reflection.cc(2389): warning C4506: no definition for inline function 'google::protobuf::Message *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::NewFromPrototype(const GenericType *,google::protobuf::Arena *)'
  | with
  | [
  | GenericType=google::protobuf::Message
  | ]
  | external/protobuf/src/google/protobuf/generated_message_reflection.cc(2389): warning C4506: no definition for inline function 'google::protobuf::Arena *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetArena(GenericType *)'
  | with
  | [
  | GenericType=google::protobuf::Message
  | ]
  | external/protobuf/src/google/protobuf/generated_message_reflection.cc(2389): warning C4506: no definition for inline function 'void *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetMaybeArenaPointer(GenericType *)'
  | with
  | [
  | GenericType=google::protobuf::Message
  | ]
  | ERROR: D:/temp/_bazel_buildkite/d7l7pdbx/external/sassc/BUILD.bazel:4:1: Couldn't build file external/sassc/_objs/sassc/external/sassc/sassc.o: C++ compilation of rule '@sassc//:sassc' failed (Exit 2)
  | external/sassc/sassc.c(10): fatal error C1083: Cannot open include file: 'getopt.h': No such file or directory

jin · 2018-03-26T20:00:07Z

cc @laszlocsomor @spomorski

laszlocsomor · 2018-03-27T08:28:57Z

@alexeagle : what's an easy, simple way to repro?

alexeagle · 2018-04-03T18:05:25Z

@laszlocsomor run the skydoc examples on windows?

alexeagle · 2018-09-11T21:50:29Z

Note, the protobuf library and rules_sass now both succeed on windows. Now it's skydoc itself that fails because it doesn't handle lack of runfiles symlinks:

ERROR: C:/users/alexeagle/projects/rules_nodejs/docs/BUILD.bazel:17:1: Generating Skylark doc for docs (9 files) failed (Exit 1)
Traceback (most recent call last):
  File "c:\users\alexea~1\appdata\local\temp\Bazel.runfiles_qthyx8\runfiles\io_bazel_skydoc\skydoc\main.py", line 330, in <module>
    main(FLAGS(sys.argv))
  File "c:\users\alexea~1\appdata\local\temp\Bazel.runfiles_qthyx8\runfiles\io_bazel_skydoc\skydoc\main.py", line 322, in main
    html_writer = HtmlWriter(writer_options)
  File "c:\users\alexea~1\appdata\local\temp\Bazel.runfiles_qthyx8\runfiles\io_bazel_skydoc\skydoc\main.py", line 211, in __init__
    self.__options.link_ext)
  File "c:\users\alexea~1\appdata\local\temp\Bazel.runfiles_qthyx8\runfiles\io_bazel_skydoc\skydoc\main.py", line 73, in _create_jinja_environment
    loader=jinja2.FileSystemLoader(_runfile_path(TEMPLATE_PATH)),
  File "c:\users\alexea~1\appdata\local\temp\Bazel.runfiles_qthyx8\runfiles\io_bazel_skydoc\skydoc\main.py", line 117, in _runfile_path
    raise AssertionError('Cannot find .runfiles directory.')
AssertionError: Cannot find .runfiles directory.
Target //docs:docs failed to build

laszlocsomor · 2018-09-17T13:13:17Z

@alexeagle , could you tell me your repo's github link and what commit you checked out, so I can repro the same error? Which Bazel version did you use?

alexeagle · 2018-09-18T13:46:17Z

rules_nodejs master
run bazel build //docs
I think I was on Bazel 0.16
Note, this is low priority for me, and possibly we should fix it in the stardoc rewrite rather than here.

filipesilva · 2018-09-21T09:57:10Z

Using Windows 10, Bazel 0.17.1, skydoc at d34c44c, and running bazel build //docs/... on https://github.com/bazelbuild/rules_typescript shows the following log:

$ bazel build //docs/...
INFO: SHA256 (https://github.com/bazelbuild/skydoc/archive/d34c44c3f4102eb94beaf2636c6cf532f0ec1ee8.zip) = 720f61e8ad98b4f62bb80c65ec9a4a27918a91e6e3fdda2d999810b4c8cbf020
INFO: Build options have changed, discarding analysis cache.
INFO: Analysed target //docs:docs (32 packages loaded).
INFO: Found 1 target...
ERROR: D:/work/rules_typescript/docs/BUILD.bazel:3:1: Generating Skylark doc for docs (6 files) failed (Exit 1)
Traceback (most recent call last):
  File "\\?\C:\Users\kamik\AppData\Local\Temp\Bazel.runfiles__u5q6cm5\runfiles\io_bazel_skydoc\skydoc\main.py", line 18, in <module>
    import gflags
  File "\\?\C:\Users\kamik\AppData\Local\Temp\Bazel.runfiles__u5q6cm5\runfiles\gflags_repo\gflags.py", line 1091
    except gflags_validators.Error, e:
                                  ^
SyntaxError: invalid syntax
Target //docs:docs failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 22.451s, Critical Path: 0.97s
INFO: 1 process: 1 local.
FAILED: Build did NOT complete successfully

laszlocsomor · 2018-09-21T13:40:34Z

@alexeagle , thanks, that helped! I could repro the issue and fix it in Skydoc. I'll send a PR shortly. I can now build //docs/... at bazel-contrib/rules_nodejs@62d309a

`skydoc/main.py` now uses Bazel's built-in runfiles library to look up data-dependencies. This commit introduces one hack: we use os.path.dirname (actually: posixpath.dirname) on a path returned by the runfiles library. This is wrong, because the runfiles path does not necessarily correlate with the actual path of the file, i.e. the path that the runfiles-lookup library returns. For example, if //foo:a and //foo:generated-b are both data-dependencies, they'll both be under the "foo" runfiles-directory, but will be in different real paths. This is fine on Linux because Bazel generates a runfiles tree with symlinks, but breaks on Windows because there Bazel only generates a runfiles manifest. Fortunately, for now, all files under `skydoc/templates` are source files, so this hack works. If anyone ever adds generated files to //skydoc/templates:templates, this hack will break. See bazelbuild#58

laszlocsomor · 2018-09-21T14:04:22Z

@filipesilva : With #108 in my local copy of skydoc, and rules_typescript/package.bzl modified to load skydoc from my copy, I can build //docs/... at bazelbuild/rules_typescript@799e364

laszlocsomor · 2018-09-21T14:04:41Z

I'm using Bazel 0.17.1 on Windows 10

laszlocsomor · 2018-09-21T14:08:32Z

@filipesilva : Which Python version are you using? The error in #58 (comment) looks like a python incompatibility issue.

laszlocsomor · 2018-09-21T14:13:44Z

Yes, indeed the Python version is the issue. I can build when I put Python 2 on the PATH, but I get the same error as #58 (comment) with Python 3.

filipesilva · 2018-09-21T15:23:26Z

@laszlocsomor just checked and I am indeed using Python 3.7.0.

Should I be on Python 2? https://docs.bazel.build/versions/master/windows.html#build-python mentions both are supported so I just went with the latest.

laszlocsomor · 2018-09-24T06:49:33Z

@filipesilva : Bazel can use both major versions of Python, but apparently the Skydoc python scripts were written for 2. I'm not happy to say this, but frankly it's probably less likely to see such breakages if you use Python 2. (That said, from now I'll make sure to always use Py 3 for my own builds to ensure all my scripts are compatible.)

* Fix runfile lookup in skydoc/main.py `skydoc/main.py` now uses Bazel's built-in runfiles library to look up data-dependencies. This commit introduces one hack: we use os.path.dirname (actually: posixpath.dirname) on a path returned by the runfiles library. This is wrong, because the runfiles path does not necessarily correlate with the actual path of the file, i.e. the path that the runfiles-lookup library returns. For example, if //foo:a and //foo:generated-b are both data-dependencies, they'll both be under the "foo" runfiles-directory, but will be in different real paths. This is fine on Linux because Bazel generates a runfiles tree with symlinks, but breaks on Windows because there Bazel only generates a runfiles manifest. Fortunately, for now, all files under `skydoc/templates` are source files, so this hack works. If anyone ever adds generated files to //skydoc/templates:templates, this hack will break. See #58

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58

filipesilva · 2018-09-24T13:48:40Z

@laszlocsomor that indeed is unfortunate but personally it makes no difference to use 2.7 or 3.7 so I will use 2.7.

I tried running your bazelbuild/rules_typescript#282 PR locally and it does finish successfully on Windows.

However, CI shows it seems to fail on everything but windows...

ERROR: /home/circleci/ts/docs/BUILD.bazel:3:1: Generating Skylark doc for docs (6 files) failed (Exit 1)

Traceback (most recent call last):
  File "/home/circleci/.cache/bazel/_bazel_circleci/143962326f06c2543998f9d4c1332492/sandbox/processwrapper-sandbox/476/execroot/build_bazel_rules_typescript/bazel-out/host/bin/external/io_bazel_skydoc/skydoc/skydoc.runfiles/io_bazel_skydoc/skydoc/main.py", line 324, in <module>
    main(FLAGS(sys.argv))
  File "/home/circleci/.cache/bazel/_bazel_circleci/143962326f06c2543998f9d4c1332492/sandbox/processwrapper-sandbox/476/execroot/build_bazel_rules_typescript/bazel-out/host/bin/external/io_bazel_skydoc/skydoc/skydoc.runfiles/io_bazel_skydoc/skydoc/main.py", line 317, in main
    html_writer.write(rulesets)
  File "/home/circleci/.cache/bazel/_bazel_circleci/143962326f06c2543998f9d4c1332492/sandbox/processwrapper-sandbox/476/execroot/build_bazel_rules_typescript/bazel-out/host/bin/external/io_bazel_skydoc/skydoc/skydoc.runfiles/io_bazel_skydoc/skydoc/main.py", line 213, in write
    '%s' % CSS_FILE)
  File "/usr/lib/python2.7/zipfile.py", line 1123, in write
    st = os.stat(filename)
OSError: [Errno 2] No such file or directory: '/home/circleci/.cache/bazel/_bazel_circleci/143962326f06c2543998f9d4c1332492/sandbox/processwrapper-sandbox/476/execroot/build_bazel_rules_typescript/bazel-out/host/bin/external/io_bazel_skydoc/skydoc/skydoc.runfile/io_bazel_skydoc/skydoc/sass/main.css'
INFO: Elapsed time: 98.490s, Critical Path: 35.30s

laszlocsomor · 2018-09-24T13:50:21Z

I'm looking at the CI failures now.

Now we can build `//docs/...` in rules_typescript and rules_nodejs on Linux too, where sandboxing is enabled by default and where the previously buggy code path is used. See bazelbuild#58

* Fix off-by-one error in substring creation. Now we can build `//docs/...` in rules_typescript and rules_nodejs on Linux too, where sandboxing is enabled by default and where the previously buggy code path is used. See #58

filipesilva · 2018-09-25T10:05:17Z

@laszlocsomor I tried using the updated skydoc on rules_typescript/rules_nodejs (as a followup to your PRs) and also ran into into the python issue there:

ERROR: D:/build/buildkite-worker-windows-java8-gd7t-1/bazel/rules-typescript-typescript/docs/BUILD.bazel:3:1: Couldn't build file docs/docs-skydoc.zip: Generating Skylark doc for docs (6 files) failed (Exit 1)
--
  | Traceback (most recent call last):
  | File "\\?\D:\temp\Bazel.runfiles_o6rauhqd\runfiles\io_bazel_skydoc\skydoc\main.py", line 19, in <module>
  | import gflags
  | File "\\?\D:\temp\Bazel.runfiles_o6rauhqd\runfiles\gflags_repo\gflags.py", line 1091
  | except gflags_validators.Error, e:

This is the same error I saw above when using Python 3.7, which I guess means https://github.com/bazelbuild/continuous-integration is using that version as well.

Also cc @meteorcloudy since last time something was needed in the CI setup (bazel-contrib/bazel-gazelle#321 (comment)) he took care of it.

meteorcloudy · 2018-09-26T13:23:37Z

Skydoc is still using gflag 2.0, which I suppose doesn't support Python 3.
Let me see if I can upgrade it to a newer version of gflag.

meteorcloudy · 2018-09-26T14:12:43Z

I got this error after upgrading gflag in Skydoc

ERROR: C:/tools/msys64/home/pcloudy/workspace/rules_nodejs/docs/BUILD.bazel:17:1: Generating Skylark doc for docs (9 files) failed (Exit 1)
Traceback (most recent call last):
  File "\\?\C:\Users\pcloudy\AppData\Local\Temp\Bazel.runfiles_6ijdo96u\runfiles\io_bazel_skydoc\skydoc\main.py", line 329, in <module>
    main(FLAGS(sys.argv))
  File "\\?\C:\Users\pcloudy\AppData\Local\Temp\Bazel.runfiles_6ijdo96u\runfiles\io_bazel_skydoc\skydoc\main.py", line 297, in main
    load_symbols = load_sym_extractor.extract(bzl_file)
  File "\\?\C:\Users\pcloudy\AppData\Local\Temp\Bazel.runfiles_6ijdo96u\runfiles\io_bazel_skydoc\skydoc\load_extractor.py", line 110, in extract
    load_symbols = self._extract_loads(bzl_file)
  File "\\?\C:\Users\pcloudy\AppData\Local\Temp\Bazel.runfiles_6ijdo96u\runfiles\io_bazel_skydoc\skydoc\load_extractor.py", line 66, in _extract_loads
    for alias, symbol in kwargs.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'

Looks like Skydoc itself need to fix Python3 compatibility.
And found this issue #57

filipesilva · 2018-09-26T14:22:26Z

@meteorcloudy should the bazel CI move to use Python 2.7 then?

meteorcloudy · 2018-09-26T14:50:20Z

@filipesilva Just confirmed we don't have Python2 installed in Windows slaves on our CI. I think the best way to go here is to make Skydoc Python3 compatible. I can investigate to see how difficult it would be.

/cc @c-parsons

laszlocsomor · 2018-09-27T07:23:40Z

@filipesilva : sorry for late response, I was away. @meteorcloudy , thanks for stepping in!

* Update skydoc repo to d9b3e3b, to fix skydoc #58 As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58 * Use kysdoc@dbb9034 * Trim spaces from sha256

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58 Closes #282 PiperOrigin-RevId: 214941694

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58 Closes bazel-contrib#282 PiperOrigin-RevId: 214941694

laszlocsomor mentioned this issue Sep 21, 2018

Fix runfile lookup in skydoc/main.py #108

Merged

laurentlb closed this as completed in #108 Sep 24, 2018

laszlocsomor added a commit to laszlocsomor/rules_nodejs that referenced this issue Sep 24, 2018

Update skydoc repo to d9b3e3b, to fix skydoc bazel-contrib#58

3933e7d

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58

laszlocsomor mentioned this issue Sep 24, 2018

Update skydoc repo to fix skydoc #58 bazel-contrib/rules_nodejs#336

Merged

laszlocsomor added a commit to laszlocsomor/rules_typescript that referenced this issue Sep 24, 2018

Update skydoc repo to d9b3e3b, to fix skydoc bazelbuild#58

8955d4b

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58

laszlocsomor mentioned this issue Sep 24, 2018

Update skydoc repo to fix skydoc #58 bazelbuild/rules_typescript#282

Closed

laszlocsomor mentioned this issue Sep 24, 2018

Fix off-by-one error in substring creation. #109

Merged

filipesilva mentioned this issue Sep 25, 2018

Update Skydoc to work on Windows bazelbuild/rules_typescript#284

Closed

filipesilva mentioned this issue Sep 25, 2018

Update Skydoc to work on Windows bazel-contrib/rules_nodejs#339

Merged

alexeagle pushed a commit to bazelbuild/rules_typescript that referenced this issue Oct 1, 2018

Update skydoc repo to 8889c43, to fix skydoc #58

f98230b

As a consequence, Bazel on Windows can now build //docs/... See bazelbuild/skydoc#58 Closes #282 PiperOrigin-RevId: 214941694

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows Support #58

Windows Support #58

mattmoor commented Sep 19, 2017

alexeagle commented Mar 26, 2018

jin commented Mar 26, 2018

laszlocsomor commented Mar 27, 2018

alexeagle commented Apr 3, 2018

alexeagle commented Sep 11, 2018

laszlocsomor commented Sep 17, 2018

alexeagle commented Sep 18, 2018

filipesilva commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018 •

edited

Loading

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

filipesilva commented Sep 21, 2018

laszlocsomor commented Sep 24, 2018

filipesilva commented Sep 24, 2018

laszlocsomor commented Sep 24, 2018

filipesilva commented Sep 25, 2018

meteorcloudy commented Sep 26, 2018

meteorcloudy commented Sep 26, 2018 •

edited

Loading

filipesilva commented Sep 26, 2018

meteorcloudy commented Sep 26, 2018

laszlocsomor commented Sep 27, 2018

Windows Support #58

Windows Support #58

Comments

mattmoor commented Sep 19, 2017

alexeagle commented Mar 26, 2018

jin commented Mar 26, 2018

laszlocsomor commented Mar 27, 2018

alexeagle commented Apr 3, 2018

alexeagle commented Sep 11, 2018

laszlocsomor commented Sep 17, 2018

alexeagle commented Sep 18, 2018

filipesilva commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018 • edited Loading

laszlocsomor commented Sep 21, 2018

laszlocsomor commented Sep 21, 2018

filipesilva commented Sep 21, 2018

laszlocsomor commented Sep 24, 2018

filipesilva commented Sep 24, 2018

laszlocsomor commented Sep 24, 2018

filipesilva commented Sep 25, 2018

meteorcloudy commented Sep 26, 2018

meteorcloudy commented Sep 26, 2018 • edited Loading

filipesilva commented Sep 26, 2018

meteorcloudy commented Sep 26, 2018

laszlocsomor commented Sep 27, 2018

laszlocsomor commented Sep 21, 2018 •

edited

Loading

meteorcloudy commented Sep 26, 2018 •

edited

Loading