Generate manual pages from brlcad-docs asciidoc repo #3
Comments
|
If using Docker is an option, then it is possible to build a Docker container that includes all the tools necessary for generating the docs (Antora, asciidoctor, etc.) Or maybe there are already such containers out there. However I am in favor of building and distributing docs as a separate package (as mentioned here) |
|
We can certainly use Docker, and we have in the past (at least I used it a number of years back), but we've never considered requiring it. In general, we have an expectation that a user can check out a repo, install a few critical tools (like a compiler and cmake), detect system libraries (e.g., X11, FOP, etc), and we nearly provide the rest (e.g., libpng, tcl/tk, libz etc). So in this perspective, we'll either need to 1) add to the list of critical tools, 2) not build docs when tools aren't available (including manual pages?), or 3) somehow provide a few more tools. There are merits to each, but our historic perspective is to do #3, which as noted is complicated here a bit by the fact that asciidoctor is in ruby. |
We don't want this to accidentally activate if -lasan is no longer a requirement someday - BRL-CAD's code isn't ready for it yet. In particular, the LeakSanitizer trips up on the rt_free_rti/rt_clean tangle as well as _db_walk_subtree. The former may need elimination of the rt_uniresource global to properly clean up, and the latter is deep in the raytracing core and will have to be approached very carefully. For convenience, here are the backtraces: 889: Test command: build/bin/gqa "-P" "1" "-Ao" "-p" "ovlp_" "brlcad/src/gtools/tests/../../libged/tests/ged_tests.g" "ovlp" 889: Test timeout computed to be: 1500 889: Trying initial grid spacing: 50 mm 889: Using grid spacing lower limit: 0.005 mm 889: Plotting overlaps to ovlp_overlaps.plot3 889: Processing with grid spacing 50 mm 44 x 48 x 39 889: Summary (50mm grid spacing): 889: list Overlaps: 889: /ovlp/r1 /ovlp/r2 count:2277 dist:1780.3mm @ (-248 350 -903.325) 889: 889: ================================================================= 889: ==672620==ERROR: LeakSanitizer: detected memory leaks 889: 889: Direct leak of 524288 byte(s) in 1024 object(s) allocated from: 889: #0 0x498087 in posix_memalign (build/bin/gqa+0x498087) 889: #1 0x7f9d3ca476c3 in alloc brlcad/src/libbu/malloc.c:137:10 889: #2 0x7f9d3ca478f6 in bu_calloc brlcad/src/libbu/malloc.c:174:12 889: #3 0x7f9d3ca9b25e in bu_ptbl_init brlcad/src/libbu/ptbl.c:45:26 889: #4 0x7f9d40d38be4 in rt_init_resource brlcad/src/librt/prep.c:855:2 889: #5 0x7f9d40d3ba65 in rt_clean_resource brlcad/src/librt/prep.c:1063:5 889: #6 0x7f9d40d3239a in rt_clean brlcad/src/librt/prep.c:1210:6 889: #7 0x7f9d40d30f80 in rt_free_rti brlcad/src/librt/prep.c:170:5 889: #8 0x7f9d380c4573 (<unknown module>) 889: #9 0x7f9d424f2526 in ged_exec brlcad/src/libged/exec.cpp:83:16 889: #10 0x7f9d424fd51b in ged_gqa brlcad/src/libged/exec_mapping.cpp:177:1 889: #11 0x4c8b17 in main brlcad/src/gtools/gqa.c:105:11 889: #12 0x7f9d3c6be0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 889: 889: Direct leak of 524288 byte(s) in 1024 object(s) allocated from: 889: #0 0x498087 in posix_memalign (build/bin/gqa+0x498087) 889: #1 0x7f9d3ca476c3 in alloc brlcad/src/libbu/malloc.c:137:10 889: #2 0x7f9d3ca478f6 in bu_calloc brlcad/src/libbu/malloc.c:174:12 889: #3 0x7f9d3ca9b25e in bu_ptbl_init brlcad/src/libbu/ptbl.c:45:26 889: #4 0x7f9d40d38c65 in rt_init_resource brlcad/src/librt/prep.c:858:2 889: #5 0x7f9d380c0db2 (<unknown module>) 889: #6 0x7f9d424f2526 in ged_exec brlcad/src/libged/exec.cpp:83:16 889: #7 0x7f9d424fd51b in ged_gqa brlcad/src/libged/exec_mapping.cpp:177:1 889: #8 0x4c8b17 in main brlcad/src/gtools/gqa.c:105:11 889: #9 0x7f9d3c6be0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 889: 889: Direct leak of 64 byte(s) in 2 object(s) allocated from: 889: #0 0x498087 in posix_memalign (build/bin/gqa+0x498087) 889: #1 0x7f9d3ca476c3 in alloc brlcad/src/libbu/malloc.c:137:10 889: #2 0x7f9d3ca478f6 in bu_calloc brlcad/src/libbu/malloc.c:174:12 889: #3 0x7f9d40ce3610 in _db_walk_subtree brlcad/src/librt/db_tree.c:1914:6 889: #4 0x7f9d40cdd664 in _db_walk_dispatcher brlcad/src/librt/db_tree.c:1992:2 889: #5 0x7f9d3ca795da in bu_parallel brlcad/src/libbu/parallel.c:574:2 889: #6 0x7f9d40cdc2a2 in db_walk_tree brlcad/src/librt/db_tree.c:2199:5 889: #7 0x7f9d41893a86 in rt_gettrees_muves brlcad/src/librt/tree.c:779:6 889: #8 0x7f9d4189bb3f in rt_gettrees_and_attrs brlcad/src/librt/tree.c:883:12 889: #9 0x7f9d4189bcba in rt_gettree brlcad/src/librt/tree.c:896:11 889: #10 0x7f9d380c0e88 (<unknown module>) 889: #11 0x7f9d424f2526 in ged_exec brlcad/src/libged/exec.cpp:83:16 889: #12 0x7f9d424fd51b in ged_gqa brlcad/src/libged/exec_mapping.cpp:177:1 889: #13 0x4c8b17 in main brlcad/src/gtools/gqa.c:105:11 889: #14 0x7f9d3c6be0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 889: 889: Direct leak of 64 byte(s) in 2 object(s) allocated from: 889: #0 0x498087 in posix_memalign (build/bin/gqa+0x498087) 889: #1 0x7f9d3ca476c3 in alloc brlcad/src/libbu/malloc.c:137:10 889: #2 0x7f9d3ca478f6 in bu_calloc brlcad/src/libbu/malloc.c:174:12 889: #3 0x7f9d40c2a4bc in rt_comb_import5 brlcad/src/librt/comb/comb.c:518:6 889: #4 0x7f9d40c79613 in rt_db_external5_to_internal5 brlcad/src/librt/db5_io.c:958:8 889: #5 0x7f9d40c79e9f in rt_db_get_internal5 brlcad/src/librt/db5_io.c:998:11 889: #6 0x7f9d40ce6b76 in rt_db_get_internal brlcad/src/librt/dir.c:131:9 889: #7 0x7f9d40cd31b4 in db_recurse brlcad/src/librt/db_tree.c:1016:6 889: #8 0x7f9d40ce34ee in _db_walk_subtree brlcad/src/librt/db_tree.c:1898:16 889: #9 0x7f9d40cdd664 in _db_walk_dispatcher brlcad/src/librt/db_tree.c:1992:2 889: #10 0x7f9d3ca795da in bu_parallel brlcad/src/libbu/parallel.c:574:2 889: #11 0x7f9d40cdc2a2 in db_walk_tree brlcad/src/librt/db_tree.c:2199:5 889: #12 0x7f9d41893a86 in rt_gettrees_muves brlcad/src/librt/tree.c:779:6 889: #13 0x7f9d4189bb3f in rt_gettrees_and_attrs brlcad/src/librt/tree.c:883:12 889: #14 0x7f9d4189bcba in rt_gettree brlcad/src/librt/tree.c:896:11 889: #15 0x7f9d380c0e88 (<unknown module>) 889: #16 0x7f9d424f2526 in ged_exec brlcad/src/libged/exec.cpp:83:16 889: #17 0x7f9d424fd51b in ged_gqa brlcad/src/libged/exec_mapping.cpp:177:1 889: #18 0x4c8b17 in main brlcad/src/gtools/gqa.c:105:11 889: #19 0x7f9d3c6be0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16 889: 889: SUMMARY: AddressSanitizer: 1048704 byte(s) leaked in 2052 allocation(s).
Oof. Writing down the details on this one, as I doubt I'll remember...
The start was turning on both the AddressSanitizer and Qt. When doing
so, most programs suddenly began reporting memory leaks. However, the
report was rather cryptic, being four entries similar to this one:
==1262202==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 21 byte(s) in 1 object(s) allocated from:
#0 0x498087 in posix_memalign (build/src/libdm/tests/dm_test+0x498087)
#1 0x7f4e90e53675 in alloc brlcad/src/libbu/malloc.c:129:10
#2 0x7f4e90e53400 in bu_malloc brlcad/src/libbu/malloc.c:167:12
#3 0x7f4e90ee1c02 in bu_strdupm brlcad/src/libbu/str.c:165:17
#4 0x7f4e87dcd3e9 (<unknown module>)
#5 0x7f4e87dcdea5 (<unknown module>)
#6 0x7f4e9650eb89 (/lib64/ld-linux-x86-64.so.2+0x11b89)
I wasn't immediately sure if the unknown module error was related to the
plugin loading, and spent a lot of time trying to find a string memory
issue in dm_init.cpp. After that proved fruitless, other than to
confirm that the error disappeared if i removed the bu_dlclose-ing of
the handles saved at initialization, I began to look for ways to decode
the "unknown module" entries. That led to the following issue:
google/sanitizers#89 which describes the
problem ASan has with dynamic libraries. Comments indicated that it
would see real issues, but won't report which dynamic library they come
from (and there are no plans to fix this anytime soon... grr...). In
fairness, valgrind can see the error too but also has the same reporting
problem; it appears to be ubiquitous.
Fortunately, we have an alternative due to the way our plugin system and
test apps work - we can simply add and remove .so files to the directory
and see how the error reporting changes to zero in on which file(s) are
triggering the problem. Doing so quickly made apparent what should have
been obvious in retrospect - two of the errors each were coming from the
Qt and swrast plugins, which had been off in earlier testing.
Since there was no backtrace beyond the bu_strdupm call itself, and
there were two errors per file, the suspect was the bu_strdup calls
initializing the "char *" names for the fb structures. The C files use
static strings (which is why the non-Qt plugins didn't show the issue),
but C++ doesn't tolerate the type mismatch. The original hack
workaround was just to bu_strdup and create a (char *) string, but as
the leak detectors correctly note this also means there's no way to
clean up the allocated memory.
As far as I can tell there is no reason for these strings to be editable
(char *) strings - the dm container's equivalents are static. This
commit removes any logic assuming if_name is dynamic, and also removes
the bu_strdup hack from Qt and swrast.
|
I think that there is no need to generate the man pages during the compilation, because they can be pre-generated and stored on the For more details see: BRL-CAD/brlcad-docs#48 |
|
I would still vastly prefer a solution that is integrated with our build. A non-self-contained solution implicitly depends on a much more complex set of external tools, and is more prone to breakage or obsolescence - particularly in time scales measured in decades. Something we can integrate with our build directly not only allows us to ensure the complete tool chain remains functional as long as we require it, it also helps to make documentation part of the "core" workflow. If documentation is a separate system, it's just that much easier to ignore it. Sean mentioned translating asciidoctor to C/C++ - it would be interesting to know what that would take (in particular, how much additional functionality beyond just the asciidoctor code itself would need to be translated to achieve a fully functional stand-alone package.) If that is a viable option and we can figure out a way to do it I could see using the external asciidoc tools as an interim solution to let documentation updating proceed, but I would like to have a viable plan in place to eventually get us back to a fully self-contained solution. |
|
I cannot comment on your preferences. However, regarding translation of asciidoctor to C/C++, despite what sean might say or wish, I don't see any chances of it happening. I am not able to do it, and I am not aware of anyone else working (or even thinking) about it. Just letting you know, so that you can make informed decisions. |
|
The other option would be to bootstrap Ruby and use a local copy to drive asciidoctor - if it's simple enough to set up a self-contained environment, the new build system work might make that viable. I'll do some experiments and see what it looks like. |
|
I agree that we want to keep generating the docs as there are numerous validation and integration benefits. That said, I think docs will eventually need to live in a submodule regardless (with the main module pulling it in when configured to build docs) just like our sample db's. Unless we're okay increasing the main repo by a gig, there're already docs not yet added like the new Intro to BRL-CAD, a rendering tutorial, some reports, and overview/showcase presentations, and their assets alone will dominate. They could be offloaded to LFS, but that's not much different than a submodule imo, just not as convenient scripting-wise and doesn't address dependency management. Solution to me has been to turn brlcad-docs into a submodule, give it a CMake build, have it detect antora/asciidoctor/ruby/node.js/etc, and have a build step generate the docs. Yes, that means we will need to pre-install things to make a release, and I think that'll be fine. Windows is the only real thorn there. Devs making releases on windows already have to install MSVC, CMake, NSIS/Wix, so add a couple more to the list. If that's hard to swallow, then we probably should adopt a system like Conan.io or Buckaroo since that's what they specifically do. The idea of manually bootstrapping ourselves is less and less appealing. |
|
If we're going to go with a brlcad-docs submodule, then would the way forward be to set up a separate brlcad-docs github project and establish the new setup there, and eventually replaces the docs dir in the main repo with the submodule when its ready? |
It already exists, that's where thew new stuff already resides, but yes exactly -- the idea would be to replace the doc dir or the doc/docbook dir entirely with it. Main repo logic would just subdir traverse down into it. |
Just be aware that only solves the manpage part. Antora is needed for the rest. Installation is trivial with Node LTS, but that's a different toolchain from what asciidoctor needs. I do think our binary releases should keep shipping all three, man+pdf+html outputs, so the brlcad-docs repo could just have a batch+shell script that actually does bootstrapping before cmake as an interim solution. |
My humble opinion is that we don't need to ship the docs with the binary, if they are available online. With PDF the situation is a bit more complex. Usually the tool asciidoctor-pdf docs/modules/man/pages/1/mged.adoc
mv docs/modules/man/pages/1/mged.pdf pdf/However, the rest of the Antora docs usually use some extra syntax and directives, besides those of asciidoc. For example including images, partials, examples, etc. So, This issue (generating PDF files from Antora docs) seems to be still a work in progress in the Antora community: https://gitlab.com/antora/antora/-/issues/349 |
|
Various Government agencies use BRL-CAD within secured environments without online access. It's a primary use case that we must maintain and support. So, while docs can be in html and hosted online, they have to also ship with BRL-CAD and function offline (i.e., at least readable). I'd come across https://www.npmjs.com/package/@nex/antora-pdf earlier as an Antora plugin, and it looked promising.. PDF isn't as important as manpage and html output -- just nice to have for better offline/printing capability. |
I understand. To summarize:
I will try to check this plugin and see how it works. |
|
I have modified the script build.sh to generate an offline HTML site as well. Regarding dependencies, do you think it is feasible for you to install podman? If you can install Podman, then I believe that I can build a Podman container that includes all the needed dependencies for generating the docs (nodejs, ruby, asciidoctor, etc.) You will just call |
This plugin is still experimental: https://gitlab.com/djencks/antora-pdf/-/tree/master/packages/antora-pdf Besides this, it extends Antora by overriding the default generator (something like: However in the new version of Antora (that is expected to come out soon), I believe that handling of extensions has been improved, such that they can add functionality incrementally, instead of overriding the whole thing. |
By the way, if you try to print a page from the browser (I have tried with Chrome) you get a wonderful preview, which can also be saved as PDF. |
The current Docbook XML files rely on XSLT and/or FOP, and are generated during compilation in the brlcad repo. A new doc management system, held in a separate brlcad-docs repo, uses Asciidoc files and Antora for html generated output and Asciidoctor for manual pages.
A desirable end-state is having docs maintained in a separate brlcad-docs repo, in asciidoc format, but generated during compilation of the brlcad repository (e.g., as a git submodule). This will require proper detection and/or supplying of Antora+Asciidoctor and any other tooling necessary to generate the desired manual page, html, and pdf outputs currently being generated by the Docbook XML system. I likely necessity for this will be CMake detection logic to report necessary tools that must be preinstalled by developers, or providing required dependencies as managed dependencies.
This issue is space for discussion on how to resolve migration issues.
The text was updated successfully, but these errors were encountered: