Versioned webpage using commit directories #101

dhimmel · 2018-01-16T18:18:34Z

Work in progress

Closes #96

Did not use jekyll-redirect-from since it seems it would use the same technique, but would be less versatile. https://stackoverflow.com/questions/9276817/301-redirect-for-site-hosted-at-github#comment57813956_32465659 redirect-template.html based off of https://stackoverflow.com/questions/9276817/301-redirect-for-site-hosted-at-github#comment57813956_32465659

Not tested on case where pathspec v exists

Requires manubot to set variables ci_source.repo_owner and ci_source.repo_name. Hence, upgrade to manubot/manubot@2745659

dhimmel · 2018-02-03T22:03:30Z

Testing this in a manubot instance at https://github.com/dhimmel/code-96. Will cherry-pick additional commits from that repo back to this PR, when things are all working.

Seems to be mostly working. Remaining issues:

ghp-import does not follow symlinks, which we require for the site to work as intended Option to preserve symlinks c-w/ghp-import#70
version directories are missing an index.html file. This should probably be a symlink or renaming of manuscript.html. Alternatively, we could add a README.
SETUP.md: do we want to purge versions pre-fork?
Does webpage.py belong here or as part of the manubot python package?

ghp-import did not push symlinks within directory. However, the versioned directory setup requires symlinks to operate as intended. Refs c-w/ghp-import#37

Uses empty branch technique from a previous Manubot setup script: https://github.com/greenelab/manubot-rootstock/blob/f82ae0dc1a059bce05a8c43c0bf5e6a58edc4f2c/initialize.sh

https://stackoverflow.com/q/44163307

dhimmel · 2018-02-12T21:00:17Z

version directories are missing an index.html file. This should probably be a symlink or renaming of manuscript.html

completed

SETUP.md: do we want to purge versions pre-fork?

Yes. completed

ghp-import does not follow symlinks, which we require for the site to work as intended

This PR now works (see https://github.com/dhimmel/code-96) even though ghp-import doesn't preserve symlinks. If ghp-import implements an option to preserve symlinks, the diffs on the webpage branch would become more simple.

@agitter ready for review

agitter · 2018-02-12T21:46:23Z

ready for review

Okay. I'll likely need a few days to get to this.

dhimmel · 2018-02-13T17:13:39Z

Here's an example commit on what upgrading OTS stamps now looks like dhimmel/code-96@8dae193. Requires a bitcoin full node.

dhimmel · 2018-02-13T20:45:28Z

I created a script https://gist.github.com/dhimmel/5cf98acc58f60ede9504422e7a0a9f41 that will take a legacy gh-pages and create version directories for past commits. This is not required, but allows you to pull out historical versions into their own directories.

agitter

Looks great overall. I only have questions and some minor comments.

agitter · 2018-02-16T13:00:25Z

webpage/README.md

+The [`v`](v) directory contains directories for each manuscript version.
+In general, a version is identified by the commit hash of the source content that created it.
+The `*.ots` files in version directories are OpenTimestamps which can be used to verify manuscript existence at or before a given time.
+[OpenTimestamps](opentimestamps.org) uses the Bitcoin blockchain to attest to file hash existence.


This is being treated as a relative link.

agitter · 2018-02-16T15:05:17Z

ci/deploy.sh

+  --checkout=gh-pages \
+  --version=$TRAVIS_COMMIT
+
+# Generate OpenTimestamps


I want to make sure I understand the timestamping strategy. We still timestamp in the same way without --wait because we want to CI deploy script to terminate quickly. However, we now have the ability to manually update all of the OTS Timestamps whenever we want as you did in dhimmel/code-96@8dae193 without rewriting history. After the upgrade, those timestamps are no longer dependent on the calendar service.

Is that correct? Do we need to document the upgrade instructions somewhere?

Yes your understanding is correct. --wait is not feasible for CI builds as it could take days IIRC. Will document.

agitter · 2018-02-16T15:09:42Z

README.md

@@ -53,6 +53,9 @@ sh build/build.sh
 # when a change is detected.
 sh build/autobuild.sh

+# Configure the webpage directory


This appears to be an optional step for a local build. If sh build/build.sh is still the minimal command needed to build the PDF and HTML manuscripts, we may want to indicate that more explicitly in the script comments or preceding text.

agitter · 2018-02-16T15:34:54Z

webpage/README.md

-+ `*.ots` files are OpenTimestamps which can be used to verify manuscript existence at or before a given time.
-  [OpenTimestamps](opentimestamps.org) uses the Bitcoin blockchain to attest to file hash existence.
+
+The [`v`](v) directory contains directories for each manuscript version.


Do we want v to be a link if it is ignored on master?

manubot#101 (review)

This build is based on 29cd993. This commit was created by the following Travis CI build and job: https://travis-ci.org/greenelab/manubot-rootstock/builds/342477457 https://travis-ci.org/greenelab/manubot-rootstock/jobs/342477458 [ci skip] The full commit message that triggered this build is copied below: Versioned webpage using commit directories (#101) Closes #96

dhimmel · 2018-02-16T20:33:51Z

Note the ots upgrade command added to the documentation in this PR may fail due to opentimestamps/opentimestamps-client#71.

slochower · 2018-02-17T01:22:37Z

I think there may be an issue with this commit. I'll be brief because I have to run, but can fill in details later if the fix is not obvious to you.

I'm running locally with sh build/autobuild.sh and viewing the local HTML output with python -m http.server in webpage/. If I update my content directory, I see the autobuild run correctly and can verify the word count increasing. However, the webpage I see in my browser is not updated. Upon looking closer, it does not seem that the html files in webpage/v are being updated correctly at all. For example, I just added the text "(try 2)" to one of my Markdown files, hit save, and saw the build complete. But if I look for that phrase in my webpage/v directory using find . -name "*.html" -exec grep -Hni "(try 2)" {} \;, I get nothing. Since PDF is built from the webpage, I don't see my changes there, either. I can't be sure which commit caused this, but I think it may be related to this feature.

I can dig deeper later, if you are unable to confirm this behavior. FWIW, here is the output of autobuild.sh:

Retrieving and processing reference metadata
## INFO
Manuscript content parts:
00.front-matter
01.recent-observations
99.back-matter
## INFO
0 unique citations strings extracted from text
0 unique standard citations    
## INFO
requests-cache starting with 0 cached responses
## INFO
requests-cache finished with 0 cached responses
## INFO
Using UTC timezone.
Dating manuscript with the current datetime: 2018-02-17T01:15:54.189811+00:00
## INFO
Reading user-provided templating variables complete:
{}
## INFO
Generated manscript stats:
{
  "reference_counts": {
    "total": 0
  },
  "word_count": 1361
}
Exporting HTML manuscript
Exporting PDF manuscript
WARNING: Ignored `-ms-text-size-adjust: 100%` at 78:5, unknown property.
WARNING: Ignored `-webkit-text-size-adjust: 100%` at 79:5, unknown property.
WARNING: Ignored `-moz-box-sizing: content-box` at 204:5, unknown property.
WARNING: Ignored `-webkit-appearance: button` at 379:5, unknown property.
WARNING: Ignored `cursor: pointer` at 380:5, the property does not apply for the print media.
WARNING: Ignored `cursor: default` at 389:5, the property does not apply for the print media.
WARNING: Ignored `-webkit-appearance: textfield` at 410:5, unknown property.
WARNING: Ignored `-moz-box-sizing: content-box` at 411:5, unknown property.
WARNING: Ignored `-webkit-box-sizing: content-box` at 412:5, unknown property.
WARNING: Ignored `-webkit-appearance: none` at 423:5, unknown property.
WARNING: Invalid or unsupported selector 'button::-moz-focus-inner,
input::-moz-focus-inner ', Unknown pseudo-element: -moz-focus-inner
WARNING: Invalid or unsupported selector '*:not("#mkdbuttons") ', (<FunctionBlock not( … )>, ':not() only accepts a simple selector')
WARNING: Ignored `-webkit-font-smoothing: subpixel-antialiased` at 486:5, unknown property.
WARNING: Ignored `-moz-border-radius: 3px` at 491:5, unknown property.
WARNING: Ignored `-webkit-border-radius: 3px
` at 492:5, unknown property.
WARNING: Ignored `-webkit-font-smoothing: subpixel-antialiased` at 528:5, unknown property.
WARNING: Ignored `cursor: text
` at 529:5, the property does not apply for the print media.
WARNING: Ignored `word-break: break-all` at 733:5, unknown property.
WARNING: Ignored `word-break: break-word` at 734:5, unknown property.
WARNING: Ignored `-webkit-hyphens: auto` at 735:5, unknown property.
WARNING: Ignored `-moz-hyphens: auto` at 736:5, unknown property.
Build complete

dhimmel · 2018-02-17T01:45:59Z

Thanks for the report @slochower.

You need to run the following to touch the webpage directory:

python build/webpage.py

Previously, index.html was a symlink to the output directory. But now index.html requires webpage.py to be updated.

Sorry to break your autobuild workflow. How useful have you found the autobuild script? Since webpage.py accepts arguments, it's not entirely clear to me how we should fix it. Probably just add python build/webpage.py with the default arguments to autobuild.sh.

https://github.com/greenelab/manubot-rootstock/blob/29cd993b0b6b9c41273e306f4c17262ffdbdf546/build/webpage.py#L13-L22

slochower · 2018-02-17T04:53:09Z

Thanks for the quick reply. Interesting change. I agree that we should probably put python build/webpage.py in autobuild.sh and I think the default arguments are fine (at least for me) because when I run autobuild.sh it's usually because I'm trying to do look at something quickly locally.

I mostly use this local workflow when I want to see my rendered document (for example, if I have a bunch of equations and figures). This comes in handy for more informal cases than writing a collaborative manuscript. Sometimes I want to write a complicated email with figures and syntax highlighting and so I start one of these Markdown documents -- but it doesn't make sense to use an existing manubot-rootstock directory for a manuscript and I don't necessarily want to setup a new GitHub repository just for a single document, so I lazily clone manubot-rootstock and just run it in local mode to generate an HTML or PDF document that I can send.

Most autobuild users will want to automatically update the webpage. See discussion at manubot#101 (comment) Update README to better document autobuild.

Most autobuild users will want to automatically update the webpage. See discussion at #101 (comment) Update README to better document autobuild.

This build is based on b7a9de8. This commit was created by the following Travis CI build and job: https://travis-ci.org/greenelab/manubot-rootstock/builds/342797312 https://travis-ci.org/greenelab/manubot-rootstock/jobs/342797313 [ci skip] The full commit message that triggered this build is copied below: autobuild.sh: include webpage.py execution (#113) Most autobuild users will want to automatically update the webpage. See discussion at #101 (comment) Update README to better document autobuild.

Most autobuild users will want to automatically update the webpage. See discussion at manubot/rootstock#101 (comment) Update README to better document autobuild.

dhimmel force-pushed the issue-96 branch 2 times, most recently from a883441 to b18f8b4 Compare January 16, 2018 22:54

dhimmel added 5 commits February 3, 2018 13:02

Structure webpage for commit directories

5bd95bc

Refs manubot#96

Move webpage code to build

a9554fc

Add git checkout webpage/v info

e4fb2f6

Print versions

8150520

dhimmel force-pushed the issue-96 branch from 9482c2f to 8150520 Compare February 3, 2018 18:02

dhimmel added 4 commits February 3, 2018 14:31

Convert webpage.py to a CLI

d0c0be4

Add checkout subprocess call

19fa719

Not tested on case where pathspec v exists

Update Travis deployment

25f5a9b

Add permalink to frontmatter

2141e2e

Requires manubot to set variables ci_source.repo_owner and ci_source.repo_name. Hence, upgrade to manubot/manubot@2745659

dhimmel force-pushed the issue-96 branch from be6668b to 2141e2e Compare February 3, 2018 21:39

dhimmel mentioned this pull request Feb 3, 2018

Option to preserve symlinks c-w/ghp-import#70

Closed

dhimmel added 5 commits February 12, 2018 14:53

pickme: ghp-import restore --follow-links

4d78501

ghp-import did not push symlinks within directory. However, the versioned directory setup requires symlinks to operate as intended. Refs c-w/ghp-import#37

Webpage: use index.html not manuscript.html

1123b2d

Update webpage README.md

3e757e7

Webpage: overwrite v/latest when not a symlink

695ab3b

SETUP: initialize empty gh-pages & output branches

6439dc3

Uses empty branch technique from a previous Manubot setup script: https://github.com/greenelab/manubot-rootstock/blob/f82ae0dc1a059bce05a8c43c0bf5e6a58edc4f2c/initialize.sh

dhimmel force-pushed the issue-96 branch from 72111ff to 6439dc3 Compare February 12, 2018 19:58

dhimmel added 2 commits February 12, 2018 15:26

Fix sequencing bug: checkout v/latest before deleting it

c9c7355

Webpage vesion checkout. Do not stage non-existent files

e936192

https://stackoverflow.com/q/44163307

agitter requested changes Feb 16, 2018

View reviewed changes

Updates based on review comments

9868a9a

manubot#101 (review)

agitter approved these changes Feb 16, 2018

View reviewed changes

dhimmel merged commit 29cd993 into manubot:master Feb 16, 2018

dhimmel deleted the issue-96 branch February 16, 2018 18:50

dhimmel mentioned this pull request Feb 17, 2018

autobuild.sh: include webpage.py execution #113

Merged

dhimmel added a commit that referenced this pull request Feb 17, 2018

autobuild.sh: include webpage.py execution (#113)

b7a9de8

Most autobuild users will want to automatically update the webpage. See discussion at #101 (comment) Update README to better document autobuild.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Versioned webpage using commit directories #101

Versioned webpage using commit directories #101

dhimmel commented Jan 16, 2018 •

edited

Loading

dhimmel commented Feb 3, 2018 •

edited

Loading

dhimmel commented Feb 12, 2018

agitter commented Feb 12, 2018

dhimmel commented Feb 13, 2018

dhimmel commented Feb 13, 2018

agitter left a comment

agitter Feb 16, 2018

agitter Feb 16, 2018

dhimmel Feb 16, 2018 •

edited

Loading

agitter Feb 16, 2018

agitter Feb 16, 2018

dhimmel commented Feb 16, 2018

slochower commented Feb 17, 2018

dhimmel commented Feb 17, 2018 •

edited

Loading

slochower commented Feb 17, 2018

Versioned webpage using commit directories #101

Versioned webpage using commit directories #101

Conversation

dhimmel commented Jan 16, 2018 • edited Loading

dhimmel commented Feb 3, 2018 • edited Loading

dhimmel commented Feb 12, 2018

agitter commented Feb 12, 2018

dhimmel commented Feb 13, 2018

dhimmel commented Feb 13, 2018

agitter left a comment

Choose a reason for hiding this comment

agitter Feb 16, 2018

Choose a reason for hiding this comment

agitter Feb 16, 2018

Choose a reason for hiding this comment

dhimmel Feb 16, 2018 • edited Loading

Choose a reason for hiding this comment

agitter Feb 16, 2018

Choose a reason for hiding this comment

agitter Feb 16, 2018

Choose a reason for hiding this comment

dhimmel commented Feb 16, 2018

slochower commented Feb 17, 2018

dhimmel commented Feb 17, 2018 • edited Loading

slochower commented Feb 17, 2018

dhimmel commented Jan 16, 2018 •

edited

Loading

dhimmel commented Feb 3, 2018 •

edited

Loading

dhimmel Feb 16, 2018 •

edited

Loading

dhimmel commented Feb 17, 2018 •

edited

Loading