Enhancement: build JSON files and HTML files for the web site

[lcn2]

Scope - About this issue

Every winning directory needs a JSON file that describes the winning YYYY/dir entry.

NOTE: This writeup is subject to changes and revisions as the understanding of this issue evolves / improves. Check back to this note from time to time. When modification are made to this comment, we will try to add a note indicating that the scope of this issue has been updated.

TODO

• Write a tool to generate .entry.json for each YYYY/dir entry.

• Remove .id files when .entry.json is ready

• Write a tool to create .inventory.md files

Once the manifest is stable, write a tool to generate .inventory.md files to hold links to all files in the winner directory.

• Write a tool to create .inventory.html from .inventory.md

NOTE: This issue is now obsolete as the inventory is now at the bottom of the YYYY/dir entry index.html files.

• Rename .inventory.html file to inventory.html

NOTE: See "UPDATE 2a" in comment 1793681758.

NOTE: This issue is now obsolete as the inventory is now at the bottom of the YYYY/dir entry index.html files.

• Come up with a method to inspect and test HTML files in place, before the results of any HTML and/or CSS changes are committed to the repo

See comment 1893176110.

• Improve the ioccc.css

NOTE: Using the existing 2020/ferguson1/index.html as an example, improve the look and feel of rendered HTML.

NOTE: Improving does not mean making it perfect, just fixing a few the more glaring problems with the look and feel of rendered HTML.

See comment 1894666815.

• Create a new issue to improve the manifest in tmp and to maintain the manifest as the files in YYYY/dir entries are changed

See issue #1933 .

NOTE: In tmp/manifest.numbers, improve the inventory_order, display_as, display_via_github, and winners_text fields.

NOTE: Build the other needed files under /tmp after exporting the CSV version (tmp/manifest.csv) of tmp/manifest.numbers

NOTE: Make adjustments to the manifest as per recent file changes.

NOTE: Update all .winner.json files as per the updated manifest.

• Write a new tool to generate the new top level location.html

• Write a new tool to generate YYYY/dir entry index.html files from YYYY/dir entry README.md and .entry.json files

• Write a new tool to re-generate top level winners.html

NOTE: The file is now authors.html and winners.html does a redirect to winners.html.

• Write a new tool to re-generate top level years.html

• Write a new tool to generate year level index.html files from YYYY/dir entry README.md and .year files

• Write a new tool to create the bzip2 tarball for each YYYY/dir entry

• Write a new tool to create the bzip2 tarball for the winners of each year.

• Write a new tool to create the bzip2 tarball for all entries.

• Write a new tool to generate the new top level thanks-for-help.html from thanks-for-help.md

• Write a new tool to generate the new top level CODE_OF_CONDUCT.html from CODE_OF_CONDUCT.md

• Write a new tool to generate the new top level bugs.html from the existing bugs.md

• Write a new tool to generate the new top level README.html from the existing top level README.md

• Write a new tool to generate the top level faq.html from the existing faq.md

• Write a new tool to generate the top level thanks-for-help.html from the existing thanks-for-help.md

• Write a new tool to generate the top level contact.html from the existing contact.md

• Write a new tool to generate the new top level judges.html from a new judges.md

• Write a new tool to generate the new top level news.html from a new news.md

• Write a new tool to generate the top level index.html from a new index.md

• Write a new tool to generate other winner level HTML files from YYYY/dir entry markdown files

NOTE: This deals with winner files other than YYYY/dir entry index.html and YYYY/dir entry README.md files.

• Write a tool to generate the sitemap.xml file

See Sitemaps on Wikipedia.

• Write a new tool to generate all of the above mentioned HTML files and JSON files for the entire site

NOTE: Add to the top level Makefile, special rules to help drive the process to generate all of the above mentioned HTML files and JSON files for the entire site.

• Create a new issue to write a tool that performs validation checks on the JSON files created by this issue

NOTE: This TODO item was moved to issue #2007

Near Final TODO

• Changes references and links from "foo.md" to "foo.html"

When all HTML files exist, it is time to change all eferences and links from "foo.md" to "foo.html".

[xexyl]

You know what's REALLY cool about this one?

We can use OUR json parser jparse to validate the files! That's probably the most useful part of it as a tool and not the library - it can validate JSON!

Now I will reply to the rest of the comment next (perhaps in between replying to other comments as well).

[xexyl]

Oh and please assign the rest of the issues to me as I reply to them (since I know I have to reply first).

[xexyl]

Scope - About this issue

Every winning directory needs a JSON file that describes the winning entry.

It does not need an auth.json file or info.json file though, right ?

NOTE: This writeup is subject to changes and revisions as the understanding of this issue evolves / improves. Check back to this note from time to time. When modification are made to this comment, we will try to add a note indicating that the scope of this issue has been updated.

On this note: would you please (this goes with the other issues) let me know if you updated an issue so that I know to look at it lest I do a major change (as these ARE major) and find out it's no longer valid? It's quite easy to get so involved that one does not think about this (and I often don't think about it anyway esp as our threads do get long with lots to discuss - which is good and I bet you think that it would be better if development was more like that ... I do anyway).

.award.json per winner directory

This JSON file will be similar, but not identical to the .info.json file produced for/by the mkiocccentry repo.

So they should be a dot file then?

While some of the JSON data will be copied over, there will be some key differences between .award.json and ``.info.json`. Foe example, JSON members such as "entry_num" and "tarball" won't apply. And there will need to be a JSON member:

Do we need a .info.json too then ?

"award" : "name of the award"

That makes sense yes.

The manifest won't have a "info_JSON", nor a "auth_JSON" file. Instead a:

"winner_json" : ".winner.json"

will be present.

You refer to the MANIFEST file in each year's directory, right ?

There will need to be a very important JSON members that contains information about the name of the entry. In addition to:

"IOCCC_contest" : "IOCCCXX",
"IOCCC_year" : YYYY,

each entry will need:

Something tells me that some of this can be automated ... that would be helpful for sure.

"dirname" : "name_of_winner_directory",
"entry_name" : "YYYY_dirname",

For example, for 1984/mullender we might have:

"IOCCC_contest" : "IOCCC01",
"IOCCC_year" : 1984,
"dirname" : "mullender",
"entry_name" : "1984_mullender",

Okay indeed I can think that at least some of this can be generated.

If the author wins more than one entry in a year the dir name would be like usual but what about entry name? Just add the number of the entry (for the winner) to the entry_name ?

BTW: The "entry_name" value will correspond to the __ HTML names found in years.html and used as entry links such as the #name in the anchor tag of winners.html.

In other words for 2020 my entries:

• ferguson1
• ferguson2

which might become:

• #ferguson1
• #ferguson2

Right ?

The list of winning authors will for an enter will be given. A JSON member whose value is an array of

"author_list" : [
    "some_author_handle_1",
    "some_author_handle_2",
    "some_author_handle_3"
]

The "some_author_handle_X" JSON strings correspond to the "author_handle" files found in the array if authors in the corresponding .auth.json file of the original submission.

I might just be too tired to process this now. What array do you refer to? Which .auth.json? I thought that just lists the winner per entry? Or am I mixing it up with the new way with IOCCC28 and beyond ?

Each winning directory will have one .award.json file.

NOTE: Later on, the .award.json files will be used by the web site builder to build information and links within the web site.

Sounds good.

We recommend that in comments below, a proposed .award.json template be drafted first, so that the overall format may be discussed and considered, before forming a set of actual .award.json files are created.

Good idea to come up with a template.

author JSON file for each person who authored an IOCCC winner

Each person who wrote at least one winning IOCCC entry will have author JSON file. The name of this file will be their authors "author_handle" (from some corresponding .auth.json file as submitted) followed by .json.

For example, assume that "Cody Boone Ferguson" has an "author_handle" of "ferguson". Then the author JSON file for "Cody Boone Ferguson" would be .".ferguson.json"_.

Now this is for the contest as a whole (more than one year) or just each year? Is it an idea to have one for both?

The contents of the author JSON file will include some of the corresponding "authors" array JSON members from some corresponding .auth.json file.

Okay yes I'm afraid I might just be too tired to parse this right now :( I'll try continuing this but I might have to resume the rest of it later.

Each author JSON file will have a list of one or more awards. There will be a JSON member whose value is a JSON array of values of "entry_name"'s. For example,

"awards" : [
    "2018_ferguson",
    "2020_ferguson1",
    "2020_ferguson2"
],

The will be a top level directory called author_list/ that contain all of the author JSON files.

Above the years I gather?

We recommend that in comments below, a proposed "author JSON file" template be drafted first, so that the overall format may be discussed and considered, before forming a set of actual author JSON files are created.

Good idea with a template here too.

data for the .award.json file and the author JSON files

Data for the .award.json files and the author JSON files will need to be scraped from several sources. One source is the original www.ioccc.org site. Another source is a SQL database that was previously used by a previous IOCCC judge to try and build the original www.ioccc.org site. The SQL database contains, among other things, information about winning entries and authors.

Given a discrepancy between the web site and the SQL database, we recommend assuming that the data in the SQL database is valid.

Here is a zip file of an SQL file containing SQL commands:

IOCCC-data.sql.zip

Here is a zip file pf a SQLite database, presumably built from the above mentioned SQL file:

IOCCC-data.sqlite.zip

Any idea how to deal with this information? I actually wonder if there's a sql to json tool out there but if not any recommended tools to look at it under macOS?

[xexyl]

I've tried to use the sql file and get errors ... was able to fix some but I don't know enough of sql to fix them all.

Going afk for a while .. maybe can work on some of this later but until the issue of not being able to fork this repo (instead of winner) I'm not sure how much good it'll do.

[xexyl]

Given a discrepancy between the web site and the SQL database, we recommend assuming that the data in the SQL database is valid.
Here is a zip file of an SQL file containing SQL commands:
IOCCC-data.sql.zip
Here is a zip file pf a SQLite database, presumably built from the above mentioned SQL file:
IOCCC-data.sqlite.zip

Any idea how to deal with this information? I actually wonder if there's a sql to json tool out there but if not any recommended tools to look at it under macOS?

Of course in my then still tired head I was sillily thinking of the database itself which would make no sense at all. It's the commands which I see you also say.

Nevertheless at least in some systems it seems to have syntax errors. It also seems to with converters I found.

I haven't tried the sqlite file though.

I want to do a bit more before my zoom meeting but my hands are so cold that it's difficult to type (I don't get cold that easily generally except for my extremities which are very often cold). Anyway I have a zoom meeting at the hour and shortly after that I'll probably leave for the day. Hopefully tomorrow I can do a bit more and maybe we can work out the problem of not being able to clone this repo.

If you do have to recreate the repo please keep this one here so we can go back for details I the four issues. Of course tomorrow I need to do some other things that I'm not looking forward to but it's for good reasons and they need to be done anyway. Still I should be able to look at some of this and hopefully with a clearer head. Was a pretty rough night.

[lcn2]

If the SQL files are not directly usable, one my be able to extract the text from the SQL directives in order to build the individual JSON files.

We effectively need to split up the SQL data into individual chunks: some for people who won, some for the each winning entry.

[xexyl]

I think this will be helpful. I also think it might be possible to write a rudimentary C tool that does it.

I will reply to all the comments tomorrow.

[lcn2]

You know what's REALLY cool about this one?

We can use OUR json parser jparse to validate the files! That's probably the most useful part of it as a tool and not the library - it can validate JSON!

Now I will reply to the rest of the comment next (perhaps in between replying to other comments as well).

That was the idea for the general parser. We will create our own style of chkentry (some other name of course) that performed semantic checks too on these new JSON files.

[xexyl]

You know what's REALLY cool about this one?
We can use OUR json parser jparse to validate the files! That's probably the most useful part of it as a tool and not the library - it can validate JSON!
Now I will reply to the rest of the comment next (perhaps in between replying to other comments as well).

That was the idea for the general parser. We will create our own style of chkentry (some other name of course) that performed semantic checks too on these new JSON files.

Makes sense (the first part) - and I recall something like this as well. And the second part sounds like a good idea too.

As for taking the sql file and turning it into json files I have some ideas. Mostly a hack (or perhaps hacks depending on the number of ways).

I'm afraid I still am not clear on everything though so I won't really be doing anything of it today. Even so I hope I can do more tomorrow.

I think I finally have the other issue resolved - until the domains have transferred.

Tomorrow I'll be a bit busy part of the day too - with things that can't be avoided but again for good reasons.

Going for the other threads now.

[xexyl]

Done with that .. now as far as templates for the json files.

Any ideas what each one should look like?

As for the hack I thought of that might allow for generating some of the data from the sql file it would kind of just read the line by pattern (insert into ...) and then parse the data to insert. That part might be more problematic though because json types are different from sql.

Database news

I'm not sure what to do about it. Do we need a json file for it? I guess not?

Database person

The fact I added an alt url for each author means that we probably need another member presumably alt_url but where that belongs I don't know - I guess the .auth.json file. But that doesn't resolve the fact that we need a template for each one. The persons database would of course end up being an array of authors - unless it's meant to be for each entry and not an entire list? What is needed?

Database mirrors

The mirrors it's suggested you don't do that now. But:

INSERT INTO mirrors VALUES(7,'Extraterrestrial','<A HREF="http://www.seti.org">SETI</A> is looking for some sites :-)','','',NULL,NULL,NULL,NULL,1);

is hilarious! Though I am very against trying to find life on other planets as we would just destroy it - just as we would destroy other planets if we went there and I don't think we have that right (not that that would ever stop humans). Still it's a hilarious thought so thank you for the laugh there!

Database judges

The judges one is easy enough once we know what information to include but I'm not sure what file it should be in nor am I sure if we want specific information not there (as noted). Obviously I would have to only have you and Leo.

Database year

As far as years go I guess it would be an array of something like (with the names of the members chosen arbitrarily):

"year": 2020,
"contest_number": 27
"name" : "...",

Or should the contest number be a string for e.g. "27th" ?

Database winners

As far as winners I guess this is different from the persons database in that it just is a list of winners and not their information. Is that right?

What about the fact that anonymous authorship is no longer allowed? I guess the previous anonymous will just be called "anonymous" ?

Database prizes

As for prizes I guess it's also an array of the different information but it seems like it might be for the list of winning entries - not a yearly basis (even if the year is there). Is that right?

Database resource

As for resource: it seems it'll also have to be an array. It seems like it's a more complicated manifest in the .info.json files.

Database alphabet_headers

This is an easy one of course.

General questions / thoughts

Should any not be used?

Should any be merged?

What to do about dead links in any of these? In some cases we might use the Internet Wayback Machine but what if there weren't any captures ?

What format / layout (ordering) do we want?

And the templates: what should each look like? I guess that's the most important thing.

Aside

You might have noticed I went from bottom to top .. I've been known also to read articles backwards with improved reading comprehension. No I can't explain that one. I also read sideways, backwards, diagonally, mirrored and other things as well as being able to (though usually I am too tired to do it) read in blocks and scan pages very very quickly finding information that I need (even if I don't know exactly what I'm looking for). I also open books to where I left off - or where the information is that I'm after. Amongst other things :-) .. definitely some gifts but gifts I love.

... going to do other things now. I can reply later depending on when you reply. Else tomorrow.

I hope you got some sleep! I did leave a quick pull request but it unfortunately went to winner.git even though I tried your idea that I thought would actually work.

Good day!

[lcn2]

See comment 1421609887 for a suggestion on making progress on this issue.

[lcn2]

Every winning directory needs a JSON file that describes the winning entry.
It does not need an auth.json file or info.json file though, right ?

The .auth.json and the .info.json files from a new IOCCC submission will not appear in the final winner directory. Certainly information from those submitted files will be used in generating JSON files for the website, but those particular files will not be used. Moreover, the JSON files this issue is talking about will be slightly different in format. They will have some new values and will omit some of the values that were in the submitted entry JSON files.

[lcn2]

So they should be a dot file then?

The reason why we want to call the per-directory JSON file something that starts with a "." (such as perhaps .winner.json - we are not firm on the exact filename, BTW, just that it will start with ".") is that submitted entries are not allowed to have files that begin with "." other than the JSON files that mkiocccentry creates.

[lcn2]

You refer to the MANIFEST file in each year's directory, right ?

No, that file will likely go away. We refer to some elements in the new per directories JSON file that will contain a manifest similar to how .info.json does for the submitted entries.

[lcn2]

If the author wins more than one entry in a year the dir name would be like usual but what about entry name? Just add the number of the entry (for the winner) to the entry_name ?

Yes.

[lcn2]

In other words for 2020 my entries:

ferguson1
ferguson2
which might become:

#ferguson1
#ferguson2
Right ?

Correct.

[lcn2]

Now this is for the contest as a whole (more than one year) or just each year? Is it an idea to have one for both?

The author handle will be for the contest as a whole.

It is why we ask people submitting new entries if they've won before and if so, what is their handle. This will help us match up a new winning entry with somebody who is previously one so we can add their new win into the list of their previous winners.

[lcn2]

Any idea how to deal with this information? I actually wonder if there's a sql to json tool out there but if not any recommended tools to look at it under macOS?

We don't know. If we were to have to do it, we would take the SQL commands and use an editor to turn the appropriate text into stuff we want for JSON file.

There may be a better way to do this and you're attempting to do something with the SQL commands or the SQL database might be more useful. It depends on how comfortable you are in working with SQL database.

[lcn2]

We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.

[xexyl]

See comment 1421609887 for a suggestion on making progress on this issue.

I'm not yet clear on how that comment will help this one unless you mean that solving this issue will simplify that one. Mind clarifying?

[xexyl]

Every winning directory needs a JSON file that describes the winning entry.
It does not need an auth.json file or info.json file though, right ?

The .auth.json and the .info.json files from a new IOCCC submission will not appear in the final winner directory. Certainly information from those submitted files will be used in generating JSON files for the website, but those particular files will not be used. Moreover, the JSON files this issue is talking about will be slightly different in format. They will have some new values and will omit some of the values that were in the submitted entry JSON files.

Okay. So I guess we should figure out what they should contain then (which I guess is part of the template suggestion).

[xexyl]

So they should be a dot file then?

The reason why we want to call the per-directory JSON file something that starts with a "." (such as perhaps .winner.json - we are not firm on the exact filename, BTW, just that it will start with ".") is that submitted entries are not allowed to have files that begin with "." other than the JSON files that mkiocccentry creates.

Right. That makes sense. Thanks for the reminder.

Does this apply to all json files? And will there be any other dot files ?

[xexyl]

You refer to the MANIFEST file in each year's directory, right ?

No, that file will likely go away. We refer to some elements in the new per directories JSON file that will contain a manifest similar to how .info.json does for the submitted entries.

Ah. Well that's good that there aren't that many then. Also that might change some of at least one of my comments in another thread.

[xexyl]

With commit 4cb0de8, we have disabled forming of the compressed tarballs from make quick_www and make www.

Updated the top level TODOs with some final TODOs that need be be done after all other all other ((top priority)) issues has been completed.

On that note the issue that decides on terms has to be done before #3 can be completed. Nevertheless next week (not tomorrow but hopefully the following days though who knows with the issue I noted yesterday) I hope to address the other things that can be addressed.

[xexyl]

With commit 80f1a1e, the year level compressed tarballs have now been built via:

bin/all-years.sh -v 3 bin/tar-year.sh -v 1 -W

The TODO list at the top has been updated.

As noted the 1984 tarball no longer shows any warnings in macOS. So that issue is resolved too though I don't know that you addressed it. I do wonder what happened .. pretty strange thing. Oh well though: it's resolved now (not that I think it was even a problem).

Afk now .. I guess I might have replied to everything after all but when back I'll check. If not I'll try replying otherwise go on to my things. Might do some pull requests today but I might not: undecided yet.

[xexyl]

QUESTION about updating entries for issue 3

Is there a command I should run whenever I update an entry? For instance earlier today I updated the try script of 2012/konno and the other day I added a try script to 1995/leo. What should I do to regenerate the tarballs? I'm sure you said already or it's in the log but it'd be helpful if you give a quick command here. It might also help if there was a command that does it for every entry. I see a command above for the year level tarballs but then you disabled the www rule from creating the entry tarballs I think.

Please advise when you get a chance. I won't be doing anything else today. Hope all is okay!

[lcn2]

QUESTION about updating entries for issue 3

Is there a command I should run whenever I update an entry? For instance earlier today I updated the try script of 2012/konno and the other day I added a try script to 1995/leo.

The manifest needs to be updated if files are added or renamed or removed.

The make www (if you want to take longer, otherwise make quick_www might run faster but perhaps less exact) will rebuild stuff provided that the .entry.json files are up to date.

Followed by a commit and push of course.

What should I do to regenerate the tarballs?

With commit 4cb0de8 the make files do NOT rebuild tarballs.

We need issue #3 and other mass edits to be finished before we can consider updating tarballs as doing so not would produce needless edits of binary compressed tar blobs.

I'm sure you said already or it's in the log but it'd be helpful if you give a quick command here. It might also help if there was a command that does it for every entry. I see a command above for the year level tarballs but then you disabled the www rule from creating the entry tarballs I think.

Please advise when you get a chance. I won't be doing anything else today. Hope all is okay!

Best wishes.

[xexyl]

QUESTION about updating entries for issue 3

Is there a command I should run whenever I update an entry? For instance earlier today I updated the try script of 2012/konno and the other day I added a try script to 1995/leo.

The manifest needs to be updated if files are added or renamed or removed.

Yes. It definitely needs to be updated again. I will see if I can do that tomorrow. That was my plan but the week has been so exhausting it might be pushed out a bit longer.

The make www (if you want to take longer, otherwise make quick_www might run faster but perhaps less exact) will rebuild stuff provided that the .entry.json files are up to date.

Thanks.

Followed by a commit and push of course.
Of course.

What should I do to regenerate the tarballs?

With commit 4cb0de8 the make files do NOT rebuild tarballs.

Yes. That was my reason for asking. But you make a valid point below.

We need issue #3 and other mass edits to be finished before we can consider updating tarballs as doing so not would produce needless edits of binary compressed tar blobs.

Good point. And in order for #3 to be done another one has to be done too - the finalising terms. I guess we kind of paused that (not intentionally but did anyway). I'll see if I can get to that in the next day or two. That will help move #3 along.

I'm sure you said already or it's in the log but it'd be helpful if you give a quick command here. It might also help if there was a command that does it for every entry. I see a command above for the year level tarballs but then you disabled the www rule from creating the entry tarballs I think.
Please advise when you get a chance. I won't be doing anything else today. Hope all is okay!

Best wishes.

Thanks. It was a fine day. Got new shoes (I have terrible feet) at an ocean town which was the closest place that has such a shoe shop (Birkenstock shoes). These might actually work for me as they mould to the feet (one foot is flat and another is arched and I have other worse problems even besides that). I had these new ones on longer than I have ever been able to have new shoes on without my feet in agony. Will take time to wear them in but might be in luck. Hopefully tomorrow or the next few days I'll be able to do more. I'd also like to work on the main computer and either salvage it or reinstall it. That's a lower priority however.

[lcn2]

With commit 09b8de5 exit code use in bin scripts has been improved.

[xexyl]

With commit 09b8de5 exit code use in bin scripts has been improved.

Thanks for the note. Do you want me to do similar (make consistent) with the update status.json script ?

As far as what I have planned here today: I'm going to update the manifest if nothing else today. I'm afraid it probably will only be that or if I do anything else it won't be much more. Not this moment. I have something I should go do but after that I hope to do it.

[xexyl]

.. actually I will not be updating the manifest until a pending problem is resolved. Or question. I'll ask it in the respective issue.

[lcn2]

With commit 09b8de5 exit code use in bin scripts has been improved.

Thanks for the note. Do you want me to do similar (make consistent) with the update status.json script ?

That is not needed at this point as the JSON format has not been finalized, the script is not integrated into the other tools (no need), not a priority for this issue, and may need to be addressed in a new issue that we are working on forming

[lcn2]

With commit aa70cf8 a number of the TODO items have been completed!

The power of the md2html configuration file driven architecture is proving its worth. :-)

[xexyl]

With commit 09b8de5 exit code use in bin scripts has been improved.

Thanks for the note. Do you want me to do similar (make consistent) with the update status.json script ?

That is not needed at this point as the JSON format has not been finalized, the script is not integrated into the other tools (no need), not a priority for this issue, and may need to be addressed in a new issue that we are working on forming

Well when you do open it please let me know as I'd like to change the script (as I wrote it) to fit the new format (or whatever else).

Then when I reply you can assign it to me. I presume we'll have to discuss things first of course.

[xexyl]

With commit aa70cf8 a number of the TODO items have been completed!

The power of the md2html configuration file driven architecture is proving its worth. :-)

Sounds good!

[xexyl]

Interesting:

$ tar ft archive-2004.tar.bz2 2>&1|grep 'tar: Ignor'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
[...]

many more of those. I presume you don't see it. I wonder what it's about. More specifically I wonder why one of the tarballs has this and the others don't. It was the only one that does it. It created another file too: ._2004 (I think it was that). Just in macOS btw. Haven't tried linux. It doesn't appear to be a problem as far as content goes but it is curious.

UPDATE 0

Exists in rocky linux too.

[lcn2]

With commit 9d3438f the missing HTML files entries (HTML files that
were not index.html) has been added. Also the manifest has been updated to put the entry compressed tarball
and the end of the Primary files list and the entry index.html file at the beginning of the Secondary files.
Added a "Jump to the top" link above the Primary files section of the index.html "#inventory" section.

[lcn2]

Interesting:

$ tar ft archive-2004.tar.bz2 2>&1|grep 'tar: Ignor'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
[...]

many more of those. I presume you don't see it. I wonder what it's about. More specifically I wonder why one of the tarballs has this and the others don't. It was the only one that does it. It created another file too: ._2004 (I think it was that). Just in macOS btw.

The macports version of GNU tar is not macOS aware. For some reason the macports GNU tar is not build for macOS.
This seems to be a macports mis-feature.

Haven't tried linux. It doesn't appear to be a problem as far as content goes but it is curious.

UPDATE 0

Exists in rocky linux too.

The rocky linux, tar on the other hand is correct in issuing warnings about attributes that are common to macOS and that do not apply to rocky linux.

With commit 21c3e89 the warnings seen by rocky linux and the macports mis-featured GNU tar should hopefully be gone.

Please check, @xexyl. Thank you.

[xexyl]

Interesting:

$ tar ft archive-2004.tar.bz2 2>&1|grep 'tar: Ignor'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.quarantine'
[...]

many more of those. I presume you don't see it. I wonder what it's about. More specifically I wonder why one of the tarballs has this and the others don't. It was the only one that does it. It created another file too: ._2004 (I think it was that). Just in macOS btw.

The macports version of GNU tar is not macOS aware. For some reason the macports GNU tar is not build for macOS. This seems to be a macports mis-feature.

Haven't tried linux. It doesn't appear to be a problem as far as content goes but it is curious.

UPDATE 0

Exists in rocky linux too.

The rocky linux, tar on the other hand is correct in issuing warnings about attributes that are common to macOS and that do not apply to rocky linux.

With commit 21c3e89 the warnings seen by rocky linux and the macports mis-featured GNU tar should hopefully be gone.

Please check, @xexyl. Thank you.

It does not work right in the archive/historic directory, no. Not in macOS and not rocky linux either. You're right though. /usr/bin/tar in macOS does work right.

[xexyl]

With commit 9d3438f the missing HTML files entries (HTML files that were not index.html) has been added. Also the manifest has been updated to put the entry compressed tarball and the end of the Primary files list and the entry index.html file at the beginning of the Secondary files. Added a "Jump to the top" link above the Primary files section of the index.html "#inventory" section.

I see that. Thanks! It will help out with working on the manifest too. Actually was it updated? Ah yes. I see that it was. Thanks!

I hope I replied to everything but if not I'll probably see it tomorrow .. or later today maybe. Not sure. What I am sure of is I need to do something that doesn't take thinking and preferably laugh.

[lcn2]

With commit 0f8a0a3 ALL HTML FILES that need to be constructed from markdown data and/or JSON files are being built by bin tools!

Of course, the content of such markdown files needs to be improved. And for formatting of HTML files has become much more important. See issue #2006.

As this issue #4 winds down to a close, issue #2006 has now become a ((top priority)).

[lcn2]

With commit 02296e2 and commit c77ec50 we now built the XML sitemap.

NOTE: We are including in the XML sitemap, all files under archive/historic, bin, inc, and well as authors.
We are note sure if these should be included in the sitemap. We think we might want to do so.

[xexyl]

Made a couple typo fixes but noticed something I'm not sure about ...

# Obfuscate defined:

tr.v. -cated, -cating, -cates.
<BR><BR>
**1a.** To render obscure.<BR>
**1b.** To darken.

But this leaves out the word. One might say it's the section / heading title but it seems not the right way to do about it. But I also can't figure out (now or not I do not know) what tr.v might mean. Ah I wonder .. maybe you mean transitive verb? Even so it'd be better I think to have the word there to properly align with the rest of the line (the -cated, -cating, -cates part). I'm also not sure most people will put together the transitive verb part. I only did because I remembered that v. can mean verb. I'm happy to offer possible alternatives but I'll wait to see what you think.

[xexyl]

With commit 02296e2 and commit c77ec50 we now built the XML sitemap.

NOTE: We are including in the XML sitemap, all files under archive/historic, bin, inc, and well as authors. We are note sure if these should be included in the sitemap. We think we might want to do so.

I think it's a very good idea to do this too.

[xexyl]

With commit 0f8a0a3 ALL HTML FILES that need to be constructed from markdown data and/or JSON files are being built by bin tools!

Of course, the content of such markdown files needs to be improved. And for formatting of HTML files has become much more important. See issue #2006.

As this issue #4 winds down to a close, issue #2006 has now become a ((top priority)).

Thanks! For the taking care of it and also the note about the other issue. Of course other html files should also be checked. That is the ones that are rendered by markdown files - those that are provided by the author shouldn't be touched, of course.

[lcn2]

With commit 2b92c89, more TODO items has been completed.

[xexyl]

With commit 2b92c89, more TODO items has been completed.

Nice.

[lcn2]

Moved the few of the remaining TODOs to issue #2239. As a result this issue is now complete!

[xexyl]

Moved the few of the remaining TODOs to issue #2239. As a result this issue is now complete!

Whee!

I presume though that it would be good for me to still go through many if not all index.html files. Is that correct? But then that issue is open so I guess I don't even need to ask: but I only thought of that after writing this and don't feel like deleting it in iOS.

[lcn2]

Moved the few of the remaining TODOs to issue #2239. As a result this issue is now complete!

Whee!

I presume though that it would be good for me to still go through many if not all index.html files. Is that correct?

Correct. The ones with:

<!-- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -->
<!-- !!! DO NOT MODIFY THIS FILE - This file is generated by a tool !!! -->
<!-- !!! DO NOT MODIFY THIS FILE - This file is generated by a tool !!! -->
<!-- !!! DO NOT MODIFY THIS FILE - This file is generated by a tool !!! -->
<!-- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -->

were generated by some tool from an markdown file.

QUESTION

Should the comment:

<!-- The original markdown content was formed by the tool: bin/gen-top-html.sh -->
<!-- The HTML was converted from markdown by the tool: bin/md2html.sh -->

be extended to also mention the markdown file from which it was formed?