add changelog from directory builder by wilzbach · Pull Request #186 · dlang/tools

wilzbach · 2016-04-25T01:01:53Z

As discussed in the phobos PR this adds the changelog builder to the tools repo.
I also modified the changed tool to accept an additional output flag.

CyberShadow · 2016-04-28T21:55:19Z

So how does this change http://wiki.dlang.org/DMD_Release_Building#Build_steps ? I don't understand how the changes in changed.d correlate with the new program.

wilzbach · 2016-04-28T22:45:47Z

copy the changelog.dd parts from the dmd/druntime/phobos stable repos to a new dlang.org/changelog/2.068.1.dd

Would be sth. like run rdmd ../phobos/changelog > changelog/2.072.0.dd (or maybe even a Makefile target).

I don't understand how the changes in changed.d correlate with the new program.

Currently the bug issues just gets copy-pastied to the bottom - see e.g. http://dlang.org/changelog/2.071.0.html

I updated the script to build from one file only to add the relative links between the entries - it will be rather easy (just two lines) to change it to "pure github" messages.
I will wait for @MartinNowak for further input & his ideas now ;-)

MartinNowak · 2016-05-01T23:30:10Z

Would be nicer to integrate both tools.

wilzbach · 2016-05-01T23:34:03Z

Would be nicer to integrate both tools.

Sure, but please don't forget the people who just want to test their changelog - they don't need to download all bugzilla fixes.
So

extra flag to disable bugzilla
extra flag to enable bugzilla
seperate scripts

CyberShadow · 2016-05-02T06:29:48Z

they don't need to download all bugzilla fixes

Is that a problem? I've never run changed.

The goal is to run it as part of the dlang.org build anyway, to preview the upcoming changelog.

wilzbach · 2016-05-02T06:38:02Z

Is that a problem? I've never run changed.

The idea was to make the bugzilla part optional/toggleable, s.t.

you (=Phobos contributor) don't have to wait for the changed script to finish (and don't need a working internet connection)
you don't need to have every git repo from dlang cloned (e.g. installer)

CyberShadow · 2016-05-02T06:42:26Z

Yep, it needs to be moved out of the installer repository if it is to be used in day-to-day builds.

How long does the bugzilla query take?

wilzbach · 2016-05-02T06:44:25Z

How long does the bugzilla query take?

Script takes about 15s to run.

MartinNowak · 2016-05-02T08:31:30Z

Yep, it needs to be moved out of the installer repository if it is to be used in day-to-day builds.

How long does the bugzilla query take?

Could be cached, but the query itself should already be very efficient (just querying some rows by primary key).

CyberShadow · 2016-05-02T08:33:51Z

Yeah, 15 seconds isn't too bad.

I suggest merging it with changed then.

Yep, it needs to be moved out of the installer repository if it is to be used in day-to-day builds.

I had misunderstood where the "installer" requirement came from. Making the git/Bugzilla query optional would resolve it.

wilzbach · 2016-10-07T17:14:28Z

changed.d

 It stores its result in DDoc form to a text file.

-Copyright: Dmitry Olshansky 2013.
+Copyright: D Language Foundation 2016.


@DmitryOlshansky are you okay with moving the copyright to the D Language Foundation?

Sure whatever. When I started this script it was a brutal hack to show that automation is possible and easy.

wilzbach · 2016-10-07T17:14:53Z

changed.d

+It is also possible to directly preview the generated changelog file:
+
 ---
-rdmd changed.d --start=2013-01-01 --end=2013-04-01


This wasn't supported anymore.

wilzbach · 2016-10-07T17:16:43Z

changed.d

+        std.getopt.config.passThrough,
+        "output|o", &outputFile,
+        "date", &nextVersionDate,
+        "version", &nextVersionString,


@MartinNowak is passing a CLI flag okay for you?
(Otherwise the new version is only written once in the file - at the Macro section)

Sure, already have the version everywhere.

wilzbach · 2016-10-07T17:19:29Z

changed.d

-    if (args.length != 2)
+    auto outputFile = "./changelog.dd";
+    auto nextVersionString = "LATEST";
+    auto nextVersionDate = "September 19, 2016";


Is there a nice way to get this with Vanilla D? I couldn't find one :/

Not really.

wilzbach · 2016-10-07T17:26:24Z

I suggest merging it with changed then.

Ok I finally got around doing this. It now checks the dmd, druntime and phobos for manual text changelog files in the discussed format (simple Git commit style with the title as single line, followed by a separating empty line and then the detailed description).

I had misunderstood where the "installer" requirement came from. Making the git/Bugzilla query optional would resolve it.

If no arguments are passed, Bugzilla isn't queried (but a friendly message about this is shown).

@CyberShadow If you still want to integrate the pending changelog into DAutoTest something like this will work then:

rdmd changed.d "v2.071.2..upstream/stable" && dmd ../dlang.org/macros.ddoc ../dlang.org/html.ddoc ../dlang.org/dlang.org.ddoc ../dlang.org/doc.ddoc ../dlang.org/changelog/changelog.ddoc changelog.dd -Df../dlang.org/changelog/pending.html

CyberShadow · 2016-10-07T21:28:57Z

@CyberShadow If you still want to integrate the pending changelog into DAutoTest something like this will work then:

Can this be a dlang.org Makefile target, part of the default target?

wilzbach · 2016-10-07T21:34:57Z

Can this be a dlang.org Makefile target, part of the default target?

Yes of course :)
But let's go step by step and first agree on the changelog building here.

MartinNowak

Looks mostly good.
Can you summarize the changelog format again here for reference? It was ddoc with meta header, correct?

MartinNowak · 2016-10-08T22:46:02Z

changed.d

+            changedRepos.each!(x => x.changes.writeTextChangesHeader(w, x.headline));
+
+            if (revRange.length)
+                w.put("$(BR)$(BIG $(RELATIVE_LINK2 bugfix-list, List of all bug fixes and enhancements in D $(VER).))\n\n");


👏 having to add $(VER) myself everytime, started to become really annoying.

MartinNowak · 2016-10-08T22:47:58Z

changed.d

+            // print the overview headers
+            changedRepos.each!(x => x.changes.writeTextChangesHeader(w, x.headline));
+
+            if (revRange.length)


revRange.length reads confusing (range + length), how about !revRange.empty?

MartinNowak · 2016-10-08T22:49:25Z

changed.d

+            revRange.getBugzillaChanges.writeBugzillaChanges(w);
+    }
+
+    w.formattedWrite("$(CHANGELOG_NAV_LAST %s)\n", previousVersion);


There is a small script to fix all the navigation.
https://github.com/dlang/dlang.org/blob/cb4129cb40b430ba5fcb508a8fe2f87fa9fbc2ac/changelog/update_nav.sh
Don't need to fully integrate that functionality for now though.

Ok let's go with the most simple, working version for now and then continue to extend it :)

MartinNowak · 2016-10-08T23:19:40Z

changed.d

+        {
+            // search for raw change files
+            alias Repo = Tuple!(string, "path", string, "headline");
+            auto repos = [Repo("dmd", "Language changes"),


We had a distinction for compiler and language changes.
Examples for noteworthy compiler changes would be a huge speedup or fixed template emission.
Any idea how to support those?

Well initially I suggested to use sth. like YAML for the entries, then we could have our own attribute and use sth. like:

title: Foo Bar category: language-change priority: 4 description: > a long description

Now we already have two attributes category and priority, so I am seriously reconsidering to use YAML. Any opinions on this @MartinNowak @CyberShadow?

MartinNowak · 2016-10-08T23:21:47Z

changed.d

+    bool hideTextChanges = false;
+    string revRange;
+
+    auto helpInformation = getopt(


Please use and provide option help.
See https://dlang.org/phobos/std_getopt.html#.getopt.

(done - see below)

MartinNowak · 2016-10-08T23:27:00Z

changed.d

+                 .map!(repo => Repo(buildPath("..", repo.path, "changelog"), repo.headline))
+                 .filter!(x => x.path.exists)
+                 .map!(x => tuple!("headline", "changes")(x.headline, x.path.readTextChanges.array))
+                 .filter!(x => !x.changes.empty);


Please keep a literal and understandable lambda argument name instead of using x, r for repo would be fine.
Can live w/ the x, but missing names are usually a code smell, b/c it requires the code reader to extract a mental model from raw code, instead of having the writer present one.

MartinNowak · 2016-10-09T11:49:51Z

changed.d

+    ChangelogEntry entry = {
+        title: (cast(string) fileRead[0..firstLineBreak]).strip,
+        description: (cast(string) fileRead[firstLineBreak..$]).strip,
+        basename: filename.baseName.stripExtension // let's be safe here


What does safe mean here? Remove comment?

MartinNowak · 2016-10-09T11:51:49Z

changed.d

                break;

-            case "enhancement":
+                case "enhancement":


wrong formatting

MartinNowak · 2016-10-09T12:01:59Z

One design problem came to my mind. There is no defined order/priority of changelog entries, right now it will just use the folders default order (which is alphabetically on my OS+FS, not sure about others).

We can leave this as a final manual polish step for now.

For the future we might adopt some P1_20161231235959_slug-name.dd scheme. With P1-P5 matching Bugzilla's highest to least priority system. It's a bit too complex to begin with and our changelogs shouldn't be that big in the beginning.

MartinNowak · 2016-10-10T19:27:45Z

For the future we might adopt some P1_20161231235959_slug-name.dd scheme. With P1-P5 matching Bugzilla's highest to least priority system. It's a bit too complex to begin with and our changelogs shouldn't be that big in the beginning.

Let's adopt a color/font-style scheme for the changelog priority instead.
Something for more/less important stuff, and red for deprecations/breakages.
I'll go over the giant 2.072.0 changelog to prioritze stuff and standardize this.

MartinNowak · 2016-11-21T09:41:30Z

Ping, any interest to continue the work on this? Didn't found time for the prioritization, feel free to propose any scheme.

wilzbach · 2016-11-22T19:26:39Z

changed.d

+}

-    return result.data;
+string toString(Month month){


While not necessary, hardcoding the Date seemed lame. Unfortunately Phobos doesn't know anything about the mapping of Dates to Month (at least I couldn't find anything).

wilzbach · 2016-11-22T19:34:16Z

Ping, any interest to continue the work on this? Didn't found time for the prioritization, feel free to propose any scheme.

Thanks for the ping.

Can you summarize the changelog format again here for reference? It was ddoc with meta header, correct?

Added it to the header of the file too.
Yep, it is similar to the git commit format - an entry consists of a title line, a blank separator line and the description, e.g.

Title

My awesome description

Let's adopt a color/font-style scheme for the changelog priority instead.
Something for more/less important stuff, and red for deprecations/breakages.
..
For the future we might adopt some P1_20161231235959_slug-name.dd scheme. With P1-P5 matching Bugzilla's highest to least priority system. It's a bit too complex to begin with and our changelogs shouldn't be that big in the beginning.

I already commented about this in a comment as it seems to me that we need some meta information per entry, so using a slightly "more complex" file format might be better, e.g. YAML is pretty easy to write & parse:

title: An ugly bug
category: language-change
priority: 5
description: >
Write your long description here

We can leave this as a final manual polish step for now.

Yep let's go step by step and go with the simplest working version and then extend it gradually :)

wilzbach · 2016-12-09T16:21:33Z

Ping @MartinNowak @CyberShadow @aG0aep6G @JackStouffer as mentioned above we seem to need some meta information per entry, so using a slightly "more complex" file format might be better, e.g. YAML.
In the original discussion it was rejected due to unneeded complexity, but (1) we need a flexibly and simple file format and (2) this PR is stalling anyways, so if no one shouts __within the next week __ I will change the PR to use YAML as file format for the changelog files because imho it's more important to have this process in place than the actual format.

CyberShadow · 2016-12-09T16:23:47Z

That sounds like a grossly overcomplicated design.

Why not just put each entry in its own file and use the filenames for sorting?

wilzbach · 2016-12-09T16:31:37Z

Why not just put each entry in its own file and use the filenames for sorting?

That would be too simple ;-)
As mentioned above Martin wants at least sorting based on priority and splitting in different categories (e.g. language changes and compiler changes).
Hence I think directly storing the meta information in the file is a lot easier to understand than such a proposed naming scheme:

P1_20161231235959_slug-name.dd scheme. With P1-P5 matching Bugzilla's highest to least priority system.

Probably it would be even P1_language_20161231235959_slug-name.dd :O

wilzbach · 2016-12-09T16:32:37Z

Why not just put each entry in its own file and use the filenames for sorting?

(just for the record: that's exactly what at the moment this PR does)

CyberShadow · 2016-12-09T16:33:28Z

sorting based on priority and splitting in different categories

But not chronological? Then why the timestamp in the filename? And I don't see a timestamp in your proposed YAML format.

CyberShadow · 2016-12-09T16:44:39Z

Probably it would be even P1_language_20161231235959_slug-name.dd :O

IMHO that's preferable to using a serialization language (of which we need a small % of features anyway). Less code is better.

But I defer the decision completely to Martin.

MartinNowak · 2016-12-15T16:40:44Z

Let's adopt a color/font-style scheme for the changelog priority instead.
Something for more/less important stuff, and red for deprecations/breakages.
I'll go over the giant 2.072.0 changelog to prioritze stuff and standardize this.

Sorry, didn't got to that, and didn't clearly state the simpler idea.
If we just use sth. like $(BREAKING_CHANGE my title), $(DEPRECATION my title), $(FEATURE my
title), $(BUGFIX my title) for the titles, that should be good enough for grouping and highlighting.
We can infer components from the repo it's in, the distinction between language and compiler changes is vague at least, and could be nicely solved by moving changelogs for language changes to the dlang.org repo (where the specs live).

As mentioned above Martin wants at least sorting based on priority and splitting in different categories (e.g. language changes and compiler changes).

Nothing people ever say is set in stone, unless they say so. Sometimes it's hard to tell from a written messages whether sth. is just an idea or a well thought through concept, which is a mistake of the writter trying to save time. If things seem unclear, at best get in direct contact to people for clarifications.

wilzbach · 2016-12-16T01:36:59Z

Sorry, didn't got to that, and didn't clearly state the simpler idea.
If we just use sth. like $(BREAKING_CHANGE my title), $(DEPRECATION my title), $(FEATURE my
title), $(BUGFIX my title) for the titles, that should be good enough for grouping and highlighting.

Okay, then this means we stick with the plain "github commit message" format :)
After all if we realize some troubles, we can still change it & we will only realize potential problems, if we are working with the single file changelog files.

So is there anything else blocking this PR?

We can infer components from the repo it's in, the distinction between language and compiler changes is vague at least, and could be nicely solved by moving changelogs for language changes to the dlang.org repo (where the specs live).

I like this idea! The changelog builder didn't check dlang.org yet, so I added that (and tools + installer).

wilzbach · 2016-12-16T02:19:48Z

Changelog directory PRs for the repos:

DMD: dlang/dmd#6325
Druntime: dlang/druntime#1716
Phobos: dlang/phobos#4228
Dlang.org: dlang/dlang.org#1524

PetarKirov · 2016-12-29T09:51:24Z

Ping @wilzbach @MartinNowak, what's blocking this PR?

wilzbach · 2016-12-29T09:58:01Z

Ping @wilzbach @MartinNowak, what's blocking this PR?

Ehm not much - I guess @MartinNowak needs to flip the switch for the initial version?

MartinNowak · 2017-01-04T17:50:08Z

Will do a final review (and merge) later today before branching and building 2.073.0-b0.

MartinNowak · 2017-01-06T21:25:13Z

See wilzbach#1

MartinNowak

Will merge the followup myself.

fixup for #186

MartinNowak · 2017-01-06T23:53:51Z

see #211

This was referenced Apr 25, 2016

generate upcoming changelog dlang/dlang.org#1279

Closed

move changelog to a folder dlang/phobos#4228

Merged

wilzbach force-pushed the add_changelog_dir_builder branch from 8ed255f to cef6a79 Compare April 28, 2016 22:40

wilzbach force-pushed the add_changelog_dir_builder branch from cef6a79 to 0466196 Compare October 7, 2016 14:05

Add changelog builder

a18ebc0

wilzbach force-pushed the add_changelog_dir_builder branch from 0466196 to a18ebc0 Compare October 7, 2016 17:05

Prettify style + cleanup

b0ed7fe

wilzbach commented Oct 7, 2016

View reviewed changes

Added flag to hide manual text file changes

2e10f1b

MartinNowak reviewed Oct 8, 2016

View reviewed changes

MartinNowak reviewed Oct 9, 2016

View reviewed changes

Address review comments

a6eb5bd

wilzbach commented Nov 22, 2016

View reviewed changes

wilzbach assigned MartinNowak Nov 22, 2016

wilzbach mentioned this pull request Dec 15, 2016

Issue 8573 - A simpler Phobos function that returns the index of the … dlang/phobos#4921

Merged

Add dlang.org,installer and tools repos to the changelog builder

dd82e0c

This was referenced Dec 16, 2016

Move changelog to a folder and use single file entries dlang/dmd#6325

Merged

Move changelog to a folder and use single file entries dlang/druntime#1716

Merged

Move changelog to a folder and use single file entries dlang/dlang.org#1524

Merged

MartinNowak approved these changes Jan 6, 2017

View reviewed changes

MartinNowak merged commit 3064d99 into dlang:master Jan 6, 2017

MartinNowak added a commit that referenced this pull request Jan 6, 2017

Merge pull request #211 from MartinNowak/pr/186

b138382

fixup for #186

Uh oh!

Conversation

wilzbach commented Apr 25, 2016

Uh oh!

CyberShadow commented Apr 28, 2016

Uh oh!

wilzbach commented Apr 28, 2016

Uh oh!

MartinNowak commented May 1, 2016

Uh oh!

wilzbach commented May 1, 2016

Uh oh!

CyberShadow commented May 2, 2016

Uh oh!

wilzbach commented May 2, 2016

Uh oh!

CyberShadow commented May 2, 2016

Uh oh!

wilzbach commented May 2, 2016

Uh oh!

MartinNowak commented May 2, 2016

Uh oh!

CyberShadow commented May 2, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wilzbach commented Oct 7, 2016

Uh oh!

CyberShadow commented Oct 7, 2016

Uh oh!

wilzbach commented Oct 7, 2016

Uh oh!

MartinNowak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak Oct 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak Oct 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MartinNowak commented Oct 9, 2016

Uh oh!

MartinNowak commented Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MartinNowak commented Nov 21, 2016

Uh oh!

MartinNowak Oct 8, 2016 •

edited

Loading

MartinNowak Oct 8, 2016 •

edited

Loading

MartinNowak commented Oct 10, 2016 •

edited

Loading

wilzbach commented Dec 9, 2016 •

edited

Loading

MartinNowak commented Jan 6, 2017 •

edited

Loading