Consider whether or not to migrate bugs.python.org source code to this repo #2
Comments
|
We have a handful of repositories to consider: https://hg.python.org/tracker/roundup/This is our fork of roundup. It should remain as up to date as possible with upstream at http://hg.code.sf.net/p/roundup/code which is also a mercurial repository. I think it's fine that it remain on hg.python.org until such a time that the upstream repo is compatible with migrating it to GitHub or similar. https://hg.python.org/tracker/python-dev/The templates, extensions, detectors, and modules that customize bugs.python.org. I'm 100% on board with bringing that over to GitHub! Other hosted trackersWe still also host: Jython's trackerhttps://hg.python.org/tracker/jython/ We could similarly migrate the customizations into this repository along side the CPython tracker. Meta trackerhttps://hg.python.org/tracker/meta/ which it sounds like goes away per #1 Setuptools trackerhttps://hg.python.org/tracker/setuptools/ which seems defunct in favor of the issue tracker at https://github.com/pypa/setuptools/issues. Perhaps we should archive/statcify the existing tracker? Roundup's TrackerNot currently source controlled. I'd like to see this migrated into this repository as well. |
|
Making this repo a replacement for https://hg.python.org/tracker/python-dev/ sounds excellent to me, since most of the events that core-workflow folks are going to be interested in should already exist at that level. I filed #4 to cover doing something about the legacy setuptools issue tracker. Running http://bugs.jython.org/ and http://issues.roundup-tracker.org/ out of this repo would also be fine by me, although I'd also ask if it might make more sense to give them their own repos (as that way the webhook triggers for CI and automated redeployments would more naturally align between the repositories and the live sites). |
|
I would prefer to leave python-dev on hg.p.o at least for now. If python-dev is moved here, the other instances should be moved as well, since I often have to apply the exact same patch on all 4 of them. |
|
While maintaining consistent workflows is definitely preferable, we already have a major inconsistency between bugs.python.org, CPython itself, and the rest of the CPython workflow automation that means we can't just send a PR for bugs.python.org the way we can for our other workflow tools, and that creates a significant barrier to contribution for the bugs.python.org instance. It also makes it significantly more difficult to set up CI for bugs.python.org, since we can't just use Travis CI the way we do for CPython and other workflow repos. While there are some rumblings along the lines of "Well maybe we should just stop using bugs.python.org entirely, then", I personally consider that to be a more drastic solution than we actually need. Instead, the lowest impact option we currently have available to mitigate the impact of the workflow inconsistency is to shift where it occurs to be between the bugs.python.org instance repo (which would move to GitHub), and the Roundup fork (which would remain on hg.python.org unless/until Roundup itself migrated to using git, whether on GitHub, GitLab, or somewhere else). This does have flow-on effects for the other Roundup instances that the PSF hosts, and definitely shifts the burden of the inconvenience down the stack to the folks maintaining the rolling fork, but has the major advantage that most bugs.python.org contributions will be able to flow through a simplified version of the contribution process documented in the CPython developer guide. It will only be when folks decide they want/need to modify Roundup itself that they'll need to start looking at Mercurial and non-PR based contribution processes (and even there, we could potentially mitigate the difference by making BitBucket the master copy of the Roundup fork and accepting PRs there, and switching hg.python.org over to being a PSF-hosted mirror). |
|
@ezio-melotti What are the 4 instances? setuptools, jython, b.p.o and what? And are these other instances providing people-power to help support their instances? If the answer is "no" then I think it might be time to ask them to consider migrating to any of the other various free open source hosting platforms that come with an issue tracker rather than have knock-on effects of trying to support them. |
|
b.p.o, meta-tracker, jython, setuptools. AFAIK no one is working on them, but b.p.o and meta are ours and setuptool is not longer used. |
|
http://issues.roundup-tracker.org itself is also hosted on bugs.python.org |
|
While it is hosted on the same machine, it's not part of the 4 instances we maintain, and it's currently maintained by the Roundup devs. |
|
OK, so maybe we should reach out to the Jython devs and let them know we are no longer going to push changes to their fork for them and they will need to have someone do it on their behalf? That will bring the number of instances down to one (assuming this replaces the meta tracker). |
|
I'm not sure that would be the best course of action. The other instances require so little maintenance that they are not a burden (at least to me). For instance, the Jython instance got a dozen of commits in the last 6 years, and those changesets are applied to all the instances without changes. |
|
@ezio-melotti but will it be a burden if we start to get more people involved? It just sounded like it was a burden in your earlier comment is all. I mean if you want to take it on to backport all changes that land in b.p.o's code then I have no objection. |
|
I'm not sure I understand exactly what you mean, so I'll try to summarize the situation and my position. TL;DR imho the vcs is not what's stopping people, we need to improve the docker image so that it's easier to set up a tracker and test things and we need more people that understand Roundup and are able to review patches. Moving to github will also make my life slightly harder for the time being. Regarding our Roundup fork:
Regarding the instances:
Regarding the tasks and where contributors would be useful:
For the future, I think we should follow these steps:
In parallel to this we can also:
|
|
So I'm hearing:
So who wants to help make step 1 happen? And are people okay with helping make that happen on hg.python.org and the current development process there? If anyone answers "yes" to the first question but "no" to the second then please speak up. |
|
Note that the dockerfile is already on GitHub at https://github.com/python/docker-bpo (thanks to @soltysh) and that should be the most recent version (there used to be a couple of clones on BitBucket too, including https://bitbucket.org/ezio_melotti/docker-b.p.o/ and the original version by @introom). Getting out of our fork is something I've been working on since we switched to Mercurial, and we are not far now, but the rest api patch will push us further apart until is backported upstream. |
|
I personally think that as long as this project is in mercurial, without any CI, and without good way to even review patches, then it is not ready to start accepting lots of patches and fixes from new contributors. I've read https://wiki.python.org/moin/TrackerDevelopment , and I still couldn't figure out how to upload a patch, let alone how to help review patches. |
|
None of the issues Ezio raised are the ones I am trying to solve with this proposal: the one I care about is that folks that are already actively contributing to CPython, the main PSF website, and to other core development workflow tools are not able to meaningfully contribute to bugs.python.org maintenance without:
That's a problem, since bugs.python.org is essentially our core workflow engine, and without a straightforward way to fix small niggling problems, folks are unlikely to be motivated to progress towards tackling the bigger structural problems with the current maintenance process - they're more likely to invest their time in working out how to eliminate bugs.python.org from the workflow entirely. |
|
@ncoghlan: What kind of tooling/workflow would you suggest then (other than just GitHub)? In the past (before the switch to GitHub) I wanted to fix things on the python.org website and couldn't figure out how to do it.
In order to create these patches they would still need to figure out how to set up and run the tracker locally, understanding how Roundup works, what to change, where, and how, how to debug it, and how to test their changes. This is IMHO a much higher wall than learning how to use mercurial (learning
The workflow is the same that we used on CPython for years: get a local clone, change the code, create a diff, upload it to the (meta)tracker, have someone review/try the patch, and, if it works, commit and push it. |
|
@ezio-melotti Given @soltysh's work on OpenShift/Kubernetes compatibility, then folks should be able to deploy demo instances from their public forks to the OpenShift Online starter tier, and potentially also to the Google/Azure/Amazon Kubernetes offerings (GKE/AKS/EKS). That's a lot more powerful than only being able to test locally, since it means folks can also share UI changes for review and comment before they deploy them. However, even without that, moving to a PR based workflow means that anyone can pull the PR down and run it up locally in Docker, rather than having to download the patch and apply it manually. (Essentially, you'd gain access to this aspect of the CPython workflow: https://devguide.python.org/gitbootcamp/#editing-a-pull-request-prior-to-merging ) It would even be possible for the CI to do a full local deployment using Docker, and then start setting up some basic automated behavioural tests using something like behave or pytest-bdd. The key stepping stone towards doing that would be to write out the kinds of things that a human should be checking for in a manual test, and then asking whether or not a test script could be checking for those things instead. Getting back to the learning curve question, though, "How to run your own modified copy of the issue tracker" and "How to check if the issue tracker is still working correctly after making changes" are important points, but they're also intrinsically part of the task "I want to make changes to the issue tracker". "How to use our legacy patch-based development workflow" isn't the same kind of learning activity: it's a "We know there are better workflows available, because we migrated CPython to them, but we simply haven't got around to modernising this one yet" task. As such, it's a lot more demotivating as a barrier to contribution than being asked to learn how to run or test the service that you're aiming to work on. As such, rather than attempting to smooth out that part of the learning curve, it makes more sense to take that step out entirely, leaving only the intrinsic barriers to entry (i.e. actually being able to run a test instance). |
|
@soltysh's Dockerfile is excellent. It's very straightforward to get up and running. Docker has improved a great deal on its support of MacOS and Windows in the past year, and installing docker is no longer the same hurdle it once was. One thing that would improve the experience using the Dockerfile would be to include some dummy data that would aid in testing that UI changes are working as expected. IMHO, the good work by @ezio-melotti and others on bugs.python.org would be even more valuable if the source and Dockerfile were all in one place and consistent with the workflows of CPython. |
|
That actually reminds me of something I've set up at work: a pipenv managed "dev environment" type arrangement, where If we updated https://github.com/python/docker-bpo to include Mercurial inside the Docker image, then the initial repository cloning and later updating could be handled as additional helper scripts inside the image. The change review process would still be fairly awkward, though :( |
|
For my 2 cents as someone who has recently tried to contribute code to add logging in with Github, here's some brief thoughts:
|
|
On Thu, 28 Jun 2018 11:17:18 -0700, Ammar Askar ***@***.***> wrote:
* As someone who has done a fair bit of web dev and now had a cursory
look at the bpo source code, it would be fairly involved to add
automated CI testing, even the type proposed above. The project is
fairly outdated and not built with this in mind. With the level of
contributions it currently receives, this is a fairly unreachable
goal. You would really need someone to volunteer a fair bit of time to
make it happen.
Yes, roundup was developed long ago, and has actually held up surprisingly
well, considering that.
roundup is a separate project from bugs.python.org, and there are a few
(not many!) developers working on it in addition to Ezio's occasional
contributions. They talked on the -dev mailing list about a possible
git migration at one point. No idea where they are at now, since
the mailing lists started requiring a "confirm you are still interested"
response and I didn't bother. I still hang out on their irc channel,
though (#roundup on oftc).
All of which is to say that a move of roundup itself to git would
benefit more than just cpython, and possibly also pull in some additional
knowledgeable folks if someone wanted to spearhead it.
That, however, is separate from the *instance* definitions and the
testing of the same. I don't see why testing the b.p.o instance would be
any different a problem than testing any other web site, but I've never
written automated tests for a web site so I really don't know.
…--David
|
Right, when talking about bpo up there, I really mean roundup + the bpo instance as a whole. |
|
I have just discovered the repo https://github.com/psf/bpo-roundup today. Is this now the official codebase for bpo, and it's no longer on mercurial? It would be great if all relevant documentation can be updated. For example, the page https://wiki.python.org/moin/TrackerDevelopment seems outdated. |
|
bpo-roundup is our fork of Roundup. Repository for bugs.p.o's Roundup instance is located at https://github.com/psf/bpo-tracker-cpython |
|
Ok then should we retire this repo (python/bugs.python.org) and transfer all open issues to psf/bpo-tracker-cpython and start pointing people there? It's confusing that we have the two repos right now. |
|
Sounds good to me. I can move issues over to the new repo. Archiving/redirecting it to the new location may require someone from the infra team. |
|
No, I don't think so. I haven't been involved in tracker development
for a while, so I may be out of date, but this is my understanding:
Our roundup setup comes in two pieces: there is our modifications to the
base roundup code (our fork, as Berker mentioned), and there is the code
that constitutes our *instance*, which is a set of template and
code files that implement bugs.python.org on top of the base roundup
code. Those are the two code repositories Berker metioned, IIUC.
It is often hard to tell where a bug or feature needs to be addressed
(the roundup codebase or the bugs.python.org codebase). Therefore
we have always had a "meta tracker" to track bugs related to
bugs.python.org. The repo you are asking about is the replacement
for the old meta-tracker, which was itself a roundup instance.
So no, unless things have changed more than I think, this repo is
doing exactly what it should be doing: holding the issues related
to bugs.python.org, and nothing else (no code). A "repo"
named 'bugs.python.org' is a much more obvious place for people to
post bugs/feature requests than the other two repos, assuming
one could even figure out which one to post to :)
You will note that the two code repos (both under 'psf', not 'python')
do not have any issues, which is as it should be.
…--David
PS: I haven't read back through it, but after a quick scan I think that
if psf/bpo-roundup is a fork of the github repo for roundup being
maintained by the roundup team, then this issue can probably be closed
once the README of this repo is updated accordingly.
|
|
Ezio can correct me if I'm wrong, but I think we are close to ditching our Roundup fork so perhaps we can now combine meta tracker and source code of bugs.p.o instance in the same repo. |
|
Even if you're still keeping Roundup fork, I just think the concept of meta-tracker is less relevant on GitHub. You can always transfer issues to the other repo easily if necessary. Additionally you're able to cross reference other GitHub repo in your PR. Having this repo as the "meta-tracker" right now feels odd to me since us core devs aren't really the maintainers of roundup or b.p.o. If anything perhaps core devs should be added to the https://github.com/psf/bpo-tracker-cpython repo so we can help maintain it. |
|
https://github.com/psf/bpo-roundup contains the Roundup code + our changes in the bugs.python.org branch ("our fork"). https://github.com/psf/bpo-tracker-cpython is the instance used for https://bugs.python.org/ These 3 instances are run on top of the code of the bpo-roundup repo. https://github.com/python/bugs.python.org (this repo) was meant to replace the old meta-tracker. The old meta-tracker tracked issues related to all the other 4 repos (back then there were actually 5 instances, our Roundup fork, and a couple of ancillary repos, but most issues were either about our fork, the b.p.o instance, or affected all the instances). Once we switch to GitHub issues, we won't need the bpo-tracker-cpython repo anymore, but bpo-roundup will have to stay around unless issues.roundup-tracker.org and bugs.jython.org also switch. Merging the issues from this repo into the bpo-roundup repo might be ok then, with the understanding that it will cover both our fork and the two remaining instances. |
(Migrating this proposal from http://psf.upfronthosting.co.za/roundup/meta/issue655 )
When CPython migrated to GitHub, the bugs.python.org code stayed on hg.python.org in order to remain consistent with upstream Roundup.
It would be nice if the bugs.python.org maintenance could be split somehow such that most CPython specific changes could be managed using the same process as other core-workflow tools, even if there were some lower level issues that still required making changes to a modified version of Roundup managed in Mercurial.
The text was updated successfully, but these errors were encountered: