GCC Git hooks

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

GCC Git hooks

Jason Merrill
At Cauldron this weekend Joel offered to adjust his git hooks
(https://github.com/brobecke/git-hooks), which are already used by gdb
and glibc, to meet GCC's needs.  Separately, Joseph volunteered to
deal with converting the gcc-www repository to git and dealing with
those hooks.

I expect that Joel's hooks should work fine for gcc-www with minimal
changes, but that the main GCC repository will need more work for
bugzilla integration.

The GCC SVN hooks initially look like pretty vanilla SVN hooks using
svnmailer (http://opensource.perlig.de/svnmailer/); the customized
part of the svnmailer.conf is just

[libstdcxx]
for_paths = .*(libstdc..-v3)/.*
to_addr = [hidden email]

[java]
for_paths = .*(boehm-gc|fastjar|gcjx|gcc/java|libffi|libjava|zlib)/.*
to_addr = [hidden email]

[gccdefault]
to_addr = [hidden email]
bugzilla_to_addr = [hidden email]

Pretty short...but the last line relies on Daniel's custom
bugzilla/svnmailer integration (attached below), and it looks like
Joel's hooks don't have anything comparable.  Any thoughts about
adjusting Daniel's bugzilla.py vs. pulling in something like
http://www.theoldmonk.net/gitzilla/ ?

Jason

bugzilla.py (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joseph Myers
On Sat, 14 Sep 2019, Jason Merrill wrote:

> At Cauldron this weekend Joel offered to adjust his git hooks
> (https://github.com/brobecke/git-hooks), which are already used by gdb
> and glibc, to meet GCC's needs.  Separately, Joseph volunteered to
> deal with converting the gcc-www repository to git and dealing with
> those hooks.

I also volunteered to look at automated means of identifying local fixes
for cvs2svn artifacts in the main repository, that could be applied as
part of any conversion.  (Primarily that's for misplaced branchpoints -
something that I find actually confusing in practice when trying to work
out which changes were on an old branch or to find when a branch was
created in order to locate relevant mailing list discussion.  There are
however other cvs2svn artifacts, less confusing in practice, that it might
also be possible to fix in an automated way.)

reposurgeon has some logic to identify and fix some such issues, but some
preliminary checks on an old partial conversion with an old reposurgeon
version indicate it's far from complete as regards the misplaced
branchpoints in the GCC repository, so I think our own logic for
identifying such fixes (commits with bad parents and appropriate
replacement parent commits) will be useful for any repository conversion
(either directly to produce a list of such fixes for the conversion
process, or indirectly to inform improvements to the logic in reposurgeon,
and for verification of the results).

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Gerald Pfeifer
In reply to this post by Jason Merrill
On Sat, 14 Sep 2019, Jason Merrill wrote:
> Separately, Joseph volunteered to deal with converting the gcc-www
> repository to git and dealing with those hooks.

This is great, thank you!

I was absolutely going to join Cauldron this week, but something personal
is consuming me all of this month (and I am also off work), so I wanted
to at least write a note with some thoughts, and this would have been one
of the items -- converting wwwdocs without waiting for anything else --
... and then I got swamped the last days.

Thank you for picking this up, and for volunteering Joseph!

Let me know how I can help, and I will do my best.  (Even if I'll be
offline most of the next three weeks, I can get online most days and
spend some time.)

Yeah. :)

Gerald
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joseph Myers
On Sun, 15 Sep 2019, Gerald Pfeifer wrote:

> Let me know how I can help, and I will do my best.  (Even if I'll be
> offline most of the next three weeks, I can get online most days and
> spend some time.)

Apart from general review of the test conversion / conversion and hook
scripts, which everyone can do, I think the main thing would be to advise
on what needs to happen to avoid breaking the www.gnu.org copy of the site
and your HTML validation system for commits.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Gerald Pfeifer
On Sun, 15 Sep 2019, Joseph Myers wrote:
> Apart from general review of the test conversion / conversion and hook
> scripts, which everyone can do, I think the main thing would be to advise
> on what needs to happen to avoid breaking the www.gnu.org copy of the site
> and your HTML validation system for commits.

I reached out to the GNU webmasters for the former.  

On the latter, let's just flip the switch whenever you're ready, and I'll
see to adjust my script within a day or two and manually perform checks
until that's done.  (If you can create a sample e-mail notification for
me, I'll use that to start looking into it.)

Thanks,
Gerald
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joel Brobecker
In reply to this post by Jason Merrill
Hello everyone,

On Sat, Sep 14, 2019 at 04:53:17PM -0400, Jason Merrill wrote:

> At Cauldron this weekend Joel offered to adjust his git hooks
> (https://github.com/brobecke/git-hooks), which are already used by gdb
> and glibc, to meet GCC's needs.  Separately, Joseph volunteered to
> deal with converting the gcc-www repository to git and dealing with
> those hooks.
>
> I expect that Joel's hooks should work fine for gcc-www with minimal
> changes, but that the main GCC repository will need more work for
> bugzilla integration.
>
> The GCC SVN hooks initially look like pretty vanilla SVN hooks using
> svnmailer (http://opensource.perlig.de/svnmailer/); the customized
> part of the svnmailer.conf is just
>
> [libstdcxx]
> for_paths = .*(libstdc..-v3)/.*
> to_addr = [hidden email]
>
> [java]
> for_paths = .*(boehm-gc|fastjar|gcjx|gcc/java|libffi|libjava|zlib)/.*
> to_addr = [hidden email]
>
> [gccdefault]
> to_addr = [hidden email]
> bugzilla_to_addr = [hidden email]
>
> Pretty short...but the last line relies on Daniel's custom
> bugzilla/svnmailer integration (attached below), and it looks like
> Joel's hooks don't have anything comparable.  Any thoughts about
> adjusting Daniel's bugzilla.py vs. pulling in something like
> http://www.theoldmonk.net/gitzilla/ ?
Looking at the configuration file, I believe the git-hooks
should have most, if not all, of the features that would be needed for
the GCC repository. In particular, there is already a way to relay
commits to third-party tools via calling of a script; GDB uses that
to interface with their Bugzilla database.

But before I say more I should point to the documentation which,
for historical reasons, is available on the GDB wiki rather than
the git-hooks GitHub repository. I will fix that, but in the meantime:

    https://sourceware.org/gdb/wiki/GitHooksUsersGuide

I'm attaching a file called project.config, which shows the current
configuration for the GDB repository, as it is might be a good
starting point for GCC's configuration.

Of interest:

  * We can see that "hooks.mailinglist" is pointing to a script.
    The purpose of that script is to determine, based on which files
    changed, which mailinglists should the commit email be sent to.

  * There is a hooks.style_checker line. Currently, GDB points to
    a script which does nothing. If GCC has some scripts they want
    to be run to validate the contents of a file, this is where
    this can be done. There is no mechanism, really, to say "don't
    run the style_checker", but it's easy to just pass a null
    style_checker as done by GDB.

  * For bugzilla integration, GDB uses the hooks.file-commit-cmd
    file. I'm attaching the email-to-bugzilla script, although
    I don't know how useful it will be for GCC. It predates the git-hooks
    and I just re-used it as is when we switched over to the git-hooks.

Given that, it seems like the git-hooks might be ready to support
all the needs of the GCC repository? We would need to:

  - write a script that determines the list of recipients based
    on the list of files being changed; that should be a trivial
    adaptation of the script used on the GDB side;

  - Adapt the script filing the commit with bugzilla

  - create a refs/meta/config "branch", and add the project.config
    file with the git-hooks configuration.

I can definitely help with the configuration setup phase in whatever
capacity you'd like. I would recommend people take a look at the list
of options currently available to see what kind of initial configuration
we might want to start with.

--
Joel

project.config (1K) Download Attachment
email_to.py (1K) Download Attachment
email-to-bugzilla (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joel Brobecker
In reply to this post by Gerald Pfeifer
Hi Gerald,

On Mon, Sep 16, 2019 at 12:15:57AM +0800, Gerald Pfeifer wrote:

> On Sun, 15 Sep 2019, Joseph Myers wrote:
> > Apart from general review of the test conversion / conversion and hook
> > scripts, which everyone can do, I think the main thing would be to advise
> > on what needs to happen to avoid breaking the www.gnu.org copy of the site
> > and your HTML validation system for commits.
>
> I reached out to the GNU webmasters for the former.  
>
> On the latter, let's just flip the switch whenever you're ready, and I'll
> see to adjust my script within a day or two and manually perform checks
> until that's done.  (If you can create a sample e-mail notification for
> me, I'll use that to start looking into it.)

You mean the email notification sent by the hooks when a commit
gets pushed? If yes, here is an example:

https://www.sourceware.org/ml/gdb-cvs/2019-09/msg00041.html

--
Joel
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joseph Myers
In reply to this post by Gerald Pfeifer
On Mon, 16 Sep 2019, Gerald Pfeifer wrote:

> On the latter, let's just flip the switch whenever you're ready, and I'll
> see to adjust my script within a day or two and manually perform checks
> until that's done.  (If you can create a sample e-mail notification for
> me, I'll use that to start looking into it.)

The email notifications are the part most likely to change substantially
at various points after the initial git conversion.  The post-receive hook
I sent uses post-receive-email as something easy to set up, but I fully
expect we might end up using the AdaCore hooks for the wwwdocs repository
as well as for the GCC sources, with some suitable configuration that is
mostly the same between the repositories.

I don't recommend using the emails to extract any metadata about the
commits because the formatting of that information is likely to change.  
Rather, I suggest using the emails only as a signal that something has
been pushed to the repository.  You can then e.g. use "git rev-parse HEAD"
before and after updating the local checkout to see what the old and new
HEAD commits were, and "git diff --name-only" to list the modified, new or
removed files (as in the posted hook).  If you want to extract information
about individual new commits (e.g. authors) then "git rev-list
$old_HEAD..$new_HEAD" can be used to list the new commits present in
$new_HEAD but not in $old_HEAD.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Gerald Pfeifer
On Mon, 16 Sep 2019, Joel Brobecker wrote:
> You mean the email notification sent by the hooks when a commit
> gets pushed? If yes, here is an example:
>
> https://www.sourceware.org/ml/gdb-cvs/2019-09/msg00041.html

Thank you, Joel!  I got a little worried how to best parse that ;-),
but then Joseph recommended against it and...

On Mon, 16 Sep 2019, Joseph Myers wrote:
> Rather, I suggest using the emails only as a signal that something has
> been pushed to the repository.  You can then e.g. use "git rev-parse HEAD"
> before and after updating the local checkout to see what the old and new
> HEAD commits were, and "git diff --name-only" to list the modified, new or
> removed files (as in the posted hook).

...luckily provided an alternative.  Thanks for that, Joseph!  I had
a look and have started to adjust my script following your recommendation.

From my perspective this should not hold off anything.  I can keep
adjusting even once the switch has taken place and validate changes
manually during that period.

When, roughly, do you expect the switch can be ready?  I assume we'll
have some sort of flag day?  (I got in contact with [hidden email]
who are asking the same questions.)

Gerald
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joel Brobecker
> > You mean the email notification sent by the hooks when a commit
> > gets pushed? If yes, here is an example:
> >
> > https://www.sourceware.org/ml/gdb-cvs/2019-09/msg00041.html
>
> Thank you, Joel!  I got a little worried how to best parse that ;-),
> but then Joseph recommended against it and...
>
> On Mon, 16 Sep 2019, Joseph Myers wrote:
> > Rather, I suggest using the emails only as a signal that something has
> > been pushed to the repository.  You can then e.g. use "git rev-parse HEAD"
> > before and after updating the local checkout to see what the old and new
> > HEAD commits were, and "git diff --name-only" to list the modified, new or
> > removed files (as in the posted hook).
>
> ...luckily provided an alternative.  Thanks for that, Joseph!  I had
> a look and have started to adjust my script following your recommendation.

Not sure if the following could be of use, but the commit emails have
some information in the email header itself.

For instance, if you look at
https://sourceware.org/cgi-bin/get-raw-msg?listname=gdb-cvs&date=2019-09&msgid=20190917115728.124469.qmail%40sourceware.org

You can see:

    X-Act-Checkin: binutils-gdb
    X-Git-Author: Tom de Vries <[hidden email]>
    X-Git-Refname: refs/heads/gdb-8.3-branch
    X-Git-Oldrev: 6f4f8f476a4e41cc7117a8e85087963c0ac3e95b
    X-Git-Newrev: fafa92ec3ca92e06fdea8f0f6a0fb08f5f906f77

(the X-Act-Checkin field gives the name of the repository)

> >From my perspective this should not hold off anything.  I can keep
> adjusting even once the switch has taken place and validate changes
> manually during that period.
>
> When, roughly, do you expect the switch can be ready?  I assume we'll
> have some sort of flag day?  (I got in contact with [hidden email]
> who are asking the same questions.)

Just for the avoidance of doubt, the only item on my list is
to move the git-hooks' doc directly into the git-hooks repository
so they can be easily found when going to the hooks' github page
(in progress, pull request sent yesterday).

--
Joel
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joseph Myers
In reply to this post by Gerald Pfeifer
On Tue, 17 Sep 2019, Gerald Pfeifer wrote:

> When, roughly, do you expect the switch can be ready?

That's a matter of how much time we want to allow for people to try out
and comment on the conversion.

> I assume we'll have some sort of flag day?

Yes, at some point we'd do a final conversion and switch the live website
over to using git.  Then, in git, commit the post-receive hook, updates to
the preprocess script so that when run in whole-site mode it knows to
ignore .git/ the way it now knows to ignore CVS/, and updates to the
website to document how to edit it using git instead of CVS.

--
Joseph S. Myers
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: GCC Git hooks

Joel Brobecker
In reply to this post by Joel Brobecker
> But before I say more I should point to the documentation which,
> for historical reasons, is available on the GDB wiki rather than
> the git-hooks GitHub repository. I will fix that, but in the meantime:
>
>     https://sourceware.org/gdb/wiki/GitHooksUsersGuide

Just a quick note to confirm that the documentation has now been
moved to the git-hooks github page: https://github.com/adacore/git-hooks

Don't hesitate to reach out to me, if you have questions, or would
like some help configuring the hooks.

--
Joel