Commit 10e12b53 authored by Leigh Stoller's avatar Leigh Stoller

Support for push webhooks for repo-based profiles:

We are running another apache server on boss, on port 51369, which
invokes a backend perl script that maps the URL path argument to the
profile, and then calls out to manage_profile to pull from the
repository and update the profile to reflect the new HEAD branch.
Using mod_rewrite in the apache config to restrict URLs to exactly
the one URL that is accepted, modulo the value of the secret token.

I had to refactor a bunch of code in manage_profile to make it easier to
add a new entrypoint for modification from a git repo. This needed to be
done for a long time, I had never cleaned up the original profile
creation code.

On the edit profile web page, there is a new row in the Repository panel
providing the Push URL, and an explanatory help modal.

There is a new slow polling timer that looks for a change to the repo
hash and causes the web page to update in place from the repo, as when a
push hook is invoked and changes the repo.
parent f9ce0452
......@@ -291,6 +291,26 @@ sub Lookup($$;$$)
}
return undef;
}
#
# Lookup by repo key.
#
sub LookupByRepoKey($$)
{
my ($class, $token) = @_;
return undef
if ($token !~ /^\w+$/);
my $query_result =
DBQueryWarn("select uuid from apt_profile_versions ".
"where repokey='$token'");
return undef
if (!defined($query_result) || !$query_result->numrows);
my ($uuid) = $query_result->fetchrow_array();
return Lookup($class, $uuid);
}
AUTOLOAD {
my $self = $_[0];
......@@ -423,6 +443,7 @@ sub Create($$$$$$)
#
$cquery .= "name=$name,profileid='$profileid'";
$cquery .= ",pid='$pid',pid_idx='$pid_idx'";
$cquery .= ",gid='$gid',gid_idx='$gid_idx'";
# And the versions table.
$vquery = $cquery;
......@@ -445,6 +466,7 @@ sub Create($$$$$$)
$vquery .= ",repourl=" . DBQuoteSpecial($argref->{'repourl'});
$vquery .= ",reponame=" . DBQuoteSpecial($argref->{'reponame'});
$vquery .= ",repohash=" . DBQuoteSpecial($argref->{'repohash'});
$vquery .= ",repokey=" . DBQuoteSpecial($argref->{'repokey'});
}
# Back to the main table.
......@@ -512,11 +534,13 @@ sub NewVersion($$)
goto bad
if (! DBQueryWarn("insert into apt_profile_versions ".
" (name,profileid,version,pid,pid_idx, ".
" gid,gid_idx, ".
" creator,creator_idx,updater,updater_idx, ".
" created,uuid, ".
" parent_profileid,parent_version,rspec, ".
" script,paramdefs,reponame,repourl) ".
"select name,profileid,'$newvers',pid,pid_idx, ".
" gid,gid_idx, ".
" creator,creator_idx,'$uid','$uid_idx',".
" now(),uuid(),'$profileid', ".
" '$version',rspec,script,paramdefs, ".
......
......@@ -42,6 +42,7 @@ WEB_BIN_SCRIPTS = webmanage_profile webmanage_instance webmanage_dataset \
webcreate_instance webrungenilib webns2rspec webns2genilib \
webrspec2genilib webmanage_reservations webmanage_gitrepo \
webmanage_images
APACHEHOOKS = apt_gitrepo.hook
WEB_SBIN_SCRIPTS= webportal_xmlrpc
LIBEXEC_SCRIPTS = $(WEB_BIN_SCRIPTS) $(WEB_SBIN_SCRIPTS)
USERLIBEXEC = rungenilib.proxy genilib-jail genilib-iocage gitrepo.proxy
......@@ -49,14 +50,14 @@ USERLIBEXEC = rungenilib.proxy genilib-jail genilib-iocage gitrepo.proxy
# These scripts installed setuid, with sudo.
SETUID_BIN_SCRIPTS = rungenilib manage_gitrepo
SETUID_SBIN_SCRIPTS =
SETUID_SUEXEC_SCRIPTS=
SETUID_SUEXEC_SCRIPTS= apt_gitrepo.hook
#
# Force dependencies on the scripts so that they will be rerun through
# configure if the .in file is changed.
#
all: $(BIN_SCRIPTS) $(SBIN_SCRIPTS) $(LIBEXEC_SCRIPTS) $(SUBDIRS) \
$(LIB_SCRIPTS) $(USERLIBEXEC) all-subdirs
$(LIB_SCRIPTS) $(USERLIBEXEC) $(APACHEHOOKS) all-subdirs
subboss:
......@@ -66,7 +67,9 @@ install: $(addprefix $(INSTALL_BINDIR)/, $(BIN_SCRIPTS)) \
$(addprefix $(INSTALL_SBINDIR)/, $(SBIN_SCRIPTS)) \
$(addprefix $(INSTALL_LIBDIR)/, $(LIB_SCRIPTS)) \
$(addprefix $(INSTALL_LIBEXECDIR)/, $(LIBEXEC_SCRIPTS)) \
$(addprefix $(INSTALL_LIBEXECDIR)/, $(APACHEHOOKS)) \
$(addprefix $(INSTALL_DIR)/opsdir/libexec/, $(USERLIBEXEC)) \
$(addprefix $(INSTALL_DIR)/apt/, $(APACHEHOOKS)) \
$(INSTALL_ETCDIR)/cloudlab-fedonly.json \
$(INSTALL_ETCDIR)/cloudlab-nofed.json
......@@ -106,4 +109,10 @@ $(INSTALL_DIR)/opsdir/libexec/%: %
-mkdir -p $(INSTALL_DIR)/opsdir/libexec
$(INSTALL) $< $@
$(INSTALL_DIR)/apt/%: %
@echo "Installing $@"
-mkdir -p $(INSTALL_DIR)/apt
-rm -f $(INSTALL_DIR)/apt/$<
ln $(INSTALL_LIBEXECDIR)/$< $(INSTALL_DIR)/apt/$<
.PHONY: $(SUBDIRS) install
#!/usr/bin/perl -w
#
# Copyright (c) 2008-2017 University of Utah and the Flux Group.
#
# {{{GENIPUBLIC-LICENSE
#
# GENI Public License
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and/or hardware specification (the "Work") to
# deal in the Work without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Work, and to permit persons to whom the Work
# is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Work.
#
# THE WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE WORK OR THE USE OR OTHER DEALINGS
# IN THE WORK.
#
# }}}
#
use strict;
use English;
use Getopt::Std;
use Data::Dumper;
use JSON;
use CGI;
# Configure variables
my $TB = "@prefix@";
my $MAINSITE = @TBMAINSITE@;
my $TBOPS = "@TBOPSEMAIL@";
my $MANAGEPROFILE = "$TB/bin/manage_profile";
# Locals
my $debug = 0;
my $token;
#
# Turn off line buffering on output
#
$| = 1;
#
# Untaint the path
#
$ENV{'PATH'} = '/bin:/usr/bin:/usr/local/bin';
delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'};
# Testbed libraries.
use lib '@prefix@/lib';
use emutil;
use libaudit;
use libtestbed;
use APT_Profile;
use User;
use Group;
#
# We want to make sure we send back a header.
#
sub SendStatus($)
{
my ($status) = @_;
if (1) {
# Error, we are done.
LogEnd($status);
}
else {
# We want to keep logging, not send an email here.
LogStop();
}
print "Content-Type: text/plain \n\n";
print "We love all profiles equally.\n";
print "Exited with $status\n";
exit(0);
}
#
# Send logs to tblogs
#
LogStart(0, undef, LIBAUDIT_LOGTBLOGS());
# The query holds the token we need to find the profile.
my $query = new CGI();
if ($debug > 1) {
my %headers = map { $_ => $query->http($_) } $query->http();
print STDERR Dumper(\%headers);
print STDERR Dumper($query);
}
#
# The profile is provided in the path.
#
if (!exists($ENV{'PATH_INFO'}) || $ENV{'PATH_INFO'} eq "") {
print STDERR "No path info\n";
SendStatus(1);
}
my $pathinfo = $ENV{'PATH_INFO'};
if ($pathinfo =~ /^\/([-\w]+)$/) {
$token = $1;
}
else {
print STDERR "Bad path info\n";
SendStatus(1);
}
if (!defined($token)) {
print STDERR "No token provided\n";
SendStatus(1);
}
if ($token !~ /^[-\w]+$/ || length($token) > 64) {
print STDERR "Bad token format\n";
SendStatus(1);
}
#
# Before calling out, find the profile.
#
my $profile = APT_Profile->LookupByRepoKey($token);
if (!defined($profile)) {
print STDERR "No profile for token $token\n";
SendStatus(1);
}
if (!defined($profile->repourl())) {
print STDERR "Not a repo based profile for token $token\n";
SendStatus(1);
}
#
# Let the parent return, no need to keep the client waiting since it
# don't care anyway.
#
my $mypid = fork();
if ($mypid) {
#
# Must end with this for the client. Does not return;
#
SendStatus(0);
}
sleep(1);
libaudit::AuditFork();
if ($debug) {
print "$profile\n";
}
#
# We are going to do the update as the profile creator.
#
my $creator = User->Lookup($profile->creator_idx());
if (!defined($creator)) {
print STDERR "Cannot lookup creator for $profile\n";
exit(1);
}
my $group = Group->Lookup($profile->gid_idx());
if (!defined($group)) {
print STDERR "Cannot lookup group for $profile\n";
exit(1);
}
if ($creator->FlipTo($group)) {
print STDERR "Could not flip to $creator\n";
exit(1);
}
my $output =
emutil::ExecQuiet("$MANAGEPROFILE updatefromrepo " . $profile->uuid());
if ($?) {
print STDERR $output;
exit(1);
}
if ($debug > 1) {
print $output;
}
LogEnd(0);
exit(0);
This diff is collapsed.
......@@ -91,6 +91,7 @@ $(function ()
fields["profile_repourl"] != "") {
fromrepo = 1;
repohash = fields["profile_repohash"];
setTimeout(function f() { CheckRepoChange() }, 10000);
}
// If this is an existing profile, stash the name/project
......@@ -1431,5 +1432,29 @@ $(function ()
});
}
/*
* Timer to ask for the current repository hash value to determine
* if it has changed. We tell the user to reload the page.
*/
function CheckRepoChange()
{
var callback = function(json) {
//console.info("CheckRepoChange", json);
if (json.code == 0 && repohash != json.value) {
// Mark as HEAD in the page.
repohash = json.value;
// Reset the list of tags and branches whenever we
// successfully update our clone.
SetupRepo();
}
setTimeout(function f() { CheckRepoChange() }, 10000);
};
var xmlthing = sup.CallServerMethod(ajaxurl,
"manage_profile", "GetRepoHash",
{"uuid" : version_uuid});
xmlthing.done(callback);
}
$(document).ready(initialize);
});
......@@ -164,8 +164,8 @@ function Do_DeleteProfile()
return;
}
$retval = SUEXEC($this_uid, $profile->pid(),
"webmanage_profile delete -t " . $webtask->task_id() .
" -- $opt " . $profile->uuid(),
"webmanage_profile -t " . $webtask->task_id() . " " .
"delete $opt " . $profile->uuid(),
SUEXEC_ACTION_IGNORE);
if ($retval != 0) {
......@@ -265,8 +265,8 @@ function Do_PublishProfile()
return;
}
$retval = SUEXEC($this_uid, $profile->pid(),
"webmanage_profile publish -t " . $webtask->task_id() .
" " . $profile->uuid(),
"webmanage_profile -t " . $webtask->task_id() . " " .
"publish " . $profile->uuid(),
SUEXEC_ACTION_IGNORE);
if ($retval != 0) {
$webtask->Refresh();
......@@ -966,8 +966,8 @@ function UpdateMaster($profile, $script, $rspec)
chmod($fname, 0666);
$retval = SUEXEC($this_uid, $profile->pid(),
"webmanage_profile update -t " . $webtask->task_id() .
" " . $profile->uuid() . " $fname",
"webmanage_profile -t " . $webtask->task_id() . " " .
"update " . $profile->uuid() . " $fname",
SUEXEC_ACTION_IGNORE);
$webtask->Refresh();
......@@ -1214,6 +1214,37 @@ function Do_GetCommitInfo()
SPITAJAX_RESPONSE($webtask->TaskValue("commitinfo"));
}
#
# Get the repo hash
#
function Do_GetRepoHash()
{
global $this_user;
global $ajax_args;
$this_idx = $this_user->uid_idx();
$this_uid = $this_user->uid();
if (!isset($ajax_args["uuid"])) {
SPITAJAX_ERROR(1, "Missing profile uuid");
return;
}
$profile = Profile::Lookup($ajax_args["uuid"]);
if (!$profile) {
SPITAJAX_ERROR(1, "Unknown profile uuid");
return;
}
if (!$profile->repourl()) {
SPITAJAX_ERROR(1, "Not a repo-based profile");
return;
}
if ($this_idx != $profile->creator_idx() && !ISADMIN()) {
SPITAJAX_ERROR(1, "Not enough permission");
return;
}
SPITAJAX_RESPONSE($profile->repohash());
}
# Local Variables:
# mode:php
# End:
......
......@@ -392,6 +392,9 @@ if (! isset($create)) {
$defaults["profile_repourl"] = $profile->repourl();
# Need this so JS code knows when HEAD changes.
$defaults["profile_repohash"] = $profile->repohash();
$defaults["profile_repopushurl"]
= "https://www.emulab.net:51369/githook/" .
$profile->repokey();
}
$defaults["profile_creator"] = $profile->creator();
$defaults["profile_updater"] = $profile->updater();
......@@ -695,13 +698,13 @@ else {
#
$webtask = WebTask::CreateAnonymous();
$webtask_id = $webtask->task_id();
$command = "webmanage_profile ";
$command = "webmanage_profile -t $webtask_id ";
if ($action == "edit") {
$command .= " update -t $webtask_id " . $profile->uuid();
$command .= " update " . $profile->uuid();
}
else {
$command .= " create -t $webtask_id ";
$command .= " create ";
if (isset($copyuuid)) {
$command .= "-c " . escapeshellarg($copyuuid);
}
......@@ -713,6 +716,9 @@ $command .= " $xmlname";
$retval = SUEXEC($this_user->uid(), $project->unix_gid(), $command,
SUEXEC_ACTION_IGNORE);
SUEXECERROR(SUEXEC_ACTION_CONTINUE);
if ($retval) {
if ($retval < 0) {
$errors["error"] = "Internal Error; please try again later.";
......@@ -735,7 +741,7 @@ if ($retval) {
}
$webtask->Delete();
}
unlink($xmlname);
#unlink($xmlname);
if (count($errors)) {
SPITFORM($formfields, $errors);
return;
......
......@@ -132,6 +132,7 @@ class Profile
function repourl() { return $this->field('repourl'); }
function reponame() { return $this->field('reponame'); }
function repohash() { return $this->field('repohash'); }
function repokey() { return $this->field('repokey'); }
function webtask_id() { return $this->field('webtask_id'); }
function profile_disabled() { return $this->field('profile_disabled'); }
function parent_profileid() { return $this->field('parent_profileid'); }
......
......@@ -130,6 +130,8 @@ $routing = array("myprofiles" =>
"Do_GetBranchList",
"GetCommitInfo" =>
"Do_GetCommitInfo",
"GetRepoHash" =>
"Do_GetRepoHash",
"GetCommitList" =>
"Do_GetCommitList")),
"status" =>
......
......@@ -151,6 +151,21 @@
<td>Est. Size:</td>
<td class="commit-size"></td>
</tr>
<tr style="white-space: nowrap;">
<td>Push URL:</td>
<td style="white-space: nowrap;">
<input readonly type="text" class="form-control input-sm"
onClick="this.select();"
style="display: inline;
padding-left: 3px; padding-right: 3px;
padding-bottom: 0px; padding-top: 0px;"
value="<%- formfields.profile_repopushurl %>">
<a href='#' class='btn btn-xs'
style="padding: 0px"
data-toggle='modal'
data-target="#webhook-help-modal">
<span class='glyphicon glyphicon-question-sign'
style='margin-bottom: 4px;'></span></a></td>
<% if (isadmin) { %>
<tr>
<td>RepoName:</td>
......@@ -362,7 +377,11 @@
data-content='Fetch from your repository, updating
your profile to reflect the current HEAD
of your master branch. Branches and tags
are updated as well (see below).'>
are updated as well (see below). You
can also set up a <em>push webhook</em>
to automate updates, see the <b>Push
URL</b> help info in the Repository panel
on your left.'>
<span class='glyphicon glyphicon-question-sign'
style='margin-bottom: 4px;'></span>
</a>
......@@ -835,6 +854,75 @@
</div>
</div>
</div>
<% if (fromrepo) { %>
<!-- web hook help -->
<div id='webhook-help-modal' class='modal fade'>
<div class='modal-dialog'>
<div class='modal-content'>
<div class='modal-body'>
<button type='button' class='close' data-dismiss='modal'
aria-hidden='true'>&times;</button>
<div class='clearfix'></div>
<p>
The <em>push URL</em> is an example of a
<a href="https://developer.github.com/webhooks/" target="_blank">
<em>webhook</em></a> supported by public
<a href="https://git-scm.com/" target="_blank">Git</a>
repositories like
<a href="github.com" target="_blank">github.com</a>,
as well as those based
on <a href="https://www.gitlab.com/" target="_blank">GitLab</a>.
</p>
<p>
Specifically, a <em>push</em> webhook is used to notify
a third party that a new commit has been created, either
by a push or by executing a commit via the repository
web interface. With <%- window.APTTILE %>, when we receive the
notification from your repository, we will fetch from
your repository, updating your profile to reflect the
current HEAD of your master branch. Branches and tags
are updated as well. When complete, we will send you an email
confirmation so you know that your profile has been updated.
</p>
<p>
Setting up a webhook is relatively straightfoward:
<ul>
<li><b>github.com</b>: Go to your repository and
click on the <b>Settings</b> option
in the upper right, then click on <b>Webhooks</b>, then
click on the <b>Add Webhook</b> menu option. Paste your
push URL into the <b>Payload URL</b> form field, leave
everything else as is, and click on the <b>Add Webhook</b>
button at the bottom of the form.
</li>
<li><b>gitlab</b>: Go to your repository and click on the
<span class='glyphicon glyphicon-cog'
style='margin-bottom: 4px;'></span> in the upper
right, then click on the <b>Integrations</b> menu option.
Paste your push URL into the <b>URL</b> form field, leave
everything else as is, and click on the <b>Add Webhook</b>
button at the bottom of the form.
</li>
<li><b>bitbucket.org</b>: Go to your repository and click on the
<span class='glyphicon glyphicon-cog'
style='margin-bottom: 4px;'></span> in the lower
left, then click on the <b>Webhooks</b> menu option, then
click on the <b>Add Webhook</b> button. Give your new
webhook a <b>Title</b> and
paste your push URL into the <b>URL</b> form field, leave
everything else as is, and click on the <b>Save</b>
button at the bottom of the form.
</li>
</ul>
</p>
<p>
</p>
</div>
</div>
</div>
</div>
<% } %>
<!-- Docstring help -->
<div id='docstring-help-modal' class='modal fade'
data-keyboard='false' data-backdrop='static'>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment