Enable Azure Pipelines #13818

foolip · 2018-10-31T14:01:54Z

Once #13769 has landed, the next step will be to enable it in the Azure DevOps control panel: https://dev.azure.com/web-platform-tests/wpt

Once enabled, it has the potential to block PRs if anything breaks, and unlike with Travis it will not be possible for anyone with write access to this repo to restart builds. This means we need to monitor it carefully, document how to turn it off, and make sure enough people have access to turn it off.

@web-platform-tests/admins, if you don't have one, can you create a Microsoft account and send me the emails you used? Then I can add you to the Azure DevOps project.

For monitoring, I'll write a script that looks at all open PRs updated from now on, and checks for failing or ever-pending Azure Pipelines checks.

This just runs infrastructure/ tests on relevant PRs, something minimal and useful before trying to enable Azure Pipelines for the first time. Part of #13299. Next step: #13818

foolip · 2018-10-31T14:27:52Z

I've enabled it now, and as part of that there was a first run, which did almost nothing:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=1&view=logs
https://api.github.com/repos/web-platform-tests/wpt/commits/1fb4c5a78d1af72228e9371845aa1bbadbb7b3aa/statuses (events can be seen there)

foolip · 2018-10-31T17:34:56Z

It's run on a number of PRs now, some including the infrastructure/ tests. If it starts misbehaving, here's how to disable it:

Go to https://dev.azure.com/web-platform-tests/wpt
Click Pipelines, then "Azure Pipelines" (that's the name of the pipeline that I chose)
Click Edit and then Triggers
Under "Pull request validation", select "Override YAML Pull Request trigger from here" and then "Disable pull request validation"

If someone could verify that the steps can be followed before it's urgent and I'm not around, that'd be great.

foolip · 2018-10-31T22:03:47Z

Node script to list Azure Pipelines status of all PRs opened since it was enabled

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// merge time of https://github.com/web-platform-tests/wpt/pull/13769
const SINCE = '2018-10-31T11:06:44Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const statuses = await paginate(octokit.repos.getStatuses, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    // note: statuses are in reverse chronological order
    const lastAzureStatus = statuses.find(status => status.context == 'Azure Pipelines');
    if (!lastAzureStatus) {
        console.log(`#${pr.number}: no status`);
        continue;
    }

    console.log(`#${pr.number}: ${lastAzureStatus.state}`);
  }
})();

It currently only lists success and "no status", for PRs that have had comments but no new commits. Nothing has failed or gotten stuck yet.

https://dev.azure.com/web-platform-tests/wpt/_build?definitionId=1 lists the builds in the Azure Pipelines UI, but I didn't want to just trust that :)

kaylangan · 2018-11-01T16:15:36Z

Hey @foolip! I'm a Program Manager for Azure Pipelines. It looks like you've been able to get up and running in other PRs. If you have any questions or suggestions, please don't hesitate to reach out!

foolip · 2018-11-01T16:55:51Z

Thanks @kaylangan! I have an email thread with @thejohnjansen about some issues I've seen, but I'll paste it here verbatim for all to see:

I just got pytest junit output working on my fork, here are results:
https://dev.azure.com/foolip/wpt/_build/results?buildId=140&view=ms.vss-test-web.test-result-details

That view is really great, so I think I'll try to get that merged, and perhaps use it for Linux testing too instead of Travis. However, I don't quite understand how to use `continueOnError: true`. It seems like the following steps run even without that, and what it actually means is that the overall status will be successful even if the step fails? (I tried using it to make sure test results were published even on test failure.)

Another minor question: the name of the test file doesn't show up in the test hierarchy, just the name I picked ("tools/wpt/ pytest") and the method names. Can this be changed?

More questions:

1) On my fork, the Checks API integration works well: https://github.com/foolip/wpt/pull/11/checks

However, on the main repo, it looks like the old statuses API is used, as there's nothing (useful) in the Checks tab: https://github.com/web-platform-tests/wpt/pull/13844/checks

The way I enabled Azure Pipelines wasn't exactly the same in both cases, but I don't know why there's this difference. I'd love to have the Checks tab populated on the main repo too.

2) What is the expected upgrade time for macOS agent? We're now using macOS-10.13, and that's fine as long Apple keeps releasing Safari Technology Preview for 10.13, but if major OS upgrades ever lag behind a lot, it might become a problem. Cirrus CI 

3) When I push changes and observe what happens, Travis and Taskcluster usually register their runs as queued/started before Azure Pipelines by several seconds. It seems like something might be unnecessarily slow here? On a repo using only Azure Pipelines, this increases the window in which the PR can be merged without having run through any checks.

That's all for now. Thanks for this great product! :D

foolip · 2018-11-01T21:07:33Z

Good news! With the help of @davidstaheli I was able to change the authentication method of the Azure Pipelines project so that the Checks API integration is used instead. That means we get the "Checks" tab nicely populated: https://github.com/web-platform-tests/wpt/pull/13856/checks

(And it'll get even nicer: https://github.com/foolip/wpt/pull/11/checks?check_run_id=28196826)

sideshowbarker · 2018-11-01T21:27:10Z

(And it'll get even nicer: https://github.com/foolip/wpt/pull/11/checks?check_run_id=28196826)

Very cool 👍

foolip · 2018-11-01T23:04:57Z

So, given that output, I'm wondering if I should try moving these tests from Travis to Azure Pipelines for Linux. @jgraham our plan was to move stuff to Taskcluster, but maybe we can leverage Azure Pipelines for the bits it seems especially good for?

foolip · 2018-11-02T16:53:11Z

Oh no, first sign of infra trouble!

In https://github.com/web-platform-tests/wpt/pull/13875/checks?check_run_id=28469086 there was this failure:

System.InvalidOperationException: Git fetch failed with exit code: 128
   at Agent.Plugins.Repository.GitSourceProvider.GetSourceAsync(AgentTaskPluginExecutionContext executionContext, RepositoryResource repository, CancellationToken cancellationToken)
   at Agent.Plugins.Repository.CheckoutTask.RunAsync(AgentTaskPluginExecutionContext executionContext, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)

I presume it will go away if I retry. This was my own PR so no big problem.

@kaylangan @thejohnjansen could you look into why this might be happening? Is there an automatic retry mechanism we can use? Taskcluster retries each task (=job in Azure Pipelines ) 5 times, I think mostly to deal with the possibility of the AWS spot instances used being killed.

Hexcles · 2018-11-02T18:26:11Z

@foolip also happened in #13877 . I think it's perhaps caused by GitHub outage: https://status.github.com/messages

foolip · 2018-11-02T18:53:00Z

Oh, that's two. If it was just the GitHub outage, wouldn't Travis and Taskcluster also fail?

If it keeps happening, then everyone feel free to disable per the steps above.

foolip · 2018-11-03T00:01:53Z

After switching from statuses to checks (#13818 (comment)) the script in #13818 (comment) no longer works.

Updated script that checks (ha) for checks instead

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// merge time of https://github.com/web-platform-tests/wpt/pull/13769
const SINCE = '2018-10-31T11:06:44Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const checks = await paginate(octokit.checks.listForRef, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    const azureRun = checks.check_runs.find(run => run.name == 'Azure Pipelines');
    if (!azureRun) {
        console.log(`#${pr.number}: no check`);
        continue;
    }

    if (azureRun.status !== 'completed') {
      console.log(`#${pr.number}: ${azureRun.status}`);
    } else {
      console.log(`#${pr.number}: ${azureRun.conclusion}`);
    }
  }
})();

Right now it lists only #13875 as failing, which seems right.

foolip · 2018-11-03T07:54:24Z

OK, another problem. On some PRs Azure Pipelines wasn't triggered at all. #13884 and #13886 are examples.

foolip · 2018-11-04T09:27:12Z

Given the lack of a check run on #13900, I now suspect builds aren't triggered for PRs from forks by default, which is also the default for Taskcluster.

foolip · 2018-11-06T17:07:56Z

I've fixed the "PRs from forks" problem and documented how in #13952.

foolip · 2018-11-07T09:25:20Z

Running the script in #13818 (comment) again now, the only failure listed is #13640, where the failures seems legitimate and the infrastructure/ tests are failing also on Travis.

A lot of PRs don't have a check because they were created before Azure Pipelines was added and just had comments to bump their updated date, and some (like #13926) are before builds for PRs from forks were enabled.

foolip · 2018-11-07T09:47:28Z

Updated script that only includes problematic "no check" cases

'use strict';

const octokit = require('@octokit/rest')();

const GH_TOKEN = process.env.GH_TOKEN;

if (!GH_TOKEN) {
  console.error('Please set the GH_TOKEN environment variable.');
  console.error('https://github.com/settings/tokens');
  process.exit(1);
}

octokit.authenticate({
  type: 'token',
  token: GH_TOKEN
});

// Time of https://github.com/web-platform-tests/wpt/issues/13818#issuecomment-436330922
const SINCE = '2018-11-06T17:07:56Z';

async function paginate(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const { data } = response;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    data.push(...response.data);
  }
  return data;
}

async function paginateSearch(method, parameters) {
  parameters = Object.assign({ per_page: 100 }, parameters);
  let response = await method(parameters);
  const items = response.data.items;

  while (octokit.hasNextPage(response)) {
    response = await octokit.getNextPage(response);
    items.push(...response.data.items);
  }
  return items;
}

(async () => {
  const prs = await paginateSearch(octokit.search.issues, {
    q: `repo:web-platform-tests/wpt is:pr is:open updated:>${SINCE}`,
  });

  for (const pr of prs) {
    const commits = await paginate(octokit.pullRequests.getCommits, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      number: pr.number,
    });

    // only look at the final commit
    const ref = commits[commits.length - 1].sha;

    const checks = await paginate(octokit.checks.listForRef, {
      owner: 'web-platform-tests',
      repo: 'wpt',
      ref,
    });

    const azureRun = checks.check_runs.find(run => run.name == 'Azure Pipelines');
    if (!azureRun) {
        // If created before the cutoff time and there are no checks, that's
        // probably because the update was just a commment and no CI has run.
        if (Date.parse(pr.created_at) < Date.parse(SINCE)) {
            continue;
        }
        // Otherwise there might be something wrong.
        console.log(`#${pr.number}: no check`);
        continue;
    }

    if (azureRun.status !== 'completed') {
      const msAgo = Date.now() - Date.parse(azureRun.started_at);
      const minAgo = Math.floor(msAgo / 60000);
      console.log(`#${pr.number}: ${azureRun.status} (started ${minAgo} min ago)`);
    } else {
      console.log(`#${pr.number}: ${azureRun.conclusion}`);
    }
  }
})();

Overall, things seem to be working just fine.

foolip · 2018-11-07T15:51:20Z

In https://github.com/web-platform-tests/wpt/runs/29626821 the /infrastructure/assumptions/ahem.html failed on macOS Chrome Dev, but not Firefox Nightly or Safari Technology Preview. That suggests there might still be a problem with font installation on macOS :/

…=testonly Automatic update from web-platform-testsAdd configuration for Azure Pipelines (#13769) This just runs infrastructure/ tests on relevant PRs, something minimal and useful before trying to enable Azure Pipelines for the first time. Part of web-platform-tests/wpt#13299. Next step: web-platform-tests/wpt#13818 -- wpt-commits: 4098fe5f1f08442a8d63db9eb107b49a2f4fe752 wpt-pr: 13769

foolip · 2018-11-12T09:47:20Z

I've run the script in #13818 (comment) again, no new failures.

foolip · 2018-11-16T16:54:09Z

From #14096 I learned that to be blocking, we also need to make the Azure Pipelines check required in the GitHub settings. I won't do that right away, but should be done before this issue is closed.

foolip · 2018-11-16T16:57:43Z

I've run the script again. #14074 is the single new failure, but appears to be correct, the failures are in the modified code and unittests are also failing on Travis. Overall, things seems to be working pretty well.

foolip · 2018-11-23T10:00:34Z

#14210 is an issue about monitoring Taskcluster on an ongoing basis. Possibly we can share some monitoring code/process across our CI systems.

foolip · 2018-11-23T23:27:28Z

Checking again for failures:

Add name constraints to CA #14206 is a legit failure, Travis has also failed in the same way
[Azure Pipelines] Run affected tests on Safari Technology Preview #14156 is my own PR, legit failure

That's all. Still working well :)

jgraham · 2019-01-08T17:01:30Z

Should this issue be closed?

foolip · 2019-01-08T17:38:07Z

The monitoring part isn't done. I could rename #14210 to cover it too, but then that might also end up in a half-done state :)

jgraham · 2019-01-08T20:07:32Z

I think this should be closed and a bespoke issue opened for whatever monitioring you are planning.

foolip · 2019-01-09T09:26:42Z

Filed #14763.

…=testonly Automatic update from web-platform-testsAdd configuration for Azure Pipelines (#13769) This just runs infrastructure/ tests on relevant PRs, something minimal and useful before trying to enable Azure Pipelines for the first time. Part of web-platform-tests/wpt#13299. Next step: web-platform-tests/wpt#13818 -- wpt-commits: 4098fe5f1f08442a8d63db9eb107b49a2f4fe752 wpt-pr: 13769 UltraBlame original commit: afe5654fe9baae07072ff2384738e54bd29c1601

foolip added infra priority:roadmap labels Oct 31, 2018

foolip self-assigned this Oct 31, 2018

foolip added a commit that referenced this issue Oct 31, 2018

Add configuration for Azure Pipelines (#13769)

4098fe5

This just runs infrastructure/ tests on relevant PRs, something minimal and useful before trying to enable Azure Pipelines for the first time. Part of #13299. Next step: #13818

foolip mentioned this issue Nov 23, 2018

Monitor Taskcluster PR and master runs #14210

Closed

foolip mentioned this issue Dec 18, 2018

Safari check runs not showing up for most PRs web-platform-tests/wpt.fyi#945

Closed

foolip changed the title ~~Enable and monitor Azure Pipelines~~ Enable Azure Pipelines Jan 9, 2019

foolip mentioned this issue Jan 9, 2019

Monitor Azure Pipelines runs #14763

Closed

foolip closed this as completed Jan 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Azure Pipelines #13818

Enable Azure Pipelines #13818

foolip commented Oct 31, 2018

foolip commented Oct 31, 2018

foolip commented Oct 31, 2018 •

edited

Loading

foolip commented Oct 31, 2018

kaylangan commented Nov 1, 2018

foolip commented Nov 1, 2018

foolip commented Nov 1, 2018

sideshowbarker commented Nov 1, 2018

foolip commented Nov 1, 2018

foolip commented Nov 2, 2018

Hexcles commented Nov 2, 2018

foolip commented Nov 2, 2018

foolip commented Nov 3, 2018

foolip commented Nov 3, 2018

foolip commented Nov 4, 2018

foolip commented Nov 6, 2018

foolip commented Nov 7, 2018

foolip commented Nov 7, 2018

foolip commented Nov 7, 2018

foolip commented Nov 12, 2018

foolip commented Nov 16, 2018

foolip commented Nov 16, 2018

foolip commented Nov 23, 2018

foolip commented Nov 23, 2018

jgraham commented Jan 8, 2019

foolip commented Jan 8, 2019

jgraham commented Jan 8, 2019

foolip commented Jan 9, 2019

Enable Azure Pipelines #13818

Enable Azure Pipelines #13818

Comments

foolip commented Oct 31, 2018

foolip commented Oct 31, 2018

foolip commented Oct 31, 2018 • edited Loading

foolip commented Oct 31, 2018

kaylangan commented Nov 1, 2018

foolip commented Nov 1, 2018

foolip commented Nov 1, 2018

sideshowbarker commented Nov 1, 2018

foolip commented Nov 1, 2018

foolip commented Nov 2, 2018

Hexcles commented Nov 2, 2018

foolip commented Nov 2, 2018

foolip commented Nov 3, 2018

foolip commented Nov 3, 2018

foolip commented Nov 4, 2018

foolip commented Nov 6, 2018

foolip commented Nov 7, 2018

foolip commented Nov 7, 2018

foolip commented Nov 7, 2018

foolip commented Nov 12, 2018

foolip commented Nov 16, 2018

foolip commented Nov 16, 2018

foolip commented Nov 23, 2018

foolip commented Nov 23, 2018

jgraham commented Jan 8, 2019

foolip commented Jan 8, 2019

jgraham commented Jan 8, 2019

foolip commented Jan 9, 2019

foolip commented Oct 31, 2018 •

edited

Loading