Syncer keeps cancelling integrations

[reststop]

Not sure I understand what's happening.

Environment::
Using a Branch and PR on Buildasaur as my test project.
Due to cocoapods issues, I added the Pods to my remote repository to eliminate 'pod setup' issues on beta version of cocoapods.

Steps::

• Create a pull request with my fix for SSH URL from clone urls + Pods
  carl/stash3-ssh-fix
• Create a new Syncer
• Start Syncer

What I am seeing::

The Bot is created, and an integration runs...
The integration shows as being cancelled
Another integration starts
That one is cancelled...
(repeats as long as Syncer is running)
Stop Syncer, and the integration, and the pending integrations all finish successfully.

I surmise that the syncer is killing the bot integration.

------------------------------------

[INFO]: Sync starting at 2016-05-12 22:11:38 +0000
[VERBOSE]: Resolving prs:
    PR 1: Get clone SSH URL by name [carl/stash3-ssh-fix -> feature/bitbucket_server] 
and branches:

and bots:
    ...
    Bot β [buildasaur] carl/stash3-ssh-fix
[INFO]: Bot β [buildasaur] carl/stash3-ssh-fix successfully enqueued Integration #14
[VERBOSE]: SyncPair PR (1:carl/stash3-ssh-fix) + Bot (β [buildasaur] carl/stash3-ssh-fix) finished sync after 0.39 seconds.
[INFO]: Rate Limit: NO INFO
[INFO]: Sync finished successfully at 2016-05-12 22:11:39 +0000, took 0.83 seconds.

------------------------------------

Build log for bot shows:
** BUILD INTERRUPTED **

First syncer output shows existing bots, and the following:

[INFO]: Successfully created bot β [buildasaur] carl/stash3-ssh-fix
[VERBOSE]: SyncPair PR (carl/stash3-ssh-fix) + No Bot finished sync after 0.647 seconds.
[INFO]: Rate Limit: NO INFO
[INFO]: Sync finished successfully at 2016-05-12 22:08:10 

Next one shows:

and bots:
    Bot β [buildasaur] carl/stash3-ssh-fix

[INFO]: Bot β [buildasaur] carl/stash3-ssh-fix successfully enqueued Integration #1
[VERBOSE]: SyncPair PR (1:carl/stash3-ssh-fix) + Bot (β [buildasaur] carl/stash3-ssh-fix) finished sync after 0.367 seconds.
[INFO]: Rate Limit: NO INFO
[INFO]: Sync finished successfully at 2016-05-12 22:08:24 +0000, took 0.779 seconds.

And the succeeding ones show:

    Bot β [buildasaur] carl/stash3-ssh-fix
[INFO]: Successfully cancelled integration 1
[INFO]: Bot β [buildasaur] carl/stash3-ssh-fix successfully enqueued Integration #2
[VERBOSE]: SyncPair PR (1:carl/stash3-ssh-fix) + Bot (β [buildasaur] carl/stash3-ssh-fix) finished sync after 0.344 seconds.
[INFO]: Rate Limit: NO INFO
[INFO]: Sync finished successfully at 2016-05-12 22:08:39 +0000, took 0.749 seconds.

No error messages are shown, just hidden in the output, that it successfully cancelled the previously enqueued integration.

[reststop]

A previous run when I was using a live project, was having API issues getting commit status, and that is what I was attempting to debug. Now this. I believe this was also happening with the prior test, I saw integrations being cancelled, and not by me.

The other issue masked this issue, but now, with no problems showing up with getting status (so far anyway), this is a real killer.

I am suspicious that it is checking for some status, or it is not posting the status on the Stash server, so it is killing what it thinks may be a long running bot. I think each integration got farther by compiling yet another set of files until all were compiled, because after the 12th integration, the remaining ones finished fast enough and the Syncer has nothing to kill.

The most recent log shows this, without any "cancelling" for integrations #13-18::

    Bot β [buildasaur] carl/stash3-ssh-fix
[INFO]: Bot β [buildasaur] carl/stash3-ssh-fix successfully enqueued Integration #18
[VERBOSE]: SyncPair PR (1:carl/stash3-ssh-fix) + Bot (β [buildasaur] carl/stash3-ssh-fix) finished sync after 0.361 seconds.

So, another issue is that the Syncer seems to think it has to create a new integration for some reason, and kill/cancel any running integration.

Any hints on what to look for on this will be very helpful.

[czechboy0]

Look here, this is where Buildasaur figures out which integrations to cancel. There must be an issue with recognizing the head commits is my guess.

[reststop]

Thanks... that will help. There has to be a reason why each time through, it decided it needs to cancel it.
(Likely not getting some info it is expecting, so I'm trying to work the logic backwards)

[reststop]

Can you expound upon "recognizing the head commits" for me.
Are these on the XCode server or in the repository?

BTW, probably not related, but if I edit the Bot and examine the Repositories section it tells me:
"The server SSH fingerprint failed to verify."
It does checkout the repository fine, but maybe that is causing some issues with status?
For regular bots, I usually tell it ot use existing SSH keys and give it my keys... which are the same as the ones I selected in the Syncer setup.

[reststop]

Stepping through the code, on first pass (first breakpoint) after re-starting the existing bot, the value of commit is "", which returns 0 entries for headCommitIntegrations.
Examining the bot, the first integrations shows 5 commits. but all succeeding integrations on the bot show 0 commits.

[reststop]

Yes, it appears headCommit is always "" in syncPRWithBot

(lldb) p pr
(BuildaGitServer.BitBucketEnterprisePullRequest) $R43 = 0x0000618000062e40 {
  BuildaGitServer.BitBucketEnterpriseIssue = {
    BuildaGitServer.BitBucketEnterpriseEntity = {}
    number = 1
  }
  title = "Get clone SSH URL by name"
  source = 0x000061800008e6a0 {
    BuildaGitServer.BitBucketEnterpriseEntity = {}
    branch = "carl/stash3-ssh-fix"
    commit = ""
    repo = 0x00006180000d4f90 {
      BuildaGitServer.BitBucketEnterpriseEntity = {}
      permissions = (read = true, write = true)
      latestRateLimitInfo = (payload_data_0 = 0x0000000100e17da0 "NO INFO", payload_data_1 = 0x0000000000000007, payload_data_2 = 0x0000000000000000, instance_type = 0x0000000100e1c790 BuildaGitServer`type metadata for BuildaGitServer.BitBucketEnterpriseRateLimit, protocol_witness_0 = 0x0000000100e1c6a0 BuildaGitServer`protocol witness table for BuildaGitServer.BitBucketEnterpriseRateLimit : BuildaGitServer.RateLimitType in BuildaGitServer)
      sshURL = 0x0000618000102c70 "ssh://git@stash.company.com/project/buildasaur.git"
      originUrlSSH = "ssh://git@stash.company.com/project/buildasaur.git"
    }
  }
  destination = 0x0000618000088980 {
    BuildaGitServer.BitBucketEnterpriseEntity = {}
    branch = "feature/bitbucket_server"
    commit = ""
    repo = 0x00006180000d4f20 {
      BuildaGitServer.BitBucketEnterpriseEntity = {}
      permissions = (read = true, write = true)
      latestRateLimitInfo = (payload_data_0 = 0x0000000100e17da0 "NO INFO", payload_data_1 = 0x0000000000000007, payload_data_2 = 0x0000000000000000, instance_type = 0x0000000100e1c790 BuildaGitServer`type metadata for BuildaGitServer.BitBucketEnterpriseRateLimit, protocol_witness_0 = 0x0000000100e1c6a0 BuildaGitServer`protocol witness table for BuildaGitServer.BitBucketEnterpriseRateLimit : BuildaGitServer.RateLimitType in BuildaGitServer)
      sshURL = 0x00006180001021c0 "ssh://git@stash.company.com/project/buildasaur.git"
      originUrlSSH = "ssh://git@stash.company.com/project/buildasaur.git"
    }
  }
}
(lldb) p pr.headCommitSHA
(String) $R44 = ""

[reststop]

OK, found this issue.

It was an API issue with the PR. The wrong key was being used to get the PR info.
The latest version of Bitbucket Server (or at least one later than ours) uses the key "latestCommit" and the older version of Stash we have uses the key "latestChangeset"

Made that change, and now I'm back to debugging the former issue:
[ERROR]: Syncing encountered a problem. Error: Failed to post a status on commit 0e3f0b953c1bdc13a52b70561ff1d9eafd87e7f3 of repo fightclub/buildasaur. Context: SyncPair: PR (1:carl/stash3-ssh-fix) + Bot (β [buildasaur] carl/stash3-ssh-fix)

We can close this, I think.

[reststop]

I have a question which I will ask in another issue, since it is germane to all of the debugging I'm doing and would help me immeasurably.

[czechboy0]

This is where it's probably failing. Again, try to step through the code with the debugger to see why it's no working. My guess is that not all the required functions of a new server support were properly implemented. Please file a ticket in the fork's issue section, as I can't do anything to help here.