I have updated my build number configuration in the VSTS build definition to $(GitVersion_NuGetVersionV2) as per the documentation in order to get the build number to match the package version being deployed.
One of my builds had the GitVersion build step included which was being executed before the msbuild step. The issue this caused is that the GitVersion build step assigned the default git version label as the build number rather than first evaluating whether the build number was environment variable which is what GitVersionTask does. As this was set by the GitVersion build step, the environment variable was lost from the build number and then GitVersionTask via msbuild could not assign the correct build number.