Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When can we expect NTLM support? #45

Open
Zleep-Dogg opened this issue Oct 14, 2015 · 38 comments
Open

When can we expect NTLM support? #45

Zleep-Dogg opened this issue Oct 14, 2015 · 38 comments

Comments

@Zleep-Dogg
Copy link

So... what does "in a month or so" mean? #20 (comment)

The month seemed short enough that I didn't want to bother implementing it myself, but we could really use the ntlm support soon... anything we can do to help the progress? I'll be happy to test things as well...

@mmajcica
Copy link

Same here. TFS 2015 is out for already some time and still it is not supported by the official tool?

@StevenBorg
Copy link

Yikes. Would really like to install the Octopus Deploy task, which is not supported in TFS 2015.1. But we need to hit TFS in a production environment for over 300 dev teams. Changing to accept Basic Auth just ain't gonna happen.

@yaakov-h
Copy link

Same issue here. It seems that as much as Microsoft have touted cross-platform builds on TFS 2015, it isn't actually supported out of the box.

Any update on this, @bryanmacfarlane?

@gravufo
Copy link

gravufo commented Jan 21, 2016

Please implement this feature! Enabling Basic Auth for on-premises servers is just ridiculous...

@bryanmacfarlane
Copy link
Contributor

Update:
Extensions will be supported on-prem in QU2. That's the official vehicle for build tasks. That will have a web UI for uploading tasks which of course works with integrated auth.

Regarding other tfs-cli commands. We wanted to implement in core CLR (xplat) but it was not ready when we created this tool. core CLR on windows will support NTLM and integrated auth (domain joined). Therefore we are not doing the node work for ntlm (which wouldn't support integrated). The core CLR port of tfs-cli will support the cmd line options back compat so really it's an internal implementation detail.

@jessehouwing
Copy link

Workaround is to use Fiddler to handle the auth for you:

@Allann
Copy link

Allann commented Mar 17, 2016

Same issue. TFS On-Prem should be as important as VSTS as many companies are not able to use VSO.

@bryanmacfarlane
Copy link
Contributor

Good news. We worked with the core CLR folks to ensure core CLR supports NTLM across Linux, OSX (and of course windows). As part of that work, we got our http clients building with core CLR and validated ntlm with linux/OSX/win.

Our new combined agent is building off of that. https://github.com/Microsoft/vsts-agent

That's paving the path for tfx to ported to core CLR. It's just a thin cmd line over our http clients anyways.

Of course we will maintain cmd line arg compat and it will encapsulate and bundler the core CLR with the tool.

Alot of work is leading up to the "right" solution. I know it seems like we're not doing work but it's building toward that.

@brettjacobson
Copy link

any update?

@bryanmacfarlane
Copy link
Contributor

Assigning to Trevor to address the plans here for cli.

Couple things have happened since I posted:

  • vso-node-api is getting ntlm added right now.
  • new agent is working with integrated and ntlm on all platforms so we've confirmed the net core work.

The difference between node api and net core is integrated on windows works. So, if this picks up node api ntlm work, it will prompt and persist domain creds. If it uses net core instead, integrated just works on windows (no need to enter or persist creds).

So, the user question is whether avoiding prompting/persisting domain creds is critical enough to avoid.

@brettjacobson
Copy link

I could live with having to enter my credentials when executing tfx login, as long as the credentials are encrypted with NTLM. (Since right now, using --auth-type basic doesn't work with https! That problem makes this problem doubly important)

@bryanmacfarlane
Copy link
Contributor

What's the issue with basic auth over https?

@mmajcica
Copy link

mmajcica commented Jun 6, 2016

There is a workaround. This post may be useful.
http://blog.jessehouwing.nl/2016/01/publish-build-tasks-to-tfs-2015-without.html?m=1

Mario
On 6 Jun 2016 17:52, "Bryan MacFarlane" notifications@github.com wrote:

What's the issue with basic auth over https?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#45 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AC4ud_NfsytstOw5-xTqENAoX-FkMz1Yks5qJEI3gaJpZM4GOqKV
.

@brettjacobson
Copy link

@bryanmacfarlane #63 basic auth + https no workie on premises

@shieldsjared
Copy link

Agree that NTLM is critical need for those using on-prem instances. Even writing extensions is bothersome because we can't easily automate publishing to our on-prem instance w/o using basic authentication (and we're not turning on basic auth!). As an alternative, providing on-premise with tokens like VSTS has would have sufficed. The code/deploy/test cycle is time consuming for extensions!

@dangmike
Copy link

Still waiting on NTLM support for our on-prem TFS servers. Getting close?

@T-Hugs
Copy link
Contributor

T-Hugs commented Jul 12, 2016

Yeah, I actually have done most of the work, but I'm really swamped with other things right now. My goal was to get it out by the end of this week. Thanks for hanging in there :)

@T-Hugs T-Hugs closed this as completed Jul 12, 2016
@T-Hugs T-Hugs reopened this Jul 12, 2016
@gravufo
Copy link

gravufo commented Jul 29, 2016

Bump. Any news on ETA?

@StevenBorg
Copy link

For the time being, you can always use Fiddler to intercept the traffic and do the authentication for you. The always brilliant JesseHouwing shows you how here: http://blog.jessehouwing.nl/2016/01/publish-build-tasks-to-tfs-2015-without.html

@StevenBorg
Copy link

Alternatively, you can package it as a VSIX extension, and upload it to the private TFS marketplace. (Or so I've heard...)

@jessehouwing
Copy link

Or use something like; http://ntlmaps.sourceforge.net/ to have something that's always on.

@gravufo
Copy link

gravufo commented Aug 1, 2016

Listen, I know the workarounds. Why don't you fix the problem at the source?
@trevorsg said he was pretty much done with the code and wanted to ship ~20 days ago. I was asking what the status is on that, not "what are the workarounds".

Considering this issue has been open for almost a year now, I think it's quite normal that people are starting to be impatient.

Thanks for providing feedback at least, though.

@T-Hugs
Copy link
Contributor

T-Hugs commented Aug 1, 2016

@gravufo - Unfortunately, the status is still the same. There's just a lot of stuff going on with finalizing TFS Dev15. This is the last week of the sprint, so it will be tricky to get this finished this week, but maybe I'll find a late night to work on it :)

@shieldsjared
Copy link

shieldsjared commented Aug 1, 2016

Well the news that dev15 is in its last sprint gives me a little less heartburn on the topic!

@gravufo
Copy link

gravufo commented Aug 1, 2016

No problem, that's all I wanted to know :)
It just helps us plan accordingly.
Thanks for the fast feedback!

@StevenBorg
Copy link

@gravufo
Christian, I don't think Jesse and I meant to imply you didn't know the workarounds yourself. You've been on the thread too long for that. Our replies, however, are now there for those searching for a solution for their first time. As they come across this thread, they'll be able to maybe get past a roadblock that would have stopped them from moving forward. I know I appreciate the workarounds when I end up at a GitHub issue link from Bing / Google / DuckDuckGo. So, think of our comments as more of a community service than an insult to your google-foo. :-)

@gravufo
Copy link

gravufo commented Aug 1, 2016

Right, I didn't take it as an insult. I just thought you were trying to answer a question that wasn't asked :)
I agree that it's great to have workarounds on Github, I love seeing them too. The first workaround was already posted in this same thread, but I guess adding the two other ones is worth mentioning.
No worries and cheers!

@T-Hugs
Copy link
Contributor

T-Hugs commented Aug 1, 2016

@shieldsjared Just to clarify, I said we're on the "last week of the sprint", not necessarily that this is the last sprint of Dev15 :)

@shieldsjared
Copy link

Ah, you're right! I guess I read "what I hoped" it said.  Excited for the next build, it's a bummer seeing all the goodness making it to VSTS before us on-prem folks.... But I get it!
Thanks for putting me back into reality :/

@renatotkr
Copy link

any updates?

@swt-das
Copy link

swt-das commented May 5, 2017

Considering that "in a month or so" was a good 20-something months ago I assume we can consider this issue abandoned. Having to hardcode credentials in automated builds is problematic for all the obvious reasons.

Could we maybe get the patch that contains the work done up to now, so that someone else doesn't have to duplicate already done work?

@PeteW
Copy link

PeteW commented May 5, 2017

Adding to the list of teams which invest in TFS on-prem that wont be able to use your CLI creation because of this significant gap in functionality

@bryanmacfarlane
Copy link
Contributor

TFS2017 on-prem supports PAT tokens (without enabling basic auth)

@swt-das
Copy link

swt-das commented May 8, 2017

@bryanmacfarlane And how do I use them in automated builds without hardcoding the token in some script?

As far as I can tokens aren't any more secure than using basic authentication and hardcoding username and password for that scenario (I just use a different method to revoke access and the windows user needs more configuration to be secure).

@bryanmacfarlane
Copy link
Contributor

@swt-das How do you use it in a build without hardcoding in a script?

The build generates a token per build which expires after the build. That token represents the project or collection service account (depending on what you set in the options tab for scope). You can assign permissions etc... to that account. It's available to task authors and ad-hoc scripts (if you check that option on the options tab).

That is the most secure option since it's per build, not persisted anywhere and masked from logs automatically. No domain credentials could be leaked and if somehow compromised it's useless when the build ends.

Example in a task (would be same variable in a script). https://gist.github.com/bryanmacfarlane/154f14dd8cb11a71ef04b0c836e5be6e#file-vsts_sample-ts

To answer your specific question on PAT tokens. That's the second best option. You are correct that you shouldn't hard code in scripts - bad. In the build definition, there's a variables tab. Each variable has a lock icon next to it. If you lock the variables, it's securely stored and available to your build scripts as a variable. Because you locked it, it will be masked from logs. If you do that, then don't do the login option of tfx (will persist) - pass as cli args on invocations. Or use the rest clients from scripts as noted above. That's the second best option because (1) it's a token - not your domain credentials if leaked - scoped to scopes of the token - not everything on your domain as you (2) it can be revoked independently of your account and (3) and it can be masked from logs. But it's not as good as the per job token since if compromised, has a lifetime longer than the build.

Regarding NTLM, we have to separate NTLM from integrated. Integrated means your locally logged on identity. Where as ntlm support in this context means supplying the user name and password. That's less secure because if compromised, it's scope is you on the domain, not revokable independent of the account and would need to be persisted (the node-api typescript module tfx uses has NTLM, not integrated).

The most secure option for automating in a build would be to add an option in the cli to use the per build token we generate. It also works and untrusted domain scenarios.

@swt-das
Copy link

swt-das commented May 8, 2017

@bryanmacfarlane I thought that "NTLM" was referring to using the domain credentials of the logged in user (I was wondering why one would limit this to NTLM and not allow Kerberos though..), similar to what Invoke-WebRequest -UseDefaultCredentials would do. If that's not the case then it's not particularly helpful for my scenario anyhow, good to know.

The solution with a per-build token is perfect and I wasn't aware of that option. Clearly that's the way forward. Is there also a sample somewhere for how this would look in a PowerShell script?

@swt-das
Copy link

swt-das commented May 10, 2017

@bryanmacfarlane So I created a task based on the sample code and the code in app/exec/build/tasks/upload.ts.

The token I get back from tl.getEndpointAuthorizationParameter('SYSTEMVSSCONNECTION', 'AccessToken', false); is 755 characters long, so we're taking the apim.getBearerHandler route. But when trying to upload the task I get a 405 error.

import vm = require('./lib/jsonvalidate');
import trace = require('./lib/trace');
import tl = require('vsts-task-lib/task');
import trm = require('vsts-task-lib/toolrunner');
import * as apim from 'vso-node-api';
import taskagentm = require('vso-node-api/TaskAgentApi');
import path = require('path');
import archiver = require('archiver');

var c_taskJsonFile: string = 'task.json';

function getTaskAgentApi() : taskagentm.ITaskAgentApi {
    let url = tl.getEndpointUrl('SYSTEMVSSCONNECTION', false);
    console.log(`Connecting to ${url}.`);
    let token = tl.getEndpointAuthorizationParameter('SYSTEMVSSCONNECTION', 'AccessToken', false);
    let auth = token.length == 52 ? apim.getPersonalAccessTokenHandler(token) :
                                    apim.getBearerHandler(token);
    let vsts: apim.WebApi = new apim.WebApi(url, auth);
    let agentApi = vsts.getTaskAgentApi(url);
    return agentApi;
}

async function createTaskArchive(taskPath : string) : Promise<archiver.Archiver> {
    let archive = archiver('zip');
    archive.on('error', function(error) {
        console.log(`Archiving error: ${error.message}`);
        error.message = `Archiving error: ${error.message}`;
        throw error;
    });
    archive.directory(path.resolve(taskPath), false);
    archive.finalize();
    return archive;
}

async function run() {
    try {
        let taskPath = tl.getInput('taskDirectoryPath', true);
        vm.exists(taskPath, `specified directory ${taskPath} does not exist.`);
        let agentApi = getTaskAgentApi();
        let tp = path.join(taskPath, c_taskJsonFile);
        var taskJson = await vm.validate(tp, `no ${c_taskJsonFile} in specified directory.`);
        let archive = await createTaskArchive(taskPath);

        console.log('Upload archive.');
        await agentApi.uploadTaskDefinition(null, archive, taskJson.id, false);
        console.log('Success uploading');
    }
    catch (err) {
        tl.setResult(tl.TaskResult.Failed, err.message);
    }
}

run();

@kasajian
Copy link

I think that there's a much simpler solution to this whole thing. This isn't about NTLM, because you probably want Kerberos anyway. What you're really asking for single-sign-on, via "Integrated Windows Authentication" -- SPNEGO.

The problem is that with NodeJS, the http stack doesn't support this -- and to use the "kerberos" module is problematic.

What I've done in other projects when I want to use NodeJS code that will do REST to unprem TFS server is to make the HTTP calls not via the HTTP module but via the libcurl module. Works great, and you don't have pass in user-name and password.. it uses the credentials of whatever you're logged in as. Of course you can pass it in. But the point is, you don't need to use insecure "basic authentication" which requires you to update the TFS server IIS box every time you upgrade TFS, nor do you need a "personal access token" -- you're already authenticated.

Here's examples that you can prove to yourself this works with regular curl.exe and with node-libcurl.

Using regular curl:
xargs -a url_tfs.txt curl -negotiate -u :
where tfs.txt is a text file containing the REST endpoint such as:
http://mylocaltfs:8080/tfs/mycol/_apis/wit/WorkItems?ids=42&fields=System.Id,System.WorkItemType,System.Title,System.State&api-version=1.0

(I put the url in a text file instead of the command-line so that I can avoid having to escape the ampersands)

If you want to know why user name is being passed as ":", then read the curl manual (curl --manual). it's a weird reason.

And here's an example of how to do this with node:

var endpoint = urlString;
var url = require("url");
var endpointUrl = url.parse(endpoint);

var Curl = require( 'node-libcurl' ).Curl;
var curl = new Curl();

curl.setOpt( 'USERNAME', '' );
//curl.setOpt( 'VERBOSE', 1 );
curl.setOpt( 'URL', endpoint );
curl.setOpt( 'HTTPAUTH', Curl.auth.NEGOTIATE );
curl.setOpt( 'NOPROXY', endpointUrl.hostname );

curl.on( 'end', function( statusCode, body, headers ) {

	if (statusCode === 200) {
		console.log(body);
		cb(null, { statusCode, body, headers } ); 
	} else {
		cb(new Error(), { statusCode, body, headers } ); 
	}

	this.close();
});

curl.on( 'error', curl.close.bind( curl ) );
curl.perform();

So what you want to do is add a new authentication type (call it "negotiate" or something), and then use this method of making REST calls instead of what's currently done.

I don't have a POST example, but with curl, libcurl, and node-libcurl being popular modules, I'm sure it wouldn't be too hard to come up with a sample that works with on-prem TFS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests