Study possible ways to get devfile content from private Bitbucket repository during factory resolving process

[skabashnyuk]

Is your task related to a problem? Please describe.

We would like to allow users to use private Bitbucket git repositories in factories.

Describe the solution you'd like

To enable that we need to somehow get the content of the file.
I see at least two directions

1. With "app token" we can call curl -u skabashnyuk:cSwXKMeR9ua3jYa4nQzh https://api.bitbucket.org/2.0/repositories/skabashnyuk/test1/src/156c52d2a02d0744c3a9185ee8c27574a41d64c8/README.md
2. Connect Keycloak with Bitbacket's OAuth2. Find a way how to use oauth token to get content of the file or create/get "app token".

Describe alternatives you've considered

Store "app token" as a secret in k8s namespace.

Additional context

#17829

[mshaposhnik]

Manual steps to make private repo clone or file access:

0. Register OAuth consumer in the Bitbucket workspace (doc: https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/)

1. Add BB identity provider into Keycloak. Note that storing tokens must be ON to allow to obtain OAuth tokens via REST.
   

2. Login via Identity provider. Get the Keycloak access token. Obtain the OAuth token using the following request (doc: https://github.com/keycloak/keycloak-documentation/blob/master/server_development/topics/identity-brokering/tokens.adoc):
   GET -H "Authorization: <Access_token>" https://keycloak-eclipse-che.192.168.39.211.nip.io/auth/realms/che/broker/bitbucket/token

OAuth Token responce must be as follows:

{"scopes": "project team account", "access_token": "s1oWBhx82xLd4dM6pV-0et4Hln2hYlt-A74NGgqvDrlwLvmjEKcJl1zunVE8w9BTvXWb5SZ5L2QOvcX8uSzloZss24bhM6713CUGRz0DeGmTD7MiPSbtgku_", "expires_in": 7200, "token_type": "bearer", "state": "authorization_code", "refresh_token": "NgPXffWYf7C2J3vGCF"}

3. Clone repo or get a particular file useing OAuth token:

git clone https://x-token-auth:<token>@bitbucket.org/<username>/<repository_name>.git

or

curl -H "Authorization: Bearer <token>" https://api.bitbucket.org/2.0/repositories/<username>/<repository_name>/src/<workspace_hash>/README.md

[mshaposhnik]

So, since OAuth 2 flow is only available on BitBucket SAAS, we was forced to get back to OAuth 1 (Called Application Links on BitBibucket).
To use the flow, here is the steps required:

1. Prepare signature keys:

SSH to your BitBucket Server instance and generate RSA key-pairs
# generate the key pairs
$ openssl genrsa -out private.pem 1024
# copy the key and pem to a location where it can be accessed
$ openssl rsa -in private.pem -pubout > public.pub`
$ openssl pkcs8 -topk8 -inform pem -outform pem -nocrypt -in private.pem -out privatepkcs8.pem 

2. Configure app link

In the Bitbucket Server as an Admin:
1. Go to Administration -> Application Links
2. Enter your Che URL in the 'application url' field and press the 'Create new link' button.
3. When  the 'Configure Application URL' window appears press the 'Continue' button.
4. In the ‘Link applications’ window fill in the following fields and press the 'Continue' button:
  - Application Name : Che
  - Service Provider Name : Che
  - Consumer key : {added by you} (Save this string, you will need it later)
  - Shared secret : {added by you} 
  - Request Token URL : {your Bitbucket Server URL}/plugins/servlet/oauth/request-token
  - Access token URL : {your Bitbucket Server URL}/plugins/servlet/oauth/access-token
  - Authorize URL : {your Bitbucket Server URL}/plugins/servlet/oauth/authorize

Add Che oAuth Info on next screen:

In the following screen press the pencil icon to edit and select the 'Incoming Authentication' menu. Fill in the following fields in the ‘OAuth’ tab:
- Consumer Key : {the consumer key from the previous step}
- Consumer Name : Che
- Public Key : {the key from public.pub file} (The key must not contain ‘-----BEGIN PUBLIC KEY-----’ and ‘-----END PUBLIC KEY-----’ rows)

3. Configure Che

In the `che.properties` file add (or change if they already exist):
- bitbucket_consumer_key : the consumer key you have entered before
- bitbucket_private_key : the key from privatepkcs8.pem file 
- (The key must not contain ‘----BEGIN PRIVATE KEY-----’ and ‘-----END PRIVATE KEY-----’ rows) (The key must be specified as a single raw)
- bitbucket_endpoint : replace the default Bitbucket url by your Bitbucket Server url

4. Restart Che

5. Initiate Oauth Login procedure:

https://<che-host>/api/oauth/1.0/authenticate?userId=<userId>&oauth_provider=bitbucket-server&request_method=POST&signature_method=rsa&redirect_after_login=<che-host>&token=<token>

If necessary, enter login information and press "Allow" on request.

6. Retrieve authorization information:

https://<che-host>/api/oauth/1.0/signature?oauth_provider=bitbucket-server&request_method=POST&request_url=https://<bitbucket-host>/rest/api/latest/projects/<project>/repos/<repo>/raw/devfile.yaml&user_id=<userId>&token=<token>

This will return the following content:

OAuth oauth_consumer_key="key123321", oauth_nonce="43e6575f5642b788", oauth_signature="m0SyHKjJMoEBZKqLq3PDLDnM5eydeB%2BsU4zj9lKYFnA%2FNgeFATaWWMtWXMW%2Fe1CYLN1fS%2FpCocWxAOaeW0P%2BPCPINr3QFsOSKYjNqldNra3MUJtgOyMZekUVHkejlyVnRPuhdVLNpdebDZND%2F9pWCsD7SDVLJ2H5kBsV%2Bf0fE4c%3D", oauth_signature_method="RSA-SHA1", oauth_timestamp="1605255164", oauth_token="hf48oD8v4LkQOpniRIOhVkBeuIqWcBcG", oauth_version="1.0"

7. Make signed query to REST api passing previous step result as an authorization header value:

curl  
-H 'Authorization: OAuth oauth_consumer_key="key123321", oauth_nonce="43e6575f5642b788", oauth_signature="m0SyHKjJMoEBZKqLq3PDLDnM5eydeB%2BsU4zj9lKYFnA%2FNgeFATaWWMtWXMW%2Fe1CYLN1fS%2FpCocWxAOaeW0P%2BPCPINr3QFsOSKYjNqldNra3MUJtgOyMZekUVHkejlyVnRPuhdVLNpdebDZND%2F9pWCsD7SDVLJ2H5kBsV%2Bf0fE4c%3D", oauth_signature_method="RSA-SHA1", oauth_timestamp="1605255164", oauth_token="hf48oD8v4LkQOpniRIOhVkBeuIqWcBcG", oauth_version="1.0"'

https://bitbucket-bitbucket.apps.cluster-cb82.cb82.example.opentlc.com/rest/api/latest/projects/TEST/repos/test1/raw/devfile.yaml

Content of the file can be returned. That can also be used for app tokens creation, SSH keys upload etc. (with corresponfing URL changes etc)

[skabashnyuk]

At this moment we may suggest the following flow

Prerequisites: Users should have one k8s namespace created.

Flow:

1. Users click on the factory link
2. Dashboard call factory API to resolve factory
3. Che server notices that there are no bitbucket secrets in the user's k8s namespace and he is not authorized against Bitbucket server.
   Che server asked dashboard to initiate OAuth1 flow against the Bitbucket server and try again.
4. Che server with help of OAuth1 token create new creates bitbucket personal access token and two k8s secretes
   4.1 Bitbacket personal access token

apiVersion: v1
kind: Secret
metadata:
  name: bitbucket-personal-access-token-secret
  labels:
    app.kubernetes.io/part-of: che.eclipse.org
    app.kubernetes.io/component: bitbucket-personal-access-token
  annotations:
    che.eclipse.org/expired-after: 12323423
    che.eclipse.org/bitbucket-userid: 34593094509
    che.eclipse.org/bitbucket-username: 'tshevchenko'
data:
  token: ZHNhZmFkZmFzZGZxMzRyMzRyMTM0

4.2 Git credential secret

apiVersion: v1
kind: Secret
metadata:
  name: git-credentials-secret
  labels:
    app.kubernetes.io/part-of: che.eclipse.org
    app.kubernetes.io/component: workspace-secret
  annotations:
    che.eclipse.org/automount-workspace-secret: 'true'
    che.eclipse.org/mount-path: /home/theia/.git-credentials
    che.eclipse.org/mount-as: file
    che.eclipse.org/git-credential: 'true'
    che.eclipse.org/bitbucket-personal-access-token-secret: 'bitbucket-personal-access-token-secret'
data:
  credentials: aHR0cHM6Ly91c2VyOnBhc3NAZXhhbXBsZS5jb20K

5. Che server with help of "bitbucket personal access token " gets content of the devfile
6. Dashboard creates a new workspace and starts it.
7. Workspace started, git-credentials-secret mounted as a file that allows to clone/push git operation with private Bitbucket repository

CC @l0rd @mshaposhnik @sparkoo @metlos @sleshchenko