How to send signed AWS Elasticsearch request?

[bestickley]

Describe the issue with documentation

How to send signed AWS Elasticsearch request?

I've come with up the below solution. Interested to hear if others have a better solution. Regardless, this needs to be documented.

// createSignedHttpRequest.ts
import { SignatureV4 } from "@aws-sdk/signature-v4";
import { HttpRequest } from "@aws-sdk/protocol-http";
import { Sha256 } from "@aws-crypto/sha256-js";
import { defaultProvider } from "@aws-sdk/credential-provider-node";

interface CreateSignHttpRequestParams {
  body?: string;
  headers?: Record<string, string>;
  hostname: string;
  method?: string;
  path?: string;
  port?: number;
  protocol?: string;
  query?: Record<string, string>;
  service: string;
}

export async function createSignedHttpRequest({
  body,
  headers,
  hostname,
  method = "GET",
  path = "/",
  port = 443,
  protocol = "https:",
  query,
  service,
}: CreateSignHttpRequestParams): Promise<HttpRequest> {
  const httpRequest = new HttpRequest({
    body,
    headers,
    hostname,
    method,
    path,
    port,
    protocol,
    query,
  });
  const sigV4Init = {
    credentials: defaultProvider(),
    region: process.env.AWS_DEFAULT_REGION as string,
    service,
    sha256: Sha256,
  };
  const signer = new SignatureV4(sigV4Init);
  return signer.sign(httpRequest) as Promise<HttpRequest>;
}

import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { IncomingMessage } from "http";
import { createSignedHttpRequest } from "./shared/createSignedHttpRequest";
const nodeHttpHandler = new NodeHttpHandler();

(async () => {
  const body = JSON.stringify({
    query: {
      match: {
        title: "Moneyball",
      },
    },
  });
  const hostname =
    "<insert-es-endpoint>.us-east-1.es.amazonaws.com";
  const signedHttpRequest = await createSignedHttpRequest({
    method: "POST",
    body,
    headers: {
      "Content-Type": "application/json",
      host: hostname,
    },
    hostname,
    path: "/node-test/_doc/_search",
    service: "es",
  });
  console.log(signedHttpRequest);
  try {
    const res = await nodeHttpHandler.handle(signedHttpRequest);
    const body = await new Promise((resolve, reject) => {
      const incomingMessage = res.response.body as IncomingMessage;
      let body = "";
      incomingMessage.on("data", (chunk) => {
        body += chunk;
      });
      incomingMessage.on("end", () => {
        resolve(body);
      });
      incomingMessage.on("error", (err) => {
        reject(err);
      });
    });
    console.log(body);
  } catch (err) {
    console.error("Error:");
    console.error(err);
  }
})();

[joshp-f]

You can use two third party libraries, one that provides an elasticsearch wrapper client and one that transforms AWS credentials for the elasticsearch client.

import { Client } from '@elastic/elasticsearch'
import { createAWSConnection, awsGetCredentials } from '@acuris/aws-es-connection';
let client:Client | null = null;
async function getClient(){
  if (!client) {
    const awsCredentials = await awsGetCredentials()
    const AWSConnection = createAWSConnection(awsCredentials)
    client = new Client({ 
      ...AWSConnection,
      node: YOUR_ES_ENDPOINT,
    })
  }
  return client;
}
async function useesclientforsomestuff() {
  const client = await getClient();
}

[cvarjao]

@bestickley, thanks for this! its was very helpful. Have you figure out how to import saved objects using Kibana REST API? For some reason, I can't get that one to work. I think I having issues with figuring out:

The request body must include the multipart/form-data type
https://www.elastic.co/guide/en/kibana/master/saved-objects-api-import.html#saved-objects-api-import-request-body

I keep getting 400:

{"statusCode":400,"error":"Bad Request","message":"[request body.file]: expected value of type [Stream] but got [string]"}

[bestickley]

@cvarjao, you’re welcome!
Unfortunately no, I don’t have experience with that. Do note that you’re referencing elastic.co’s docs instead of the opendistro for elasticsearch docs which AWS supports (https://opendistro.github.io/for-elasticsearch/) not sure if that makes a difference

[cvarjao]

For future reference, I am using form-data library, and I forgot to properly set the filename property when appedning to the FormData object.

Building on the original suggestion, here are the highlight of my changes:

const FormData = require.main.require("form-data");
[...]
const form = new FormData();
form.append("file", fs.readFileSync(vizualizationFile,{encoding:'utf8'}), {contentType:'application/json', filename:'applications.ndjson'});
await executeSignedHttpRequest({
  body: form.getBuffer(),
  headers: {
    host: hostname,
    ...form.getHeaders()
  },
  path: '/_plugin/kibana/api/saved_objects/_import'
})

[dsmrt]

👋 @bestickley,

First off, thank you for the solution you put together here. It's been a huge help!

Everything has been working as expected except for wildcards in the index name. If we take your example and add a wildcard to the index (where the path looks like this: path: "/node-*/_doc/_search"), I get the following error:

{
  "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
}

If anyone's come across this error and knows how to fix it, let me know. Any help is appreciated.

[bestickley]

Hi @dsmrt, you're welcome! I've never run across that error.

[dsmrt]

For future reference ...

It looks like there's already a issue created for the signature v4 wildcard ("/node-*/_doc/_search") bug here: #1896

[dsmrt]

Furthermore, you can fix the wildcard issue by running the index name/target through an encoder function (detailed here, from v2: #1896 (comment)) or like this:

  const encodeTarget = (target: string):string => {
    var output = encodeURIComponent(target);
    output = output.replace(/[^A-Za-z0-9_.~\-%]+/g, escape);

    // AWS percent-encodes some extra non-standard characters in a URI
    output = output.replace(/[*]/g, function (ch) {
      return '%' + ch.charCodeAt(0).toString(16).toUpperCase();
    });
    return output;
  }

[efelipe402]

@bestickley thanks for the steps to send a signed request to elasticsearch, I'm using serverless framework and I was able to reproduce all steps describe, but I found some errors like could find a is-crt-available in aws @AWS-SDK

then I installed @aws-sdk/util-user-agent-node, aws-crt libraries and now it compiles with a warning

and the lambda prints the request signed, but shows and errors that could no find host and the host url is open to internet, I can access to it through browser as well to kibana without credentials with no problem

what do you recommend me, I've been searching for a month to connect it, with a lambda using auth, with no sucess.
only it works using postman requests to the open elasticsearch, all IAM permissions are set, only signed requests is what it holds me back.

thanks a lot.

this is my package dependencies

[bestickley]

@efelipe402, I cannot deep dive into your specific issue, but since you said "only it works using postman requests to the open elasticsearch", I'd print the request that's being sent in your Lambda to see how it differs from postman.
Additionally, you may want to look into opensearch-js, a fork of that latest elasticsearch js client that's compatible with opensearch. This issue appears to be gaining some traction to enable IAM authentication. Unfortunately it's not ready yet but hopefully will be in around a month (based on this comment).
For a quicker solution, check out the comment above.

[efelipe402]

thanks @bestickley, I will try it, I even tried out
https://stackoverflow.com/questions/38599786/proper-signing-of-requests-to-aws-resources-through-http

but throws and error this, even putting hard coded keys like secret and access key.

this signed request is a headache, what versions of nodejs packages did you use in order to make it work in your example or can you share the github repo ?

[bestickley]

@efelipe402, I can't share the repo, but hopefully this helps. Node.js version is in my package.json

import { FoundLog, Log } from "./log";
import { SearchResponse } from "./searchResponse";
import { GqlArgs } from "./index";
import { BatchGetItemCommand, DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { unmarshall } from "@aws-sdk/util-dynamodb";
import { Logger } from "src/shared/logger";
import { Got } from "got";
import { SearchLogConnection } from "./searchLogConnection";

const logger = new Logger();

interface SearchLogsParams {
  args: GqlArgs;
  ddbClient: DynamoDBClient;
  httpClient: Got;
}

export async function searchLogs({
  args,
  ddbClient,
  httpClient,
}: SearchLogsParams): Promise<SearchLogConnection> {
  const { ddbTable, esDomainEndpoint, esIndex } = getEnvVars();
  const { foundLogs, totalFound } = await searchElasticsearch({
    args,
    esDomainEndpoint,
    esIndex,
    httpClient,
  });
  let logs: Log[] = [];
  if (foundLogs.length) {
    logs = await getLogsDdb({ ddbClient, ddbTable, foundLogs });
  }
  return { logs, totalFound };
}

interface SearchElasticsearchParams {
  httpClient: Got;
  esDomainEndpoint: string;
  esIndex: string;
  args: GqlArgs;
}

async function searchElasticsearch({
  args,
  esDomainEndpoint,
  esIndex,
  httpClient,
}: SearchElasticsearchParams): Promise<{
  foundLogs: FoundLog[];
  totalFound: number;
}> {
  const { query, from = 0, size = 10 } = args;
  const res = await httpClient({
    allowGetBody: true,
    body: JSON.stringify({
      from,
      size,
      query: {
        multi_match: {
          fields: ["message", "note"],
          query,
        },
      },
      sort: [{ createdAt: "desc" }],
    }),
    headers: {
      "content-type": "application/json",
      host: esDomainEndpoint,
    },
    hostname: esDomainEndpoint,
    pathname: `/${esIndex}/_search`,
    protocol: "https:",
    responseType: "json",
  });
  logger.debug("res.body: ", res.body);
  const resBody = res.body as SearchResponse<Log>;
  return {
    foundLogs: resBody.hits.hits.map((h) => h._source),
    totalFound: resBody.hits.total.value,
  };
}

interface GetLogsDdbParams {
  ddbClient: DynamoDBClient;
  ddbTable: string;
  foundLogs: FoundLog[];
}

async function getLogsDdb({
  ddbClient,
  ddbTable,
  foundLogs,
}: GetLogsDdbParams): Promise<Log[]> {
  const keys: Record<string, { S: string }>[] = [];
  for (const foundLog of foundLogs) {
    keys.push({
      PK: { S: `LOG#${foundLog.id}` },
      SK: { S: foundLog.createdAt },
    });
  }
  const command = new BatchGetItemCommand({
    RequestItems: {
      [ddbTable]: {
        Keys: keys,
      },
    },
  });
  logger.debug(command);
  const res = await ddbClient.send(command);
  logger.debug(res);
  if (!res.Responses) {
    throw new Error("res.Responses is undefined from BatchGetItemCommand");
  }
  const logs = res.Responses[ddbTable].map((l) => {
    const item = unmarshall(l);
    item.createdAt = item.SK;
    item.id = item.PK.replace("LOG#", "");
    return item as Log;
  });
  return logs;
}

function getEnvVars() {
  if (!process.env.ES_DOMAIN_ENDPOINT) {
    throw new Error("process.env.ES_DOMAIN_ENDPOINT not set");
  }
  if (!process.env.ES_INDEX) {
    throw new Error("process.env.ES_INDEX not set");
  }
  if (!process.env.DDB_TABLE) {
    throw new Error("process.env.DDB_TABLE not set");
  }
  return {
    ddbTable: process.env.DDB_TABLE,
    esDomainEndpoint: process.env.ES_DOMAIN_ENDPOINT,
    esIndex: process.env.ES_INDEX,
  };
}

{
  "name": "lumberyard-lambdas",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "ts-node esbuild.ts",
    "test": "jest",
    "test:debug": "node --inspect-brk node_modules/.bin/jest --runInBand"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@aws-cdk/custom-resources": "^1.90.0",
    "@types/adm-zip": "^0.4.33",
    "@types/aws-lambda": "^8.10.72",
    "@types/jest": "^26.0.20",
    "@types/node": "^14.14.28",
    "aws-lambda": "^1.0.6",
    "esbuild": "^0.8.46",
    "eslint": "^7.20.0",
    "eslint-config-prettier": "^7.2.0",
    "eslint-plugin-prettier": "^3.3.1",
    "jest": "^26.6.3",
    "prettier": "^2.2.1",
    "ts-jest": "^26.5.2",
    "ts-node": "^9.1.1",
    "typescript": "^4.2.2"
  },
  "volta": {
    "node": "14.15.4"
  },
  "dependencies": {
    "@aws-crypto/sha256-js": "^1.1.0",
    "@aws-sdk/client-cloudfront": "^3.6.0",
    "@aws-sdk/client-cloudwatch-logs": "^3.5.0",
    "@aws-sdk/client-dynamodb": "^3.5.0",
    "@aws-sdk/client-lambda": "^3.5.0",
    "@aws-sdk/client-s3": "^3.5.0",
    "@aws-sdk/protocol-http": "^3.5.0",
    "@aws-sdk/signature-v4": "^3.5.0",
    "@aws-sdk/util-dynamodb": "^3.7.0",
    "@types/cookie": "^0.4.0",
    "@types/jsonwebtoken": "^8.5.0",
    "adm-zip": "^0.5.3",
    "agentkeepalive": "^4.1.4",
    "cookie": "^0.4.1",
    "got": "^11.8.1",
    "jsonwebtoken": "^8.5.1",
    "jwks-rsa": "^1.12.2"
  }
}

[macromania]

After spending a couple of nights on this, I ended up following working for me. I'm sharing here in case this is useful in the meantime...

Context
I'm using CDK to setup my cluster and have a Lambda to query. I use IAM principals to access the cluster.
For CDK part, I ended up adding following grants to my Lambda

// Just for development purposes, this is a small cluster
const searchDomain = new opensearch.Domain(this, "SearchDomain", {
    version: opensearch.EngineVersion.OPENSEARCH_1_0,
});

const searchHandler = new nodeLambda.NodejsFunction(this, "SearchHandler", {
    bundling: this.bundlingOptions,
    entry: `../src/search.ts`,
    runtime: lambda.Runtime.NODEJS_14_X,
    tracing: lambda.Tracing.ACTIVE,
    logRetention: 14,
    timeout: cdk.Duration.seconds(30),
    description: "Handles OpenSearch Queries",
    environment: {
        DOMAIN_ENDPOINT: searchDomain.domainEndpoint,
        DOMAIN_REGION: this.region,
    },
});

// Read only grant didn't work. So using Read&Write works at the moment
searchDomain.grantPathReadWrite("docs/*", searchHandler);

Implementation
Following Lambda code works within this setup.

import { APIGatewayEvent } from "aws-lambda";
import { ServiceResult } from "lambda-result";
import { HttpRequest } from "@aws-sdk/protocol-http";
import { SignatureV4 } from "@aws-sdk/signature-v4";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { Sha256 } from "@aws-crypto/sha256-browser";
import * as AWS from "aws-sdk";
import { IncomingMessage } from "node:http";

const host = process.env.DOMAIN_ENDPOINT!;
const region = process.env.DOMAIN_REGION!;
const index = "docs";
const creds = new AWS.EnvironmentCredentials("AWS");

exports.handler = async (event: APIGatewayEvent) => {
    const query = {
        query: {
            match: {
                id: "abcd",
            },
        },
    };

   // Use POST instead of GET for search
    const request = new HttpRequest({
        headers: {
            "Content-Type": "application/json",
            host: host,
        },
        hostname: host,
        method: "POST",
        path: `${index}/_search`,
        body: JSON.stringify(query),
    });

    const signer = new SignatureV4({
        credentials: creds,
        region: region,
        service: "es",
        sha256: Sha256,
    });
    const signedRequest = await signer.sign(request);

    const client = new NodeHttpHandler();

    try {
        //@ts-ignore
        const { response } = await client.handle(signedRequest);

        if (response.statusCode >= 400)
            return ServiceResult.Failed({
                errorMessage: `An error occurred while receiving search results`,
                errorType: "SearchError",
            });

        const body: string = await new Promise((resolve, reject) => {
            const incomingMessage = response.body as IncomingMessage;
            let body = "";
            incomingMessage.on("data", (chunk) => {
                body += chunk;
            });
            incomingMessage.on("end", () => {
                resolve(body);
            });
            incomingMessage.on("error", (err) => {
                reject(err);
            });
        });

        const result = JSON.parse(body);

        return ServiceResult.Succeeded(result.hits);
    } catch (e) {
        console.error(e);
        return ServiceResult.Failed({
            errorMessage: `${e}`,
            errorType: "SearchException",
        });
    }
};
    

Packages Used

{
  "devDependencies": {
    "@types/aws-lambda": "^8.10.85",
    "@types/node": "^16.11.7",
    "aws-sdk": "^2.1048.0",
    "typescript": "^4.4.4"
  },
  "dependencies": {
    "@aws-crypto/sha256-browser": "^2.0.1",
    "@aws-sdk/credential-provider-node": "^3.40.0",
    "@aws-sdk/node-http-handler": "^3.40.0",
    "@aws-sdk/protocol-http": "^3.40.0",
    "@aws-sdk/signature-v4": "^3.40.0",
    "lambda-result": "^1.0.1"
  }
}

Things to Note

1. I tried making GET request to search domain but didn't work. I kept getting 403. This is why I ended up using POST
2. To get POST endpoint working, I had to use grantPathReadWrite("docs/*", searchHandler); as I couldn't find an easier way in CDK to just give es:HTTPPost permission to Lambda. So, slightly I'm giving more privilege to my Lambda even though it never writes to the path
3. I tried Opensearch JS Client https://opensearch.org/docs/latest/clients/javascript/ but really couldn't wrap my head around how to get the correct credentials and connection setup for my client. Docs don't explain the IAM principle approach in great detail at the writing of this comment.

[windbeneathyourwings]

@bestickley

Thank you for contributing to this discussion. For no other reason than finding this gem I was able to retrofit your code to communicate directly via the browser through cors with a public aws open search domain. The portion regarding creating v4 signature was invaluable. I was able to to use your code as an example to generate a signature for a cognito identity pool and proxy the request through express.

For others who might have the same problem this what I did.

interface CreateSignHttpRequestParams {
  body?: string;
  headers?: Record<string, string>;
  hostname: string;
  method?: string;
  path?: string;
  port?: number;
  protocol?: string;
  query?: Record<string, string>;
  service: string;
  cognitoSettings: CognitoSettings,
  authFacade: AuthFacade
}

I added two inputs here to include cognito settings and auth facade to have easy access to the jwt.

export async function createSignedHttpRequest({
  body,
  headers,
  hostname,
  method = "GET",
  path = "/",
  port = 443,
  protocol = "https:",
  query,
  service,
  cognitoSettings,
  authFacade
}: CreateSignHttpRequestParams): Promise<HttpRequest> {
  const httpRequest = new HttpRequest({
    body,
    headers,
    hostname,
    method,
    path,
    port,
    protocol,
    query,
  });
  const sigV4Init = {
    credentials: fromCognitoIdentityPool({
      client: new CognitoIdentityClient({ region: cognitoSettings.region }),
      identityPoolId: cognitoSettings.identityPoolId,
      logins: {
        [`cognito-idp.${cognitoSettings.region}.amazonaws.com/${cognitoSettings.userPoolId}`]: () => authFacade.getUser$.pipe(map(u => u ? u.id_token : undefined), take(1)).toPromise()
      }
    }),
    region: cognitoSettings.region,
    service,
    sha256: Sha256,
  };
  const signer = new SignatureV4(sigV4Init);
  return signer.sign(httpRequest) as Promise<HttpRequest>;
};

I altered the function that generates the signed request using an identity pool credential provider instead. This is how I have been able to communicate directly with other aws services like s3 directly in the browser. I had a lot of trouble figuring out how to do this with open search since the sdk does not provide an abstractions to for searching that includes this info like s3 does. This approach seems to work.

In my component (angular) I was than able to successfully create the signed request and proxy it using express.

    const body = JSON.stringify({
      query: {
        match_all: {}
      },
    });
    const hostname =
      "search-domain-xxx.us-east-1.es.amazonaws.com";
    const signedHttpRequest = new Promise((resolve, reject) => {
      createSignedHttpRequest({
        method: "POST",
        body,
        headers: {
          "Content-Type": "application/json",
          host: hostname,
          // authority: "classifieds-ui-dev.auth.us-east-1.amazoncognito.com",
          // "Origin": "http://localhost:4200",
          // "Access-Control-Allow-Origin": "*",
          // "Access-Control-Allow-Headers": "*",
          // "Access-Control-Allow-Methods": "*"
        },
        hostname,
        path: '/classified_ads/_search',
        // port: 4000,
        protocol: 'https:',
        service: "es",
        cognitoSettings: this.cognitoSettings,
        authFacade: this.authFacade
      }).then(signedHttpRequest => {
        console.log('signedHttpRequest', signedHttpRequest);
        // return nodeHttpHandler.handle(signedHttpRequest).then();
        delete signedHttpRequest.headers.host;
        const url = `/opensearch${signedHttpRequest.path}`;
        console.log('url', url);
        this.http.post(url, signedHttpRequest.body, { headers: signedHttpRequest.headers, withCredentials: true }).pipe(
          tap(res => resolve(res))
        ).subscribe();
        return ''; //this.http.request(signedHttpRequest);
      });
    }).then(res => {
      console.log('completed request', res);
    });

On the back-end I'm using Angular serverside rendering with express. Therefore, I was able to use the npm package express-http-proxy to proxy the request out to the search domain and implicitly support cors in the process of doing so.

main.server.ts

  // Open search AWS proxy
  server.use('/opensearch', proxy('https://search-domain-xxxx.us-east-1.es.amazonaws.com', {
    proxyReqOptDecorator: proxyReqOpts => {
      proxyReqOpts.headers['host'] = 'search-domain-xxxx.us-east-1.es.amazonaws.com';
      return proxyReqOpts;
    }
  }));

An important note is that the v4 sig requires a host header. Therefore, the host header is included in the signature created in the browser but removed as part of the request to the proxy. This is done because the browser does not allow the host in the header like that. However, it must be part of v4 sig. The solution was to remove it than include it again inside the proxy before the request is passed to the open search domain.

Another point is that for Angular I have an interceptor that automatically adds the jwt to outbound request. This jwt should not be included in request to the open search proxy. Therefore, I added it to the exclusion list in my http interceptor. This is only applicable to Angular but interceptors seem to be a common pattern for many frameworks.

import { Injectable } from '@angular/core';
import { HttpInterceptor, HttpHandler, HttpRequest, HttpEvent } from '@angular/common/http';
// import { OktaAuthService } from '@okta/okta-angular';
import { AuthFacade } from 'auth';
import { Observable } from 'rxjs';
import { concatMap, take } from 'rxjs/operators';

@Injectable()
export class TokenInterceptor implements HttpInterceptor {

  constructor(private authFacade: AuthFacade/*, private oktaAuth: OktaAuthService*/) {}

  intercept(req: HttpRequest<any>, next: HttpHandler) {

    return this.authFacade.token$/*this.getAccessToken()*/.pipe(
      take(1),
      concatMap(t => {
        if (t && req.url.indexOf('cloudfront') === -1 && req.url.indexOf('cloudinary') === -1 && req.url.indexOf('carquery') === -1 && req.url.indexOf('gateway.marvel.com') === -1 && req.url.indexOf('hereapi.com') === -1 && req.url.indexOf('/opensearch') === -1) {
          const authReq = req.clone({
            // headers: req.headers.set('Authorization', `Bearer ${t}`)
            headers: req.headers.set('Authorization', t)
          });
          return next.handle(authReq)
        } else {
          return next.handle(req);
        }
      })
    );

  }

  /*getAccessToken(): Observable<string | undefined> {
    return new Observable((observer) => {
      this.oktaAuth.isAuthenticated().then((isAuthenticated: boolean) => {
        if(isAuthenticated) {
          this.oktaAuth.getAccessToken().then((token: string) => {
            observer.next(token);
            observer.complete();
          });
        } else {
          observer.next(undefined);
          observer.complete();
        }
      });
    });
  }*/

}

Also this merely a proof of concept. I now have a better understanding of the pieces that are needed to factor them into my application as modules or part of other libraries that already exist.

[efelipe402]

bestickley

@joshp-f @bestickley Thanks a lot for sharing I was able to sign the request using the library
from '@acuris/aws-es-connection'. :)

[zshzbh]

Hey everyone,

Thanks everyone for posting your own thoughts on this topic.

May I know if this issue still needs to be opened for discussion?

Cc: @cvarjao @dsmrt @macromania @joshp-f @efelipe402

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.