GraalVM and Clojurescript compatibility

[jeroenvandijk]

Thank you for developing and sharing this library!

I would love to use (parts) of this library in other contexts than the normal JVM:

• A native image with GraalVM
• A NodeJS environment (via Clojurescript or Sci)

In these more exotic environments, it is not possible to use this library due to its dependencies on JVM specifics or incompatibilities with GraalVM. The consequence is that one cannot benefit from all the work that has gone into this library. In order to increase the reach of this library I would like to suggest to consider extracting parts of this library and make it CLJC compatible.

To give a specific example, I'm building a GraalVM command line tool that needs to call the AssumeRole API (and others). There is no JVM library that is compatible with GraalVM that allows me to do this. We have solved this by shell-ing out to the python AWS cli tool. Now we see this same issue in an AWS Lambda context. There are new constraints that require a new non-clj "solution".

If parts of this library were available to me in these environments, I wouldn't have to shell out or find some other workaround. With minimal code, I would be able to do the API call myself.

I think the most important parts of this library are ready to go as is. The part that I need most is

• [cognitect.aws.signers]
  The url-encoding needs a platform specific implementation (e.g. js/encodeURIComponent(s))

Ideally, there would be a function that would "export" the request that will be done. E.g. (aws/invoke sts {:op :AssumeRole :request { :RoleArn "role-arn-here", :RoleSessionName "session-name-here"}}) would give me something like*:

{:remote-addr 127.0.0.1, :headers {accept-encoding gzip, authorization AWS4-HMAC-SHA256 Credential=ABC/20200130/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=698cb22807c63f8417d99ebaa234eb7173b000a2256a7e01ee00171e3b974963, content-length 92, content-type application/x-www-form-urlencoded; charset=utf-8, host localhost, user-agent Jetty/9.4.15.v20190215, x-amz-date 20200130T124646Z}, :async-channel #object[org.httpkit.server.AsyncChannel 0x4232f55e /127.0.0.1:8080<->/127.0.0.1:56095], :server-port 80, :content-length 92, :websocket? false, :content-type application/x-www-form-urlencoded, :character-encoding utf-8, :uri /, :server-name localhost, :query-string nil, :body #object[org.httpkit.BytesInputStream 0x730737c BytesInputStream[len=92]], :scheme :http, :request-method :post}
BODY: Action=AssumeRole&Version=2011-06-15&RoleSessionName=session-name-here&RoleArn=role-arn-here

Thank you for your attention! Feel free to ask me for more clarification on this subject.

(*) I intercepted this request by overriding the endpoint and running a local server:

(def sts (aws/client {:api :sts 
                      :credentials-provider (credentials/basic-credentials-provider
                        
                                             {:access-key-id     "ABC"
                                             :secret-access-key "XYZ"})     
                      :endpoint-override {:protocol :http
                                          :hostname "localhost"
                                          :port     8080}}))

[dchelimsky]

#20 proposes the same solution, yes?

[jeroenvandijk]

#20 is very similar indeed. However I'm not only interested in using a different HTTP client (and transmit it over ssh), I'm also suggesting to consider extracting bits that are platform-agnostic to a library so that those parts can be used on other platforms (e.g. GraalVM or NodeJS). The parsing of XML and JSON, signing etc. would need to be implemented on these other platforms.

Btw for those who is interested, I found a way to export the request map

[holyjak]

I have once tried to build aws-api into a GraalVM native binary and another problem that I run into was, I believe, related to the use of locking, likely due to https://clojure.atlassian.net/browse/CLJ-1472.

[digash]

I was very excited to use Clojure again. I've ran into this #120 trying to use babashka as AWS Batch Lambda, bummer, back to python.

[viesti]

On babashka, there's been talk about a babashka AWS pod in the Clojurians Slack #babashka channel. The idea being that one could expose the AWS libs from for example Go to babashka. I did a trial in which I included the babashka Lambda layer and awscli Lambda layer and just shelled out to awscli, seemed to work well enough for small scripting at least :)

[galdolber]

For anyone interested, I'm working on a version that runs on GraalVM native image.

Significant changes:

• service descriptor has to be provided (this allows to remove docs and unused functions to reduce the binary size)
• remove all shared resources: http-client, credentials-provider and region-provider have to be provided
• retry support is handled outside the invoke api (from the http-client you provide)
• embed default endpoints
• Remove all reflection
• Replace json implementation with cheshire
• Replace defmulti usage on cognitect.aws.client
• Remove core.async dependency
• Remove clojure.logging dependency
• Extract doc and validate to different namespaces
• Use patched data.xml that avoids reflection
• Client is a simple map, no longer a deftype

Sample:

(ns demo.core
  (:require [cognitect.aws.client.api :as aws]
            [cognitect.aws.credentials :as credentials]
            [cognitect.aws.region :as region]
            [org.httpkit.client :as http]
            [cheshire.core :as json]))

(set! *warn-on-reflection* true)

(defn http-client [{:keys [uri scheme server-port server-name] :as req}]
  @(http/request (assoc req :url (str (name scheme) "://" server-name ":" server-port uri))))

(def credentials-provider (delay (credentials/default-credentials-provider http-client)))

(def region-provider (delay (region/default-region-provider http-client)))

;; This is only processe on compile time, the file is not needed on production
;; Descriptor file from https://raw.githubusercontent.com/aws/aws-sdk-js/master/apis/lambda-2015-03-31.normal.json
(def lambda-api (json/parse-string (slurp "lambda-2015-03-31.normal.json")))

(def lambda
  (delay (aws/client
          {:api :lambda
           :http-client http-client
           :service lambda-api
           :credentials-provider @credentials-provider
           :region-provider @region-provider})))

(defn -main [& _]
  (println (aws/invoke @lambda {:op :ListLayers})))

It's experimental, undocumented and not 100% compatible with aws-api, but it works, and you can check it out here:
https://github.com/galdolber/aws-api-lite/

[BrunoBonacci]

FYI: for anyone looking to compile aws-api with GraalVM native-image you can find a working example at:
https://github.com/BrunoBonacci/graalvm-clojure

[kwojcik-blockether]

Regarding GraalVM it seems that the library works great when both static-load of libraries are used and .edn files are included in resources. Example here.

Found some reflection warnings though which should be fixed:

Reflection warning, clojure/data/xml/jvm/name.clj:35:1 - call to static method decode on java.net.URLDecoder can't be resolved (argument types: unknown, java.lang.String).
Reflection warning, clojure/data/xml/jvm/name.clj:38:1 - call to static method encode on java.net.URLEncoder can't be resolved (argument types: unknown, java.lang.String).
Reflection warning, cognitect/aws/credentials.clj:254:24 - reference to field toInstant on java.lang.Object can't be resolved.
Reflection warning, cognitect/aws/protocols/rest.clj:29:26 - call to method endsWith on java.lang.Object can't be resolved (no such method).
Reflection warning, cognitect/aws/protocols/rest.clj:31:46 - call to method substring on java.lang.Object can't be resolved (no such method).
Reflection warning, cognitect/aws/protocols/rest.clj:33:32 - call to method replace on java.lang.Object can't be resolved (no such method).
Reflection warning, cognitect/aws/protocols/rest.clj:34:32 - call to method replace can't be resolved (target class is unknown).
Reflection warning, cognitect/aws/protocols/rest.clj:61:18 - call to method contains can't be resolved (target class is unknown).

@dchelimsky will you accept a PR fixing those warnings?

[dchelimsky]

@dchelimsky will you accept a PR fixing those warnings?

Sorry, but no. We don't accept PRs for this project: https://github.com/cognitect-labs/aws-api#contributing

[dchelimsky]

I addressed all of the warnings in aws-api here (will be part of next release): b65d61d

[jeroenvandijk]

@dchelimsky Thank you! We use the latest version in https://github.com/babashka/pod-babashka-aws

@BrunoBonacci Thanks for your example. The last bit needed to get this library working in Babashka