[TT-8735] FIPS mode for Tyk

[erweinst]

Is your feature request related to a problem? Please describe.
We are using Tyk as our main api gateway in a numerous products running in production. It is a containerized version we compile from the code base and push it to an internal docker registry.
As for a security auditing process we are running, we need to ensure all components are FIPS compliant.
Nginx has that option available: https://docs.nginx.com/nginx/fips-compliance-nginx-plus/

Describe the solution you'd like
We'd like to have Tyk with FIPS mode.

Describe alternatives you've considered
Unfortunately we don't have alternatives. We must have this FIPS mode on or we need to migrate to another gateway solution.

[buger]

For the ones who need more context, here are good article explaining what it is in Go context:

• https://gokulchandrapr.medium.com/go-crypto-and-kubernetes-fips-140-2-fedramp-compliance-66d852ccccd2
• https://kupczynski.info/posts/fips-golang/

As for Tyk itself, it feels that supporting FIPS will be compiling Tyk with a modified version of Go.

Also I found relatively recent news that BoringCrypto support was merged as experimental feature to Go upstream golang/go#51940, and it is possible to use it starting from Go 1.19.

At the moment we do have plan for ugrading to newer Go in next major release, but also have not yet decided if it will be 1.18 or 1.19. This request will defo change the weight in side of 1.19, but no guaratees of it yet.

Also keep in mind that we try to prioritise features based usage data, customer value and overall feedback. So if you are existing Tyk customer, pls also ping us to a support@tyk.io, it will help us with prioritization. If not, why are you still not a customer? 😄 But if serious, in any case write to us, and maybe you can be our early tester and help us build this functionality.

Thank you!

[erweinst]

Hey @buger, thanks for your response!
I'm trying to upgrage the go version and use the GOEXPERIMENT=boringcrypto flag but I think that BoringCrypto isn't installed correctly (the assertion line fails).
Here is a snippet from the Dockerfile:

# Install and configure go
ENV GOPATH /go
# todo: upgrade golang version
ENV GOLANG_VERSION 1.19.1

ENV GOLANG_FILE go$GOLANG_VERSION.linux-amd64.tar.gz
RUN apk add --update build-base libc6-compat && \
    wget -q https://golang.org/dl/$GOLANG_FILE && \
    tar -xf $GOLANG_FILE -C /usr/local/

RUN	export PATH="/usr/local/go/bin:$PATH"; \
	go version

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
WORKDIR $GOPATH
WORKDIR $HOME/go/src/github.com/TykTechnologies

RUN mkdir /plugins
COPY plugins /plugins
RUN git clone --depth 1 --branch v4.0.3 https://github.com/TykTechnologies/tyk.git  &&\
	cd tyk && \
	CGO_ENABLED=1 NOGOPLUGIN_ENABLED=0 go build -tags 'goplugin' && \
	cp -r /plugins . && \
	cd plugins && \
	GOEXPERIMENT=boringcrypto && \
	go build -buildmode=plugin -o logRouteMiddleware.so && \
	cd .. && \
	go tool nm tyk > tags.txt && \
    grep '_Cfunc__goboringcrypto_' tags.txt 1> /dev/null
` ``

[buger]

Have not tried this Dockerfile personally yet, but I see a mistake that you set GOEXPERIMENT variable before building plugin, and not for tyk itself.

So the last line can look like:

RUN git clone --depth 1 --branch v4.0.3 https://github.com/TykTechnologies/tyk.git  &&\
        export GOEXPERIMENT=boringcrypto && \
	cd tyk && \
	CGO_ENABLED=1 NOGOPLUGIN_ENABLED=0 go build -tags 'goplugin' && \
	cp -r /plugins . && \
	cd plugins && \
	go build -buildmode=plugin -o logRouteMiddleware.so && \
	cd .. && \
	go tool nm tyk > tags.txt && \
    grep '_Cfunc__goboringcrypto_' tags.txt 1> /dev/null

[erweinst]

That was the thing! missed that 'go build' line. Thank you @buger

[erweinst]

Hey @buger,
What about the crypto imports? I saw in the source code that it is using
import crypto/tls as it should be import crypto/tls/fipsonly to ensure it is using only fips verified libs.
Any plans with releasing such version?

[buger]

@erweinst

As I can see it is not really about replacing all import, but just adding this import anywhere to code, look at comment here https://go.googlesource.com/go/+/dev.boringcrypto/src/crypto/tls/fipsonly/fipsonly.go

E.g. it should have import _ "crypto/tls/fipsonly" somewhere.
Since FIPS loaded only when some condition met, like ENV var, it does make sense inject this import dynamically, before running go build. Like this:

awk 'NR==4{print "\t_ \"crypto/tls/fipsonly\""}1' main.go > /tmp/main.go && mv -f /tmp/main.go main.go
go build ....

[buger]

@erweinst Just following up, how its going?
Want to get some early feedback, since we thinking about making it included into our upcoming release.

[erweinst]

Hey @buger, the compilation part and the symbols validation part looks ok.
We are still not using SSL to communicate with Tyk, and once we will start to, we will also have E2E tests to ensure that we are allowing only specific protocol with specific cyphers. Still some work to do here..
For this issue, I think you can close it :)
Thanks!

[DavidTal]

Hello @buger,
I have tried the above, it Tyk instance seems to be working yet I get the next errors in the logs:
level=error msg="Context variable type is not supported: "

I could not find any relevant information about this,
Do you know what is the issue?

Thanks!

[buger]

@DavidTal this error comes from URLRewite missconfiguration, some variable which you use there has issues.

[buger]

@erweinst @DavidTal I did some deeper investigation, and in general in our to get FIPS compilance it is require to ensure that application use specific ciphers for all incoming and outgoing communications. And tyk allows dynamically configure most of it, without re-compilation. I created very early draft for such docs, what do you think?
https://gist.github.com/buger/0042d4528217abcb0ad87b35e74c5114

Thanks!

[erweinst]

@buger that is awesome! A couple of notes/questions:

1. It would be great to have an examples folder and have a "fedramp configuration" setup there
2. We plan to have AWS NLB in front of Tyk and in front of any downstream service. What should we do in such case?
3. Regarding redis or any other service Tyk uses - it also should have in-transit encryption. Like discussed here Redis encryption in transit doesn't work #4613. Worth adding it to the gist

[buger]

1. Agree, thats the next step. My guide above contains all needed low level data, but if smth is off and not clear to you let me know.
2. In this case it is all about where you configure TLS. Default setup for NLB as I understand directly route TCP traffic to the target application, which means that TLS configuration will be on Tyk side. So if Tyk configure properly, all traffic going though NLB to Tyk will be FIPS compilant.
3. As part of FIPS even communication with Database, and between database nodes should be protected. In the linked ticket , as I understand, you made it work by adding ssl_insecure_skip_verify. But in general thats not the real issue, real issue is how to make Tyk talk to Redis with only allowed TLS ciphers? As I mentioned in a guide, configuring DB to be FIPS compilant is hard, and much easier use Cloud solutions. For example AWS offers FIPS for ElasticCache, and you do not even have to use GovCloud for that https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/elasticache-compliance.html. Basically when you configured it properly, it expose endpoint on domain like that elasticache-fips.us-east-2.amazonaws.com, which ensures that TLS will be done FIPS compilant.

Hope it makes sense!

[DavidTal]

Hello @buger

After upgrading to branch v4.0.3 as suggested I have noticed some behavior changes.
For example, when triggering end point without a valid token, in the previous versions the response was:
"error": "Key not authorised"

Yet in the current v4.0.3 version we get:
"error": "Access to this API has been disallowed"

In the code I see that the auth_manager.go was changed, then KeyAuthorised function was removed. Does it explain the difference in the response code?

Another issue that I see is the message in the log:
level=error msg="Context variable type is not supported: "
Does not appear in previous versions but does in v4.0.3. The error appears because the tyk_meta does not exist, although the session token was triggered. It looks like that it take more time to the v4.0.3 TYK to find the tyk_meta information.

Thanks!

[buger]

@DavidTal would be great to move this to a separate github issue, since it is not relevant to FIPS 😅

Overall it depends on from which version you are migrating, it is OK, to have some small changes in logging between major versions. But this specific message depeends on context, because "Key not authorised" message is still used by OpenID, oAuth and Coprocess plugins, so it depends on auth method you have, and can be some other symptom https://github.com/search?q=repo%3ATykTechnologies%2Ftyk+%22Key+not+authorised%22&type=code

As for tyk_meta, it depends where it was used. If it used in URLRewrite it is a one thing, if it used on body transformation, it may require setting enable_session field in body transform configuration

tyk/apidef/api_definitions.go

Line 234 in a36064a

EnableSession bool `bson:"enable_session" json:"enable_session"`

[DavidTal]

Hi @buger
Currently we are using TYK version v2.5.4, is the above solution relevant to this version?
Or should we upgrade to v4.0.3 ?
We would prefer not to upgrade the TYK version.
Thanks

[buger]

@DavidTal this is VERY OLD version, and not supported for a few years at least. So even from this point of view, better migrate to 4.0.X branch. But overall feels that this option were available in 2.5 version https://github.com/TykTechnologies/tyk/blob/v2.5.5/apidef/api_definitions.go#L83

[buger]

@erweinst I made my PR passing tests, and generated unstable images/packages under v5.0.0-fips tag, in our unstable channel: https://packagecloud.io/tyk/tyk-gateway-unstable?page=1

Keep in mind this is early RC and we do not yet know in which release it will land, and how we will treat it commercially, but i'm positive.

Thanks!

[andyo-tyk]

To give an update on this - we are looking to add FIPS compliance to the Tyk stack towards the end of this year. I'll update this channel as we make progress.

Thanks for supporting Tyk!

[pprasad]

awk 'NR==4{print "\t_ "crypto/tls/fipsonly""}1' main.go > /tmp/main.go && mv -f /tmp/main.go main.go
go build ....
i'm getting below error after enabling above statement for Fips on tyk v5.1.0
time="Dec 11 06:31:58" level=error msg="http: proxy error: remote error: tls: handshake failure" api_id=xxxxxxxxxxxxxxxxxxxxxx api_name="Authorize API" mw=ReverseProxy org_id=default prefix=proxy server_name=xxxxxxxxxx user_id=-- user_ip=x.x.x.36 user_name=

[thecloudhermit]

To give an update on this - we are looking to add FIPS compliance to the Tyk stack towards the end of this year. I'll update this channel as we make progress.

Thanks for supporting Tyk!

@andyo-tyk any update?

[andyo-tyk]

Sorry for the much later than planned update on this - and unfortunately it's not the news I wanted to be able to share.

Following significant discussion with our legal team, we've identified that Tyk is unable to provide a formal FIPS version of the OSS Gateway due to the legal implications this could entail.

Please see this docs page for the details and, if you're a licensed customer of Tyk please speak to your Account Manager if you require further information.