Cant ping internal IP on GKE node - But am i really supposed to?

[NicholaiStaalung]

Expected Behavior

Validate access to node in GKE cluster

Current Behavior

0 packets received

Steps to reproduce

Follow steps 0-2 for installing feast on GKE in https://github.com/gojek/feast/blob/master/docs/getting-started/install-feast.md#google-kubernetes-engine

Specifications

• Version: Master (Think master is still 0.3)
• Platform: Installing on GKE from local terminal
• Subsystem: python 3.6.8, helm 2.16.0, kubectl client 1.16.1, kubectl server 1.15.5

Possible Solution

I guess im not supposed to access the internal IP from my local, since its an internal IP. And i guess i need the access for the python sdk, which installs locally, to communicate with the node. Im not sure of the solution here.

[woop]

Hi @NicholaiStaalung

You should be able to connect to an open port on a service in GKE, but you are correct in that you should not be able to ping an Internal IP on a GKE node.

[NicholaiStaalung]

Hi @NicholaiStaalung

You should be able to connect to an open port on a service in GKE, but you are correct in that you should not be able to ping an Internal IP on a GKE node.

Okay. I figured already and tried pinging the external. However, the issue persists through the guide, as we assign the internal IP to $FEAST_IP and use it for the python sdk to configure core and serving URL. See the below images from the installation guide

Assigning the internal ip as $FEAST_IP and pinging it (Which was unsuccessful)

Here we update the values.yaml file with $FEAST_IP which is the internal IP (But as i mentioned in #350 , there are no feast.example.com to be updated, which i think is a mistake)

Here we update the python sdk config file with the URL's, which is still based on the internal ip

And showing the output of feast version in the guide for the GCP setup it clearly isn't the IP that were showed when you pinged (1st image). It looks more like a local router IP to me.

I did try using the external node ip from GCP as $FEAST_IP but i gave me errors, which is another issue to open. However, i suspect that there should be a mix of internal and external ip usage here which will solve that issue, so i wont open one just yet.

And maybe i missed something about Nodeport. I expected it to be part of the values.yaml file, but i dont see it in there.

[woop]

Hi @NicholaiStaalung,

Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

[NicholaiStaalung]

Hi @NicholaiStaalung,

Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

Okay. Thanks for getting back to me.

[couch-potato08]

Hi @NicholaiStaalung,

Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

Hi, I tried following the documentation for installation on a GKE cluster. I ran into the same errors mentioned by @NicholaiStaalung. And it also fails at this step:
helm install --name feast -f my-feast-values.yaml .
Could you give an estimated date by which users can expect the updated guide?

[lgvital]

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

Hey @woop, we're evaluating feast as our feature store. Should we be using the latest version, or should we opt for an earlier, more stable version, given that you're in the middle of a refactor?

[woop]

Hi @NicholaiStaalung,
Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.
I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.
I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

Hi, I tried following the documentation for installation on a GKE cluster. I ran into the same errors mentioned by @NicholaiStaalung. And it also fails at this step:
helm install --name feast -f my-feast-values.yaml .
Could you give an estimated date by which users can expect the updated guide?

Hi @couch-potato08 apologies on the slow response! We've been a little bit slow over the holiday period.

We've only tested with Helm 2. Are you using Helm 3?

I am doing a run through the guide at the moment to see if its working.

[woop]

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

Hey @woop, we're evaluating feast as our feature store. Should we be using the latest version, or should we opt for an earlier, more stable version, given that you're in the middle of a refactor?

Hi @lgvital the latest release would be the best one for you to use to ensure forward compatibility. It allows for project isolation and we've cleaned up the API somewhat to simplify things.

The example notebook on feast.dev is still unfortunately describing the 0.3 flow (without projects). I have just created an issue (#414) to update it, but it should be very easy to get working with the current release. Essentially a project should just be created using the Python SDK, then configured before ingestion and retrieval.

[woop]

So it seems like there are numerous problems here

• InternalIP should only be used if you are using the Private IP of the GKE node (non public). Which is presumably why you are experiencing problems
• If you are using ExternalIP you might be running into problems with a firewall.
• For some reason the values.yaml is missing all feast.example.com URLs which should be replaced.

Just to be clear, I dont think you need to use a mix of internal and external IP. I think it should all be "external" if you are using GKE with public IPs, or "internal" if you have a private network. Neither of these ranges are the Cluster IP ranges.

[NicholaiStaalung]

So it seems like there are numerous problems here

• InternalIP should only be used if you are using the Private IP of the GKE node (non public). Which is presumably why you are experiencing problems
• If you are using ExternalIP you might be running into problems with
• For some reason the values.yaml is missing all feast.example.com URLs which should be replaced.

Just to be clear, I dont think you need to use a mix of internal and external IP. I think it should all be "external" if you are using GKE with public IPs, or "internal" if you have a private network. Neither of these ranges are the Cluster IP ranges.

Thanks @woop
Could you clarify your point number 2 regarding the external IP? It seems there is something missing

[woop]

So it seems like there are numerous problems here

• InternalIP should only be used if you are using the Private IP of the GKE node (non public). Which is presumably why you are experiencing problems
• If you are using ExternalIP you might be running into problems with
• For some reason the values.yaml is missing all feast.example.com URLs which should be replaced.

Just to be clear, I dont think you need to use a mix of internal and external IP. I think it should all be "external" if you are using GKE with public IPs, or "internal" if you have a private network. Neither of these ranges are the Cluster IP ranges.

Thanks @woop
Could you clarify your point number 2 regarding the external IP? It seems there is something missing

Oops, added clarification. I mean a firewall.

[woop]

If you have a public IP exposed and you are using NodePort, then connecting from the Feast CLI from your local environment should be possible.

[NicholaiStaalung]

Thanks. Ill try it out

[woop]

I probably won't be able to get to updating these notebooks and the guide until at least next week. Hopefully I can have everything fully updated then.

[NicholaiStaalung]

No worries on my behalf. I technically have everything i need from the docker-compose installation. I just need to move it at some point.

[couch-potato08]

Hi @NicholaiStaalung,
Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.
I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.
I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

Hi, I tried following the documentation for installation on a GKE cluster. I ran into the same errors mentioned by @NicholaiStaalung. And it also fails at this step:
helm install --name feast -f my-feast-values.yaml .
Could you give an estimated date by which users can expect the updated guide?

Hi @couch-potato08 apologies on the slow response! We've been a little bit slow over the holiday period.

We've only tested with Helm 2. Are you using Helm 3?

I am doing a run through the guide at the moment to see if its working.

Thank you for the response @woop. As the guide instructed, I'm using Helm 2 (Helm v2.16.1). Also, how to proceed with the values.yaml file, since there are no instances of feast.example.com in it.

[lgvital]

I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.

Hey @woop, we're evaluating feast as our feature store. Should we be using the latest version, or should we opt for an earlier, more stable version, given that you're in the middle of a refactor?

Hi @lgvital the latest release would be the best one for you to use to ensure forward compatibility. It allows for project isolation and we've cleaned up the API somewhat to simplify things.

The example notebook on feast.dev is still unfortunately describing the 0.3 flow (without projects). I have just created an issue (#414) to update it, but it should be very easy to get working with the current release. Essentially a project should just be created using the Python SDK, then configured before ingestion and retrieval.

Thanks for the clarification on using latest version. The images in feast/infra/charts/feast/charts/feast-core/values.yaml and feast/charts/feast-serving/values.yaml both reference 0.3.2. Should I update these to reference 0.4.2 instead?

So it seems like there are numerous problems here

InternalIP should only be used if you are using the Private IP of the GKE node (non public). Which is presumably why you are experiencing problems
If you are using ExternalIP you might be running into problems with a firewall.
For some reason the values.yaml is missing all feast.example.com URLs which should be replaced.
Just to be clear, I dont think you need to use a mix of internal and external IP. I think it should all be "external" if you are using GKE with public IPs, or "internal" if you have a private network. Neither of these ranges are the Cluster IP ranges.

So, just to summarize your recommendations if I'm using GKE, I should always be using "ExternalIp" with GKE with public IPs. To connect to feast CLI from local environment, I should set up NodePort, similar to what is listed in values-demo.yaml:

  service:
    type: NodePort
    grpc:
      nodePort: 32090

[lgvital]

Got it to work. I did need to use ExternalIp and expose the services using NodePort like above. I also ran into firewall issues in GKE as mentioned by @woop.

I followed GKE's docs and ran the following:

gcloud compute firewall-rules create feast-core-port --allow tcp:32090

This needs to be repeated for batch + online services with different port numbers. Lmk if there is an easier/preferred way to go about this. Thanks for the help!

[woop]

Got it to work. I did need to use ExternalIp and expose the services using NodePort like above. I also ran into firewall issues in GKE as mentioned by @woop.

I followed GKE's docs and ran the following:

gcloud compute firewall-rules create feast-core-port --allow tcp:32090

This needs to be repeated for batch + online services with different port numbers. Lmk if there is an easier/preferred way to go about this. Thanks for the help!

Awesome :)

Which version are you using at the moment? 0.4.2?

[woop]

Hi @NicholaiStaalung,
Thanks for detailing all of this out. The IP mentioned above is definitely the "external" IP of the Node. I think I copied the feast version from the Minikube guide, which is why it's showing an internal.
I think this guide still has too many rough edges to safely try out. We are all heavily focused on code refactors now, so updating the docs is taking a back seat since that will have to be done as part of the new release.
I think the safest place to focus for the time being is the docker compose file, so I will move my comments over to that thread and see if we can't help you there.

Hi, I tried following the documentation for installation on a GKE cluster. I ran into the same errors mentioned by @NicholaiStaalung. And it also fails at this step:
helm install --name feast -f my-feast-values.yaml .
Could you give an estimated date by which users can expect the updated guide?

Hi @couch-potato08 apologies on the slow response! We've been a little bit slow over the holiday period.
We've only tested with Helm 2. Are you using Helm 3?
I am doing a run through the guide at the moment to see if its working.

Thank you for the response @woop. As the guide instructed, I'm using Helm 2 (Helm v2.16.1). Also, how to proceed with the values.yaml file, since there are no instances of feast.example.com in it.

I will update these as soon as possible. Please give me a bit of time.

[woop]

No worries on my behalf. I technically have everything i need from the docker-compose installation. I just need to move it at some point.

Great :)

[lgvital]

Got it to work. I did need to use ExternalIp and expose the services using NodePort like above. I also ran into firewall issues in GKE as mentioned by @woop.
I followed GKE's docs and ran the following:
gcloud compute firewall-rules create feast-core-port --allow tcp:32090
This needs to be repeated for batch + online services with different port numbers. Lmk if there is an easier/preferred way to go about this. Thanks for the help!

Awesome :)

Which version are you using at the moment? 0.4.2?

Yes, 0.4.2. Creating and setting a project was pretty simple as well.

[lgvital]

Ended up having issues with ingestion. Opened #427. Unsure if it's related to this issue.