Create API for GCE Metadata

[JustinBeckwith]

There's a great metadata service for getting access to information on the VM. It would be great to have a structured API for this:
https://cloud.google.com/compute/docs/metadata?hl=en

[JustinBeckwith]

For reference, the Go implementation:
https://github.com/GoogleCloudPlatform/gcloud-golang/blob/master/compute/metadata/metadata.go

[dhermes]

We already use the metadata service to auto-detect the current project on GCE. As a first start we could move that functionality into a sub-package.

@tseaver @jgeewax Any preferences on what the subpackage should be named?

[stephenplusplus]

So far, our APIs assume you're talking to the googleapis.com API, regardless of how you auth. We have recently landed support for Resource Manager and are working on Logging, when things get more complex. Specifically, a portion of our API-- creating a Project (RM) or a Sink (Logging)-- requires user auth, as opposed to from a service account. This can quickly confuse our users, since it's up to our docs to explain how, when, & where to use auth and run our code. So this method is another example where we have a method that may or may not work when actually called.

Are there other actions that can be performed only when a user is running on a GCE instance? Possibly enough that could warrant creating a new module that has a dream-suite of utilities for someone running inside GCE?

[JustinBeckwith]

Not that I know of, but it's entirely possible. This will be a common thread though for the compute environment based APIs. For example, there are App Engine APIs we need to build veneers for that dig out App Engine specific information that only work when running inside of App Engine:

https://cloud.google.com/appengine/docs/python/refdocs/google.appengine.api.modules.modules

I think those use standard auth mechanisms though based on this:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

[stephenplusplus]

There's a library that looks like it was built to be a GAE suite of tools:

This library will only work inside App Engine Managed VMs, or with the Managed VM SDK.

• https://github.com/GoogleCloudPlatform/appengine-nodejs

It doesn't look like it's been updated recently, so I'm not sure if it has any plans for expansion in sight. However, that seems like the right approach, as it's obvious to the user that their GAE app will benefit from this GAE-only toolkit. That could set the precedent as well for GCE, even if it starts as just a module with one function, "getMetadata". Heck, I'll write it!

// @jgeewax any thoughts?

[JustinBeckwith]

Yeah - we really need to take appengine-nodejs down (we don't want people using that). It's talking to the App Engine v1 APIs with node, which was never really intended.

The App Engine APIs I'm talking about are available via a One Platform API:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

[stephenplusplus]

The API seems pretty straightforward, so I wouldn't mind writing a library for it. Just to review, my proposal here is:

1. Write a separately-installable GCE-env-only Node module.

   var env = require('gcloud-compute-engine');
   env.getMetadata(function(err, metadata) {});

2. Write a separately-installable GAE-env-only Node module.

   var env = require('gcloud-appengine');
   env.getApps(function(err, apps) {});
   
   var app = env.app('my-site');
   app.getModules(function(err, modules) {});
   
   var appVersion1 = app.module('1');
   appVersion1.delete(function(err) {});

The whole point being, where confusion is possible, simply narrowing the scope of what a user can expect from a given library we offer.

[JustinBeckwith]

Interesting - why separate these specifically into their own NPM modules? Splitting this up seems kind of confusing, unless we choose to split up the whole module.

Instead, I could see adding something like:

let gcloud = require('gcloud');
gcloud.appEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.appEngine.getModules(function(err, modules) {
      console.log(modules);
    });
  } else {
    console.log('not running in app engine!');
});

That way I can write code to specifically handle the case where I'm not in a particular environment. Same for GCE:

let gcloud = require('gcloud');
gcloud.computeEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.computeEngine.getMetadata(path, function(err, res) {
      console.log(res);
    });
  } else {
    console.log('not running in compute engine!');
  }
});

If isAvailable doesn't sound right, getExecutionEnvironment is another option:

let gcloud = require('gcloud');
gcloud.getExecutionEnvironment(function(err, env) {
  if (!err && env === gcloud.environments.appEngine) {    
      console.log("I'm in app engine!");
  } else {
      console.log('not running in app engine!');
  }
});

[JustinBeckwith]

@jmdobry and @jonparrott interested in your thoughts here too.

[theacodes]

• I don't want to create another appengine specific API, ever. They can just use environment variables for stuff like getting the current module.
• getExecutionEnvironment is the 100% wrong way to do this, I think. You're either on GCP or not. Being on GAE shouldn't matter vs being on GKE or GCE. You're either in GCP or not.
• The metadata API is incredibly useful, and having an API to do things like watch for changes would be extremely useful for users. I don't understand the desire to put it in a separate npm module as it's applicable to all of our hosting platforms. I suppose node.js doesn't have anything like Python's namespace packages.

[stephenplusplus]

why separate these specifically into their own NPM modules?

One reason is simply to shrink the overhead of this repo. The bigger this library grows, it's simply harder to maintain and put out a new release. If we stabilize at 1.0, then catch a breaking change in one of (smallest possible number here) APIs we support, we have to force a major increment upgrade (+1.0) on all of our users to have them benefit from smaller releases with features and fixes.

That aside, I prefer this split to avoid confusion. I think a user should be able to install a library and have everything work. Then, if they need something more specific to their application, they can install another library. I mentioned earlier the Logging and Resource Manager API muddying things up a bit earlier, but in those cases, we at least give the user the majority of the API regardless of what environment they're in.

With these extra APIs for App and Compute Engine, I think the separation will make it more clear for a user to understand what to use given their environment. Also, it will help users who may only need the GAE/GCE functionality by offering that library as separately installable.

[stephenplusplus]

I don't want to create another appengine specific API, ever.

Why is that?

You're either in GCP or not.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

[theacodes]

Why is that?

Because from my point of view, App Engine is not a special snowflake in our hosting options.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

I'm just saying that at most, we should have

var metadata = require('gcloud').metadata();

metadata.isAvailable(function(err) { ... });

Ideally, there should be little need to distinguish between GAE, GCE, and GKE. Even if that is desired, the metadata API is the wrong place to do it.

[stephenplusplus]

Ideally, there should be little need to distinguish between GAE, GCE, and GKE

That makes sense to me. The idea of being able to stop/delete a module programmatically sounded cool, though.

But, if all we want is a "try to get metadata" method, this is fine with me?:

var gcloud = require('gcloud');

gcloud.getMetadata(function(err, metadata) {
  if (/*not in a place we can get metadata*/) {
    err.message = 'Not in a place we can get metadata.';
  }
});

[theacodes]

FYI, I believe the ideal API surface for the metadata server should look something like this:

var metadata = require('gcloud').metadata();

// get instance tags.
metadata.instance.tags(function(err, tags) {
  console.log(tags);
  // ['one', 'two', 'blue', 'shoe']
});

// Get project-level custom metadata.
metadata.project.custom(function(err, kv) {
  console.log(kv);
  // {color: "red", active: "yes"}
});

// watch for changes to the instance tags.
metadata.instance.tags.watch(function(err, tags) {
   //...
});

[theacodes]

That makes sense to me. The idea of being able to stop/delete a module programmatically sounded cool, though.

That should be done in the with the App Engine Admin API. :)

But, if all we want is a "try to get metadata" method, this is fine with me?:

Yep, or just have all of the methods of metadata do one of the following:

• Give an error with a status code of 503.
• Give a special error that indicates unavailability.
• Give the generic error for socket timeout or connection failed.

FYI, IMO the timeout for hitting the metadata server should be set relatively low.

[stephenplusplus]

I think your examples show more than I thought was involved with my "just try to get the metadata" understanding. I still would do this separate if it was my own project, but I can see the importance of the docs all being in one place.

// watch for changes to the instance tags.

I definitely wouldn't have thought of anything like that myself. Any other things we should support?

[theacodes]

That's a good start. Writing metadata is also a useful thing to have. See https://cloud.google.com/compute/docs/metadata

Let me know if there's anything I can do to help.

As a heads up, please use metadata.google.internal as the hostname. Using just metadata will break on kubernetes. ;) Feel free to cc me on any PRs for this.

[stephenplusplus]

Definitely, thanks as always!

[JustinBeckwith]

Hey on the App Engine API bit - we absolutely need to have an App Engine Management API available. There is an entire REST API surface of commands customers will use:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/

When I say App Engine API - I don't mean things like mail and datastore, I'm talking about APIs that:

• get all modules in an app
• deploy a new module
• start/stop versions
• etc...

That merits an SDK in the same way VM creation and management on our platform needs a client library.

[thesandlord]

Is this being worked on? If not, I'm happy to submit a PR. I think the API surface @jonparrott described is a good place to start.

[stephenplusplus]

Thanks for offering! Just for the sake of mentioning, I whipped up this quick API that has a basic, similar surface recently: https://www.npmjs.com/package/gcp-metadata. It's lacking features that @jonparrott originally proposed, such as a watcher. And it's missing all of the stuff @JustinBeckwith suggested.

Since this issue was originally discussed, we've modularized our repo and deprecated google-cloud, the package that bundled all of them. This discussion is now effectively:

• Should we create and maintain a new module in this repository, i.e. @google-cloud/metadata?
• What features should it have?

And if the answer to #­1 is 👍 , then a PR to get the ball rolling would be awesome 💃

@lukesneeringer for all of the answers.

[lukesneeringer]

Another thing that has changed since the issue was discussed is that we can auto-generate GAPICs now. This sounds like a thing that should be auto-gen only so we do not have as high of a maintenance burden on it.

[stephenplusplus]

This issue was moved to googleapis/nodejs-compute#12

[stephenplusplus]

Re-opening, as this wasn't really an issue about the Compute API. Is there anything to do here? @alexander-fenster ?

[JustinBeckwith]

👋 I think gcp-metadata is covering this quite nicely. @stephenplusplus we can probably close this out :)