Added feature to pick from multiple sample backends.

[revanth0212]

Description

This PR introduces a new extension and a new feature that allows developers to choose which Magento Backend they want to use while developing.

While creating a new app, developers will be prompted to choose a Magento backend from the sample backends provided by the PWA Team or enter their own backend URL. The sample backends are fetched at runtime from a secure location hosted and maintained by the PWA Team.

This PR also introduces a new Target that accepts env validator functions. These functions will receive the project env in the JSON format and they can run validations and report errors to the buildpack's build process. The extension added in the PR uses this target to add a validator that checks the backend URL and provides options if needed.

Related Issue

Closes PWA-96

Verification Stakeholders

@zetlen
@jimbo
@dpatil-magento

Specification

We need to update the docs about this change. @jcalcaben can you advise where I need to update these docs.

Verification Steps

Scaffold test

1. Checkout this branch.
2. DEBUG_PROJECT_CREATION=1 <Path to PWA Studio Package>/packages/create-pwa/bin/create-pwa
3. You should be able to see options from the new API.

Magento 2.3.3 with Venia sample data installed from the above image is fetched from the API.

Build process

1. Checkout this branch.
2. Change the MAGENTO_BACKEND_URL to an unreachable URL.
3. Run yarn build. The build process should fail and suggest some sample backends.

If the backend is fine, nothing should change from how develop works.

Checklist

• I have added tests to cover my changes, if necessary.
• I have added translations for new strings, if necessary.
• I have updated the documentation accordingly, if necessary.

[revanth0212]

I am looking at AWS docs to use a DNS name instead of this raw path. It will be changed in the future (before this PR is merged).

[sirugh]

Is this a lambda? Does it read configuration somewhere? I'm a bit surprised we don't just maintain a list in the extension.

[revanth0212]

The plan is to pull these backend URLs dynamically so we don't need to publish every single time a backend changes, which we will have to if these are part of the package. The URL above is an AWS Gateway URL. The plan is to use a domain-specific URL so in the future if we move between cloud providers (Azure, Ethos, etc), we can keep using the same URL but the implementation in the back can change.

[sirugh]

so we don't need to publish every single time a backend changes

Something will have to change somewhere whether that's a file in s3, a lambda script, or a local file in a pwa studio extension/package (like create-pwa). And as we saw in the demo, there is already a file for hosting these, but within buildpack.

I personally prefer keeping this file locally within create-pwa or buildpack and just publishing a new version of those packages if the backend changes. It's straightforward and requires no knowledge of lambdas, aws routing, etc. There is also no concern with api downtime or worry about attack vectors with a public route.

If the team want to keep this solution as is, I'd really like to see some documentation with steps to update this file. I also think we should understand the costs of this approach. How much per request does this lambda/endpoint cost us?

Edit: To be clear, I appreciate the approach as an example for how we could interact with a micro-service. It's a neat example for sure. I just think it's overkill for us.

[revanth0212]

The points you have mentioned are perfectly reasonable. I did have a thought of making it a package to avoid the cost of hosting and also avoid potential DoS security issues. But the biggest problem with publishing on NPM is that developers don't generally run yarn install every time they build or might be using an older version of the package causing older environments to be used instead. The cost of using AWS Lambda and AWS S3 is minimal and we have enabled throttling as an initial security check.

[sirugh]

But the biggest problem with publishing on NPM is that developers don't generally run yarn install every time they build or might be using an older version of the package causing older environments to be used instead.

Two questions:

1. How often does the getSampleBackends service actually need to be used? It should just be used when someone spins up their project for the first time, right?

2. Is the getSampleBackends API used during the build or do an up-check to see if the backend is accessible during build?

I think I see that it uses it on initial scaffold but then only later if the up-check fails.

[revanth0212]

1. How often does the getSampleBackends service actually need to be used? It should just be used when someone spins up their project for the first time, right?

Along with bootstrapping, It will be used every time someone adds the venia-sample-backends extension and builds their project.

2. Is the getSampleBackends API used during the build or do an up-check to see if the backend is accessible during build?

It is used during the build stage if the extension is installed. It will be used to fetch the sample backends, if the current backend is inactive. We won't be calling the service if the backend mentioned in the ENV file is active.

[davemacaulay]

What happens if this URL goes down? Will the yarn install / build hang for X seconds?

Also if we're maintaining this micro server we need to setup monitoring to ensure we can be aware of uptime issues. We should also consider any security implications of having this service running.

[revanth0212]

Nice question @davemacaulay. I have used a wrong URL to simulate the service failure and you can see here, if the service is down, the script will use the internal URLs shipped as part of the sampleBackends.json file.

You are right about monitoring. I'll create a ticket to investigate the monitoring capabilities of our team/AWS.

Coming to security, we are not giving the public access to the Lamda function or the S3 object. It is an AWS Gateway Service that we are using. Also, the Lamda function that we are using only has read access to that particular object.

[dpatil-magento]

@revanth0212 It would nice if we can move the lambda setup to aws prod account and check if any Audits fail. Rest all looks good.

[PWAStudioBot]

	Messages
📖	Associated JIRA tickets: PWA-96.
📖	DangerCI Failures related to missing labels/description/linked issues/etc will persist until the next push or next nightly build run (assuming they are fixed).
📖	Access a deployed version of this PR here. Make sure to wait for the "pwa-pull-request-deploy" job to complete.

Generated by 🚫 dangerJS against 98abb1b

[revanth0212]

@revanth0212 Please check following :

1. root .env is copied to scaffold instead considering build env vars.
2. When backend and API are down better to avoid this message in yarn build as we see null response - ......is inactive. Please consider using one of these other backends: []

Fixed 1 and added a different error message in the case of 2. If backends is empty for any reason (failure of API etc), the error message will be <MAGENTO_BACKEND> is inactive. Please consider using an active backend.

[dpatil-magento]

@revanth0212 There is prettier failure and last open question, other than these its good to go.

During yarn build say when customer real backend is down (worst case), suggesting to use demo backend instance would not be good idea?

Error: Environment has 1 validation errors:
 (1) https://production-test.test.com/ is inactive. Please consider using one of these other backends:

 [{"name":"2.4.1-cloud-meta-package","description":"Magento 2.4.1 with Venia sample data installed","url":"https://master-7rqtwti-mfwmkrjfqvbjk.us-4.magentosite.cloud/"},{"name":"2.4-develop","description":"Magento 2.4-develop with Venia sample data installed","url":"https://venia-develop-ccewfjy-mfwmkrjfqvbjk.us-4.magentosite.cloud/"}]

[revanth0212]

@revanth0212 There is prettier failure and last open question, other than these its good to go.

During yarn build say when customer real backend is down (worst case), suggesting to use demo backend instance would not be good idea?

Error: Environment has 1 validation errors:
 (1) https://production-test.test.com/ is inactive. Please consider using one of these other backends:

 [{"name":"2.4.1-cloud-meta-package","description":"Magento 2.4.1 with Venia sample data installed","url":"https://master-7rqtwti-mfwmkrjfqvbjk.us-4.magentosite.cloud/"},{"name":"2.4-develop","description":"Magento 2.4-develop with Venia sample data installed","url":"https://venia-develop-ccewfjy-mfwmkrjfqvbjk.us-4.magentosite.cloud/"}]

Good question. This extension is for developers who are trying out PWA Studio, not for production instances. It is expected that they will remove the extension from their package.json. Once it is removed, they won't be prompted with options in case of backend failure.

[dpatil-magento]

QA Approved.

[sirugh]

It is expected that they will remove the extension from their package.json.

@revanth0212 is this documented?

[revanth0212]

It is expected that they will remove the extension from their package.json.

@revanth0212 is this documented?

I haven't included this in the docs. @jcalcaben any idea where can this go if docs are the right option to place this info?

I say expected because the package itself is named as sample backends so people would know. As I type this, I can imagine someone missing to remove this from package.json. Ill do one thing, ill post a warning in the console that they are using a non-prod package/extension and they should consider removing it before going to prod as we do with the default Braintree token. How does that sound?

[revanth0212]

I have added a console log to indicate that the package being used is a Dev only package and one has to remove it before going to Prod.

[dpatil-magento]

QA re-approved.