Build reference test site

[schlessera]

We should work on a reference test site that includes typical website content and try to use or touch most or all of the functionality of the AMP plugin as a real-world website would.

This will either simplify or be a requirement for the following features:

• Add performance metrics to CI [#4569]
• Add unified sanitizer smoke test [4549]
• Add automated visual regression testing [4671]

[amanintech]

I will be happy to do that. Can this issue be assigned to me?

[schlessera]

Hey @amanintech, thanks for the interest in working on this.

We're currently not sure whether this can be done by an outside contributor, as it might require quite a lot of infrastructural knowledge to pull off. But I guess it depends on how we approach it.

Can you give me a few details about how you intend to work on this and what you'd want to build exactly?

[amanintech]

I would do something like exactly as you mentioned with a Wordpress website having the latest AMP plugin. I can add a CI that updates the plugin automatically and send the lighthouse performance report. We can host real content by pulling it from a news website in form of feed. I have worked on a repo where I also developed a theme having a feed like a feature. https://kievents.co/sid/

[westonruter]

To make this maintainable in the project and usable inside CI builds I think the test site should be in a Docker container.

[schlessera]

Yes, or maybe even a docker composer setup (instead of a single container), as we'll want to test persistent object caching as well with different servers like Memcached or Redis.

It should be the "source" version of building such a site, not the end result, so that we can replicate it wherever we need it.

Is this something you can tackle, @amanintech ?

[amanintech]

Yes, I surely can try. Where this container should be finally deployed? And will the site have any public visibility ?

[westonruter]

I think the point of the container is that it won't be deployed publicly. It would be spun up during and inside CI builds and then destroyed.

[amedina]

This is definitely something we need to do to evaluate baseline performance metrics for sites running the plugin on a set of specific conditions in relation to as set of essential metrics.

Priorities are:

1. Define content of reference test site so that the AMP content generation capabilities of the plugin are fully exercised.
2. Define tests to measure the performance gains between using or not using the AMP optimizer.
3. Define tests to measure set of essential metrics
4. Enable the execution of standalone Optimizer tests
5. Integrate measurement of essential metrics into CI build pipeline

After these are achieved:

1. Document structure of reference test site so that any publisher can set up a similar one and add it to their CI pipeline

[amanintech]

I think the point of the container is that it won't be deployed publicly. It would be spun up during and inside CI builds and then destroyed.

Honestly, I have never done that before. But I am ready to learn and give it a shot if that's ok. Could you share any tutorial to help me get started?

[schlessera]

@amanintech I think what @amedina meant to say is that we - as in: the internal AMP plugin team - have to do a lot of work upfront until this is ready for a first implementation. I'm afraid this is not a good issue for now if you are interested in contributing.

[amanintech]

Got it

[schlessera]

@amedina I'm renaming this issue back to what it is, as this was specifically for defining the reference site, independent of what it will be used for. The title you've changed it into would fall under #4569.