Installable Components

[dmolsen]

I've been interested in the idea of installable patterns for PL2 for a while now. Once InstallerUtil was done I saw that it was more flexible than just installing PatternEngines and StarterKits. Now that there is interest in components and they appear to generally work "installable patterns" can be expanded to "installable components." This would allow the use of something like composer require phase2/tabbed-panel in a project which would install Phase 2's version of a tabbed panel. Alternatively, a StarterKit could consist entirely of component dependencies. Or we allow a "fetch component" feature to keep components out of a dependency tree. All of this is almost doable today. The biggest issues are...

• where do component files go?
• how do we gracefully handle prompting the user for where to install them?
• what to do with asset files related to components?

I'd like to make the following proposal about Packagist-based components:

• component authors can define a default path for a component in composer.json. e.g. atoms/general.
• on install PL2 would look at the default path and:
  • try to match it to disk and respect numbering of folders. if that didn't work it...
  • try use each directory in the default path as keywords and compare against a user-supplied mapping in ./config. if that didn't work...
  • prompt the user to type in an install location

Now this works really well for PL2 things like patterns, data, info files. Where it breaks for me are assets like images, javascript, css and sass/less. How magical do we think this needs to be? Plugin assets that are meant to modify the the PL viewer are installed in a way that's pretty damn magical but no one really cares about the final, standardized location for those plugins.

Is this something worth worrying about? Ideas for how to address non-PL2 assets?

[EvanLovely]

I am incredibly excited and supportive on this!! On my phone right now, so I'll just reply with one thing: let's do a folder per pattern/component with all assets inside. We've separated assets in a folder per type for so long, but it's no longer necessary as we have the file extension to tell the difference and we often treat these as source files that are globbed by things like Gulp and compiled to another location. I could see this working great:

• atoms/
  • forms/
    • button/
      • button.twig
      • _button.scss
      • button.js
      • button.yml
      • button.md
      • button~alt.yml
      • button~big.yml

[dmolsen]

This is a "just in case" note re:Node. Regular, normal use of Pattern Lab PHP that is shipped out of the Pattern Lab org should in no way rely on Node. I feel really strongly about that. If a feature is officially supported it should work via PHP. Examples include browser reload and source watch in PL1. Both could have used Node but I spent the time to get those working natively in PHP. This limits the language dependencies someone needs to install to get up and running with this.

Of course, Editions or StarterKits put out outside the org can support Node, Gulp, Grunt, Sass or whatever. Just for this org the PHP projects should be vanilla PHP on the build side. I want to provide an easy-to-use base that people can build whatever on top of. That's been the ethos from the beginning. Also see the work done to make this project work sans Apache even. I really tried hard to limit dependencies.

That said, maybe this issue needs to be defined simply around supporting a default path option in a package install. Example below. Any other clean-up/moving of assets can be implemented at a later date. So PL PHP wouldn't support "components" per se but it's obvious it can be bent that way.

composer.json...
"extra": {
   "patternlab": {
       "defaultPath": "atoms/general",
       "sourceDir": [
           { "*": "\\defaultPath" }
       ]
   }
}

The above would move everything in ./dist in the package to the user-defined sourceDir and try to install in the atoms/general. If it doesn't exist it'd look for the user mapping in ./config and if that isn't found it would actively prompt the user for where the files should be installed.

An array of defaultPaths might be another good option so multiple default path keys and values could be set-up for the other possible install locations:

• baseDir,
• publicDir,
• sourceDir,
• scriptsDir, and
• dataDir.

This reminds me that I need to add an out for ./dist in packages. Let users redefine that in case they want to use something other than ./dist when shipping packages.

[aleksip]

@dmolsen You beat me to opening this issue! :) So glad to learn that you are interested in this topic and have been thinking about it already.

I just published a blog post containing a lot of my thoughts on how this could be done.

• I think that component files (projects!) should go in a folder of their own if inside Pattern Lab. But it should be possible to have them outside the main Pattern Lab folder too.
• Component projects should contain a file defining a mapping between component project folders and Pattern Lab folders. composer.json could be used for this purpose, but ideally I'd love for the component projects to be as language and tool neutral as possible. Component projects could be supported by the Node version of PL too. So maybe we could come up with a PL-specific file containing the mapping. A possible component-project.json could contain a PL section just like composer.json. Then again, non-PHP projects could still read the info from a composer.json file.

[EvanLovely]

@aleksip Take a look at how The Financial Times does their system, Origami: http://origami.ft.com They have each component in its registry in its own repo with its own issue key and the ability to release multiple versions of that component! Some examples:

• http://registry.origami.ft.com/components/o-gallery@1.7.6
• http://registry.origami.ft.com/components/o-share@2.1.3

They each have their own origami.json. Perhaps we should go with components.json?

[aleksip]

@EvanLovely Origami looks very impressive, even a bit overwhelming at first. But yes, a component project manifest file would be something neutral. A composer.json could be off-putting for Node developers, for example... :) We could just copy the extra section from composer.json.

The way I'm thinking file/folder mapping could work is that PL would use component project repos 'as is', whether they are inside or outside of the PL project. So there would not be any copying of files. The mapping to sourceDir would be applied when PL generates the styleguide. Not sure how difficult this would be to implement. Also not sure if copying is required as part of the architectural design of PL? But the way I see it, using the files directly from the component project directories (with VCS metadata) would be the Holy Grail: no need to keep files in sync!

[EvanLovely]

I definitely think Origami is the most thorough and extreme approach that we could take and is probably overkill for what we're trying to do. It's great inspiration though!

I think that keeping this info in a composer.json would be off putting to Node devs as well, just as keeping it in package.json would be off putting to PHP devs.

Additionally I am very against any copying of files as I learned that's very confusing for others on the project as they could easily be editing a file that will get overwritten later.

I think a components.json would work well and would work if all components were in one repo or there was a single component per repo or any combination of the two.

[dmolsen]

From reading @aleksip's post "components" are definitely something that is outside the scope of Pattern Lab. At least in the sense that core PL needs to explicitly support something called components. I can see a few changes that should happen with core:

• support default paths "Default Path" for installing files #30
• support user-defined ./dist Support User-defined ./dist  #31
• support array for sourceDir Support Array for patternDir #32
• support a sourceDir outside of PL root Support for sourceDir to be Outside of PL root #33
• allow a pattern to define its type and subtype Allow a pattern to define its own type and subtype #34
• related assets would be moved to ./public if at level 3 Move non-PL component files in _patterns to ./public #75

If a schema like Origami is developed then a plug-in could be created with a rule to read in the data and with custom templates to display that data. The KSS plugin sort of works this way. I would expect that one would end up leaning on NPM/Composer for dependency management.

To be clear, ./vendor should not be seen as a viable source directory for patterns.

[aleksip]

@dmolsen Thanks for all the work in thinking about this and opening those related issues!

My blog post probably made component projects seem more complex than what I'm currently thinking they could be. My current definition of a 'component project' is:

• A project that contains one or more 'component folders' and a manifest file.

And the definition of 'component folder':

• A folder that contains a component.

Both definitions are intentionally very loose, as this allows for flexibility. The component(s) themselves can be defined in the manifest file.

From PL's point of view 'component folders' just mean 'folders below patternSubType'. The manifest file could contain 1:1 to what could also be in the extra section of a composer.json. So basically PL already has 'component projects' covered and we can indeed just continue discussion from a PL point of view.

However, Composer is not the recommended way to install StarterKits, and php core/console --starterkit --install doesn't use Composer either, it only parses the PL extra data from composer.json. So I still see a possibility to easily add support for a possible component-project.json, if such a thing will ever exist.

Ok, now getting to the actual issue(s)... :)

The big question still remaining: would it be possible, instead of copying files, to use files directly from StarterKit/pattern project folders containing repository metadata?

Will #32 be implemented by copying files or by using a merged in-memory directory, linking to the contents of different real directories?

I think #30, #31 and #34 could all be solved by a single JSON schema, as a PL extra section in composer.json or some other file. ;)

[dmolsen]

@aleksip -

However, Composer is not the recommended way to install StarterKits,

I'm not sure I've explained why this is the case. It touches on your interest in retaining VCS data. Based on what I've experienced here with our CMS, theme developers want a starting point from a moment in time. They're not terribly interested in having "components" under version control or tracked with dependencies. I would say that's a mistake but that's why StarterKits are able to be pulled via ZIP without any of that overhead. They're meant as literally a starting point and then never worrying about conflicts or whatever. That may or may not work for "components."

It also avoids a simple mistake like "composer update" and boom you're getting prompted " ./source exists. Merge or replace?" and "Shit, I pressed 'r'. Somebody help me with Git!"

You can still use Composer to install StarterKits (releases or branches) but definitely trying to help people avoid a few gotchas.

Will #32 be implemented by copying files or by using a merged in-memory directory, linking to the contents of different real directories?

I was thinking it would be a config.yml var/array that could be automagically updated on install or by hand after. Then just use that array as a foreach to loop over and find patterns. So I think that would be "linking to the contents of different real directories" as the answer.

I think #30, #31 and #34 could all be solved by a single JSON schema, as a PL extra section in composer.json or some other file. ;)

I agree.

[aleksip]

I'm not sure I've explained why this is the case.

Didn't know the story behind the non-recommendation. Would be interesting to learn more about the Pattern Lab + CMS setup and development workflow in that type of use case.

Even without components and many of the changes discussed in this issue, there is a use case in which a StarterKit is used more like an independent project, with changes committed to its own repo. Composer's create-project has a --keep-vcs option, maybe the same feature could be added to the StarterKit install command?

I was thinking it would be a config.yml var/array that could be automagically updated on install or by hand after.

That seems like a great solution! Maybe add a --starterkit --update-mappings or something similar to automagically update possible changes? And maybe a --generate --read-mappings or similar for those who really want to be sure to always have the latest mappings?

[dmolsen]

Composer's create-project has a --keep-vcs option, maybe the same feature could be added to the StarterKit install command?

PRs welcome ;) I could see a Git wrapper plugin that adds that --keep-vcs flag to the fetch command for those interested in that. It's totally possible to add commands or modify existing ones via plugins. The current version of fetch is pretty dumb and I don't want to go down the path of managing multiple VCS options in core.

If one were to go with Composer managing it then I think that a dist override #31 that allows for the root of a package to be copied (instead of ./dist as it is now) to some path #30 would also allow VCS to be copied on install of a branch of a package. So if someone wants VCS they can go the Composer route.

Maybe add a --starterkit --update-mappings or something similar to automagically update possible changes? And maybe a --generate --read-mappings or similar for those who really want to be sure to always have the latest mappings?

I think we may be talking past one another a little bit. Can you describe the process by which you expect people to add something that needs to be tracked as an additional patternDir that PL should iterate over? Also, in what scenario do they need to "to be sure [they] have the latest mappings."? For example, why would that original saved directory path change in such a way that a new mapping would have to be loaded? I'm assuming "mapping" is just the list of available pattern directories PL should iterate over. To me that can be in vendor/. This is another way of addressing the VCS concern.

[aleksip]

PRs welcome ;)

Good to know! Wouldn't want to do a PR for something like this without your pre-approval of the idea. Also good to learn about the possibility of a plugin approach.

I think we may be talking past one another a little bit.

Totally possible! :) So the way I see how mappings could work is that there would not be any restrictions on how a downloaded/configured 'StarterKit' is organized. The mapping file would be a list of source to target mappings, with the targets being specific PL directories.

So if I have a Drupal theme organized like this:

drupal_theme/
├── components/
│   ├── atomic-component-set/
│   ├── single-component-1/
│   └── single-component-2/
├── css/
└── pattern-lab/
    ├── _data/
    ├── _layouts/
    └── _meta/

The theme could contain a mapping file with something like this (just a very quick sketch):

{
  "extra": {
    "patternlab": {
      "starterKitMappings": [
        {
          "type": "patternDir",
          "from": "components/atomic-component-set"
        },
        {
          "type": "pattern",
          "from": "components/single-component-1",
          "to": "[patternDir]/atoms/forms/textarea"
        },
        {
          "type": "pattern",
          "from": "components/single-component-2",
          "patternType": "molecules",
          "patternSubType": "forms",
          "patternName": "comment-form"
        },
        {
          "type": "assets",
          "from": "css",
          "to": "[sourceDir]/css"
        },
        {
          "type": "sourceDir",
          "from": "pattern-lab"
        }
      ]
    }
  }
}

This would make it possible for basically anything with a mapping file to be a Pattern Lab StarterKit.

There could be changes made to the Drupal theme which affect the mappings. And when such changes are made, I could just run --starterkit --update-mappings. Or generate with --read-mappings.

A stand-alone single-component-2 from the example could also be used in Pattern Lab on its own. It would contain a mapping file like this:

{
  "extra": {
    "patternlab": {
      "starterKitMappings": [
        {
          "type": "pattern",
          "from": ".",
          "patternType": "molecules",
          "patternSubType": "forms",
          "patternName": "comment-form"
        }
      ]
    }
  }
}

It would be a StarterKit containing just one pattern.

I'm voting for changing #32 back to 'Support array for sourceDir', as the way I've understood it, sourceDir more or less == StarterKit. With support for flexible mapping and some responsibility on the developer this would be a very powerful feature!

[dmolsen]

I'm going to stay with the mechanics of install, placement and where things live. Sorry. I see where you're going with the schema and have comments but I'd like to nail down these issues first.

So we have a Drupal theme with the above layout. Could I run composer install drupal/theme? Then files are in ./vendor, right? Or if I use a starterkit --fetch with a --targetDir option it would be in ./targetDir. Suffice it to say they're installed inside the PL project. I haven't run InstallerUtil against them yet (magically).

So the question becomes, do you need these files to be in ./source or is it preferable to leave them alone where they are in ./vendor or ./targetDir? Instead, on generate, we'd use "mappings" to tell PL "go here for files to pull into the overall output." And we'd know the list of mappings because we could come up with a mappings.json that tell us the paths to the composer.json or someSchema.json that has the mappings. This would always be read if available so no need for a flag.

Alternatively, if they're outside PL root there could be a --fetch --mappings --targetDir ../../foo (that's a bastardization of --fetch but I'm being lazy). Again, files are left in place.

I'm assuming type in the composer.json has to be Drupal-specific so maybe this is where a specific JSON file in the root of the package would be beneficial. So I could check for PL-compatible type in composer.json or someSchema.json in the package directory to know to run my installer on in-edition install via Composer or --starterkit. Otherwise the --fetch would just scan composer.json or someSchema.json for the mappings.

Or we allow supported PL types to be added via a config override/addition in an edition's composer.json...

[aleksip]

I'm going to stay with the mechanics of install, placement and where things live. Sorry. I see where you're going with the schema and have comments but I'd like to nail down these issues first.

Perfectly fine! First things first. :)

So the question becomes, do you need these files to be in ./source or is it preferable to leave them alone where they are in ./vendor or ./targetDir?

I can't think of any reason why I'd want to dictate the location inside PL. As a general rule I think it would ideal for it to be possible to 1) use StarterKits from (almost) anywhere, 2) use them 'as is' (no copying) and 3) have the possibility to install inside PL with or without VCS metadata. (Is is possible to install dependencies with VCS metadata with Composer?)

a mappings.json that tell us the paths to the composer.json or someSchema.json that has the mappings. This would always be read if available so no need for a flag.

So the 'mappings' you refer to is basically just a list of StarterKit directories? If so, couldn't it just be a list in config.yml? Edited by hand or updated automagically by --install-starterkit. A well-packaged StarterKit could even have a post-package-install in its composer.json to call the same automagic code. Editing by hand would work for StarterKits outside PL, although it is of course always nice to have a command for that.

I originally understood that the config.yml var/array you referred to was a 'virtual sourceDir' hierarchy of mappings merged from all the starterKitMappings. So that is why I thought it would need to be updated in case starterKitMappings change. But if starterKitMappings are read and merged every time, then updating 'mappings' is of course not necessary!

So I could check for PL-compatible type

If components.json or composer.json exists and containsstarterKitMappings wouldn't that be enough to assume that it is a PL compatible StarterKit, regardless of type?

[dmolsen]

As a general rule I think it would ideal for it to be possible to 1) use StarterKits from (almost) anywhere, 2) use them 'as is' (no copying) and 3) have the possibility to install inside PL with or without VCS metadata.

👍 We're in agreement then. Though the caveat would be that I don't see PL itself handling VCS on install. That's a serious rabbit hole.

Is is possible to install dependencies with VCS metadata with Composer?

Yes! 😄 This is exactly how I work on Pattern Lab. See the composer.json for the development edition to see how I include the dev branch of each part of the base project. Then I can git whatever to my heart's content and do it within the overall framework.

So the 'mappings' you refer to is basically just a list of StarterKit directories? If so, couldn't it just be a list in config.yml?

I might have been overthinking this a little bit with mappings.json but I think in general the idea is the same. I save just a simple reference and then, on generate, use that reference to read in the full info. If that's config.yml or mappings.json/yml it's about the same. At the moment I'm not entirely sure I want to limit to StarterKits only. I do like the idea of grabbing individual components. e.g. ./vendor/phase2/carousel-component/ and ./vendor/phase2/marquee-component/ could be really simple components. We've played with that level of specificity at WVU. Might be impractical but I'd like to make sure it's possible.

If components.json or composer.json exists and contains starterKitMappings wouldn't that be enough to assume that it is a PL compatible StarterKit, regardless of type?

I phrased that awkwardly. Currently PL checks for a PL-compatible type on install before doing other things. Some types just need the type declared. They don't need anything else in composer.json. So I was saying check for type in Composer or the mere existence of someSchema.json. If either is available then it meant the package was PL-compatible.

There is value in being able to declare these sort of options once in a way that could be used by PL/PHP or PL/Node. I'm warming to the extra schema.

[aleksip]

That's a serious rabbit hole.

True... --starterkit --install --keep-vcs would be nice, but it is not required.

This is exactly how I work on Pattern Lab.

I really have to start using Composer this way too.

At the moment I'm not entirely sure I want to limit to StarterKits only. I do like the idea of grabbing individual components.

What do you think of the idea of just using StarterKits for this? That is, with starterKitMappings and support for multiple StarterKits one could create a StarterKit containing just one pattern/component? (See the example a few comments back.)

I'm warming to the extra schema.

:)

[dmolsen]

Created issue #75 which I see as related to this overall conversation.

[aleksip]

It's totally possible to add commands or modify existing ones via plugins.

@dmolsen I'd like to have a go, but not sure how to do this. PatternLab\Console::loadCommands() just seems to scan for command classes in Pattern Lab Core?

[sghoweri]

@EvanLovely @aleksip @dmolsen I don't want to jump the gun too prematurely but I think I might be on to something here... I managed to get a little POC semi-working today with PL working with a couple Patterns getting managed in a separate directory by Lerna.

Gonna try to iron out a couple kinks tomorrow but if this holds up I'll have a bunch more to share / ideas to bounce around.

[aleksip]

@sghoweri Sounds interesting! Looking forward to learning more about your experiments.

[mlnmln]

@EvanLovely Any updates here? I got similar needs, also currently working on a proof of concept with lernajs. Might be able to contribute.

[sghoweri]

@EvanLovely aren't we essentially doing this with the Bolt v1 Epic setup using the Lerna, Yarn workspaces and the new recursive Twig namespaces config?

The fully customizable directory paths work isn't technically required for this, right?

Cc @remydenton