Roadmap to 2.0.0

[DaftMonk]

For Angular Fullstack 2.0.0, I'm making some major changes to the way the project is structured. I want to get feedback from the community on the direction I'll be going.

Here's the relevant links, with explanation of the changes below

New generator that will take the role of the fullstack sub-generators: https://github.com/DaftMonk/generator-ng-component.

New project structure:
https://github.com/DaftMonk/modular-fs

Changes to build process and structure:

The server and client sides are now using a modular file structure, and the build process has been improved to handle it.

• Angular client now uses the Google's project recommendations:
  https://docs.google.com/document/d/1XXMvReO8-Awi1EZXAXS4PzDzdNvV6pGcuaF4Q9821Es/pub
• Component scss is automatically injected into app.scss.
• Templates are now compiled into a single javascript payload using grunt-angular-templates.

The Express server now follows similar structure:

• Each server component folder would contain something like this:
  • name.js (for routes)
  • name-model.js
  • name-model_spec.js
  • name-controller
  • name-controller_spec.js
• Server component models and routes are automatically loaded.
• Better 404 handling. Asset folders will now send a 404 if there is no defined resource, which will override the catch-all redirect to index.html.
• Client and server side tests now look for _spec.js files to run their tests.
• Views folder is now meant purely for express rendered views. Having express and angular share views files only complicated things in the build process and in the app. Even if you use a preprocessor like jade for your html views, they should compile down into html so that they can be served as an html to js payload.

Changes to the generator:

I will be removing all current angular-fullstack sub generators, and creating a separate generator for angular components. The .yo-rc.json file will allow you to configure project and template paths. I've started work on this here: https://github.com/DaftMonk/generator-ng-component.

The new ng-component generator will support scaffolding routes with the angular-ui router as well as ngRoute.

The angular-fullstack generator will save configurations in the .yo-rc.json file as well, so you can reuse your configurations between projects.

A link to the new project structure is provided above. You can clone the new project structure for yourself, npm and bower install, and try it out. Please tell me what you think!

[fiznool]

I like the way this is going. I haven't been using the current generator for long, so my insight is limited, but for what it's worth, here are my thoughts.

• Building the HTML templates to a single js file is a great step forward.
• A generator for a server side endpoint would be nice. Something like yo angular-fullstack:endpoint, which would scaffold a routes file, controller and model/schema, using rails-inspired naming conventions (show/index/create/update/destroy) for the controller methods.
• Server-side relative routes files would work well with the new modular filesystem. So the individual routers in the modules define everything from / and it is the responsibility of the main app router to wire these all up to the actual route URL. See http://start.jcolemorrison.com/building-an-angular-and-express-app-part-2/ for more info (scroll down to Getting the Data to Express).
• Do the changes to the front end mean that this project won't be following the main generator-angular any more?
• Maybe it's just me but I prefer 'client' and 'server' for the client-side and server-side files, rather than 'app' and 'lib'. It's just a bit easier to distinguish between the two when they are so explicit.
• On the topic of naming conventions I also prefer .spec to _spec. This is really just personal preference, though :)

[snowman-repos]

I agree with all of Tom's points (and yeah, having 'lib' as the server config folder threw me a bit the first time I used it - every time I use it now the first thing I do is rename that folder to ‘server’).

I particularly like generator-ng-component, it makes a lot of sense because in this full-stack context you're likely to want to create controllers and views etc. together to tie in with the API, more like rails & ember.

[DaftMonk]

Thanks for the feedback!

Server side endpoint generators would be great, and I have been wanting to add them for a while. I wasn't planning on including that in 2.0, but I will reconsider that now.

More clear separation between server/client folders makes sense. So I would probably adjust the folder structure like this

- client/     - was app
- server/     - was lib
  - app.js    - was server.js in root
  - config/   - was in root
  - components/
  - views/

I was considering continuing to support the generator-angular structure by allowing you to configure a yo-rc.json file for it, but I actually think it would be too hard to do that without making customization of templates very complicated. So this new structure will be quite different from generator-angular, but they are moving in this direction anyway.

I didn't realize the new express routing system worked that way. Would using that be preferable to the way the routes are loaded in the example project?

I would think it would be harder to automate loading of the routes with the express 4 method. You might need to have the generator inject the routes for you. or have them injected through a grunt task.

[kevinawoo]

I would prefer the new style, as I can then create modules

- server/
    - [module]/
        - index.js
        - routes.js
        .....
    - [module2]/

even better would be to have app.js bootstrap each module which could be done by walking the folders.

[fiznool]

I think the benefit of arranging the routes like described in the blog post above is that for each component they are not tied to the absolute endpoint URL. At the moment, the thing endpoint is defined as /api/awesomeThings, whereas using modular routing, the 'global' routes file would define api/awesomeThings and link this to the thing component, which would then define everything from /. The benefit here being that the thing component is no longer tied to the absolute endpoint URL, and can hence be dropped into another project without any refactoring needed.

I also like the fact that each component controller can be scaffolded with show/index/create/update/destroy methods automatically. This should hopefully promote best RESTful practices from the outset. Of course, if you then don't need some of the endpoints you can just delete them - or, maybe even the generator asks which ones you want to create, defaulting to them all. By making the routing modular, the absolute endpoint won't need to be injected into the controller by the generator, instead the controller just becomes a simple template, as the relative routes will always be the same.

On that last point, is it difficult to take an existing server/routes.js file, parse it and add a new route? Would it be easier if you defined the routes in a json file instead, a la Backbone? For example:

{
  "/api/awesomeThings": "thing"
}

The json file could then be parsed and all keys added to an app.route definition, and all values used to lookup the appropriate component in the server/components folder.

[fiznool]

I think it's a good decision to separate from the generator-angular project. It means that this generator can (hopefully) iterate more quickly than the upstream one.

For example, it would mean that protractor could be supported from the outset. I'm a big fan of protractor, and I think it should be in the generator from 2.0.0. At the moment, generator-angular uses generator-karma to create the config files, but this is still producing a karma e2e config. The karma generator is actually very simple, and I believe it could be rewritten as part of this generator, with the benefit of supporting protractor instead of the older karma e2e.

I'm happy to help out with this, I've had some experience hacking about with the karma stuff and I have protractor working with the current version of the angular-fullstack generator.

[fiznool]

One more thought - Angular seems to support Jasmine 2.0 so it might be a good idea to use that instead of the 1.x branch.

[DaftMonk]

Excellent points. The karma e2e should definitely be replaced with Protractor, and any help with that would be great. If you want, you can submit a PR to the project structure repo and I'll use it in the new scaffold.

I updated the project to use Jasmine 2.0. It's so nice to have done for async tests.

There seem to be tradeoffs with the routing. The more automated the loading is of component routes, the harder it is to customize or exclude route definitions. If we have a less automated approach, like having the generator insert the

  var component = require('./component');
  ...
 `app.use('api/component', component);

then you can exclude routes easier, but if you add a component from another project, or you delete a component, you have to update those references. I'm not sure about which way to go.

[kevinawoo]

I've been working on the component side of the routing. I'll have a PR if it works.

I've been walking the components/ folder to determine their URL route names.

My rules:

• every folder under components/ is a new route mounted to /
• every folder components/(subFolder|*) should mount to / and take name subfolder/*
• dangling files will be mounted to it's containing folder's URL path
• each new component folder should have blueprints for GET, POST, PUT, DELETE

- server/
    - components/ 
        - api/                   # bootstraps all subfolders URL /api
            - thing1/            # sets up GET, POST... URL /api/thing1
                - routes.js
                - models.js      # structure for database schemas
                - controller.js
                - index.js       # allows for easy `var route = require('thing1')
            - thing2/            
                - routes.js
                - models.js      
                - controller.js
                - index.js       
            - index.js 
        - thing3.js              # compressed version of thing2/*  URL /thing3
        - signin/                # module for sigin   URL /sigin
            - routes.js          # the actually routes
            - models.js
            - controller.js      # permissions and other stuff
            - index.js           # just builds the component
    - routes.js                  # bootstraps anything in components
- util
    load.js                      # bootstraps routes

URLs

/api/thing1/:id    # get
/api/thing2
/api/thing3
/api/*             # throw 404
/signin/

We could use this to drop in any "component" folder and its automatically picked up as a subpath.

Any thoughts?

Side note, could you change the name of components to modules? I'm having the hardest time typing it 500 times. haha..

[fiznool]

I've created an example repo to explain how I envisaged the routing structure. @kevinawoo I think it's similar to what you are proposing, except there is walking of the components/ folder - instead everything is explicitly wired up in the routes.js file. It's pretty easy to try this out in the REST console or using curl, just git clone the repo, npm install and then node app.js.

Notice that the component doesn't know anything about the base route, and doesn't care either. Everything is self-contained inside the thing folder. I've named the files thing.controller.js and thing.model.js as otherwise it can be really confusing when you've got lots of components, all with controllers named controller.js!

For this example, the routes are defined in code rather than in a config object as previously mentioned, or using the walking method proposed by @kevinawoo. I've come round to the idea that explicitly defining these in code is the most flexible and best way forward, as it allows users to customise to their heart's content. This fits in with my own personal idea of a generator being a set of best practices, but at the same time being flexible enough to change if you need/want to.

[fiznool]

@DaftMonk I'd be happy to submit a PR for protractor support. What sort of timeline are you looking at for 2.0? I might not have much time over the next couple of days but can probably tackle this at the weekend.

[kevinawoo]

@fiznool You may be right about keeping it explicit and simple for a generator. The less magic that happens behind the screens, the easier it'll be to understand and keep clean. It almost comes down to scale. The app I'm using your generator for, I have plans for it to be fairly large. However smaller apps to midsize apps, @fiznool method would be best.

I think @fiznool gave the best suggestion. It also wouldn't be very hard if I wanted to do my 'walk' with what he has set up. 👍 on the naming convention, it may be redundant, but oh does it make those little editor tabs more definitive.

As for a subgenerator for routing, is it possible to do inserts in routes.js in a predefined area for better conflict management? I was thinking something similar to grunt-bower-install.

[kevinawoo]

I would also like to note that having the 404.js under config/ confused me, as those are routes/components/views in my mind.

It makes sense to me to have the client/ handle the 404.html and other subsequent error pages, and the 404.js to be a part of components/ that route.js can require and use. They both could sit in a customErrorHandler/. 404.html can then take full advantages of compass, minification, and bower and be built into a single serve html.

Little utility files like load.js might be best to sit in server/util so that other sections of the server will have access to those functions easily.

[fiznool]

Something like the bower install task for updating the routes file is a nice idea. Somebody has blogged about how he went about injecting code into an existing file using a 'yeoman hook', might be of some use?

[DaftMonk]

@fiznool That example is really helpful. I like that routing method, and agree that we should leave the magic out of loading those routes for now. I think it would probably be best to only do injection on routes created by the generator, when the route is initially scaffolded.

We might want to automatically require the component route files though. We could do something similar to https://www.npmjs.org/package/gulp-load-plugins for loading the component route files, so you don't have a ton of require statements.

I'd like to aim for releasing 2.0 late next week.

@kevinawoo I used 'components' rather than 'modules' on the client side to distinguish from angular modules. On the server side, I just did it to be consistent.

I think you're right about the 404 page, makes more sense to have that be in server/components/errorHandler.

The 404 should be handled on the server side in this case. For angular to handle it, we would have to redirect back to the app at index.html, which could cause an infinite loop if assets are missing on the index.html page.

In your example, for components, what does index.js do vs routes.js?

[fiznool]

@DaftMonk yes, or something similar to your loading pattern in the existing generator, where you bootstrap all of the models in server.js? The only thing is, I was wanting to get away from 'magic' and be a bit more explicit, in case people wanted to customise a bit.

We could slim it down to:

app.use('/api/thing', require('./components/thing'));

[DaftMonk]

Sure, that would work.

[kevinawoo]

If a folder contains index.js it allows the require statement to reference the folder as a import. So it really just serves as a starting point for exposing public functions. Similar to python's __init__.py.

routes.js should only contain routes.

So I took that idea and had the components/index.js collect itself's routes.js, then walk containing files and folders, picking up exports. It then pushes them in an stack to be popped by its containing parent. All end-of-the-line exports needed to be route stacks, and any middle handlers would app.use('/componentName', require(compoment)(app)) where app is passed from the parent.

Finally server/routes.js just app.use('/', require('./components')(app)) and also handled /* to redirect to 404.

[kevinawoo]

@DaftMonk, you're right about angular not handling the 404, infinity prob isn't a good thing. I am guessing there is still an option to still to do a dist/ build of the client/ folder. I always thought of dist/ to be portable, server agnostic and was "compiled" from client/. The problem I see with having 404.html file in the server/view, is there's a disconnect about 404.html existing in dist/.

It may be good to do dist/404.html or dist/errorPages/404.html and be inlined. Some options are:
https://github.com/jgallen23/grunt-inline-css
https://www.npmjs.org/package/grunt-inline
https://www.npmjs.org/package/grunt-static-inline

For client/ it could be a package folder, similar to Mac applications. 404.html/ is a folder to contain 404.styles.scss, 404.scripts.js, 404.html. Then grunt can compass, jshint, jsmin, cssmin, inline, htmlmin the folder, then replace 404.html/ folder with 404.html file.
It almost sounds like a good job for gulp:
https://www.npmjs.org/package/gulp-inline-css
https://www.npmjs.org/package/gulp-inline-source

[remicastaing]

@kevinawoo, I really like your proposition for building a REST API in a modular manner. But I'm wondering if it's a good option to give every folder it's own model. What if the thing2 model relies on the thing1 model for some operation? For exemple, in a blog system: I use the 'comment' model to create a comment and the 'comment' model uses some method of the 'article' model to increment the number of comments.

[kevinawoo]

Can we move @fiznool components/thing to components/api/thing so it better mirrors the URL mountpoints?
https://github.com/kevinawoo/ng-component-router-example

@remicastaing I like what @fiznool said about it being customizable, so each newly generated components should be fully separate. Then you may customize it however you see fit.
I think @DaftMonk is setting up a "yeoman-hook" so it can just be dropped in quickly and cleanly.

I would probably use something like this if we were talking about a user:

- components/
    - models/
        - thing.base.model.js       # base user model
        - thing2.base.model.js      # extends thing2
        - thing3.base.model.js      # [optinal] if you may think it should be extendable 
    - api/                          # these models public exposures depend on permissions
        - apiModels/
            - thing.api.model.js    # strips thing1.base model, (remove user's physical address, keeping everything else)
            - thing2.api.model.js   # strips thing2.base (showing only username)
        - thing/
            - index.js
            - thing.model.js        # specific just to thing (generate md5 hash for verification)
            - thing.controller.js
        - thing2/
            - index.js
            - thing2.model.js       # created from thing2.base (needs a captcha verifcation?)
            - thing2.controller.js
    - thing3/
        - index.js
        - thing3.model.js           # this should never be exposed, internal use only (user's social security #)
        - thing3.controller.js

[DaftMonk]

Made some more adjustments to the project structure based on feedback. I'm really happy with the direction this is going.

@kevinawoo I like that structure for larger apps, but for medium/small apps, I think having that extra folder layer for the api could be kind of annoying.

[DaftMonk]

@stevelorimer That makes sense, they are very related. However, I'm not sure it makes sense to call it session, since we're using tokens now, we don't really have sessions any more.

Still I wonder what to do with the config/access middleware. Should that be a config or a component?

@remicastaing Thats a very interesting idea. Could you give me an example of what the account model might look like?

[fiznool]

Does having initializations for user and thing in 'component-name'.fixture.js make sense? They're being run from the /index.js file, is that ok or should there be some kind of separate install script that runs them?

Something I'm wrestling with right now is not persisting the fixtures to the database during testing. Ideally for testing the server-side I would like a pristine database, and setup my own fixtures in my test specs.

For this reason, I think it might be better just to have a single script that sets up the fixtures, and this is run for development and production modes but not for test mode.

What should be done about components that don't quite fit into the category of controller/model. For example, the authenticate.controller is used for logging in right now. There's also a config/access.js file that is a middleware used to check the role of the user before letting them access an api url.

I'd personally consider these to be core pieces of the server and so they should go in a separate place to components/. I view components as being reusable chunks that can be reused on different projects. What you are describing is essentially boilerplate middleware so I would just leave them in config.

[fiznool]

Regarding the fixtures, right now this is what I'm doing for the 1.x branch.

// config/env/{development|production}.js
module.exports = {
  ...
  dummydata: true
  ...
};

// server.js
if(config.dummydata) { require('./server/config/dummydata'); }

I guess this could be extended further, so you could have different dummy data for development, test and production, if you wanted to.

[DaftMonk]

@fiznool Ok, so I'll move the fixture data back into a single file: sampledata.js, and have it required in the server based on the config.

I see what you mean about the access file being more of a server wide configuration helper. Still, I wonder, do you think there would be a place for 'services', cron jobs etc, or 'middleware' in the components?

Also I'm still not sure how to get right the api urls and folder structure for logging in and logging out. Since we're using access tokens I'm thinking something like this:

login:
POST to auth/access_tokens
with the following data {username, password},

logout:
DELETE auth/access_tokens/{accessToken}

folder structure

components/auth/index.js
components/auth/access_token/index.js
components/auth/access_token/access_token.controller.js

On another note, I may also want to add an option to support socket.io in the generator, creating a socket file in the server root that loads component based sockets, same as the routes.

[fiznool]

@DaftMonk sorry for the silence...

Still, I wonder, do you think there would be a place for 'services', cron jobs etc, or 'middleware' in the components?

I guess there could be instances where application-wide things like middleware and cron jobs go in components. Or they could go in another folder. Are you thinking of building a generator for these? If not then I guess we could leave it up to the user to decide where to put them.

Also I'm still not sure how to get right the api urls and folder structure for logging in and logging out.

I like the proposal above, very RESTful. It makes sense since we are asking the server to create and destroy our tokens. I would personally drop the access_ and just make the URL auth/tokens for a little more succinctness, but this is a very minor thing.

On another note, I may also want to add an option to support socket.io in the generator, creating a socket file in the server root that loads component based sockets, same as the routes.

I saw the socket.io addition to the repo. Very cool! Looking forward to playing with that. I can see it being very useful on the project I am working on at present.

[DaftMonk]

I've added social Auths to the passport branch https://github.com/DaftMonk/modular-fs/tree/passport. Any feedback on it would be appreciated.

I had to add back in sessions because the twitter passport strategy requires them to keep identify the user between the twitter login and the app callback. Maybe adding connect-mongo as a session store is too heavy for this purpose. Any drawbacks to just using the in memory session store for this?

@fiznool I've split up the 'components' to make it more clear what it should be used for. Everything will still be modularized by folders, but the api and auth folders will hold components that are more app specific. Any components that are mostly reusable between apps should go in the components folder.

[steve-lorimer]

Since you're going mean stack route, and therefore have mongo already, I
wouldn't necessarily think connect-mongo was too heavy weight?

On 12 May 2014 11:38, Tyler Henkel notifications@github.com wrote:

I've added social Auths to the passport branch. Any feedback on it would
be appreciated.

I had to add back in sessions because the twitter passport strategy
requires them to keep identify the user between the twitter login and the
app callback. Maybe adding connect-mongo as a session store is too heavy
for this purpose. Any drawbacks to just using the in memory session store
for this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/192#issuecomment-42790138
.

[DaftMonk]

Well if we only use sessions to to save some info for a few seconds, and then remove it, we might not want to save it to the database. Mainly I would prefer to leave it out, if possible, because occasionally connect-mongo gives warnings: #85.

But if there are other use cases for having a session store, or performance issues from using the in memory store, it works well enough that we could leave it in.

[DaftMonk]

Just had a look at FeathersJS, http://feathersjs.com/#example, and from first glance, it looks like it would be a great addition to the app. It's a very lightweight framework built on express that allows you to bridge the gap between your express rest api and socket.io.

When I get the chance, I'll try updating the socket.io branch to use it.

[kevinawoo]

Awesome, FeathersJS looks like a great idea. Hate to ask, but any anticipated completion time of 2.0?

[kjellski]

Oh my, FeathersJS would be extremely great! 👍

[DaftMonk]

@kevinawoo Won't have much time to work on the generator for the next few days, still getting moved into a new apartment. I might be able to get at least a pre-release out by the end of the month.

[DaftMonk]

Hmm, after toying around with feathers it wasn't as easy to add to the project as I thought it would be. I might be doing something wrong, but I was having trouble using some of the express 4.0 router functions with it.

It would be great if someone else could take a stab a integrating it, but I really want to get the current set of features released asap, so I'm going to hold off on adding feathers for now.

[remicastaing]

I forked @DaftMonk's passport branch and I did some refactoring inside server\auth.

I also divided the user model in two models: user and account. One user can have multiple accounts.

Take a look at: https://github.com/remicastaing/modular-fs/tree/multisocial

[kjellski]

@remicastaing I've had a quick look, looks really good, made some comments. Take them with a grain of salt, had no time to really test it practically... and thanks for making this!

[remicastaing]

Moin @kjellski, thanks for having a look. Time now for me to add some specs files.

[outrightmental]

Hey all, I have some time over the next month to help out. Anything in particular that could use a mini-project?

[steve-lorimer]

Yay!

[dancancro]

Congrats on the new release. Would you like to help out and update this? Send me your email address and I'll add you as an editor.

http://www.dancancro.com/comparison-of-angularjs-application-starters/

On Jul 3, 2014, at 5:31 AM, Steve Lorimer notifications@github.com wrote:

Yay!

—
Reply to this email directly or view it on GitHub.

[steve-lorimer]

Hi Tyler

I've just started using the fullstack generator after a fair while away
from all things angular, and it's AWESOME! I just wanted to say props to
you.

Cheers
Steve

On 3 July 2014 16:44, Dan Cancro notifications@github.com wrote:

Congrats on the new release. Would you like to help out and update this?
Send me your email address and I'll add you as an editor.

http://www.dancancro.com/comparison-of-angularjs-application-starters/

On Jul 3, 2014, at 5:31 AM, Steve Lorimer notifications@github.com
wrote:

Yay!

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#192 (comment)
.