Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sdk browser #236

Merged
merged 12 commits into from
Jan 11, 2017
Merged

Sdk browser #236

merged 12 commits into from
Jan 11, 2017

Conversation

emirotin
Copy link
Contributor

Same as before but from a local branch

@emirotin
Copy link
Contributor Author

Some strange error with browser tests so far, looks like env is not defined: https://monosnap.com/file/kT2wwHdiBf7hEcS2WYtqFhq5a5gvZY.png

@emirotin
Copy link
Contributor Author

Hm, started seeing it locally as well...

@emirotin
Copy link
Contributor Author

emirotin commented Nov 25, 2016

PhantomJS 2.1.1 (Linux 0.0.0) LOG: 'Running in the browser, env:', undefined

That's killing me, just ran locally and works fine

@emirotin
Copy link
Contributor Author

In addition AppVeyor tests failed because of HTTP timeout

@emirotin
Copy link
Contributor Author

emirotin commented Dec 6, 2016

So I've fixed the browser env issue, but still the AppVeyor tests fail because of HTTP timeout. Any idea how to deal with it? Just try increasing the timeout in tests definition? Linux/macOS tests don't seem to suffer from it and the full suite runs on my machine.

@emirotin
Copy link
Contributor Author

emirotin commented Dec 6, 2016

Yay, Travis passed. The timeout is already 30 minutes but it's the test timeout, I think there might be a separate fetch timeout that makes it throw. Will investigate.

fs = require('fs')
path = require('path')

exports.loadEnv = ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to replace this with https://www.npmjs.com/package/dotenv ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, absolutely. I was aware of it but was lazy to properly look into the docs :)

# });
###
exports.getAll = (callback) ->
Promise.try ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably be Promise.try(settings.getAll)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not rewrite the majority of code, just reindented almost every module to make it a factory instead of the static export. But will still go through your comment (I wish the test suite took less than 20 minutes to run...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This specific one I would ignore for two reasons:

  1. uniformity with the rest of the code in this module
  2. not thinking about if the settings module methods are bound or not

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm ok with leaving this one as-is, btw something that would be huge imo is if you can get the coffeescript headers added to the js files as I think that would cause github to exclude them from the diff as autogenerated files


deps = {
settings,
request: getRequest(opts),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no need for the commas

exports.create = (title, key, callback) ->

# Avoid ugly whitespaces
key = key.trim()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If key isn't a string this will throw, breaking the expectation that a promise will always be returned

device: deviceModel.get(uuid)
pubNubKeys: configModel.getPubNubKeys()
.then (results) ->
return logs.subscribe(results.pubNubKeys, results.device)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be nicer to do .then ({ pubNubKeys, device }) ->

# });
###
exports.isPassed = (callback) ->
token.getProperty('twoFactorRequired').then (twoFactorRequired) ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be .then(_.negate)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of the browser bundle size I try to use as few lodash as possible, so I will keep it as is.

@emirotin
Copy link
Contributor Author

emirotin commented Dec 6, 2016 via email

@Page-
Copy link
Contributor

Page- commented Dec 7, 2016

https://github.com/github/linguist#linguist
This library is used on GitHub.com to detect blob languages, ignore binary or vendored files, suppress generated files in diffs, and generate language breakdown graphs.

https://github.com/github/linguist#generated-file-detection
Not all plain text files are true source files. Generated files like minified js and compiled CoffeeScript can be detected and excluded from language stats. As an added bonus, unlike vendored and documentation files, these files are suppressed in diffs.

And the code that actually does the detection for coffeescript: https://github.com/github/linguist/blob/master/lib/linguist/generated.rb#L139-L176

@emirotin
Copy link
Contributor Author

emirotin commented Dec 7, 2016 via email

@emirotin
Copy link
Contributor Author

emirotin commented Dec 8, 2016

Hey @pimterry this is an important PR for the changes planned to a bunch of our interfaces.
It's ready to merge and ship but doesn't pass the tests on windows at the CI server.
Should you have any idea how to fix it I will be very happy as it's basically a month old and at the moment it's the only thing that prevents merging.

Other comments are welcome as well, of course, but for your convenience here's the overview:

  • ignore build/**/*.js files, they're generated by CS (and should bow be auto-collapsed in the files list)
  • the majority of lib/**/*.coffee files have identical changes — conversion from singleton modules to factory methods with dependency injection
  • the rest of the changes mostly consists of the browser tests through karma, generic normalization (like standard npm scripts), and updating the dependencies

javascript: 'build/**/*.js'
directories:
doc: 'doc/'
build: 'build/'

gulp.task 'coffee', ->
gulp.src(OPTIONS.files.app)
.pipe(coffee()).on('error', gutil.log)
.pipe(coffee(header: true)).on('error', gutil.log)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jviotti it'd be awesome to gradually add this to our other modules as well (as changes are made to them), it lets the .js files get suppressed in the diff and is overall so much easier to review :D

@Page-
Copy link
Contributor

Page- commented Dec 8, 2016

You've committed a yarn.lock file btw @emirotin

@emirotin
Copy link
Contributor Author

emirotin commented Dec 8, 2016 via email

@emirotin
Copy link
Contributor Author

emirotin commented Dec 8, 2016 via email

@pimterry
Copy link
Contributor

pimterry commented Dec 9, 2016

@emirotin: Sure, I can take a look at this. Not immediately though, as I don't actually work for Resin yet - I've just been keeping an occasional eye on interesting things since the summit. I start properly on Monday. Once I'm all set up with the basics then I'll take a proper look at this.

@emirotin
Copy link
Contributor Author

emirotin commented Dec 9, 2016 via email

@pimterry
Copy link
Contributor

I've got this all set up and working locally, and had a reasonable look at the changes, but I can't see any specific issues that would break this. I think it's pretty likely that the timeouts that are breaking these tests aren't from this change, and they're part of a wider existing issue instead.

The failures here seem to be a variable mix of ETIMEDOUT, which don't match between the two identical builds. This has been happening for months on and off for a while on all sorts of other Resin-SDK builds too:
https://ci.appveyor.com/project/resin-io/resin-sdk/build/1.0.853/job/mu1d9u2t86yf6web#L501,
https://ci.appveyor.com/project/resin-io/resin-sdk/build/1.0.854/job/71otvx4lh0p4dybe#L428,
https://ci.appveyor.com/project/resin-io/resin-sdk/build/1.0.891/job/yvn4diu3not0917r#L325, https://ci.appveyor.com/project/resin-io/resin-sdk/build/1.0.898/job/4827gdwyu322qjd8#L380, https://ci.appveyor.com/project/resin-io/resin-sdk/build/1.0.902/job/buv52hhr48wdjqe2#L311.

Doesn't look like the builds for this repo have been reliably passing since July: https://ci.appveyor.com/project/resin-io/resin-sdk/history.

My best guess for why this is only failing on Windows and not on Travis's linux builds is because the built-in connect timeout is much stricter on Windows (see nodejs/node-v0.x-archive#2324). Tricky to confirm this though, and it doesn't look like it's possible to do the easy short-term fix of boosting the connect timeout beyond the OS default (you can only shorten it).

This does suggest that the API endpoints are intermittently really slow (occasionally up to 12 seconds to get a response), which I can't easily reproduce locally. Do we have any stats or monitoring for those? Is that plausible? Alternatively something else could have been causing intermittent timeouts for months, but that's pretty surprising too.

I'm going to keep looking into this for now as it needs fixing either way, but it might not need to block this PR - it doesn't make it any worse than it already is. If you re-run the builds a couple of times they'll probably come around to green again 😭

@emirotin
Copy link
Contributor Author

Hey @pimterry thanks for the great investigation!
I didn't know about the past failures and assumed the change is somehow related to the switch from request to node-fetch.
OK then merging this sounds good for me, I'll run my final tests and add a bit of documentation about the browser support.

@Page- did I address your review?
@jviotti can I have an approval, does it look good to you?

@jviotti
Copy link
Contributor

jviotti commented Dec 12, 2016

I'm glad you resolved it! Looks good to me!

@Page-
Copy link
Contributor

Page- commented Dec 13, 2016

So the yarn.lock introduces a shrinkwrap but only for certain people and tests? I think I'd rather have the tests run against what will be installed when a user npm installs as that's the most common case and I'd want to make sure that works. If we want to switch to shrinkwrapping, then I'd like it for it to always be shrinkwrapped rather than only sometimes

@emirotin
Copy link
Contributor Author

emirotin commented Dec 13, 2016 via email

switch to the factory pattern
allow loading test env from the .env file
lazy-load all the models
update to mocha v3
fix deprecation warnings
granular lodash imports
use a more modern .asCallback name
add yarn.lock
use /v2/ api for tests
add CS preamble
@emirotin
Copy link
Contributor Author

@jviotti @pimterry what do you think about adding the support for yarn?
See Page's and mine comments above.
Basically it's the recommended way of using it - people who have it will have exactly the same deps as us when publishing which means more guarantees.
People who don't will have the same experience as before relying on npm's resolution at the install time.

@emirotin
Copy link
Contributor Author

emirotin commented Dec 16, 2016

Sounds like we should just try yarn and see if we have any problems.
Yarn is simple:

  • install it (globally) from npm
  • to get the shrinkwrapped set of deps run yarn install
  • to get the latest compatible deps run yarn upgrade (it will update the lock file which should be committed then)

I'm going to merge this PR as soon as all the tests pass.

@emirotin
Copy link
Contributor Author

emirotin commented Dec 16, 2016

Looks like good to go?:)

@emirotin
Copy link
Contributor Author

As we have some good convo WRT using a better fetch polyfill here I'll wait and not merge it before we have it sorted in all the sub-deps.

@pimterry
Copy link
Contributor

pimterry commented Jan 6, 2017

With all the recent changes finally finished up and #245 merging them in, I think this really is good to go now. Is there anything else left over, or can we merge this @emirotin?

I think immediate next steps are:

  • Updating the list of interdependencies
  • Using this in the CLI
  • Moving the data services in the UI to start using this

I'll get started on those shortly. Let me know if there's anything else related to this I should check out.

@emirotin
Copy link
Contributor Author

emirotin commented Jan 6, 2017

Before merging I'd like to do several things:

  • generalize the mocking code and move it to a separate module (plan to do today)
  • write a bit of docs about the SDK browser usage specifics and experimental nature
  • finalize a sample app that demonstrates the browser usage with browserify (I already have it, just needs some polishing)

Things to do before or after the release:

  • create the equivalent example with webpack
  • think about encoding the exclusion of node-only deps (like stream) in this module rather than in the consumer - not sure if possible without the own builds - see below (for context: to reduce the size of the browser bundle these deps need to excluded and with browserify it's done in the consumer module's config)
  • provide browser builds dist (at least a UMD module, but maybe also the version that doesn't bundle some bigger common deps, like Bluebird)

@pimterry pimterry mentioned this pull request Jan 6, 2017
This now uses the new injection dependency approach, so we build each
of these libraries and glue them together, rather than them internally
instantiating their own local versions of one another.
@emirotin
Copy link
Contributor Author

Looks great.
Couple of nits:

  • let's provide both unminified and minified bundles
  • let's squash some commits where relevant

Other than that let's merge as soon as the tests pass (unless other guys have any objections, of course).

One bug solved en route here, highlighted by minification: we depended on the
function name of errors thrown by Resin-Request, which isn't dependable
(e.g. in prescence of minification). We now depend on .code instead,
which should work reliably.
@pimterry
Copy link
Contributor

@emirotin I've now squashed the bundle commits in place here (sorry, should've done that originally), and there's a minified+unminified PR up at #253

@pimterry
Copy link
Contributor

I've re-run the one last failed Travis job, just to be sure. Once I've got that passing, based on @emirotin's last comment not getting any objections, I'm going to merge this 🎉. Last chance to shout now if you disagree!

@pimterry pimterry merged commit 593bd1f into master Jan 11, 2017
@pimterry
Copy link
Contributor

🎉🎉🎉

@emirotin
Copy link
Contributor Author

emirotin commented Jan 11, 2017 via email

@pimterry
Copy link
Contributor

@emirotin Sure, done. I'm going to get started on trying to pull this into resin-cli now, we'll see how it goes.

@pimterry
Copy link
Contributor

Ah, not done -prepublish means I have to wait for a full local build to complete before I can publish, and that's still running. En route though! 😄

@emirotin
Copy link
Contributor Author

emirotin commented Jan 11, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants