Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

endpoint for predictions #183

Open
gyachdav opened this issue Oct 14, 2016 · 74 comments
Open

endpoint for predictions #183

gyachdav opened this issue Oct 14, 2016 · 74 comments
Assignees
Labels

Comments

@gyachdav
Copy link

Hello there pokedata, it has been a while 😆

I didn't see any spec in your API that will allow the map team to retrieve predictions. Are predictions included in the sighting data? is there a parameter that distinguish predictions from sightings?

Same questions apply to the model that holds prediction data.

Please respond ASAP.

@swathi-ssunder
Copy link
Collaborator

@gyachdav - Hello, the pokemon sightings from the API(that are fetched from the db) do not include the predictions.

@sacdallago
Copy link
Member

Yes, because they should be generated on the fly :)

I guess once you can run the predictions, aka call the right function, and get an output, it will be pretty straightforward to implement the route.. Although: you could already implement the route, and imagine to have some magic predictShit(lat, lng, timestamp) function that returns the predictions and you later expose this to a GET request :)

@bensLine
Copy link

Btw. predictions for predict(lat, lng, timestamp) will be available as an array of the following objects:
{"pokemonId":"16","confidence":"0.242","latitude":11.6088567,"longitude":48.1679286}

@vivek-sethia
Copy link
Collaborator

Refer #187 . Changes will need to be made after the predictions are made available later.

@gyachdav
Copy link
Author

referencing PokemonGoers/PredictPokemon-2#67 again since I did not get any answer so far.

According to @Aurel-Roci the integration of predictions into this project is complete. On the demo app I do not see any prediction.

What is the status then? Is there a demo URL that will show that this integration is really done?

@PokemonGoers/pokedata @PokemonGoers/predictpokemon-2

@gyachdav
Copy link
Author

@sacdallago please make sure this is handled in the next 24 hours thanks

@sacdallago
Copy link
Member

Guys, can you give us an endpoint? :)

@vivek-sethia
Copy link
Collaborator

vivek-sethia commented Oct 28, 2016

Referring to this, the route is already implemented.

As mentioned in the above comments, we only need to know which function needs to be invoked and were waiting on @PokemonGoers/predictpokemon-2 for this.

Correct me if I am missing something here.

@Aurel-Roci
Copy link

The function is mentioned in the comments above:
function: predict(lat, lng, timestamp)
response : {"pokemonId":"16","confidence":"0.242","latitude":11.6088567,"longitude":48.1679286}

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - And how are we supposed to include your project? Is it available as an npm package?

@Aurel-Roci
Copy link

@vivek-sethia Yes it is available as a package, I mentioned that in the issue with adding the java in the docker container. I thought this was already known by you guys since @jonas-he mentioned the package.

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - seems like some miscommunication. Will integrate the package and commit in a while.

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - I am trying to integrate and get the following error about the file prediction_feature_config.json.
Is there something that needs to be done additionally?

2016-10-28T18:05:13.742Z started prediction script, init DC
fs.js:634
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: ENOENT: no such file or directory, open 'prediction_feature_config.json'
    at Error (native)
    at Object.fs.openSync (fs.js:634:18)
    at Object.fs.readFileSync (fs.js:502:33)
    at fileToJson (PokeData/node_modules/predict-pokemon/dataSet_creator.js:364:23)
    at Object.DC.init (PokeData/node_modules/predict-pokemon/dataSet_creator.js:30:22)
    at PokeData/node_modules/predict-pokemon/prediction.js:31:8
    at Object.<anonymous> (PokeData/node_modules/predict-pokemon/prediction.js:241:3)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)

@Aurel-Roci
Copy link

Aurel-Roci commented Oct 28, 2016

@vivek-sethia Yeah, sorry forgot to tell you that you have to unzip json/pokestop_groups.7z file in the prediction npm package location. It will take some time to initialize the module. Let me know if it works

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - I already extracted it after referring to PokemonGoers/PredictPokemon-2#67 (comment).

And still got this error, so wanted to know if there is anything else?

@Aurel-Roci
Copy link

Aurel-Roci commented Oct 28, 2016

@vivek-sethia I cant seem to reproduce that error. The only thing that comes to my mind is to add / before the file name /prediction_feature_config.json, like so.

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - Where exactly am I supposed to make this change ?

@Aurel-Roci
Copy link

@vivek-sethia in node_modules/predict-pokemon/prediction.js. Line 31

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - Doesn't work.
Well, do I have to copy the extracted pokestop_groups.json to some other location?

@Aurel-Roci
Copy link

@vivek-sethia from what I see from the stacktrace the problem is with prediction_feature_config.json. For some reason it cannot find it, which I dont understand

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - Exactly, I already verified that the prediction_feature_config.json file exists at the expected location.
I have committed the change. Maybe you could check once if there is something missing.

@Aurel-Roci
Copy link

@vivek-sethia the require('predict-pokemon'), should be require('predict-pokemon').predictor.

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - Already tried that too. Same error as above.

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci - Maybe you could try the code yourself and see it it works.

@semioniy
Copy link

@vivek-sethia I guess we found an error, stay tuned, we try to solve it.

@Aurel-Roci
Copy link

Aurel-Roci commented Oct 28, 2016

@vivek-sethia Thanks to @semioniy, this is what we found, first what has to be done is move the content of the predict-pokemon package from Project/node_modules/predict-pokemon directly into the project directory. For some reason the paths do not seem to work correctly in the package. Also change the tzwhere like described here https://www.npmjs.com/package/predict-pokemon

@vivek-sethia
Copy link
Collaborator

@Aurel-Roci Can you please make this change and test it yourself? Seems to be much easier that way. You can amend my commit if needed.

@bensLine
Copy link

bensLine commented Nov 7, 2016

Oh, good :)
The file was to big for git, if there is no limitation for npm packages we should be able to do that.

@sacdallago
Copy link
Member

@bensLine for the reference: you can use unzip to unzip a zip file in your package before the packe is being installed, by defining this directive in the preinstall script of the package.json. This is fancy AF. Or you can do the unzipping in prepublish, but that way you would have the unzipped files in your repo, although you could .gitignore them. There are PLENTY way to handle this :) but it should be done on your side!

@bensLine
Copy link

bensLine commented Nov 8, 2016

@sacdallago thanks for the info, I never used JS and npm before the seminar, so every pointer is very welcome :) someone of my team @PokemonGoers/predictpokemon-2 will look into it.

@sacdallago
Copy link
Member

Since PokemonGoers/PredictPokemon-2#74 (comment), @swathi-ssunder version bump, docker hub trigger: we see if it works?
@bensLine

@swathi-ssunder
Copy link
Collaborator

Updated to latest version #196. Hopefully everything works this time 🙏

@jonas-he
Copy link
Collaborator

@sacdallago its merged and docker build finished successfully. However DNS is not working right now. Maybe because there is still an error (->container shuts down) like we had in the past?

@sacdallago
Copy link
Member

Merci! @jonas-he will have a look at the docker cloud log in the mornin'

@sacdallago
Copy link
Member

Not over yet! 🗡

[pokedatadevelop-1]2016-11-13T01:16:24.075715817Z npm info it worked if it ends with ok
[pokedatadevelop-1]2016-11-13T01:16:24.077662800Z npm info using npm@2.14.2
[pokedatadevelop-1]2016-11-13T01:16:24.080686356Z npm info using node@v4.0.0
[pokedatadevelop-1]2016-11-13T01:16:25.659310042Z npm info prestart pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-13T01:16:25.708214646Z npm info start pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-13T01:16:25.716176367Z 
[pokedatadevelop-1]2016-11-13T01:16:25.716226527Z > pokemongo-api@0.0.1 start /usr/src/app
[pokedatadevelop-1]2016-11-13T01:16:25.716235727Z > NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-13T01:16:25.716241644Z 
[pokedatadevelop-1]2016-11-13T01:16:35.636328011Z 2016-11-13T01:16:35.633Z started prediction script, init DC
[pokedatadevelop-1]2016-11-13T01:16:51.489398555Z fs.js:549
[pokedatadevelop-1]2016-11-13T01:16:51.489440515Z   return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
[pokedatadevelop-1]2016-11-13T01:16:51.489452597Z                                                   ^
[pokedatadevelop-1]2016-11-13T01:16:51.489458631Z 
[pokedatadevelop-1]2016-11-13T01:16:51.489463581Z Error: ENOENT: no such file or directory, open '/usr/src/app/node_modules/predict-pokemon/feature_sources/../json/pokestop_groups.json'
[pokedatadevelop-1]2016-11-13T01:16:51.489468642Z     at Error (native)
[pokedatadevelop-1]2016-11-13T01:16:51.489473087Z     at Object.fs.openSync (fs.js:549:18)
[pokedatadevelop-1]2016-11-13T01:16:51.489477925Z     at Object.fs.readFileSync (fs.js:397:15)
[pokedatadevelop-1]2016-11-13T01:16:51.489482905Z     at fileToJson (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:139:23)
[pokedatadevelop-1]2016-11-13T01:16:51.489488102Z     at /usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:7:26
[pokedatadevelop-1]2016-11-13T01:16:51.489541341Z     at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:142:3)
[pokedatadevelop-1]2016-11-13T01:16:51.489554023Z     at Module._compile (module.js:434:26)
[pokedatadevelop-1]2016-11-13T01:16:51.489559155Z     at Object.Module._extensions..js (module.js:452:10)
[pokedatadevelop-1]2016-11-13T01:16:51.489563271Z     at Module.load (module.js:355:32)
[pokedatadevelop-1]2016-11-13T01:16:51.489567232Z     at Function.Module._load (module.js:310:12)
[pokedatadevelop-1]2016-11-13T01:16:51.489571127Z     at Module.require (module.js:365:17)
[pokedatadevelop-1]2016-11-13T01:16:51.489575012Z     at require (module.js:384:17)
[pokedatadevelop-1]2016-11-13T01:16:51.489579314Z     at /usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:52:30
[pokedatadevelop-1]2016-11-13T01:16:51.489583692Z     at Array.forEach (native)
[pokedatadevelop-1]2016-11-13T01:16:51.489588267Z     at Object.DC.init (/usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:38:32)
[pokedatadevelop-1]2016-11-13T01:16:51.489593029Z     at /usr/src/app/node_modules/predict-pokemon/prediction.js:48:8
[pokedatadevelop-1]2016-11-13T01:16:51.641364127Z 
[pokedatadevelop-1]2016-11-13T01:16:51.650631189Z npm info pokemongo-api@0.0.1 Failed to exec start script
[pokedatadevelop-1]2016-11-13T01:16:51.651150535Z npm ERR! Linux 4.4.0-42-generic
[pokedatadevelop-1]2016-11-13T01:16:51.654334520Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
[pokedatadevelop-1]2016-11-13T01:16:51.655635753Z npm ERR! node v4.0.0
[pokedatadevelop-1]2016-11-13T01:16:51.657848977Z npm ERR! npm  v2.14.2
[pokedatadevelop-1]2016-11-13T01:16:51.659788031Z npm ERR! code ELIFECYCLE
[pokedatadevelop-1]2016-11-13T01:16:51.661028434Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js`
[pokedatadevelop-1]2016-11-13T01:16:51.662301128Z npm ERR! Exit status 1
[pokedatadevelop-1]2016-11-13T01:16:51.663278508Z npm ERR! 
[pokedatadevelop-1]2016-11-13T01:16:51.664938311Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'.
[pokedatadevelop-1]2016-11-13T01:16:51.666558686Z npm ERR! This is most likely a problem with the pokemongo-api package,
[pokedatadevelop-1]2016-11-13T01:16:51.667410798Z npm ERR! not with npm itself.
[pokedatadevelop-1]2016-11-13T01:16:51.669910098Z npm ERR! Tell the author that this fails on your system:
[pokedatadevelop-1]2016-11-13T01:16:51.671229682Z npm ERR!     NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-13T01:16:51.674070397Z npm ERR! You can get their info via:
[pokedatadevelop-1]2016-11-13T01:16:51.675422797Z npm ERR!     npm owner ls pokemongo-api
[pokedatadevelop-1]2016-11-13T01:16:51.677335732Z npm ERR! There is likely additional logging output above.
[pokedatadevelop-1]2016-11-13T01:16:51.690191312Z 
[pokedatadevelop-1]2016-11-13T01:16:51.694651110Z npm ERR! Please include the following file with any support request:
[pokedatadevelop-1]2016-11-13T01:16:51.704430089Z npm ERR!     /usr/src/app/npm-debug.log

@sacdallago
Copy link
Member

@bensLine @Aurel-Roci

@Aurel-Roci
Copy link

@sacdallago did you call unzipFiles() first? Like here https://www.npmjs.com/package/predict-pokemon. If it still works after that, I can do something with callbacks, but I would like your feedback on it.

@swathi-ssunder
Copy link
Collaborator

@sacdallago , @Aurel-Roci - Even after unzipFiles(), the status is same as in #183 (comment)

@bensLine
Copy link

@Aurel-Roci I added a post install script to unzip the files. can you please release the package again to test it. PokemonGoers/PredictPokemon-2#75

@Aurel-Roci
Copy link

@swathi-ssunder the new version is up

@jonas-he
Copy link
Collaborator

see #198

@swathi-ssunder
Copy link
Collaborator

@Aurel-Roci , @bensLine

Error: Cannot find module 'fstream'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:286:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/scripts/postinstall.js:3:15)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
npm info predict-pokemon@0.2.3 Failed to exec postinstall script
npm ERR! Linux 3.19.0-66-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v4.0.0
npm ERR! npm  v2.14.2
npm ERR! code ELIFECYCLE

npm ERR! predict-pokemon@0.2.3 postinstall: `node scripts/postinstall.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the predict-pokemon@0.2.3 postinstall script 'node scripts/postinstall.js'.
npm ERR! This is most likely a problem with the predict-pokemon package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node scripts/postinstall.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls predict-pokemon
npm ERR! There is likely additional logging output above.
npm info preuninstall predict-pokemon@0.2.3
npm info uninstall predict-pokemon@0.2.3
npm info postuninstall predict-pokemon@0.2.3
npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
The command '/bin/sh -c npm install' returned a non-zero code: 1
The command "docker build -t pokemongoers/pokedata ." failed and exited with 1 during .

@Aurel-Roci
Copy link

@swathi-ssunder package dependency was missing, added now. Sorry for all this

@swathi-ssunder
Copy link
Collaborator

@sacdallago - The docker build was successful this time, the DNS still does not work though!

@jonas-he
Copy link
Collaborator

jonas-he commented Nov 16, 2016

so i looked into this and it turns out that after executing npm start this is the output:

npm info it worked if it ends with ok
npm info using npm@2.14.2
npm info using node@v4.0.0
npm info prestart pokemongo-api@0.0.1
npm info start pokemongo-api@0.0.1

> pokemongo-api@0.0.1 start /usr/src/app
> NODE_ENV=production node app.js

2016-11-16T16:39:27.282Z started prediction script, init DC
4837ms to construct shortcut table
Average timezones per 1° lat x 1° lng: 4.414128352490422
2016-11-16T16:39:35.486Z created training arff file
2016-11-16T16:39:35.487Z requesting http://predictemall.online:65014/api/pokemon/sighting/ts/2016-09-14T08:00:00.000Z/range/1d?token=I0TPIIpCLH8lR8iDrCMV
npm info poststart pokemongo-api@0.0.1
npm info ok 

after this the process exits. Dont know exactly whats causing this. might be that it exits because the request to the predictemall site fails (i get a bad gateway when trying it in browser).

@jonas-he
Copy link
Collaborator

sorry it turns out that my docker container exits because it couldnt connect to the database. It was still configured with mlab database, but i guess it was taken down, so authentication error -> process exits. To know what the real problem is well have to wait on @sacdallago

@sacdallago
Copy link
Member

It seems to be up and running to me :) Can you confirm that the predictions are visible?

@jonas-he
Copy link
Collaborator

jonas-he commented Nov 18, 2016

@sacdallago are you talking about the dockerhub thing or predictemall.online ? The first is down for me the latter is up. However I see no possibility to request predictions

@sacdallago
Copy link
Member

@jonas-he 😮 how didn't I reply to that. Sorry :(

Yes, yes. Everything is up where it should be, and down where it should be down :) Why don't you see a possibility to request predictions?

@jonas-he
Copy link
Collaborator

@sacdallago i can turn prediction toggle on/off and change the time range but nothing happens on the map as well as in terms of network requests (i see requests for sightings but not for predictions).

Another thing is that there is still some dummy running, which is generating tons of pokemobs in munich.

@samitsv
Copy link
Collaborator

samitsv commented Dec 18, 2016

updated the api for prediction, #199, but one of the build fails https://travis-ci.org/PokemonGoers/PokeData/jobs/184915843

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants