Model location not detected in OSX #36

bmmalone · 2017-02-01T16:42:09Z

Originally from @tmargus in Issue #34:

create_orf_profiles.py can't pick up location of models. Where in configuration files these can be specified or program should figure out itself?
Anyway, by hard coding models into
create_orf_profiles.py
periodic_models_str = "/Users/tmargus/git/rp-bp/rpbp_models/periodic/start-high-low-low.stan"
non_periodic_models_str = "/Users/tmargus/git/rp-bp/rpbp_models/nonperiodic/no-periodicity.stan"

This is related to appdirs. It is not immediately clear why they are not found correctly.

tmargus · 2017-02-01T18:54:45Z

pipe install -e .
created appropriate folders for models under
~/Library/Application\ Support/rpbp/rpbp_models/
but all subfolders like periodic/, nonperiodic/, trnslated/, and untranslated/ were empty.

I copied models to their destination

@:rpbp_models$ cp nonperiodic/*  ~/Library/Application\ Support/rpbp/rpbp_models/nonperiodic/
@:rpbp_models$ cp periodic/*     ~/Library/Application\ Support/rpbp/rpbp_models/periodic/
@:rpbp_models$ cp translated/*   ~/Library/Application\ Support/rpbp/rpbp_models/translated/
@:rpbp_models$ cp untranslated/* ~/Library/Application\ Support/rpbp/rpbp_models/untranslated/

but it didn't help. It still don't find models.
estimate-metagene-profile-bayes-factors: error: argument --periodic-models: expected at least one argument
Maybe it remembers what it have copied.
Tonu

bmmalone · 2017-02-01T19:30:08Z

Hi Tonu,

Internally, the .stan files (which are just plain text files) are (supposed to be) compiled and saved as pickle (.pkl) files in the respective folders. The pipeline scans those folders for .pkl during execution, so it wouldn't detect the original stan model files.

It seems as though the pickling and saving isn't working as expected.

Could you please try completely removing the ~/Library/Application\ Support/rpbp/ folder (and everything underneath it), and then re-installing with pip3 install --verbose --log log.rpbp-install.txt . ?

In particular, towards the end of that, there should be some lines like:

INFO : pickle-stan rpbp_models/nonperiodic/no-periodicity.stan /home/bmalone/.local/share/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl

The second argument to pickle-stan is exactly where the pickled model should be saved. Also, in case there are any error messages during compilation, they will show up there.

Could you please let me know (upload the file, etc.) if there is anything interesting going on there? Thanks.

Have a good day,
Brandon

tmargus · 2017-02-01T19:39:15Z

Hi Brnadon,
installation script cant cover space between Application and Supprot splits it into two
should write like ../Application\ Supprot/.. but miss the backslash.

INFO     : pickle-stan rpbp_models/nonperiodic/no-periodicity.stan /Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl
    usage: pickle-stan [-h] stan out
    pickle-stan: error: unrecognized arguments: Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl

Tonu

bmmalone · 2017-02-01T19:42:00Z

Ah, okay. could you please try manually this command (from the main rpbp folder):

pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl"

If that works, then I can quickly put together a fix.

Thanks,
Brandon

tmargus · 2017-02-01T19:48:14Z

Hi, works fine! ``` pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" DIAGNOSTIC(S) FROM PARSER: Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. Warning (non-fatal): assignment operator <- deprecated in the Stan language; use = instead. INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_1c0d8baec383282006d5731a3770192a NOW. ``` Tonu

…

On 1. veebr 2017, at 20:42, bmmalone ***@***.***> wrote: Ah, okay. could you please try manually this command (from the main rpbp folder): pickle-stan "rpbp_models/nonperiodic/no-periodicity.stan" "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" If that works, then I can quickly put together a fix. Thanks, Brandon — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXfyp59WFyqCKvwrjcYiTYiwg16uyEdks5rYOAIgaJpZM4L0F9r>.

This addresses Issue #36.

bmmalone · 2017-02-01T20:01:23Z

Great! I have pushed a new version which includes a small fix to handle the space. It is working as expected locally. Could you please give it a shot and let me know how it goes? Thanks for your persistence.

tmargus · 2017-02-01T20:30:08Z

it still not clean.
I compiled these *.stan models to *.pkl manually and now program finds them.
However, estimate-metagene-profile-bayes-factors is running to errors.

FileNotFoundError: [Errno 2] No such file or directory: '/Users/tmargus/Library/Application'
Is it caused how pkl's are compiled?

Error message:

INFO     root     2017-02-01 20:59:25,406 : estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models /Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl --nonperiodic-models /Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET
INFO     root     2017-02-01 20:59:25,406 : calling
INFO     riboutils.estimate_metagene_profile_bayes_factors 2017-02-01 20:59:26,079 : Estimating Bayes factors for lengths: 16,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34
  0%|                                                                                                                                                                                                                                                                                                                                            | 0/18 [00:00<?, ?it/s]multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 344, in __call__
    return self.func(*args, **kwargs)
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/joblib/parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 72, in estimate_profile_bayes_factors
    periodic_models = [pickle.load(open(pm, 'rb')) for pm in args.periodic_models]
  File "/Users/tmargus/anaconda/lib/python3.5/site-packages/riboutils/estimate_metagene_profile_bayes_factors.py", line 72, in <listcomp>
    periodic_models = [pickle.load(open(pm, 'rb')) for pm in args.periodic_models]
FileNotFoundError: [Errno 2] No such file or directory: '/Users/tmargus/Library/Application'

During handling of the above exception, another exception occurred:
...................

bmmalone · 2017-02-01T20:40:40Z

Hi Tonu,

Sorry, this is essentially the same problem, just in another place. The call to estimate-metagene-profile-bayes-factors also needs the spaces escaped or quoted.

If you'd like to run it by hand to get a feel for the output, you can use:

estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl" --nonperiodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET

The same problem will crop up later during a call to estimate-orf-bayes-factors, so the entire pipeline still won't work. I will (attempt to) track down all of the places this is occurring and try to have a patch in the next hour or so.

Have a good day,
Brandon

tmargus · 2017-02-01T20:49:22Z

aa... ok! Now this is command is working! it creates file c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz I will be back tomorrow Tonu

…

On 1. veebr 2017, at 21:40, bmmalone ***@***.***> wrote: estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl" --nonperiodic-models "/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl" --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --log-file log_1.txt --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET

bmmalone · 2017-02-02T00:07:49Z

Hi Tonu,

Okay, good :) the space is definitely the problem, then.

At least for the models, it turns out there is a single function in a pre-requisite project, riboseq-utils, which is always used to do the scanning for pickle files I mentioned earlier. I updated the function in this commit: dieterich-lab/riboseq-utils@4d32fc4 to always shell-escape the model filenames with quotes.

Locally, it appears to work; however, you will need updated versions of both riboseq-utils and misc. I'm not really sure of the easiest way to do that... maybe either uninstall them with pip3 uninstall riboutils misc, then re-run pip3 install . for rpbp, or directly clone the repos and install both of them with pip. Unfortunately, the integration with them is still fragile since we have not added things to pypi, yet.

Please take a look and let me know how it turns out. In particular, in the logs of the commands (like the one you posted above), the paths to the models should be quoted.

Have a good day,
Brandon

tmargus · 2017-02-02T12:04:59Z

Hi Brandon,

I uninstalled misc and riboseq-utils wit pip, download new git repos fro misc and riboseq-utils and installed them with pip3 install .
Then rerun install in rp-bp/ folder pip3 install -e .
It should do the thing.

I did not change *.pkl models in ~/Library/Application\ Support/rpbp/rpbp_models/.
running command
process-all-samples c-elegans-test.yaml --overwrite --num-cpus 4 --logging-level INFO --log-file log_2.txt
still ends with error but I don't see estimate-metagene-profile-bayes-factors in report.
It still complains about pkl files Some input files ["'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl'", "'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl'"] are missing.
but now the names are double qouted ["'pkl'"]. Could it be the reason of error?

Part of Error message.

INFO     root     2017-02-02 11:40:02,017 : extract-metagene-profiles /Users/tmargus/projects/rpbp/data/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /Users/tmargus/projects/rpbp/data/WBcel235.79.chrI/WBcel235.79.chrI.bed.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 4 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET  --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
INFO     root     2017-02-02 11:40:02,017 : calling
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,495 : [extract-metagene-profiles]: /Users/tmargus/anaconda/bin/extract-metagene-profiles /Users/tmargus/projects/rpbp/data/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /Users/tmargus/projects/rpbp/data/WBcel235.79.chrI/WBcel235.79.chrI.bed.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 4 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10841/10841 [00:00<00:00, 337473.09it/s]
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,627 : Reading annotations
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,655 : Constructing canonical translation initiation ORF data frames
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,683 : Constructing canonical translation termination ORF data frames
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,691 : Finding translation initiation site matches
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,736 : Finding translation termination site matches
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,763 : Extracting metagene profiles
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,764 : Profiles will be created for lengths: 12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42,46,48
INFO     riboutils.extract_metagene_profiles 2017-02-02 11:40:02,824 : Writing metagene profiles to disk
INFO     root     2017-02-02 11:40:02,918 : gunzip -t /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz
INFO     root     2017-02-02 11:40:02,918 : calling
WARNING  misc.utils 2017-02-02 11:40:02,925 : [utils.call_if_not_exists]: This function is deprecated. Please use the version in misc.shell_utils instead.
WARNING  misc.utils 2017-02-02 11:40:02,925 : Some input files ["'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl'", "'/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl'"] are missing. Skipping call: 
estimate-metagene-profile-bayes-factors /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /Users/tmargus/projects/rpbp/data/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 4 --periodic-models '/Users/tmargus/Library/Application Support/rpbp/rpbp_models/periodic/start-high-low-low.pkl' --nonperiodic-models '/Users/tmargus/Library/Application Support/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl' --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --stdout-logging-level NOTSET --stderr-logging-level NOTSET --log-file log_2.txt --file-logging-level NOTSET

Cheers,
Tonu

bmmalone · 2017-02-02T12:20:32Z

Hi Tonu,

Sorry, I forgot to push the updates to "misc" last night. Could you please re-pull and install it. Sorry about that.

Also, you are right that the reason it can't find the file is because it is looking for a filename which begins with a literal quote; the update to "misc" fixes that behavior.

Have a good day,
Brandon

tmargus · 2017-02-02T12:50:35Z

It worked now! Great!
Thanks a lot.
Tonu

bmmalone · 2017-02-02T13:57:28Z

Hi Tonu,

Great! Thanks so much for iterating on this. I'm going to close this issue now, but please don't hesitate to re-open it if necessary, or file any others if anything else comes up. We consider it very important that our software is both usable and based on solid theoretical principles.

That said, the fix for this issue will not help for other paths with spaces. However, all of the other paths are easily controlled, so hopefully that is not much of an issue. Still, I have preemptively opened a new issue (#41) that at least documents this behavior.

Thanks, again.

Have a good day,
Brandon

This is again related to Issue #36. The change log includes more details.

bmmalone self-assigned this Feb 1, 2017

bmmalone mentioned this issue Feb 1, 2017

missing misc.bio_utils.bam_utils #34

Closed

bmmalone added a commit that referenced this issue Feb 1, 2017

FIX model compilation failed with spaces in path

235a1a1

This addresses Issue #36.

bmmalone mentioned this issue Feb 2, 2017

Spaces in path names #41

Closed

bmmalone closed this as completed Feb 2, 2017

bmmalone mentioned this issue Feb 14, 2017

STAR bam symlink problems #51

Closed

tmargus mentioned this issue Feb 21, 2017

space in path #55

Closed

bmmalone added a commit that referenced this issue Feb 21, 2017

FIX compiling Stan models with spaces in paths

64cc2f1

This is again related to Issue #36. The change log includes more details.

bmmalone mentioned this issue Jun 5, 2017

Models not detected in linux #72

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model location not detected in OSX #36

Model location not detected in OSX #36

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017 via email

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017 via email

bmmalone commented Feb 2, 2017

tmargus commented Feb 2, 2017 •

edited

Loading

bmmalone commented Feb 2, 2017

tmargus commented Feb 2, 2017

bmmalone commented Feb 2, 2017

Model location not detected in OSX #36

Model location not detected in OSX #36

Comments

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017 via email

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017

bmmalone commented Feb 1, 2017

tmargus commented Feb 1, 2017 via email

bmmalone commented Feb 2, 2017

tmargus commented Feb 2, 2017 • edited Loading

bmmalone commented Feb 2, 2017

tmargus commented Feb 2, 2017

bmmalone commented Feb 2, 2017

tmargus commented Feb 2, 2017 •

edited

Loading