Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Illegal instruction #19

Open
NicolasC0 opened this issue Nov 28, 2023 · 24 comments
Open

Illegal instruction #19

NicolasC0 opened this issue Nov 28, 2023 · 24 comments
Assignees

Comments

@NicolasC0
Copy link

NicolasC0 commented Nov 28, 2023

Thank you so much for developing mixer!

I believe I've installed and set up the singularity container for mixer correctly, however, when I try to run the job I get "Illegal instruction" error such as this:

/var/spool/slurmd/job67969116/slurm_script: line 31: 14435 Illegal instruction     $PYTHON /tools/mixer/precimed/mixer.py fit1 $MIXER_COMMON_ARGS $EXTRACT --trait1-file $COMORMENT/mixer/reference/sumstats/ADHD2022_munge.sumstats.gz --out $RESULTS/fit1/ADHD/ADHD.fit1.$REP

There is no problem with the singulrity execution:

(base) [10721897@aguia4:/temporario/10721897]$ singularity exec --home=$PWD:/home $SIF/mixer.sif bash
Singularity>

neither with using python

(base) [10721897@aguia4:/temporario/10721897]$ singularity exec --home=$PWD:/home $SIF/mixer.sif bash
Singularity> python
Python 3.8.18 | packaged by conda-forge | (default, Oct 10 2023, 15:44:36)
[GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The error appears when trying to pass any arguments to mixer.py:

(base) [10721897@aguia4:/temporario/10721897]$ singularity exec --home=/temporario/10721897:/home /temporario/10721897/mixer/singularity/mixer.sif python /tools/mixer/precimed/mixer.py fit1
Illegal instruction

I believe the server achitecture is compatible (Architecture: x86_64, CPU op-mode(s): 32-bit, 64-bit, Model name: Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz), but I'm very new to this level of programming and have never used or installed a container/singularity before.

Has anyone had this problem?

@espenhgn
Copy link
Contributor

espenhgn commented Nov 29, 2023

Hi @NicolasC0,
Thanks for raising this issue. Does the most basic Python statement work? E.g.:

singularity exec --home=$PWD:/home $SIF/mixer.sif python -c "print('hello world')"

What is the singularity version you have, i.e., what is the output of:

singularity --version

I don't know, but it may help to add the --cleanenv flag, such as:

export PYTHON="singularity exec --home=$PWD:/home --cleanenv $SIF/mixer.sif python"

@NicolasC0
Copy link
Author

NicolasC0 commented Nov 29, 2023

Thank you for the quick response!

python seems to be working

singularity exec --home=/temporario/10721897:/home /temporario/10721897/mixer/singularity/mixer.sif python -c "print('hello world')"
hello world

and the singularity version is 3.8.7-1.el7. Could this be the issue?

I tried the --cleanenv flag but still got the same error

@espenhgn
Copy link
Contributor

espenhgn commented Nov 30, 2023

Ok. I don't think the singularity version should be a problem. The container is built using version 3.7.4, and I use it w. version 3.7.3. Version 2.x would cause problems due to changes in the sif format. I have only tested from Linux hosts, however.

Does invoking the mixer --help function work?

singularity exec --no-home --cleanenv mixer.sif python /tools/mixer/precimed/mixer.py --help

Other basic imports/checks (via bash) you may test:

singularity exec --no-nome --cleanenv mixer.sif bash # start bash, then issue:
plink --version
simu_linux --help
python -c "import numpy"
python -c "import sys; sys.path.append('/tools/mixer/precimed/mixer'); import libbgmb"
python -c "import ctypes; ctypes.CDLL('/tools/mixer/src/build/lib/libbgmb.so')

@NicolasC0
Copy link
Author

Thank you for the suggestions.

The --help functions does work:
image

When issuing --plink version and simu_linux --elp both work as well:
image

But there's an error when issuing
python -c "import sys; sys.path.append('/tools/mixer/precimed/mixer'); import libbgmb"
python -c "import ctypes; ctypes.CDLL('/tools/mixer/src/build/lib/libbgmb.so'):

image

@espenhgn
Copy link
Contributor

espenhgn commented Dec 1, 2023

Thanks again for testing.
I'm sorry though, the last two suggestions contained a typo, and should've been:

python -c "import sys; sys.path.append('/tools/mixer/precimed/mixer'); import libbgmg"
python -c "import ctypes; ctypes.CDLL('/tools/mixer/src/build/lib/libbgmg.so')"

You can roll back to the older version of the mixer.sif file, via for instance the "raw download" from commit 46cd4ae6c188bff4a9115c7ad7ed5c53c10c2e62

The file was updated via #2, which may inadvertently have broken things so it would be good to test.

@NicolasC0
Copy link
Author

Thank you again for the suggestions, I apologize for the delay in my response!

I also get Illegal instruction error when running those commands
image

Should I also try to test the older version of the mixer.sif file?

@espenhgn
Copy link
Contributor

espenhgn commented Dec 7, 2023

Hi. It seems like this last command at least exposed which exact file may have caused the issue. I don't have an Intel-based machine at hand, however, and can't reproduce it, but will check with someone else in our group.

If you have time, you may of course test the older mixer.sif file -- functionally it should be the same (and may work on your end).

Also, please post your operating system / version.

@ofrei
Copy link
Contributor

ofrei commented Dec 7, 2023

@NicolasC0 Sorry about these troubles!

Model name: Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz)

This is a fairly old cpu, however I'm still surprised about the illegal instruction error. According to this page the E7- 2870 CPU supports Intel® SSE4.1, Intel® SSE4.2 (part of westmere CPU family). I think this may not work due to -march=native setting here.

On the machine were we built mixer.sif container we have avx2 instructions set - so I suspect this makes mixer.sif incompatible with E7- 2870 .

ofrei@comorment:~$ lscpu | grep Flags
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc 
rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid 
sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c 
rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsbase bmi1 
avx2 smep bmi2 erms invpcid xsaveopt arat

Base on https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html I think to try fixing this we need to re-generate mixer.sif containier patching CMakeLists.txt with -march=westmere option.
@espenhgn could you try this out? If build fails then I might need to change some codes to avoid using CPU features beyon sse4.2 instructionset.

@espenhgn espenhgn self-assigned this Dec 12, 2023
@espenhgn
Copy link
Contributor

Hi @NicolasC0. I've pushed a rebuilt container using the-march=westmere option here (as per @ofrei suggestion): https://github.com/comorment/mixer/blob/espenhgn/issue19/singularity/mixer.sif

Can you see if it works for you?

@NicolasC0
Copy link
Author

Thank you so much for the changes, I believe the "illegal instruction" problem is fixed! Though now it seems like it can't quite identify the sumstats trait file ("ValueError: Input file --trait1_file does not exist: /temporario/10721897/mixer/reference/sumstats/ADHD2022_munge.sumstats.gz") even though the file does exist and is in the correct path. I've tried moving it to different directories, removing and adding it back, tried with different files to no avail. I'm unsure if it's something I'm overlooking when running the script or preparing the files?

@espenhgn
Copy link
Contributor

espenhgn commented Jan 2, 2024

Hi again, @NicolasC0.
Sounds promising! The missing file issue is likely due to how directories are mounted to the virtual machine.
Using the argument --home=/temporario/10721897:/home, and referencing files in the directory /temporario/10721897 on the host machine, then you must link files relative to that as the VM sees only /home/, so you'd end up with
--trait1-file /home/mixer/reference/sumstats/ADHD2022_munge.sumstats.gz as argument as so forth.

You may also use the SINGULARITY_BIND environment variable to mount additional directories on the host than the reference data directory, see https://docs.sylabs.io/guides/3.0/user-guide/bind_paths_and_mounts.html

@NicolasC0
Copy link
Author

It seems to be working! thank you so much for your great help! I have only one last question, the log files show mixer version 1.2.0 and I was wondering if that was correct? The main file is at around 1.2GB and to pull the repository I used git 2.9.5 and git-lfs 3.4.1.

@espenhgn
Copy link
Contributor

Excellent. The version 1.2.0 reported sounds correct as this is hardcoded into the Mixer master branch. Better to think of the current master as a v1.3 development version; while the v1.3.0 tag used in this repo is for the codes etc. found here.

@NicolasC0
Copy link
Author

Oh I see, so it is a development version. Thank you immensely for your great help!

@liyehaha
Copy link

Hi, I have face the same issue, and try the sif file in tag espenhgn/issue19. I have met the new issue here, please help and check this. Thanks very much.

System Info:
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
kernel: 4.15.0-20-generic
singularity: singularity-ce-3.11.3

ISSUE:

INFO:root:init(lib_name=/tools/mixer/src/build/lib/libbgmg.so, context_id=0)
INFO:root:init_log(SCZ.fit.rep.log)
INFO:root:log_message(***********************************************************************

  • mixer.py: Univariate and Bivariate Causal Mixture for GWAS
  • Version 1.2.0
  • (c) 2016-2020 Oleksandr Frei, Alexey A. Shadrin, Dominic Holland
  • Norwegian Centre for Mental Disorders Research / University of Oslo
  • Center for Multimodal Imaging and Genetics / UCSD
  • GNU General Public License v3

Call:
./mixer.py fit1
--out SCZ.fit.rep
--bim-file /home/reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim
--ld-file /home/reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld
--trait1-file SCZ.sumstats.gz
--extract /home/reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep.snps
--threads 16
)
INFO:root:init(lib_name=libbgmg.so, context_id=0)
Traceback (most recent call last):
File "/tools/mixer/precimed/mixer.py", line 23, in
args.func(args)
File "/tools/mixer/precimed/mixer/cli.py", line 647, in execute_fit1_or_test1_parser
results = init_results_struct(libbgmg, args)
File "/tools/mixer/precimed/mixer/cli.py", line 639, in init_results_struct
results['options']['sum_weights'] = float(np.sum(libbgmg.weights))
File "/tools/mixer/precimed/mixer/libbgmg.py", line 168, in weights
return self._get_vec_impl(self.cdll.bgmg_retrieve_weights, np.float32, self.num_tag, trait=None)
File "/tools/mixer/precimed/mixer/libbgmg.py", line 410, in _get_vec_impl
self._check_error(func(*args))
File "/tools/mixer/precimed/mixer/libbgmg.py", line 418, in _check_error
raise RuntimeError(self.get_last_error())
RuntimeError: runtime_error: arg <= 0

@espenhgn
Copy link
Contributor

Hi @liyehaha. What is the CPU of the host system you run on?

@Licun1
Copy link

Licun1 commented Apr 26, 2024

I'm his friend. Thanks for your quick response. CPU:e5-2650, a fairly old cpu. We have runned this program with a dektop: CPU, Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz. Running slowly, but it runs normally. Additionally, I have a question, when I runnig the program as below:

for i in $(seq 1 $file_count); do
    unset REP
    unset EXTRACT
    REP="rep${i}"
    EXTRACT="--extract ./reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.$REP.snps"
    echo $REP
    echo $EXTRACT
    $PYTHON /tools/mixer/precimed/mixer.py fit1 $MIXER_COMMON_ARGS $EXTRACT --trait1-file ana2.sumstats.gz --out ana2.fit.$REP
    $PYTHON /tools/mixer/precimed/mixer.py fit1 $MIXER_COMMON_ARGS $EXTRACT --trait1-file ra2.sumstats.gz --out ra2.fit.$REP
    $PYTHON /tools/mixer/precimed/mixer.py fit2 $MIXER_COMMON_ARGS $EXTRACT --trait1-file ana2.sumstats.gz --trait2-file ra2.sumstats.gz --trait1-params ana2.fit.$REP.json --trait2-params ra2.fit.$REP.json --out ana2_vs_ra2.fit.$REP

    $PYTHON /tools/mixer/precimed/mixer.py test1 $MIXER_COMMON_ARGS --trait1-file ana2.sumstats.gz --load-params ana2.fit.$REP.json --out ana2.test.$REP
    $PYTHON /tools/mixer/precimed/mixer.py test1 $MIXER_COMMON_ARGS --trait1-file ra2.sumstats.gz --load-params ra2.fit.$REP.json --out ra2.test.$REP
    $PYTHON /tools/mixer/precimed/mixer.py test2 $MIXER_COMMON_ARGS --trait1-file ana2.sumstats.gz --trait2-file ra2.sumstats.gz --load-params ana2_vs_ra2.fit.$REP.json --out ana2_vs_ra2.test.$REP
done
rep1
--extract ./reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep1.snps
^[OPINFO:root:__init__(lib_name=/tools/mixer/src/build/lib/libbgmg.so, context_id=0)
INFO:root:init_log(ana2.fit.rep1.log)
INFO:root:log_message(***********************************************************************
* mixer.py: Univariate and Bivariate Causal Mixture for GWAS
* Version 1.2.0
* (c) 2016-2020 Oleksandr Frei, Alexey A. Shadrin, Dominic Holland
* Norwegian Centre for Mental Disorders Research / University of Oslo
* Center for Multimodal Imaging and Genetics / UCSD
* GNU General Public License v3
***********************************************************************
Call: 
./mixer.py fit1 \
	--out ana2.fit.rep1 \
	--bim-file ./reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.@.bim \
	--ld-file ./reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.@.run4.ld \
	--trait1-file ana2.sumstats.gz \
	--extract ./reference/ldsc/1000G_EUR_Phase3_plink/1000G.EUR.QC.prune_maf0p05_rand2M_r2p8.rep1.snps \
	--threads 4 
)
INFO:root:__init__(lib_name=libbgmg.so, context_id=0)
INFO:root:log_message(--fit-sequence: ['diffevo-fast', 'neldermead']...)
INFO:root:log_message(fit_type==diffevo-fast...)
INFO:root:log_message(--diffevo-fast-repeat=0: UnivariateParams(_pi: 0.002870810951456501, _sig2_beta: 9.911545604177664e-06, _sig2_zero: 1.0438623722670584))
INFO:root:log_message(--diffevo-fast-repeat=1: UnivariateParams(_pi: 0.0003735118666917421, _sig2_beta: 5.83360153814499e-05, _sig2_zero: 1.0529398817596527))
INFO:root:log_message(--diffevo-fast-repeat=2: UnivariateParams(_pi: 0.0003145592187998604, _sig2_beta: 5.4045287428896244e-05, _sig2_zero: 1.04240709265637))
INFO:root:log_message(--diffevo-fast-repeat=3: UnivariateParams(_pi: 0.001968172007481384, _sig2_beta: 1.627414022059366e-05, _sig2_zero: 1.0423671161675183))
INFO:root:log_message(--diffevo-fast-repeat=4: UnivariateParams(_pi: 0.0007632882157393912, _sig2_beta: 2.946135690552829e-05, _sig2_zero: 1.0570225315422859))
INFO:root:log_message(--diffevo-fast-repeat=5: UnivariateParams(_pi: 0.00026478553099695866, _sig2_beta: 6.0679640550734944e-05, _sig2_zero: 1.0537550534725206))
INFO:root:log_message(--diffevo-fast-repeat=6: UnivariateParams(_pi: 0.0010119927461300507, _sig2_beta: 1.4520046748033912e-05, _sig2_zero: 1.059092153431677))
INFO:root:log_message(--diffevo-fast-repeat=7: UnivariateParams(_pi: 0.0006071820514972005, _sig2_beta: 4.575763501429385e-05, _sig2_zero: 1.0343178701085438))
INFO:root:log_message(--diffevo-fast-repeat=8: UnivariateParams(_pi: 0.00035539704035027794, _sig2_beta: 6.44351117377325e-05, _sig2_zero: 1.0446494966557394))
INFO:root:log_message(--diffevo-fast-repeat=9: UnivariateParams(_pi: 0.0009174404580525105, _sig2_beta: 2.8702664297173653e-05, _sig2_zero: 1.0526239638912476))
INFO:root:log_message(--diffevo-fast-repeat=10: UnivariateParams(_pi: 0.0003738171296923532, _sig2_beta: 6.971415793180406e-05, _sig2_zero: 1.0449850808985977))
INFO:root:log_message(--diffevo-fast-repeat=11: UnivariateParams(_pi: 0.0006210027209726347, _sig2_beta: 4.553614425140047e-05, _sig2_zero: 1.0343842555045686))
INFO:root:log_message(--diffevo-fast-repeat=12: UnivariateParams(_pi: 0.0005136176008835462, _sig2_beta: 2.8869999599367165e-05, _sig2_zero: 1.0500248093595417))
INFO:root:log_message(--diffevo-fast-repeat=13: UnivariateParams(_pi: 0.00039280733981807793, _sig2_beta: 4.679338973576257e-05, _sig2_zero: 1.0584739780206323))
INFO:root:log_message(--diffevo-fast-repeat=14: UnivariateParams(_pi: 0.0016723903758663736, _sig2_beta: 1.0172568695052486e-05, _sig2_zero: 1.0496721234826145))
INFO:root:log_message(--diffevo-fast-repeat=15: UnivariateParams(_pi: 0.0005767661062806921, _sig2_beta: 4.8104924762630844e-05, _sig2_zero: 1.0445081101996496))
INFO:root:log_message(--diffevo-fast-repeat=16: UnivariateParams(_pi: 0.000350923171513242, _sig2_beta: 5.6396621629083244e-05, _sig2_zero: 1.0321777390989346))
INFO:root:log_message(--diffevo-fast-repeat=17: UnivariateParams(_pi: 0.0031006673080079907, _sig2_beta: 8.545288824132161e-06, _sig2_zero: 1.0236133073580136))
INFO:root:log_message(--diffevo-fast-repeat=18: UnivariateParams(_pi: 0.00017467846986412346, _sig2_beta: 0.00014689591841989555, _sig2_zero: 1.0488457098292294))
INFO:root:log_message(--diffevo-fast-repeat=19: UnivariateParams(_pi: 0.0005006924250656558, _sig2_beta: 4.068694479749773e-05, _sig2_zero: 1.0380542591442647))
INFO:root:log_message(fit_type==diffevo-fast done (UnivariateParams(_pi: 0.0005767661062806921, _sig2_beta: 4.8104924762630844e-05, _sig2_zero: 1.0445081101996496),    message: Optimization terminated successfully.
   success: True
       fun: 156850.43324762565
         x: [ 4.355e-02 -9.942e+00 -7.457e+00]
       nit: 7
      nfev: 360
    cost_n: 107788.375
   cost_df: 3
      cost: 156850.43324762565
       BIC: 313735.63027012313
       AIC: 313706.8664952513
 cost_fast: 156850.43324762603
    params:        pi: 0.0005767661062806921
            sig2_beta: 4.8104924762630844e-05
            sig2_zero: 1.0445081101996496))
INFO:root:log_message(fit_type==neldermead...)
INFO:root:log_message(fit_type==neldermead done (UnivariateParams(_pi: 0.00045151000283363363, _sig2_beta: 5.7099401875814416e-05, _sig2_zero: 1.0454670204367547),        message: Optimization terminated successfully.
       success: True
        status: 0
           fun: 156846.24928933248
             x: [ 4.446e-02 -9.771e+00 -7.702e+00]
           nit: 100
          nfev: 240
 final_simplex: (array([[ 4.446e-02, -9.771e+00, -7.702e+00],
                       [ 4.446e-02, -9.771e+00, -7.702e+00],
                       [ 4.446e-02, -9.771e+00, -7.702e+00],
                       [ 4.446e-02, -9.771e+00, -7.702e+00]]), array([ 1.568e+05,  1.568e+05,  1.568e+05,  1.568e+05]))
        cost_n: 107788.375
       cost_df: 3
          cost: 156846.24928933248
           BIC: 313727.2623535368
           AIC: 313698.49857866496
     cost_fast: 156850.8135569445
        params:        pi: 0.00045151000283363363
                sig2_beta: 5.7099401875814416e-05
                sig2_zero: 1.0454670204367547))
INFO:root:log_message(Calculate AIC/BIC w.r.t. infinitesimal model (fast cost function)...)
INFO:root:log_message(fit_type==infinitesimal...)
INFO:root:log_message(fit_type==infinitesimal done (UnivariateParams(_pi: 1.0, _sig2_beta: 2.7504553312292243e-08, _sig2_zero: 1.0432549965678353),        message: Optimization terminated successfully.
       success: True
        status: 0
           fun: 156854.0897630793
             x: [-1.741e+01  4.235e-02]
           nit: 29
          nfev: 64
 final_simplex: (array([[-1.741e+01,  4.235e-02],
                       [-1.741e+01,  4.235e-02],
                       [-1.741e+01,  4.234e-02]]), array([ 1.569e+05,  1.569e+05,  1.569e+05]))
        cost_n: 107788.375
       cost_df: 2
          cost: 156854.0897630793
           BIC: 313731.35537607316
           AIC: 313712.1795261586
     cost_fast: 156854.08976307942
        params:        pi: 1.0
                sig2_beta: 2.7504553312292243e-08
                sig2_zero: 1.0432549965678353))

Why does this repetition range from 0 to 19, a total of 20 repetitions, while there are 22 reference genome files (22 chromosomes)?

@espenhgn
Copy link
Contributor

espenhgn commented May 3, 2024

Thanks for the update. I think the e5-2650 CPU is now so outdated that we will not support it (officially); supporting only AVX. The i5-7500 CPU supports Intel® SSE4.1, Intel® SSE4.2, Intel® AVX2 and is nearly current.

Similar to what I did earlier, it may be possible to use the -sandybridge compiler flag instead of -westmere.
But I'm a bit surprised the build I linked above didn't work as westmere provides a subset of sandy bridge extensions, meaning, it ought to work on that CPU.

For your latter question, the repetitions are over random seeds in each run, not chromosomes. It's not super clear from the MiXeR documentation, but in the code, the random seed is set as seed=(args.seed + repeat) where repeat corresponds to one of the 20 repetitions.

@Licun1
Copy link

Licun1 commented May 3, 2024

Thanks for your answer. I have completed this run, but I encountered another issue.

Singularity> python /tools/mixer/precimed/mixer_figures.py one --json ana2.fit.json ra2.fit.json --out ana2_and_ra2.fit --trait1 ana2 ra2 --statistic mean std --ext svg
generate ana2_and_ra2.fit.csv from 2 json files...
Done.
Skip generating power plots, data not available. Did you include --power-curve in your "python mixer.py fit" command?
--json argument is a wild-card (contains *), skip figures generation
Singularity> python /tools/mixer/precimed/mixer_figures.py one --json ana2.test.json ra2.test.json --out ana2_and_ra2.test --trait1 ana2 ra2 --statistic mean std --ext svg
generate ana2_and_ra2.test.csv from 2 json files...
Done.
HAS POWER? True
HAS POWER? True
/tools/mixer/precimed/mixer/figures.py:338: UserWarning: FixedFormatter should only be used together with FixedLocator
  plt.gca().set_xticklabels(labels=['10K', '100K', '1M', '10M', '100M'])
/tools/mixer/precimed/mixer/figures.py:340: UserWarning: FixedFormatter should only be used together with FixedLocator
  plt.axes().set_yticklabels(labels=['0', '20', '40', '60', '80', '100'])

Generated ana2_and_ra2.test.power.svg
--json argument is a wild-card (contains *), skip figures generation
Singularity> python /tools/mixer/precimed/mixer_figures.py two --json-fit ana2_vs_ra2.fit.json --json-test ana2_vs_ra2.test.json --out ana2_vs_ra2 --trait1 ana2 --trait2 ra2 --statistic mean std --ext svg
generate ana2_vs_ra2.csv from 2 json files...
Done.
Generated ana2_vs_ra2.svg

It seems there's an issue with power analysis. Can you help me address this problem?

@espenhgn
Copy link
Contributor

espenhgn commented May 6, 2024

This seems normal, so what is the exact issue?
The output also suggests: Skip generating power plots, data not available. Did you include --power-curve in your "python mixer.py fit" command?

Please also have a look at this section

@Licun1
Copy link

Licun1 commented May 8, 2024

Sorry for late response.

python /tools/mixer/precimed/mixer_figures.py one --json ana2.fit.json ra2.fit.json --out ana2_and_ra2.fit --trait1 ana2 ra2 --statistic mean std --ext svg
#Singularity> python /tools/mixer/precimed/mixer_figures.py one --json ana2.fit.json ra2.fit.json --out ana2_and_ra2.fit --trait1 ana2 ra2 --statistic mean std --ext svg
generate ana2_and_ra2.fit.csv from 2 json files...
Done.
Skip generating power plots, data not available. Did you include --power-curve in your "python mixer.py fit" command?
--json argument is a wild-card (contains *), skip figures generation#

python /tools/mixer/precimed/mixer_figures.py one --json ana2.test.json ra2.test.json --out ana2_and_ra2.test --trait1 ana2 ra2 --statistic mean std --ext svg   
#Singularity> python /tools/mixer/precimed/mixer_figures.py one --json ana2.test.json ra2.test.json --out ana2_and_ra2.test --trait1 ana2 ra2 --statistic mean std --ext svg
generate ana2_and_ra2.test.csv from 2 json files...
Done.
HAS POWER? True
HAS POWER? True
/tools/mixer/precimed/mixer/figures.py:338: UserWarning: FixedFormatter should only be used together with FixedLocator
  plt.gca().set_xticklabels(labels=['10K', '100K', '1M', '10M', '100M'])
/tools/mixer/precimed/mixer/figures.py:340: UserWarning: FixedFormatter should only be used together with FixedLocator
  plt.axes().set_yticklabels(labels=['0', '20', '40', '60', '80', '100'])

Generated ana2_and_ra2.test.power.svg
--json argument is a wild-card (contains *), skip figures generation#

when I run these two lines commands, I reveived a message "--json argument is a wild-card (contains *), skip figures generation", and then I hadn't obtained these two svg format figures. what's the meanning of "--json argument is a wild-card (contains *)"?

@espenhgn
Copy link
Contributor

Hi. These messages you are receiving are for information, and this discussion belongs in another issue, not with the original illegal instruction problem reported here.

The arguments to each script are explained by invoking the help option (output truncated):

$ $PYTHON /tools/mixer/precimed/mixer_figures.py one --help
Matplotlib is building the font cache; this may take a moment.
usage: mixer_figures.py one [-h] [--argsfile ARGSFILE] [--out OUT] [--ext {png,svg} [{png,svg} ...]] [--zmax ZMAX] [--statistic {point_estimate,mean,median,std,min,max} [{point_estimate,mean,median,std,min,max} ...]] [--json JSON [JSON ...]] [--trait1 TRAIT1 [TRAIT1 ...]]
                            [--power-thresh POWER_THRESH] [--power-figsize POWER_FIGSIZE [POWER_FIGSIZE ...]]

options:
  -h, --help            show this help message and exit
  ...
  --json JSON [JSON ...]
                        json file from a univariate analysis. This argument does support wildcards (*) or a list with multiple space-separated arguments to process more than one .json file. This allows to generate a combined .csv table across many traits.
...

Wildcards are commonly used in the shell to refer to e.g., files matching a pattern.

@espenhgn
Copy link
Contributor

espenhgn commented May 13, 2024

As for the "Illegal instruction" issue, also occurs with the Docker image now pushed to ghcr.io:` with Mac hosts (M1 chip):

$ export ISHELL="docker run --platform=linux/amd64 --rm -it -v ${MIXER}/usecases:/work --entrypoint=bash ghcr.io/comorment/mixer:espenhgn-issue20"
$ISHELL
root@668b80e03b5e:/tools# python -c "import ctypes; ctypes.CDLL('/tools/mixer/src/build/lib/libbgmg.so')"
Illegal instruction

Only a subset of instructions are supported:

root@668b80e03b5e:/tools# lscpu | grep Flags
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint

The fix appears to be to build the image locally.

@Licun1
Copy link

Licun1 commented May 14, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants