Simplify installation of pych with Brew or Pip

[buddha314]

The installation procedure for pych is detailed here: http://pychapel.readthedocs.io/introduction.html#installation It requires tracking down libraries, making links, etc. These tasks could be greatly simplified by homebrew or pip.

Either way, you cannot do pip install pyChapel then run `pych --check'.

[lydia-duncan]

I've had success with pip install pyChapel in the past, but maybe not with pych --check.

To be honest, I don't know what the pych executable is specifically intended for - you should be able to use python to compile programs with the pyChapel directives once pip install has run, if I recall correctly.

Though again, I didn't push a release after the latest batch of changes, so the build might be busted for a version of Chapel after the one listed.

[lydia-duncan]

Ah, crud, I think you're right that it isn't a full install . . .

[buddha314]

I'm all about the crud. You'll grow to hate me. I took from the docs your need pych to use @Chapel(sfile=''), but I don't have a full example. If you post one I will try it with just pip install

[buddha314]

Actually, here is an example: http://pychapel.readthedocs.io/introduction.html#hello-world Fails for me.

[lydia-duncan]

:(

My kingdom for a pyChapel BTR person . . .

[buddha314]

A glimpse into my ignorance: What is BTR?

[lydia-duncan]

Build-Test-Release

[lydia-duncan]

Also, don't hesitate to open issues! It's always exciting to see someone play with pyChapel :)

[buddha314]

var hesitation = 0;

[buddha314]

And for what it's worth, my two platforms of choice are OS X and Ubuntu. Homebrew is for OS X

[russel]

Whilst I recently switched to Homebrew on my macOS El Capitan laptop, I used to be a staunch supporter of MacPorts. All too often people provide for one and not the other.

As for Linux I am Debian Sid and Fedora Rawhide, it's always good to see packages appear for them. If in Debian, it gets into Ubuntu automatically, albeit up to six months later.

[russel]

@lydia-duncan build-test-release should be automated. Or am I misunderstanding the issue?

[lydia-duncan]

@russel - it definitely should, but transitioning to that point is not automated, and that's the part that is hurting us most.

[buddha314]

pych could be really helpful to me later this week / early next week. Do you have a sense of when this issue might be addressed? Thanks!

[awallace-cray]

I am unfamiliar with pych, but I will try turning http://pychapel.readthedocs.io/introduction.html#installation into a Homebrew formula, and report progress here today or tomorrow.

Meanwhile, any suggestions welcome!

[buddha314]

I wish you the best of Slack, Reverend Bob...

[awallace-cray]

Will prepare a script + written instruction to manually build Pychapel on Linux from source, with Chapel from current dev source (master branch). After that, will try for same on OS X. Cheers.

[awallace-cray]

Hit problems with Pip/Python dependencies. Here is a preliminary result.
https://gist.github.com/awallace-cray/f70f3f00fa08ba4756b5966a69ebbc3b
I will continue with the Ubuntu 16.04 instructions tomorrow.

[awallace-cray]

Based on yesterdays result, here is a slightly-nicer package of Pychapel build scripts for Ubuntu.
https://gist.github.com/awallace-cray/4eefef0ae7cd8a71191351d0af254bfe

This is a shell archive ("shar") file. Like a tar file, except you expand it by running it as a sh script in an empty directory. Lets you move multiple files easily using just copy-paste.

This shar contains the following
pychpl-1-sudo.sh : System installs with sudo. One-time only for new machine.
pychpl-2-full-pych.sh : Pychapel, full install. One-time-only for new user/env.
pychpl-3-full-chpl.sh : Chapel, full install. One-time-only for new user/env.
pychpl-5-rebuild.sh : Pychapel and Chapel rebuild from existing source in the current workspace
venv.sh : source this file to activate the virtual environment (ie, to just run the last-built "pych")

All these scripts have been lightly tested on AWS Ubuntu server 16.04 VMs.

Note: the Pychapel full install script still includes a source patch for these Ubuntu VMs. I cannot just commit that patch to pychapel "master" because it appears to break the build on the Linux machines we use internally.

I will not be able to work on Pychapel again until next week. Hope this helps!

[buddha314]

Thanks @awallace-cray ! I haven't been able to try this yet. Do you have a sense of next steps and/or timing for this ticket?

[awallace-cray]

I intended to start on Mac OS X version later today. If successful, that would lead to a Homebrew formula change, meaning your brew install chapel would bring pych with it. It can take several days just to change the official homebrew formula. However you could benefit sooner (again, if I am successful) by using a less-automatic interim procedure.

We still want some combination of pip install pychapel plus (something) to be easier to use and less Python environment-dependent than it seems to be now. That seems like a longer-term effort, at least to me.

[awallace-cray]

@buddha314 - Pychapel on Mac OS X will not be easily fixed. Will go back to work on improving Ubuntu, perhaps pip install + (easy instructions) or (scripts). Please work with the interim Ubuntu procedure (above) and let me know how it goes - that would be helpful. Thanks.

[buddha314]

Thanks Mr. Dobbs, I'll give that a shot in the next 24 hours and get back to you.

[buddha314]

Okay, immediate error. I downloaded this file

> wget https://gist.githubusercontent.com/awallace-cray/4eefef0ae7cd8a71191351d0af254bf/raw/785736844a007461625ef695e6ebccd09c5fb220/pychpl-shar.sh

> bash pychpl-shar.sh

x - created lock directory _sh16355.
x - extracting pychpl-1-sudo.sh (text)
x - extracting pychpl-2-full-pych.sh (text)
x - extracting pychpl-3-full-chpl.sh (text)
pychpl-shar.sh: line 333: uudecode: command not found
sed: couldn't flush stdout: Broken pipe
restore of pychpl-3-full-chpl.sh failed
pychpl-3-full-chpl.sh: MD5 check failed
x - extracting pychpl-5-rebuild.sh (text)
pychpl-shar.sh: line 400: uudecode: command not found
sed: couldn't flush stdout: Broken pipe
restore of pychpl-5-rebuild.sh failed
pychpl-5-rebuild.sh: MD5 check failed
x - extracting venv.sh (text)
x - removed lock directory _sh16355.

Can't find uudecode, is that via pip or apt?

[buddha314]

All right, found it with

sudo apt-get install sharutils

Now I'll run the extracted scripts.

[buddha314]

I ran the scripts and they completed with minimal failures. However, the pych command is not in my path and I expected. Perhaps I'm doing it wrong? Do I need to source something?

[awallace-cray]

Did you source venv.sh from your interactive shell?
venv.sh : source this file to activate the virtual environment (ie, to just run the last-built "pych")

[buddha314]

Before or after I run the setup scripts? Not sure of the sequence.

[buddha314]

ah, after, but I thought I tried that. I ran all of them to let the SubGenius sort them out, was that the right strategy?

Now I'll need some sample code to try this against.

[awallace-cray]

Setup scripts build stuff that stays on disk, mostly under ~/.virtualenvs/pychapel.
So, those are first.

source venv.sh merely activates that stuff (prepends things in your PATH and similar).

After source venv.sh, pych is in your PATH

[awallace-cray]

There is a hello world in http://pychapel.readthedocs.io/introduction.html#hello-world

[buddha314]

Users are annoying, I'll say it for you.

 python hello.py
/usr/bin/ld: /tmp/tmpordxvK.a(tmpordxvK.tmp.a.o): relocation R_X86_64_PC32 against symbol `chpl_vmtable' can not be used when making
 a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "hello.py", line 11, in <module>
    hello_world_inline()
  File "/home/buddha314/.virtualenvs/pychapel/local/lib/python2.7/site-packages/pych/extern.py", line 212, in wrapped_f
    raise MaterializationError(self)
pych.exceptions.MaterializationError: Failed materializing ({'anames': [],
 'atypes': [],
 'bfile': None,
 'dec_fp': 'hello.py',
 'dec_hs': '8d35108ffe2ad173a697734a3e9938e1',
 'dec_ts': 1501629524,
 'doc': '\n            writeln("Hello, world");\n        ',
 'efunc': None,
 'ename': 'hello_world_inline',
 'lib': 'inline-chapel-8d35108ffe2ad173a697734a3e9938e1-1501629524.so',
 'module_dirs': [],
 'pfunc': <function hello_world_inline at 0x7fa29debe8c0>,
 'pname': 'hello_world_inline',
 'rtype': None,
 'sfile': None,
 'slang': 'chapel',
 'source': None}).

[buddha314]

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:        17.04
Codename:       zesty

[lydia-duncan]

That last error is probably due to your Chapel compile. Either it wasn't compiled with the quickstart and CHPL_LIBMODE=shared settings, or you don't currently have those settings in your path. Tony, is that on his end or on your script's end?

[awallace-cray]

The pychpl-shar scripts compiled Chapel with quickstart and =shared settings.
I did not use those in the venv.sh script, because they didn't need to be there according to my test.
But I do not think they would do any harm, so ...

Please add the following at the bottom of venv.sh:

# Chapel env
WORK=$(cd $(dirname ${BASH_SOURCE[0]}) ; pwd)
source $WORK/chapel-src/util/quickstart/setchplenv.bash
export CHPL_LIBMODE=shared

Thanks!

[awallace-cray]

@buddha314 : confirmed:
I had the same problem on my Ubuntu 16.04 EC2 VM when I tried python hw.py per the instructions I gave.
I added the above "Chapel env" clause to the end of venv.sh, logged out and in again, source'd the modified venv.sh, and python hw.py worked as expected.

Apologies: pych unit tests work without adding the Chapel env clause. I must not have tested python hw.py in the same env.

[buddha314]

Hmmm... sounds like somebody owes me a shiny new penny....

I'll try to day, thanks!

[awallace-cray]

Updated gists: Rudimentary Pychapel build scripts for Ubuntu 16.04, updated 2017-08-02 09:42,

• pychpl-0-README.txt - as text files
• pychpl-shar.sh - packaged with shar (sudo apt-get install sharutils)

[buddha314]

For posterity, I followed these steps in an empty directory. I ran all the shell scripts except 5.

> sudo apt-get install sharutils
> wget https://gist.githubusercontent.com/awallace-cray/4eefef0ae7cd8a71191351d0af254bfe/raw/9f32762ce1de75393872b8c8c3926af737
318811/pychpl-shar.sh
> bash pychpl-shar.sh
> bash pychpl-1-sudo.sh
> bash pychpl-2-full-pych.sh
> bash pychpl-3-full-chpl.sh

Created `hello.py'

from pych.extern import Chapel

@Chapel()
def hello_world_inline():
    """
    writeln("Hello, world");
    """
    return None

if __name__ == "__main__":
    hello_world_inline()

Then back to

> source venv.sh
> python hello.py
Hello, world

For giggles, I ran it twice. As I expected, the first time it took a moment to compile but the second time it ran immediately.

Next it to try this with external Chapel source code, but I'm worried about this ticket: #77

I'm not sure if the current ticket should be closed until the installation process is self-contained. What does the SubGenius think?

[awallace-cray]

No, this process is not yet worthy of religious icons like ourselves.

[buddha314]

Absolutely not, no.

[awallace-cray]

This docker image might be of some interest.

This is a developmental prototype docker image which includes Chapel + Pychapel, built with the same preliminary build scripts discussed above. The idea is to keep your Pychapel project files on some workstation that can run docker image. Then you run pych or python from inside the docker image, using a shared directory to access your Pychapel project files.

You could keep the Chapel and Pychapel source trees, and even the Python virtual env, outside the docker- another shared directory. A new user would have to provision this docker image before using, by running the build scripts from inside the docker. That docker image would be much much smaller.

[buddha314]

Is there a reason to prefer a docker image over a homebrew installation?

[awallace-cray]

No. This Docker was a concept experiment. I'm working on a Homebrew prototype.

[awallace-cray]

Updated gists: Rudimentary Pychapel build scripts for Ubuntu 16.04, updated 2017-08-10 10:35,

• pychpl-0-README.txt - as text files
• pychpl-shar.sh - packaged with shar (sudo apt-get install sharutils)

[awallace-cray]

The temporary patch I was using, was merged onto master on 2017-08-11.
#82

[buddha314]

That means the ubuntu install works better, right? How is OS X coming?

[awallace-cray]

Not as well. Spent about 2 days. Recorded here: #46

[awallace-cray]

Created a set of rudimentary build scripts for Pychapel on Mac OS X.
https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f

• Includes a couple more temporary source patches.

Next: see if this can be made into a usable Homebrew formula.

Meanwhile, it would be great if others could give this a try and reply here w feedback.

[awallace-cray]

https://gist.github.com/awallace-cray/31da7c2f2333e56cb7949f94d5fcea5f no longer works with current Chapel master (2018-01-23).

• the long hard-coded path to the Chapel libs in pychpl-3-full-chpl.sh lines 71, 76 is no longer correct.
  • the correct path is
    $CHPL_HOME/lib/darwin/clang/arch-native/loc-flat/comm-none/tasks-fifo/tmr-generic/unwind-none/mem-cstdlib/atomics-intrinsics/gmp-none/none/none/fs-none/*
• possibly other changes, TBD
  Nope, that's all.

Updated gist: Rudimentary Pychapel build scripts for Mac OS X 10.11 updated 2018-01-23 17:38

• Changed the lib path and re-tested
  pychpl-0-README.txt

Note: This only worked after I uninstalled homebrew and re-installed it according to the instructions in pychpl-0-README.txt. I believe, but did not prove by testing, that the problem was my homebrew install of gcc@6; especially symlinks like /usr/local/bin/gcc (gcc, g++, and gcov) that it appears I put in there manually.

[awallace-cray]

Updated gists: Rudimentary Pychapel build scripts for Ubuntu 16.04, updated 2018-01-24 10:36

• Changed the lib path and re-tested
• pychpl-0-README.txt - as text files
• pychpl-shar.sh - packaged with shar (sudo apt-get install sharutils) DELETED