Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

leveldown Package Support #14

Open
reconbot opened this issue Jan 14, 2016 · 8 comments
Open

leveldown Package Support #14

reconbot opened this issue Jan 14, 2016 · 8 comments

Comments

@reconbot
Copy link
Member

I've come across my first popular package that can't cross compile.

leveldown@1.4.3

While I'm not sure if this repo should be the support page for such problems long term, I think it's the perfect place to deal with this particular package. It's being used by @HipsterBrown via pouchdb.

The C code looks for some platform features that might be missing from our little tessel. I don't know enough to debug it.

../deps/leveldb/leveldb-1.18.0/port/atomic_pointer.h:211:2: error: #error Please implement AtomicPointer for this platform.
 #error Please implement AtomicPointer for this platform.

My google foo find me the file in question, a forum thread with possible fixes.

I took a step back for a moment and noticed they're not using node pre-gyp. They're using something called prebuild, and they claim to support the arm chipset.

I'm investigating further but I'd love help from anyone in the know.
@reconbot
Copy link
Member Author

I did a regular old npm install after setting all the cross compiling env vars. I had to manually make the tar of the package after that. I have the tar up on s3 for testing leveldown leveldown@1.4.3.

@reconbot
Copy link
Member Author

@HipsterBrown is tribute to test it out

@HipsterBrown
Copy link

Smoke Test

My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');

var db = new PouchDB('./db');

db.put({
  "_id": "test",
  "desc": "This is a test doc."
});

db.info()
.then(function(info) {
  console.log(info);
});

console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js

CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in ~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node instead of ~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The fs.copySync also required a try/catch block to catch silent errors that were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
  return new Promise(resolve => {
    // For every binary module in use...
    binaryModulesUsed.forEach(details => {
    var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
    var tempTargetModulePath = path.join(tempBundlePath, details.path);
    var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);

    try {
      fs.copySync(sourceBinary, tempTargetBinary);
    } catch (err) {
      console.error(err);
    }
   });
   // All binary modules have been replaced, resolve.
   return resolve();
  });
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args   '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in <module>
    sys.exit(gyp.script_main())
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 545, in script_main
    return main(sys.argv[1:])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 538, in main
    return gyp_main(args)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 514, in gyp_main
    options.duplicate_basename_check)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 130, in Load
    params['parallel'], params['root_targets'])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
    variables, includes, depth, check, True)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
    ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
    p_stdout, p_stderr = p.communicate('')
  File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but glad to get a successful deploy onto Tessel.

@reconbot
Copy link
Member Author

I did wierd things when making the package.
On Jan 15, 2016 1:10 AM, "Nick Hehr" notifications@github.com wrote:

Smoke Test My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');
var db = new PouchDB('./db');
db.put({
"_id": "test",
"desc": "This is a test doc."
});
db.info()
.then(function(info) {
console.log(info);
});
console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js
CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in
~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node
instead of
~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The
fs.copySync also required a try/catch block to catch silent errors that
were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
return new Promise(resolve => {
// For every binary module in use...
binaryModulesUsed.forEach(details => {
var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
var tempTargetModulePath = path.join(tempBundlePath, details.path);
var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);

try {
  fs.copySync(sourceBinary, tempTargetBinary);
} catch (err) {
  console.error(err);
}

});
// All binary modules have been replaced, resolve.
return resolve();
});
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue
nodejs/node#1440.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=node.lib',
gyp info spawn args '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in
sys.exit(gyp.script_main())
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 545, in script_main
return main(sys.argv[1:])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 538, in main
return gyp_main(args)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 514, in gyp_main
options.duplicate_basename_check)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 130, in Load
params['parallel'], params['root_targets'])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
variables, includes, depth, check, True)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
build_file_data, PHASE_EARLY, variables, build_file_path)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
expanded = ExpandVariables(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
p_stdout, p_stderr = p.communicate('')
File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but
glad to get a successful deploy onto Tessel.

—
Reply to this email directly or view it on GitHub
#14 (comment).

@reconbot reconbot mentioned this issue Jan 15, 2016
Closed
This was referenced Jan 15, 2016
Closed
Closed
@johnnyman727
Copy link
Contributor

@reconbot not sure if I can be helpful here but I'd like to understand the issue(s). Is there a problem with how T2 handles prebuild and an issue with the underlying atomicpointer library? (Along with the two CLI issues @HipsterBrown caught)

@reconbot
Copy link
Member Author

I think our issue is with prebuild and not atomicpointer. Give it a whirl
and try to build the leveldown package and see what happens.

On Mon, Jan 18, 2016 at 11:15 AM, Jon notifications@github.com wrote:

@reconbot https://github.com/reconbot not sure if I can be helpful here
but I'd like to understand the issue(s). Is there a problem with how T2
handles prebuild and an issue with the underlying atomicpointer library?
(Along with the two CLI issues @HipsterBrown
https://github.com/HipsterBrown caught)

—
Reply to this email directly or view it on GitHub
#14 (comment).

Francis Gulotta
Director of Web Connected Devices
Mobile: 917-805-3079

@johnnyman727
Copy link
Contributor

Ah I see - well I probably won't be much help on this one. @tcr might have some advice.

@nolanlawson nolanlawson mentioned this issue Jun 7, 2016
Open
@nolanlawson
Copy link
Contributor

FWIW I also tried the sqlite3 package (PouchDB's other Node-supported backing database), but no luck so far: #27

@reconbot reconbot changed the title Package Support leveldown Package Support Jun 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reconbot @nolanlawson @johnnyman727 @HipsterBrown