Skip to content
This repository has been archived by the owner on Dec 12, 2024. It is now read-only.

enable hooks for fetching code behind solidity import statements #80

Closed
1 task
serapath opened this issue Dec 19, 2018 · 15 comments
Closed
1 task

enable hooks for fetching code behind solidity import statements #80

serapath opened this issue Dec 19, 2018 · 15 comments
Assignees
Labels
#compiler specification category
Milestone

Comments

@serapath
Copy link
Contributor

serapath commented Dec 19, 2018

@todo

  • enable hooks for fetching and caching code behind solidity import statements

about imports

@alincode
Copy link

alincode commented Dec 23, 2018

@alincode alincode self-assigned this Dec 23, 2018
@alincode
Copy link

alincode commented Dec 23, 2018

keywords

  • combineSource
  • resolver
  • import parser

@alincode
Copy link

alincode commented Dec 31, 2018

@serapath

readCallback function can't be a async function, because emscripten doesn't support it very well.

but I think another workaround is provided "getReadCallback" API.

example 1:

let readCallback = await getReadCallback(sourceCode);
let output = await compiler(sourceCode, readCallback);

example 2:

let localSources = [{
       path: 'lib.sol',
       content: 'library L { function f() internal returns (uint) { return 7; } }'
     }];
 
let readCallback = await getReadCallback(sourceCode, localSources);
let output = await compiler(sourceCode, readCallback);

https://github.com/ethereum-play/solc-js/pull/1/commits/1db7eb24755acf1446487e7693073e2ca1be7301#diff-3b8192acb0d491039ab68dea0b430bfcR62

https://github.com/ethereum-play/solc-js/pull/1/commits/1db7eb24755acf1446487e7693073e2ca1be7301#diff-3b8192acb0d491039ab68dea0b430bfcR39

@serapath
Copy link
Contributor Author

serapath commented Jan 1, 2019

what about

const output = await compile(sourceode, async (importpath) => await fetch(importpath))

async function compile (source, cb) {
  const X = await getAllImports(source)
  const files = {}
  for (p of X) files[path] = await cb(p)
  const output = solcjs(source, X)
  return output
}

@serapath
Copy link
Contributor Author

serapath commented Jan 1, 2019

Are you very familiar with C++ ? ...because one other option would be to ask the solidity team whether a pull request to make imports async would be welcome :-)

@alincode
Copy link

alincode commented Jan 2, 2019

@serapath
no, I don't familiar with C++.
but about asked solidity team "support async read callback".
in the past, someone asked the same question, the answer is NO.

@chriseth answered this question in a long time ago.
ethereum/solc-js#140

@alincode
Copy link

alincode commented Jan 7, 2019

example1: await compiler(sourceCode, localSources)

let compiler = await solcjs(version);

const sourceCode = `
  pragma solidity >0.4.99 <0.6.0;

  import "lib.sol";

  library OldLibrary {
    function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
    function f(uint8 a) public returns (bool) {
        return OldLibrary.someFunction(a);
    }
  }`;

let localSources = [{
  path: 'lib.sol',
  content: 'library L { function f() internal returns (uint) { return 7; } }'
}];

let output = await compiler(sourceCode, localSources);

example2: await compiler(sourceCode);

let compiler = await solcjs(version);

const sourceCode = `
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/SafeMath.sol';

  library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
  }`;

let output = await compiler(sourceCode);

@serapath
Copy link
Contributor Author

serapath commented Jan 9, 2019

I would still propose that we hide the fact that the compiler is synchronous and use an API that enables to switch to async in the future if they ever enable it, so I would love to see something like:

const output = await compile(sourcecode, async (importpath) => await fetch(importpath))

async function compile (source, cb) {
  const X = await getAllImports(source)
  const files = []
  for (p of X) {
    var content = await cb(p)
    files.push({ path: p , content})
  }
  // let localSources = [{
    // path: 'lib.sol',
    // content: 'library L { function f() internal returns (uint) { return 7; } }'
  // }];
  const output = solcjs(source, files)
  return output
}

@alincode
Copy link

alincode commented Jan 9, 2019

=== solc-js ====

const compile = require('solc-js')

async function compile (source, resolver) {
  const localSources = await resolver.getLocalSources();
  const readcallback = await resolver.combineSource(source, localSources)
  const output = solcjs(source, readcallback);
  return output;
}

=== user test case ====

const resolver = new resolver(localSources);
let output = await solc.compile(source, resolver);

@serapath
Copy link
Contributor Author

serapath commented Jan 9, 2019

const compile = require('solc-js')
/*
function compile (sourcecode, getContent) {
  var allimports = parse(sourcecode)
  var localSources = []
  // e.g. runs 10x when 10 imports in sourcecode
  for (path of allimports) {
    localSources.push({ path, content: await getContent(path) })
  }
  return solcjs(sourcecode, localSources)  
}
*/

const myDB = { } // maybe it looks like:

const sourcecode = `
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/a.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/b.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/c.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/d.sol';
  import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/e.sol';
  import 'lib1.sol'
  import 'lib2.sol'
  import 'lib3.sol'
  import 'lib4.sol'
  import 'lib5.sol'

  library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
  }

  contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
  }`

test('sourcecode with 10 imports calls callback 10 times', function (t) {
  t.plan(10)
  const output = await compile(sourcecode, async (path) => {
    t.assert(true)
    console.log(path)
    var output
    if (myDB.contains(path)) output = await myDB.get(path)
    else output = new Promise(resolve => {
      resolver.import(path, null, ({ content }) => resolve(content)
    })
    return output
  })
})

/*
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/a.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/b.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/c.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/d.sol';
  'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/e.sol';
  'lib1.sol'
  'lib2.sol'
  'lib3.sol'
  'lib4.sol'
  'lib5.sol'
*/

const output = await compile(sourcecode, async (path) => 
    myDB.contains(path) ? await myDB.get(path) : await getImportContent(path)
)

// so "getImportContent(...)" is a replacement for
new Promise(resolve => resolver.import(path, null, ({ content }) => resolve(content))

@ethereum ethereum deleted a comment from alincode Jan 9, 2019
@alincode
Copy link

alincode commented Jan 11, 2019

https://github.com/ethereum-play/solc-js/pull/2

@serapath now, it like this. (import for loop not finish yet, so just check getContent function interface.)

describe('compiler', () => {

  const version = 'v0.5.1-stable-2018.12.03';

  it('with local import', async () => {
    let compiler = await solcjs(version);
    compiler.should.be.a('function');

    const sourceCode = `
    pragma solidity >0.4.99 <0.6.0;

    import "lib.sol";

    library OldLibrary {
      function someFunction(uint8 a) public returns(bool);
    }

    contract NewContract {
      function f(uint8 a) public returns (bool) {
          return OldLibrary.someFunction(a);
      }
    }`;

    let myDB = new Map();
    myDB.set('lib.sol', 'library L { function f() internal returns (uint) { return 7; } }');

    const getImportContent = async function (path) {
      return myDB.has(path) ? myDB.get(path) : await solcResolver.getImportContent(path);
    };

    let output = await compiler(sourceCode, getImportContent);
    let item = output[0];
    item.should.have.all.keys('name', 'abi', 'sources', 'compiler', 'assembly', 'binary', 'metadata');
    item.metadata.analysis.should.have.all.keys('warnings', 'others');
  });

  it('with github import', async () => {
    let compiler = await solcjs('v0.4.25-stable-2018.09.13');
    compiler.should.be.a('function');
    const sourceCode = `
    import 'https://github.com/OpenZeppelin/openzeppelin-solidity/contracts/math/SafeMath.sol';

    library OldLibrary {
        function someFunction(uint8 a) public returns(bool);
    }

    contract NewContract {
        function f(uint8 a) public returns (bool) {
            return OldLibrary.someFunction(a);
        }
    }`;

    let output = await compiler(sourceCode);
    let item = output[0];
    item.should.have.all.keys('name', 'abi', 'sources', 'compiler', 'assembly', 'binary', 'metadata');
    item.metadata.analysis.should.have.all.keys('warnings', 'others');
  });

});

solc-js still will a little depend solc-resolver, because I will check.

https://github.com/ethereum-play/solc-js/blob/feature/getContent/src/lib/getCompile.js#L18

if (solcImport.isExistImport(sourcecode)) {
   if (getImportContent == undefined) {
      // solution1: auto fixed getImportContent
      getImportContent = solcResolver.getImportContent;
      // solution2: just throw error, don't use solcResolver inside the compiler.
   } else if (typeof getImportContent !== 'function') {
      throw Error('getContent should be a funcion.');
    }
    readCallback = await solcImport.getReadCallback(sourcecode, getImportContent);
}

or you don't want I auto fixed getImportContent(just throw error), then I can remove solc-js depend solc-resolver. but that also meaning, if sourcecode exist import, then compile function two parameter must be require.

now

* if not exist import
  * await compiler(sourceCode);
* if exist import
  * await compiler(sourceCode);  // auto fixed getImportContent;
  * await compiler(sourceCode, getImportContent);

or you change lik this

* if not exist import
  * await compiler(sourceCode);
* if exist import, getImportContent is require.
  * await compiler(sourceCode, getImportContent);

@serapath
Copy link
Contributor Author

I think we should make another module which imports/requires solc-js and many resolvers, and exports a compile function, which takes sourcecode and uses the correct resolvers, so users dont have to bother

@serapath
Copy link
Contributor Author

const solcjs = require('solc-js')
const resolve_http = async x => await fetch(x).then(r => r.text())

const resolve_swarm = require('reaolve-sawrm')

module.exports = sourcecode => {
  const imports = getAllImports(sourcecode)

  for (path of imports) {
    if ( isSwarm(path)) resolve_swarm(path)
    // ...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
#compiler specification category
Projects
None yet
Development

No branches or pull requests

2 participants