Using the phantomjs scripts (or forking them) #6

Coretool · 2016-02-14T12:49:03Z

I think I know where to start working now: @piranna

Make changes to their preconfigure.sh script (add backends, etc) (here)
Make changes to their webkit config (they removed things like web audio)
Build the whole stuff and hope that we didn't break anything =)

The text was updated successfully, but these errors were encountered:

piranna · 2016-02-14T19:24:53Z

I've updated the script mimicking how PhantonJS works and qtbase compiles without problem, but qtwebkit requires to install some programs (gperf bison ruby flex) and requires to have qtbase compiled with ICU support. I enabled it but require to also compile and add libicu... Don't know if it makes sense to go so long or if we should move directly to content_shell instead of QtWebkit. Maybe we could continue experimenting with QtWebkit to see if we manage to make it work and later see if it compensates to move to content_shell, what do you think? Do you want that I upload my modified script and try to add ICU support yourself, or do you prefer that I add ICU support first?

Coretool · 2016-02-14T21:01:02Z

Hm... I think content_shell would be extremly interesting to work and prototype with... I think for a first attemp it is to much work if there are better ways of achivinh the same :) I think we should leave it as it easy and look forward to the content-shell !

Coretool · 2016-02-14T21:03:42Z

However, I would say that we leave GUI related code in this repo no mather which way we take (?).

piranna · 2016-02-14T23:40:02Z

There were previous attempts with content_shell and seems @formula1 was able to compile and make it work, but only at default resolution of 800x600 and trying to change that it showed nothing... :-/ We could investigate it, I only agreed to experiment with QtWebkit since it seemed it would be easier to have a working prototype before to move to other things. Who knows, maybe ICU is just the only last one needed thing, at least by using qtbase instead of full qt it got to compile a lot faster... :-)

However, I would say that we leave GUI related code in this repo no mather which way we take (?).

I agree on that. Maybe we could do it on different branches before deciding what alternative to follow, isn't it?

formula1 · 2016-02-15T04:19:41Z

@netlovers was the real hero when it came to content shell but I don't think he was able to finish it. My own attempts involved webkit.js, headless-gl and falsifying a bunch of browser objects. Didn't work out as planned though : / I am able to get it running without errors but it renders blank pngs. https://github.com/formula1/node-webkit-canvas

I saw @piranna (I see you everywhere man, you have no idea) at the servo repo which may be of interest to us. They implemented a CEF interface to hook into (https://github.com/servo/servo/tree/master/ports/cef). I'm aware it's probably a bad thing to add more language dependencies (as it stands, c compiler, python and now possibly rust) But they seem to be doing good work over there.

Heres a CEF sample if you didn't find one https://github.com/acristoffers/CEFSimpleSample

piranna · 2016-02-15T04:49:54Z

@netlovers was the real hero when it came to content shell but I don't
think he was able to finish it.

Sorry, I'm really bad with names...

My own attempts involved webkit.js, headless-gl and falsifying a bunch of
browser objects. Didn't work out as planned though : / I am able to get it
running without errors but it renders blank pngs.
https://github.com/formula1/node-webkit-canvas

At least it was a good idea :-) Probably there was sonething missing...

I saw @piranna (I see you everywhere man, you have no idea) at the servo
repo which may be of interest to us.

I have an idea about it, what I don't have it's a life :-(

They implemented a CEF interface to hook into (
https://github.com/servo/servo/tree/master/ports/cef). I'm aware it's
probably a bad thing to add more language dependencies (as it stands, c
compiler, python and now possibly rust) But they seem to be doing good work
over there.

Yeah, it's written in Rust. Being this a independent project from NodeOS
don't matter too much as far as it does what we need, since probably we'll
distribute precompiled builds.

Heres a CEF sample if you didn't find one
https://github.com/acristoffers/CEFSimpleSample

CEF is somewhat deprecated in benefict of content_shell since it has less
dependencies and lower requeriments. Anyway, it's better
https://github.com/acristoffers/CEF3SimpleSample since it uses CEF3 ;-)

Coretool · 2016-02-15T06:56:03Z

Yeah... I agree with trying it once again with ICU (fingers crossed ) Both CEF and content shell will require a bunch of work to make them run as we want (as far as I know)

piranna · 2016-02-18T08:21:15Z

Ok, not sure how or why, but I managed to compile it using ICU :-P There are some problems though:

it last forever to compile, I've left it all the night compiling in a dual core machine, so we need to publish prebuild images or people will kill themselves after killing us due to anger :-P
qtwebket tests for ICU only works for dynamic one instead of static one, so I enabled to build both of them. This has the problem is using the dynamic one by default, so we need to find a way to tell qmake to use the static one on the final build or better also on the tests
qmake is a sh*t, it does too much magic to my taste and there's no way to discover the configuration flags, old ./configure scrips are better by far (and gyp is as bad in this aspect...).
I was not able to test it because I compiled it without support for XBC and my machine seems don't have enabled framebuffer, so this need to be check later. Could you be able to download and test this yourselves?
Final build are 55mb without ICU (and the ICU database are 24mb more...), so we definitely needs to publish prebuilds images :-P

Since it's using ICU as a dynamic library I've added a run.sh script. The ldd seems to be fairly small...

[piranna@SquaredHead:~/Proyectos/noGUI]
 (master) > LD_LIBRARY_PATH=build/icu/lib ldd build/nogui/noGui 
    linux-vdso.so.1 =>  (0x00007ffdf1155000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3a77942000)
    libicui18n.so.56 => build/icu/lib/libicui18n.so.56 (0x00007f3a774c4000)
    libicuuc.so.56 => build/icu/lib/libicuuc.so.56 (0x00007f3a7712b000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3a76f27000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3a76d1f000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3a76b00000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3a7677e000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3a76476000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3a7625e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a75e94000)
    /lib64/ld-linux-x86-64.so.2 (0x000056256ab56000)
    libicudata.so.56 => build/icu/lib/libicudata.so.56 (0x00007f3a744b1000)

Problems is it have been compiled by using Ubuntu glibc and worst than that, glibc is not able to create purely statically linked binaries as musl does, so we'll need to find a way to compile it. I think the best option would be to convert the NodeOS cross-compiler as an independent project.

Coretool · 2016-02-18T08:43:49Z

WOW ! Good job ! Could you push your scripts ? I think most of these problems should be "solveable". Sure, I'll test it ;)

piranna · 2016-02-18T09:15:26Z

I've already pushed them. I have tested it and done some minor tune-ups but
still didn't work, seems plugins doesn't work when statically linked so
linuxfb plugin don't get loaded. It seems it needs to be used a macro on
the code to load it Do you want I push these changes?
El 18/2/2016 9:43, "John Green" notifications@github.com escribió:

WOW ! Good job ! Could you push your scripts ? I think most of these
problems should be "solveable". Sure, I'll test it ;)

—
Reply to this email directly or view it on GitHub
#6 (comment).

Coretool · 2016-02-18T13:15:50Z

There's a 'special' method/function/lib/whatever-qt-calls-that to include static plugins ... I will push an updated version of noGui in the next days :) (Totally forgot to tell you about that ) I think that's what is missing !

piranna · 2016-02-18T13:51:55Z

Seems I was able to include the linuxfb plugin, or at least now I get an
error about not having permissions to open /dev/fb0! :-D Could you be able
to look hos make it to use linuxfb by default (no -platform argument)
instead of xcb?
El 18/2/2016 14:15, "John Green" notifications@github.com escribió:

There's a 'special' method/function/lib/whatever-qt-calls-that to include
static plugins ... I will push an updated version of noGui in the next days
:) (Totally forgot to tell you about that ) I think that's what is missing !

—
Reply to this email directly or view it on GitHub
#6 (comment).

piranna · 2016-02-18T15:37:37Z

Yay! :-D I've managed to include the linuxfb plugin inside the static binary. I'm not able to see anything but seems is a network error in my part. Good thing is, the mouse support and icon comes by free, and the reload button clicking the mouse secondary button is a plus... ;-)

Things to do now:

make ICU static
use musl to generate purely static binaries
use full-screen framebuffer and adapt to framebuffer resizes
add option to exit - it captures all the keyboard inputs and it's not possible to go out
use linuxfb platform by default

Coretool · 2016-02-18T15:42:00Z

Yeah! That's cool !

For exiting and co. I thought about keyboard shortcuts (something like ctrl + q or so )
As @lite20 said, it is possible to change the fb size (-> I'll add code to get the screen size )

To use the fb by default... What about a js script (or even wrapper) for that ?

piranna · 2016-02-18T20:29:07Z

For exiting and co. I thought about keyboard shortcuts (something like
ctrl + q or so )

Ok, although well need to find a good one not used by anyone...
Ctrl+Alt+Sup?

As @lite20 said, it is possible to change the fb size (-> I'll add code
to get the screen size )

Ok.

To use the fb by default... What about a js script (or even wrapper) for
that ?

It's currently being used a bash script, but I want it native.

Coretool · 2016-02-18T20:53:40Z

ok, 2. ok and 3. ok =)

mitsukaki · 2016-02-19T19:25:48Z

As @lite20 said, it is possible to change the fb size (-> I'll add code to get the screen size )

If you need anything, let me know.

BTW, I'm watching your progress like a hawk :P I'm always here so if you need something (excluding compiling because my pc just can't XD) then lemme know :D

Coretool · 2016-02-20T08:40:35Z

@piranna It looks like the size of the linuxfb is set using env variables...

(http://doc.qt.io/qt-5/embedded-linux.html#linuxfb)

I could query the display size and set the size using a function from stdlib.h...

piranna · 2016-02-20T08:50:57Z

@piranna It looks like the size of the linuxfb is set using env
variables...

(http://doc.qt.io/qt-5/embedded-linux.html#linuxfb)

It's a shame that it can't be set programatically nor it gets the correct
value, but this link is awesome :-D

I could query the display size and set the size using a function from
stdlib.h...

How are you planning to do that?

Coretool · 2016-02-20T08:55:12Z

QApplication has got a method to get the screen size

And stdlib has got putenv()

piranna · 2016-02-20T10:02:08Z

And set them before calling QtWebkit? It could work...
El 20/2/2016 9:55 AM, "John Green" notifications@github.com escribió:

QApplication has got a method to get the screen size

And stdlib has got putenv()

—
Reply to this email directly or view it on GitHub
#6 (comment).

Coretool · 2016-02-20T10:10:14Z

As long as everything is called in one process it should work, since putenv() sets them only for the process...

piranna · 2016-02-20T12:01:59Z

Yeah, but it's strange there not an API for this...
El 20/2/2016 11:10 AM, "John Green" notifications@github.com escribió:

As long as everything is called in one process it should work, since
putenv() sets them only for the process...

—
Reply to this email directly or view it on GitHub
#6 (comment).

piranna · 2016-02-20T12:30:54Z

I've fully read the page and all of us we should do so. Seems EGLFS uses by
default a full-screen window based on the framebuffer size, so probably
this could work, problem is how to enable EGL support on NodeOS in a
generic way, al least software rendered... :-/
El 20/2/2016 13:01, "piranna@gmail.com" piranna@gmail.com escribió:

Yeah, but it's strange there not an API for this...
El 20/2/2016 11:10 AM, "John Green" notifications@github.com escribió:

As long as everything is called in one process it should work, since
putenv() sets them only for the process...

—
Reply to this email directly or view it on GitHub
#6 (comment).

piranna · 2016-02-20T12:36:49Z

Ok, we need to check this
https://bugs.tizen.org/jira/plugins/servlet/mobile#issue/PTREL-785
El 20/2/2016 13:30, "piranna@gmail.com" piranna@gmail.com escribió:

I've fully read the page and all of us we should do so. Seems EGLFS uses
by default a full-screen window based on the framebuffer size, so probably
this could work, problem is how to enable EGL support on NodeOS in a
generic way, al least software rendered... :-/
El 20/2/2016 13:01, "piranna@gmail.com" piranna@gmail.com escribió:

Yeah, but it's strange there not an API for this...
El 20/2/2016 11:10 AM, "John Green" notifications@github.com escribió:

As long as everything is called in one process it should work, since
putenv() sets them only for the process...

—
Reply to this email directly or view it on GitHub
#6 (comment).

Coretool · 2016-02-20T14:33:42Z

(I knew that EGLFS uses the full screen, I tested noGui on it...)
So you wanna use mesa ? I think we should track down the linuxfb thingy first (and "freeze" it) and then start searching for other ways to achieve the rendering :/

piranna · 2016-02-20T15:17:39Z

(I knew that EGLFS uses the full screen, I tested noGui on it...)

Do you think is an issue only of framebuffer?

So you wanna use mesa ? I think we should track down the linuxfb thingy
first (and "freeze" it) and then start searching for other ways to achieve
the rendering :/

In the long run I want to use EGL and Mesa, but it's ok, let's freeze a
framebuffer version and later we'll move to other things.

Coretool · 2016-02-20T15:54:52Z

It don't know, if it is only framebuffer related ... It's possible too that we just don't have this "auto adjustment" when using static binaries

So... are your build scripts complete ? I will add the framebuffer code in the next days and afterwards I think.. (just maybe ^^) we have got a first release :) ( Of course ... I needs to be packaged and so on)

piranna · 2016-02-20T16:01:30Z

It don't know, if it is only framebuffer related ...

Ok, let's forget about this and we'll check it later if we move to EGL.

It's possible too that we just don't have this "auto adjustment" when
using static binaries

No, it doesn't have relationship, static binaries are only about having all
it's code inside the binary instead of stored in external libraries.

So... are your build scripts complete ?

Sort of :-) I didn't have time to test loading a web page, though..

I will add the framebuffer code in the next days and afterwards I think..
(just maybe ^^) we have got a first release :) ( Of course ... I needs to
be packaged and so on)

One step at a time ;-)

Coretool · 2016-02-21T12:35:59Z

Well then ^^

Coretool · 2016-03-23T20:15:47Z

Wow ! That's great news ! Sorry, that I'm so inactive at the moment but school's running pretty hot right now :P I'll look into this asap.

piranna · 2016-03-23T20:33:41Z

Yeah, I've been a bit inactive lately too due to work and some showstoppers, but seems things are moving forward :-)

Coretool · 2016-04-10T19:42:05Z

I finally looked into it ^^ ! It seems like they (the developers of these addons) just remove the option from the response... I know that qt got the QNetworkReply class, maybe it allows me to do the same with qt... (I think it grants access to the header)

Coretool · 2016-04-10T20:17:06Z

@piranna

piranna · 2016-04-10T20:50:16Z

Do you think you would be able to easily override this on noGUI and submit your changes soon?

Coretool · 2016-04-10T21:05:45Z

Puh... Yeah, it shouldn't be that hard but my time is limited. I'll try my best to do it this week !

piranna · 2016-04-10T22:19:15Z

Ok, take your time and keep going ;-)
El 10/4/2016 11:05 PM, "John Green" notifications@github.com escribió:

Puh... Yeah, it shouldn't be that hard but my time is limited. I'll try
my best to do it this week !

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6 (comment)

piranna · 2016-04-10T22:20:08Z

Remember enable first the support for X11, probably this will make it
easier to test.
El 11/4/2016 12:19 AM, "piranna@gmail.com" piranna@gmail.com escribió:

Ok, take your time and keep going ;-)
El 10/4/2016 11:05 PM, "John Green" notifications@github.com escribió:

Puh... Yeah, it shouldn't be that hard but my time is limited. I'll
try my best to do it this week !

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6 (comment)

Coretool · 2016-04-11T04:31:44Z

Ah right ! That was a thing too :0...

mitsukaki · 2016-04-11T04:49:31Z

Just a question. Did y'all manage to open any page in the last noGUI? Just
curiosity :P while you guys work on this, I've begun familiarizing myself
with content shell. Hopefully i'll be of use when we get there!
On Apr 10, 2016 10:31 PM, "John Green" notifications@github.com wrote:

Ah right ! That was a thing too :0...

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6 (comment)

piranna · 2016-04-11T05:01:50Z

Just a question. Did y'all manage to open any page in the last noGUI?

No I didn't, but also didn't hace time to give it an oportunity.

mitsukaki · 2016-04-11T05:04:56Z

Oh hahaha! That's alright. Good job anyway on all your progress!
On Apr 10, 2016 11:01 PM, "Jesús Leganés Combarro" notifications@github.com
wrote:

Just a question. Did y'all manage to open any page in the last noGUI?

No I didn't, but also didn't hace time to give it an oportunity.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6 (comment)

Coretool · 2016-04-11T09:35:57Z

It doesn't work on our static linuxfb versiob... however I seems to work with X11

piranna · 2016-04-11T09:44:20Z

It doesn't work on our static linuxfb versiob... however I seems to work with X11

Have you managed to open a web page with noGUI on X11? If so maybe linuxfb it's more a network issue...

Coretool · 2016-04-11T10:02:38Z

Well... X11-non-static works...

piranna · 2016-04-11T10:09:44Z

Well... X11-non-static works...

Ok, that's something :-) Have you modified the code? Do you have it somewhere?

Coretool · 2016-04-11T10:28:06Z

I didn't modify anything... just use X11 as target platforn

piranna · 2016-04-11T10:33:07Z

I was talking about to enable the support for X11 target platform.

Coretool · 2016-04-11T10:34:35Z

You wanna enable it for nodeos or for noGui ? NoGui code is platform indipendent...

piranna · 2016-04-11T10:37:35Z

For noGUI, and just only for development purposes. I think this way it will be easier to test...

Coretool · 2016-04-11T10:47:52Z

Yeah... noGui seems to work in a non-static X11 context. I haven't tested it in a static context yet.

piranna · 2016-04-11T11:29:48Z

Ok, if we can show a url in a non-static X11, we can continue working from it. Do you have somewhere the changes to enable X11 target?

Coretool · 2016-04-11T15:45:23Z

You don't have to change anything ... using qtStudio is the easiest way, just press "run" ^^ If you try to run noGui with a normal qt version (something like ./noGui ) you will hit an error that tells you the target (=platform) names (I don't know them all)

Since qt 5.6 there seems to be something new to static versions ... I think it got easier, I will have to look at it (as far as I've seen, it seems to be much easier now)

Coretool · 2016-04-11T16:12:46Z

=> so just compile a normal, shared version of noGui and start it with ./noGui ... Of course, if you do not have X11 that won't work... but I think this isn't case

piranna · 2016-04-11T16:14:21Z

=> so just compile a normal, shared version of noGui and start it with ./noGui ... Of course, if you do not have X11 that won't work... but I think this isn't case

Of course on desktop that's not the case, that's just why I tell you about doing so for testing purposes :-)

Coretool · 2016-04-11T19:34:44Z

Ah, well I can comfirm then, that it works. I will add some logging and this x frame thingy in the near future...

piranna · 2016-04-11T19:56:56Z

Cool, keep us informed of tour progresses :-)
El 11/4/2016 9:34 PM, "John Green" notifications@github.com escribió:

Ah, well I can comfirm then, that it works. I will add some logging and
this x frame thingy in the near future...

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6 (comment)

piranna · 2016-08-06T16:20:31Z

I think these links would be of interest:

ariya/phantomjs#12948 (comment)
http://wizardyhnr.blogspot.com.es/2016/04/how-to-build-cross-compliatio.html

Coretool · 2016-08-06T16:43:48Z

Hm... So if I understood correctly they managed to build qt static ! This cross-compile system looks good and it'll be trivial to make the changes we need. Thanks for the link, lucky that I've got holidays so I've got enough time to try that (and other things) out !

piranna · 2016-08-06T17:57:33Z

So if I understood correctly they managed to build qt static !

We need to see what build arguments they are using, that's the interesting part.

Using the phantomjs scripts (or forking them) #6

Using the phantomjs scripts (or forking them) #6

Comments

Coretool commented Feb 14, 2016

piranna commented Feb 14, 2016

Coretool commented Feb 14, 2016

Coretool commented Feb 14, 2016

piranna commented Feb 14, 2016

formula1 commented Feb 15, 2016

piranna commented Feb 15, 2016

Coretool commented Feb 15, 2016

piranna commented Feb 18, 2016

Coretool commented Feb 18, 2016

piranna commented Feb 18, 2016

Coretool commented Feb 18, 2016

piranna commented Feb 18, 2016

piranna commented Feb 18, 2016

Coretool commented Feb 18, 2016

piranna commented Feb 18, 2016

Coretool commented Feb 18, 2016

mitsukaki commented Feb 19, 2016

Coretool commented Feb 20, 2016

piranna commented Feb 20, 2016

Coretool commented Feb 20, 2016

piranna commented Feb 20, 2016

Coretool commented Feb 20, 2016

piranna commented Feb 20, 2016

piranna commented Feb 20, 2016

piranna commented Feb 20, 2016

Coretool commented Feb 20, 2016

piranna commented Feb 20, 2016

Coretool commented Feb 20, 2016

piranna commented Feb 20, 2016

Coretool commented Feb 21, 2016

Coretool commented Mar 23, 2016

piranna commented Mar 23, 2016

Coretool commented Apr 10, 2016

Coretool commented Apr 10, 2016

piranna commented Apr 10, 2016

Coretool commented Apr 10, 2016

piranna commented Apr 10, 2016

piranna commented Apr 10, 2016

Coretool commented Apr 11, 2016

mitsukaki commented Apr 11, 2016

piranna commented Apr 11, 2016 via email

mitsukaki commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

Coretool commented Apr 11, 2016

piranna commented Apr 11, 2016

piranna commented Aug 6, 2016

Coretool commented Aug 6, 2016

piranna commented Aug 6, 2016