TODO list

[jarun]

Rolling TODO list thread No. 3. Previous thread is #83.

---

Possible improvements we'd like to see:

• Support all options at omniprompt
• Highlight searched keywords in abstract
• Ubuntu PPA
• Show YouTube-specific results abstract (more details)
• Document APIs in NumPy format [owner @jarun]
• Refactor and write unit tests
• Option-related improvements (too lazy to clarify, just read the comment)

Suggestions and PRs welcome!

[zmwangx]

Several ideas, all of which should be almost backwards compatible, but when implemented should be more user-friendly going forward:

1. Toggling of options in the prompt (mentioned this before somewhere)?

2. I would also like to support long command names so that commands are not so abstract to new users, especially the kind of new users who have little experience with REPL-styled utilities in general (not sure if this type of users are actually in our target audience, but it doesn't hurt). Examples:

   n(ext)
       navigate to the next page
   p(revious)
       navigate to the previous page
   o(pen) [INDEX]
       open the result corresponding to INDEX in the browser (open the
       current search instead if INDEX is omitted)
   f(irst)
       jump to the first page
   g(oogle) KEYWORD [KEYWORD...]
       initiate a new search for the given keywords with the original
       options
   e(xit), q(uit), double <ENTER>
       exit gooogler
   ?, h(elp)
       show omniprompt help
   INDEX
       equivalent to open INDEX
   KEYWORD [KEYWORD...]
       equivalent to google KEYWORD [KEYWORD...]
   

3. Completion (using readline) for 1 and 2.

4. googler without arguments enter the REPL instead of printing usage and exit. I always felt a bit weird that we have an interactive interface but users need to do at least one query noninteractively (by which I mean in command line args) to access it. It's like a Google that's only accessible through either Chrome's omnibox or the search bar of an ongoing query, but not from the google.com splash page.

   We can add a --help option for help text (standard anyway), and disable n, p, and f if there's no existing query.

[zmwangx]

A bit of clarification on long command names that I forgot to mention: I personally like the type of interfaces where users should be able to enter a partial command as long as it is unambiguous, e.g., n, ne, nex, next all work as the next command. Same for options.

We could also give the interactive part of our program an OO interface, reminiscent of cmd.Cmd from PSL. We might even be able to just extend cmd.Cmd, which is not powerful enough yet, especially if we want to support options.

[jarun]

1. Toggling of options in the prompt (mentioned this before somewhere)?

At this time we don't have any other than filter. I'm good but I think it'll work differently for different for different options.

The rest of the ideas are very useful. 👍

[zmwangx]

At this time we don't have any other than filter. I'm good but I think it'll work differently for different for different options.

I mean real options like --time. Of course, only a subset of options should work in the prompt (excluded ones are --json, --colors, --tld and such). We'll also need --no* counterparts to some options.

[jarun]

👍

[zmwangx]

Note to self: our URL construction could receive an OO refresh too to make some points above easier and avoid awkward and brittle URL updating code like

url = url.replace('start=%d&' % oldstart, 'start=%d&' % start, 1)

I'll do this and point 4 in #87 (comment) tonight.

[jarun]

👍

[zmwangx]

By the way, is "Document APIs in NumPy format [owner @jarun]" considered done?

[jarun]

Yes, I think we can close it. Most important functions are covered.

[zmwangx]

Checked off. Don't want to create an "archive" entry anymore. Just remove checked off items when we roll the thread.

[jarun]

We can remove the item.

[zmwangx]

I think we can leave it checked off. It gives off a sense of accomplishment... And It will disappear the next time we roll the thread anyway, so it won't get in the way for long.

[jarun]

👍
I guess you should add all the items you identified.

[zmwangx]

Added (sort of).

[jarun]

okies :)

[jerkovicl]

is it possible to make it available to install via pip?

[zmwangx]

@jerkovicl No. See #64.

[dufferzafar]

Adding a setup.py doesn't really mean that you have to upload this to PyPI. It will also allow you to do this: pip install git+https://github.com/jarun/googler.git/#egg=googler (I think)

People just want pip because that's what they're comfortable using and I really think you should considering adding support for it. There's also magical things like pipsi.

[jarun]

Given the utility is a standalone one (given base Python3 is installed), probably it doesn't matter anymore. We have no further plans on this line.

[zmwangx]

Adding a setup.py doesn't really mean that you have to upload this to PyPI. It will also allow you to do this: pip install git+https://github.com/jarun/googler.git/#egg=googler

Except it's overkill and doesn't work with older pip. Moreover, IIRC updating a git head package from within pip is a nightmare.

Just do this: https://github.com/jarun/googler#downloading-a-single-file.

[abhayghatpande]

Is there anyway to avoid scrolling to the bottom of the search results? Something like less? It will avoid having to page up, to read the first few results. I observe this behaviour on OS X, not sure if it is applicable for Linux given some of the screenshots I see.

[zmwangx]

Automatic scrolling is done by the terminal emulator, there's nothing we can do about it. We certainly can't bundle a curses TUI. However, if you submit a PR that implements optional piping through a pager, we could review it.

A workaround is to set the number of results per page to something suitable to your screen.

[abhayghatpande]

Fair enough. Will look into it. Thanks for the immediate response.

[jarun]

The workflow remains the same with the pager or limiting the number of results per page. Whether you page through (using Enter hopefully) 4 results per page or press n to fetch the next 4 results (using -n 4) I don't see much difference.

[knkarthik]

#109 Doesn't actully fix the following, right?

googler without arguments enter the REPL instead of printing usage and exit. I always felt a bit weird that we have an interactive interface but users need to do at least one query noninteractively (by which I mean in command line args) to access it. It's like a Google that's only accessible through either Chrome's omnibox or the search bar of an ongoing query, but not from the google.com splash page.

[zmwangx]

@knkarthik Please be more specific. What are you trying to do that's not possible?

[knkarthik]

Sorry. What I meant is googler without arguments still enters the REPL instead of printing usage and exit. I thought #109 fixed this.

[jarun]

What I meant is googler without arguments still enters the REPL instead of printing usage and exit.

That's the behaviour introduced by #109. Earlier googler used to show help and exit if no arguments were passed.

[zmwangx]

I'll add that there are already a good number of popular article/full text/whatever extractors in Python.

[jorgesumle]

I was wondering if anyone was interested in adding bs4 based parsers for popular websites, that will parse things like Wikis, forums and stackexchange etc to make them work well on w3m/elinks.

That's what things like RSS and Atom is for. Just ask those popular websites, comment systems, whatever... to implement RSS, so that you can easily access their contents/updates, it is not very difficult to implement. If they make browsing difficult, then they don't probably want you to do that. But if that's the case, then you must boycott them or webscrap their website to regain your freedom. I'd help you if you want to do that for a company that fights against our freedom, such as Google, and if I'd find that useful, or if you are willing to pay me for the job.

Fight against DRM. Some anti-DRM websites:

• https://www.eff.org/issues/drm
• https://defectivebydesign.org/
• https://media.libreplanet.org/u/libreplanet/tag/no-drm-in-web-standards/

[zmwangx]

@jorgesumle This is an entirely out of touch and out of place rant. First, websites have every right to their content, and user-generated content with rights signed away; you'd be a fool to think you should have "freedom" over someone else's work, if they don't grant it to you. Secondly, websites have absolutely zero obligation to make their websites work well in a text-based browser; graceful degradation / progressive enhancement sure is nice, but people need to be payed to do extra work. Thirdly, you're probably misinterpreting DRM (say, in the form of Encrypted Media Extensions); writing badly structured HTML that's hard to parse has nothing to do with DRM. I get that you're probably frustrated by recent developments in DRM on the web, but it has nothing to do with what @aditya3098 requested. So please stop.

[zmwangx]

I'll also mention that both MediaWiki and StackExchange have APIs that grant you access to their native markup (Wiki markup and Markdown, respectively).

[vviikk]

This might be far reaching and also sound like I'm nuts. I use Fish shell that gives be CLI auto completion, and sometimes things like git checkouttab would do a quick fetch and list me all the branches. Would be so nice if we could have something similar for googler!. Love it anyway. Thanks.

[jarun]

You'll find the scripts in https://github.com/jarun/googler/tree/master/auto-completion

[vviikk]

oooh. And also if we google "USD45 in GBP" it will convert currencies. Hope we can have this in googler. Even complex calculations is calculated by google.

[vviikk]

@jarun But isn't that just for the options? Does it return search results?

[zmwangx]

convert currencies

We decided that's out of scope for googler. See #97.

[jarun]

Aah, you are looking for auto-completion search completion. Nope, we don't have it.

[zmwangx]

@piggyslasher

search completion

Search completion API is actually pretty simple:

$ curl 'https://www.google.com/complete/search?client=psy-ab&q=git'
["git",[["git\u003cb\u003ehub\u003c\/b\u003e",0],["git",0],["git\u003cb\u003elab\u003c\/b\u003e",0],["git\u003cb\u003e stash\u003c\/b\u003e",0]],{"q":"THnbq2A3yeMOKRqVYkA-eHqtIek","t":{"bpc":false,"tlw":false}}]
$ curl 'https://www.google.com/complete/search?client=psy-ab&q=goo'
["goo",[["goo\u003cb\u003egle\u003c\/b\u003e",0,[131]],["goo\u003cb\u003egle maps\u003c\/b\u003e",0,[131]],["goo\u003cb\u003egle translate\u003c\/b\u003e",0],["goo\u003cb\u003egle docs\u003c\/b\u003e",0]],{"q":"A7UvvlTFeq-jF_cJ1aNpcm8BByY","t":{"bpc":false,"tlw":false}}]

Those strings of interest, translated to human-readable form, are:

• git<b>hub</b>, git<b>lab</b>, git<b> stash</b>;
• goo<b>gle</b>, goo<b>gle maps</b>, goo<b>gle translate</b>, goo<b>gle docs</b>.

You should be able to parse these with ease, and write a completer based on that. As for myself, I'm afraid I'm not motivated enough to implement this, but it does look like an interesting weekend hack.

[zmwangx]

Actually, since this is both simple and interesting, I just went ahead and did it. @piggyslasher If you're a Zsh user, you can go ahead and try the topic branch https://github.com/zmwangx/googler/tree/complete-queries. There's an additional script to install; see notes in https://github.com/zmwangx/googler/tree/complete-queries#shell-completion. I may add bash support later.

Anyway, here's a screencast. Pretty cool.

@jarun Feel like ever including this feature on master? If so, I'll integrate this into googler itself (put this behind a hidden option, say --complete-query), and add bash and possibly fish support.

[jarun]

@zmwangx I will check this. Need a few days. Thanks for checking it out!

[vviikk]

You sir, are a star. The both of you actually! Thanks. I use fish, but I’ve been meaning to go back to zsh. This might just be the right time.

[jarun]

@piggyslasher I use fish ;). Care to translate it to fish and submit a PR?

[zmwangx]

@jarun Sounds like you approve of this idea? I learned some fish and added support to the same branch: https://github.com/zmwangx/googler/tree/complete-queries. I also integrated the completer into googler itself (a hidden option --complete) so there's no separate script to install. Screencast (fish version):

[jarun]

Sure I approve! I am not just finding time to get back to my usual stuff. Killer schedule... but I will be back.

Please raise the PR. It's extremely useful.

[vviikk]

Dayum.... This is wicked. Thanks guys. Let me try if it works with the zsh-autosuggestions. That would pretty much make it work like the chrome address bar!

[vviikk]

@jarun was troubleshooting a bash script I was writing, and it's almost the end of the road for me with fish shell sadly. I am giving zsh one more try and so far, I've been pleasantly impressed. Will try to see if I can get live auto completion to work with zsh-autosuggestions.

[ashutoshmjain]

selecting index is not opening in browser .. Is there an .googerrc file where I need to define the browser ?

Figured it out .. I exported links ..

Thanks

[jarun]

No config file. You can use the env variable BROWSER. What's your browser? Is it set as your system default browser?

[nathan-b]

I hope this is the right place to add a feature request...if not I can open a new issue!

I'd love a command line option to just return the number of results for a query. For instance:

$ googler --count --exact --np foo
92,500,000
$

[jarun]

@nathan-b please refer to #144.

[xulayen]

在windows上上面可以安装使用吗

[xulayen]

how can i user googler on windows paltform ?

[jarun]

Install Python and invoke googler. Use option -C if you are using cmd.

[jarun]

Rolled at #209.