Website for searching in awesome.

[lockys]

Hi, dear all
First, thank all authors of awesome list, it makes the world better.

I don't know that if there are similar projects for these.
I am trying to exact awesome lists into JSON in my leisure time(still working on it) and build a simple website for searching awesome. Hope it's a helpful thing :p

I am looking forward to someone to PR any awesome list in JSON format like this because the format of awesome lists are different so that it's not so easy to write a general program to extract them into JSON I think.

It's not an issue actually and can be closed undoubtedly.
if you get interested, below is the repos:
https://github.com/lockys/awesome-search

the JSON repo:
https://github.com/lockys/awesome.json

site:
https://awesomelists.top

tks.

[inputsh]

We've previously discussed about this, but you're actually the first one who started working on it! So, thanks!

[parro-it]

Cool project!

[inputsh]

The problem I have with this JSON idea is that this list itself receives pull requests on a daily basis. It's pretty much impossible to even estimate the amount of changes happening on each and every list included here.

So, may I suggest that instead of manually adding JSON files you find some solution using GitHub's API? It might not be easy, but it definitely is necessary because manually implementing each and every change is a mission impossible.

[sindresorhus]

Should be possible to do some best-effort parsing of the Markdown. It's just sections and links, even though there are slight differences. I think that's a much better approach than trying to convince lots of people to manually maintain a JSON file.

[lockys]

@alejandronanez Thank for your constructive opinions.
@sindresorhus Yes, I agree with you. Writing a general program to parse them will be definitely better and more efficient than convincing people to contribute. Also, thank you for taking time to reply 👍

[sindresorhus]

@dthree and @wooorm have some experience with parsing Markdown and might be able to point you in the right direction.

[wooorm]

Thanks for the ping, @sindresorhus!

@lockys Cool project 👍 I’m building a tool called mdast which I think could help. mdast exposes a syntax tree for markdown.

I hacked this script together:

var fs = require('fs');
var mdast = require('mdast');
var toString = require('mdast-util-to-string');
var toc = /^((table of )?contents|packages)$/i;

mdast().use(function () {
  return function (ast) {
    var sections = {};
    var sectionName;
    var section;
    var contents;

    ast.children.forEach(function (node) {
      if (node.type === 'heading' && node.depth !== 1) { // Ignore first-level headings.
        contents = toString(node).trim();

        // Ignore TOC lists.
        if (!toc.test(contents)) {
          sectionName = contents;
          section = [];
        }
      } else if (section && node.type === 'list') {
        // set the section (only when followed by a list).
        sections[sectionName] = section;
        node.children.forEach(function (item) {
          // get the link in the paragraph in the list-item.
          var head = item.children[0].children[0];

          contents = toString(item).split(/-/);

          section.push({
            'name': contents[0].trim(),
            'description': contents.slice(1).join('').trim() || null,
            'url': head && head.type === 'link' ? head.href : null,
            'cate': sectionName
          });
        });
      }
    });

    console.log(JSON.stringify(sections, null, 2));
  }
}).process(fs.readFileSync('readme.md', 'utf-8'));

In the sindresorhus/awesome-nodejs repo, it produces your desired output (with descriptions added on top 😉):

{
  "Mad science": [
    {
      "name": "webtorrent",
      "description": "Streaming torrent client for Node.js and the browser.",
      "url": "https://github.com/feross/webtorrent",
      "cate": "Mad science"
    },
    {
      "name": "GitTorrent",
      "description": "Peertopeer network of Git repositories being shared over BitTorrent.",
      "url": "https://github.com/cjb/GitTorrent",
      "cate": "Mad science"
    },
    ...
    {
      "name": "RequireBin",
      "description": "Shareable JavaScript programs powered by npm and browserify.",
      "url": "http://requirebin.com",
      "cate": "Tools"
    },
    {
      "name": "Tonic",
      "description": "Embed a Node.js environment on any website.",
      "url": "http://blog.tonicdev.com/2015/09/30/embedded-tonic.html",
      "cate": "Tools"
    }
  ]
}

Now, I hope this helps you. I suggest checking out how mdast works (there are a lot of docs). Good luck, and ping me if you have any Qs 😄

[lockys]

@sindresorhus thanks for finding people who experiencing in markdown 😄
@wooorm mdast is pretty remarkable and helpful for me! It really reduces the pain of parsing markdown. I will try that! Thanks for your kindness and help 💯

[inputsh]

I have a coupon for a free .me domain that I could give to you for this project.

Unfortunately, awesome.me is currently taken, so if anyone has any ideas...

[gepser]

@Aleksandar-Todorovic what about http://awesomelists.me/?

[inputsh]

@gepser sounds good (and it's available). What do you think @lockys ?

[alejandronanez]

@Aleksandar-Todorovic
http://awesomethings.me

I like http://awesomelists.me too :)

[lockys]

@Aleksandar-Todorovic
wow, it's good. I love http://awesomelists.me as well.
I don't know how to express my gratitude, ha!
Thank for your generosity 😃

[inputsh]

Up and running on awesomelists.me.

Don't need to thank me, thank the people from Montenegro that gave me a coupon to their domain for free. I'm already using one (r3bl.me, so I had no idea what to do with it. 😄

Fun fact: .me is actually the biggest exporting product of the country of Montenegro.

[dthree]

👍 on using mdast. @wooorm did an excellent job and it's been working perfectly for me in complicated markdown parsing.

[lockys]

Thanks all you guys for helping this project 👍

[ghost]

Wow amazing work bro

[Sljux]

Great work!

On Sun, Nov 29, 2015 at 3:34 PM, Deejavu notifications@github.com wrote:

Wow amazing work bro

—
Reply to this email directly or view it on GitHub
#427 (comment)
.

[RichardLitt]

Actually, I talked to the guy who runs aweso.me, and he said he might be open to routing it over, as long as he still controls the domain. Just an option!

[lockys]

@RichardLitt Thank you :)
We have decided https://awesomelists.top as our final domain name.
BTW, Awesome Search is better than the previous version for now.
Also, we set a cron job to crawl every awesome list including the awesome repo every day.
Therefore, it will be up-to-date I think!
Take a look if you get interetsed :-)

[pokeball99]

@lockys #470 maybe of use for finding list not on the main awsome list, but as @sindresorhus said, they may be incomplete or such

[lockys]

@pokeball99, thanks you for telling, I have checked your issue before.
But I would use sindresorhus/awesome as only source.
The reason is that the lists which @sindresorhus decided to put on awesome would be a more mature lists as you mentioned.

[kbk0125]

Hey, if anybody is still interested in this, I took my own crack at it! https://www.rtfmanual.io/awesome/. Used the x-ray package from Node to scrape the site. Happy to share the key code snippet if anybody is interested. It is nice and hacky. Inspired by this

[veggiemonk]

I don't know if you might be interested in this, for awesome-docker i've done it like this:

<script src="https://cdnjs.cloudflare.com/ajax/libs/fetch/2.0.1/fetch.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/showdown/1.5.4/showdown.min.js"></script>
<script>
      const converter = new showdown.Converter();
      const text      = fetch('https://raw.githubusercontent.com/veggiemonk/awesome-docker/master/README.md')
                        .then(res => res.text())
                        .catch(err => console.error(err))
                        .then( text => {
                          document.getElementById('md').innerHTML = converter.makeHtml(text);
                        });
</script>

For searching, a simple Ctrl-F or Cmd-F would work. It might not be UI friendly, but I don't think people looking for these kind of resources would mind. Let me know what you think and what kind of tradeoffs is worth it. Thanks

[sindresorhus]

We have a section for Awesome related tools now. Submit what you make there. https://github.com/sindresorhus/awesome#related :)