Allow for sel to select and wrap multiple element results

[Isaac-Flath]

Fix bug in read_html.

I added tests to ensure it is grabbing something when various selectors are passed.

[jph00]

Hmm interesting. I see why you're doing it this way, although I might need a tiny bit of convincing that just concatenating the strings is the right approach...

[Isaac-Flath]

The most important thing I want to fix is a bug that prevents a function from working (see first screenshot). There are two options for fixing it after more research.

• select_one : page = str(soup.select_one(sel)): This gets the first thing that matches the CSS selector if there are multiple on the page. This mirrors the current code more closely.
• select : page ''.join(str(el) for el in soup.select(sel)) : This will be more conservative, and if a CSS selector appears multiple times on the page, it will get them all. I think there are 2 good reasons to use this:
  • If you don't realize there's a second (or more) match to the CSS selector in the URL, this error is on the side of bringing in what you need + more. Selecting one could err on the side of not bringing in what you need because it only grabbed the first one.
  • It allows for nifty stuff that I am not 100% sure will be useful (but I think it has potential). See the second screenshot where I use select to grab all the .listing-descriptions from our quarto blog. If you use select_one is just gives the first one instead of all of the .listing-descriptionss for each blog post.

Bug Screenshot

Select vs Select_one

[Isaac-Flath]

We could also do both, and have a multi=False arg, so it defaults to the current intended behavior but allows for the multiple selection option also. Maybe that's best?

[jph00]

That's a good idea. And maybe use \n as the join str?

[jph00]

And how about also having an optional wrap_tag, e.g. if set to document each returned element is wrapped with <document>...</document>?

[Isaac-Flath]

I've implemented both of them. I am not super happy with the code, but I am not sure it can be improved significantly since the wrap_arg injection needs to happen after the HTML to markdown conversion. If you have ways this can be done cleaner, that'd be helpful.

[jph00]

I'm not super happy with the code either, but I also can't see a better way to do it offhand. So I'll just merge it now, but if you come up with something better later, feel free to put in another PR, of course! :)