Add a parameter to be parsed as item's datetime stamp #4

martinrotter · 2024-01-17T11:04:10Z

No description provided.

Owyn · 2024-01-17T20:26:52Z

Would that be useful? 🤔
Since RSSGuard already sorts new entries right even without dates martinrotter/rssguard#621 (comment)

RetroAbstract · 2024-01-19T11:48:44Z

Would that be useful? 🤔

Yes. It would be useful in being able to limit the initial amount of articles CSS2RSS-based feeds fetch upon the first fetch after feed creation (or the first fetch after the database is cleaned & optimized) with article filters and properly limit the amount of articles in general with this upcoming feature.

Right now the article filter I use successfully limits the initial amount of non-CSS2RSS feed articles fetched but cannot limit the initial amount of CSS2RSS feed articles fetched as they all have the same timestamp, and the feature has unexpected behaviour as when for example limiting a CSS2RSS feed to:

keeping 1 article, deleting important and non-read articles: RSS Guard will oscillate at random between the first and second item found at every fetch.
keeping 2 articles, deleting important and non-read articles: RSS Guard will oscillate at random between the first and second item found at every fetch for the first article, and oscillate at random between the first, second and third item found at every fetch for the second article.

Owyn · 2024-01-19T12:07:18Z

Well, that can be done of course as a yet another parameter, but my worry is - will people actually bother to use it and match one more element with a new css selector every feed for this? (even if they'd want to limit the number of fetched articles with this mentioned feature)

martinrotter · 2024-01-19T12:11:29Z

well I myself would use that feature

it would be nice if the provided element was somehow magically parsed into date/time object, there is excellent python package "dateparser" which provides such automagic string -> date/time conversion function

RetroAbstract · 2024-01-19T12:22:15Z

Well, that can be done of course as a yet another parameter, but my worry is - will people actually bother to use it and match one more element with a new css selector every feed for this? (even if they'd want to limit the number of fetched articles with this mentioned feature)

I would, yes.

Owyn · 2024-01-21T14:08:53Z

@martinrotter articles keep getting remarked as non-read when their date changes even a little :-(

so with this feature they will be forever unread (cuz many dates are relative)

2024-01-21.16-47-14.mp4

Tried in 4.6.3 - same result

Owyn · 2024-01-21T21:26:01Z

Try the new update with the Date parsing selector (6th argument) - aim it at text nodes with the date or or elements with title or alt attributes containing Date

You'd need "No duplicates by URL" filter to not let items get remarked as unread every time currently (dunno why it's not a standard checkbox option)

function filterMessage() {
  if (msg.isAlreadyInDatabase(MessageObject.SameUrl)) {
    return MessageObject.Ignore;
  }
  else {
    return MessageObject.Accept;
  }
}

RetroAbstract · 2024-01-23T12:52:22Z

New date selector works relatively well, thanks alot :)

Couldn't get it to work for these pages even if they do have dates to their items:

https://jwfan.com/?cat=21
https://support.microsoft.com/en-us/topic/windows-10-update-history-8127c2c6-6edf-4fdf-8b9f-0f7be1ef3562

martinrotter · 2024-01-23T12:58:42Z

Calling script like this fails:

python3 scripts/css2rss.py ".clanek" "" "" "" "" ".date"

I only need to specify "item" selector and date selector (in this case with "date" class)

114, in <module>
    if sys.argv[2][0] == '!':
       ~~~~~~~~~~~^^^
IndexError: string index out of range

RetroAbstract · 2024-01-23T13:03:30Z

Calling script like this fails:

python3 scripts/css2rss.py ".clanek" "" "" "" "" ".date"

I only need to specify "item" selector and date selector (in this case with "date" class)
114, in <module>
    if sys.argv[2][0] == '!':
       ~~~~~~~~~~~^^^
IndexError: string index out of range

I think you need to use ~ instead of "" to skip selectors you don't need.

martinrotter · 2024-01-23T13:04:19Z

Calling script like this fails:

python3 scripts/css2rss.py ".clanek" "" "" "" "" ".date"

I only need to specify "item" selector and date selector (in this case with "date" class)
114, in <module>
    if sys.argv[2][0] == '!':
       ~~~~~~~~~~~^^^
IndexError: string index out of range
I think you need to use ~ instead of "" to skip selectors you don't need.

OK, I am really beginner when it comes to CSS selectors.

Owyn · 2024-01-23T13:06:19Z

Calling script like this fails:

python3 scripts/css2rss.py ".clanek" "" "" "" "" ".date"

both "" and ~ should work to skip elements, I'll fix it 👍
tho ~ should be easier to write and read

martinrotter · 2024-01-23T13:10:28Z

Tilde works.

Perhaps anyone can help me. I have website, which has one huge

element which contains elements (which represent titles) and elements (which contain dates/times) like this:

<p>

<a href="....." class="x">Title 1</a>
<span class="y">2024-01-07</span>

<a href="....." class="x">Title 2</a>
<span class="y">2023-07-29</span>

<a href="....." class="x">Title 3</a>
<span class="y">2022-01-07</span>

</p>

@Owyn @RetroAbstract Can anyone from you experts tell me how to instruct css2rss to spill out feed from this correctly now with date?

Owyn · 2024-01-23T13:17:18Z

<span class="y">2022-01-07</span>

date selector should point at this span element
e.g. with .y as the 6th argument

Owyn · 2024-01-23T13:22:25Z

New date selector works relatively well, thanks alot :)

Couldn't get it to work for these pages even if they do have dates to their items:

https://jwfan.com/?cat=21 https://support.microsoft.com/en-us/topic/windows-10-update-history-8127c2c6-6edf-4fdf-8b9f-0f7be1ef3562

as for the 2nd link I'm not sure what you're trying to match there... update history? there isn't a separate date element there

martinrotter · 2024-01-23T13:33:57Z

<span class="y">2022-01-07</span>
date selector should point at this span element e.g. with .y as the 6th argument

Would this also work if input element for date is messed with auxiliary text like this?

<p>

<a href="....." class="x">Title 1</a>
<span class="y">2024-01-07 SOMEMORETEXT</span>

<a href="....." class="x">Title 2</a>
<span class="y">2023-07-29 SOMEMORETEXT</span>

<a href="....." class="x">Title 3</a>
<span class="y">2022-01-07 SOMEMORETEXT</span>

</p>

Owyn · 2024-01-23T13:35:48Z

date is messed

nope:

martinrotter · 2024-01-23T13:40:40Z

date is messed

nope:

are these exceptions muted in your script? because running my script like this

css2rss.py ".a" "~~" "~~" "~~" "~~" ".y"

does not produce any exceptions which is weird

anyway, thanks for your help with this :)

Owyn · 2024-01-23T13:42:28Z

are these exceptions muted in your script? because running my script like this

They are redirected into the article's body, check it :-)

Because erroring a whole feed just because an auxiliary feature like date-parsing broke would've been too much

RetroAbstract · 2024-01-23T13:48:30Z

New date selector works relatively well, thanks alot :)
Couldn't get it to work for these pages even if they do have dates to their items:
https://jwfan.com/?cat=21 https://support.microsoft.com/en-us/topic/windows-10-update-history-8127c2c6-6edf-4fdf-8b9f-0f7be1ef3562

as for the 2nd link I'm not sure what you're trying to match there... update history? there isn't a separate date element there

Yes, I can create feeds for both sites with CSS2RSS, but can't get the date selector to work with them.

The "time" element does not work for https://jwfan.com/?cat=21, as well as "published", "entry-meta", "entry-meta > published" etc.. articles have as time-stamp the moment RSS Guard fetches the feed.

For https://support.microsoft.com/en-us/topic/windows-10-update-history-8127c2c6-6edf-4fdf-8b9f-0f7be1ef3562, you're right there isn't a separate date element, my mistake. Would the script be able to only take into consideration the date of each update and discard the rest? For example with the "January 9, 2024—KB5034122 (OS Builds 19044.3930 and 19045.3930)" update - Keep the date, use it for the date selector and discard everything after it?

martinrotter · 2024-01-23T13:50:15Z

Well, what I try to parse is this URL https://antivirus.22web.org/clanky.htm

Here is RAW input: https://www.pastebin.cz/en/p/VVFQEN9

Here is how I call it

cat raw-input.txt | python3 css2rss ".clanek" "~~" "~~" "~~" "~~" ".date"

Sadly with the the output "date_published" attribute is empty and error message is not seen in contents of article either.

Owyn · 2024-01-23T13:55:10Z

The "time" element does not work for https://jwfan.com/?cat=21,

It works for me 🤔 have you tried the script line from my screenshot?

error message is not seen in contents of article either.

it should be at the very bottom of it in item_description

martinrotter · 2024-01-23T13:59:52Z

it should be at the very bottom of it in item_description
Well, maybe I am blind.
out.json

RetroAbstract · 2024-01-23T14:12:13Z

The "time" element does not work for https://jwfan.com/?cat=21,

It works for me 🤔 have you tried the script line from my screenshot?

Works now, thanks 😄

Owyn · 2024-01-23T14:28:50Z

it should be at the very bottom of it in item_description
Well, maybe I am blind.
out.json

https://support.microsoft.com/en-us/topic/windows-10-update-history-8127c2c6-6edf-4fdf-8b9f-0f7be1ef3562

python css2rss.py "#supLeftNav > div > ul:nth-child(2) > li:nth-child(2)" ~ ~ ~ ~ ".supLeftNavLink"

or

check after 4fe2c89

now it should write when the date element wasn't found at all (it was probably the case - cuz then it won't write any parsing errors - since there's nothing to parse)

Owyn · 2024-01-23T15:12:25Z

cat raw-input.txt | python3 css2rss ".clanek" "~~" "~~" "~~" "~~" ".date"

that's because the date element isn't inside .clanek element but next to it...

martinrotter · 2024-01-23T15:49:19Z

Yes i know, how to tell it that it is next then? Pls

Owyn · 2024-01-23T15:56:13Z

Yes i know, how to tell it that it is next then? Pls

Don't tell it that it's next, just choose a root element (1st argument) a level higher

RetroAbstract · 2024-08-02T22:59:12Z

Would need some help trying to get the dates for items on eBay.

For example, the search term "nintendo switch" sorted by newly listed, the time of listing for each article is found in:

<span class="s-item__detail s-item__detail--secondary">
  <span class="s-item__dynamic s-item__listingDate">
   <!--F#f_0-->
   <span class="BOLD">Aug-2 15:37</span>
   <!--F/-->
  </span>

When I use any of the above (s-item__detail, s-item__detail--secondary, s-item__dynamic, s-item__listingDate, BOLD) as the 6th selector in the feed's post processing script box with CSS2RSS, it fails to find date elements.

Any help with this would be much appreciated. 🙂

Owyn · 2024-08-03T13:25:46Z

it fails to find date elements.

because there are no dates for ebay ads they masked as listed items, that's why it says so

python css2rss.py "li.s-item" "span.s-item__listingDate" ~ "a[href*=UserReviews]" ~ "span.s-item__listingDate"

also, date parsing now uses your local timezone, not UTC when unspecified (I've just noticed it was wrong here)

and there's now a more detailed message saying how many items exactly didn't get their date parsed or found so you'll see when it's just few and not all of them failing

#4 (comment)

RetroAbstract · 2024-08-04T11:08:15Z

it fails to find date elements.

because there are no dates for ebay ads they masked as listed items, that's why it says so

python css2rss.py "li.s-item" "span.s-item__listingDate" ~ "a[href*=UserReviews]" ~ "span.s-item__listingDate"

also, date parsing now uses your local timezone, not UTC when unspecified (I've just noticed it was wrong here)

and there's now a more detailed message saying how many items exactly didn't get their date parsed or found so you'll see when it's just few and not all of them failing

Thanks. I tried python css2rss.py "li.s-item" "span.s-item__listingDate" ~ "a[href*=UserReviews]" ~ "span.s-item__listingDate", but don't get the same results as the ones shown in the screenshot you shared.

Maya and Beautiful Soup are installed.

What URL are you using?

Update: The user agent I had set to launch with RSS Guard was the cause of me not getting the same results. Changed it and now it works fine. 🙂

Owyn · 2024-08-04T11:16:21Z

but don't get the same results as the ones shown in the screenshot you shared.

Are you sure you're actually using the line I wrote above? because it clearly chooses published date there as the title, there's no way it'd randomly choose the actual title instead - you'd need to carefully select elements for that

anyway, now latest version should clearly say how many date elements got matched and parsed ( in the feed comment field where you press "fetch now" )

the url is the one you gave me,
site content might also differ based on the user's country

RetroAbstract · 2024-08-04T12:04:01Z

Are you sure you're actually using the line I wrote above? because it clearly chooses published date there as the title, there's no way it'd randomly choose the actual title instead - you'd need to carefully select elements for that

100% sure it's the same line you wrote, copied and pasted:

I also updated css2rss to your latest commit.

site content might also differ based on the user's country

I thought this too, tried the UK and CA eBays but same results.

Owyn · 2024-08-04T12:09:27Z

Perhaps RSSGuard version also matters, I'm still using 4.3.3 nowebengine version since there's still light web-viewer there which looks better

newer versions should have an option to switch between full and lite browser and also some alternative ways to fetch websites for parsing as I've heard

RetroAbstract · 2024-08-04T12:37:11Z

Perhaps RSSGuard version also matters, I'm still using 4.3.3 nowebengine version since there's still light web-viewer there which looks better

Tried on 4.3.3 No Web Engine and indeed, I now get the results you do.

newer versions should have an option to switch between full and lite browser

4.7.2 No Web Engine is my daily driver, tried alterning Use legacy article formatting in Feeds & Articles > Articles toggled on and off but no difference.

and also some alternative ways to fetch websites for parsing as I've heard

I think those are related to the Discover feeds feature which to my knowledge cannot be used with CSS2RSS.

martinrotter · 2024-08-05T10:58:38Z

Perhaps RSSGuard version also matters, I'm still using 4.3.3 nowebengine version since there's still light web-viewer there which looks better

newer versions should have an option to switch between full and lite browser and also some alternative ways to fetch websites for parsing as I've heard

BTW, lightweight variant of browser is still available in latest RSS Guard, you just have to enable it in settings.

Owyn · 2024-08-05T11:36:32Z

BTW, lightweight variant of browser is still available in latest RSS Guard, you just have to enable it in settings.

I know, the old one just looks better for me tho so I still use it:
4.3.3:

4.7.3 (after full database + setting restore - article columns got messed up anyway)

edit: - apparently now there's "legacy article view" which almost brings back the old look (except link colors are now different (shows em as all visited now for some reason?))
4.7.3 legacy article view:

Might actually update now, thanks for the legacy view 👍🏻

martinrotter · 2024-08-05T11:38:14Z

Feel free to report the "visited link color" if it bugs you, could be a bug.

Owyn added the enhancement New feature or request label Jan 17, 2024

Owyn added a commit that referenced this issue Jan 21, 2024

add Date selector #4

a9ab5fb

Owyn added a commit that referenced this issue Jan 21, 2024

add Date selector #4

808d982

Owyn added the done label Jan 21, 2024

Owyn mentioned this issue Jan 22, 2024

[FR]: few more simple standard options for not remarking articles UnRead martinrotter/rssguard#1279

Closed

Owyn added a commit that referenced this issue Jan 23, 2024

fix skipping arguments with "" #4

0387d8b

Owyn added a commit that referenced this issue Aug 3, 2024

use local timezone for parsed dates

ee05090

#4 (comment)

RetroAbstract mentioned this issue Aug 5, 2024

[BUG]: RSS Guard 4.7.3 Lite doesn't recognize dates for CSS2RSS eBay feed items martinrotter/rssguard#1465

Closed

Add a parameter to be parsed as item's datetime stamp #4

Add a parameter to be parsed as item's datetime stamp #4

Comments

martinrotter commented Jan 17, 2024

Owyn commented Jan 17, 2024

RetroAbstract commented Jan 19, 2024

Owyn commented Jan 19, 2024 • edited Loading

martinrotter commented Jan 19, 2024 • edited Loading

RetroAbstract commented Jan 19, 2024

Owyn commented Jan 21, 2024

Owyn commented Jan 21, 2024

RetroAbstract commented Jan 23, 2024

martinrotter commented Jan 23, 2024 • edited Loading

RetroAbstract commented Jan 23, 2024

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024 • edited Loading

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024

Owyn commented Jan 23, 2024

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024

RetroAbstract commented Jan 23, 2024

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024

martinrotter commented Jan 23, 2024

RetroAbstract commented Jan 23, 2024

Owyn commented Jan 23, 2024

Owyn commented Jan 23, 2024

martinrotter commented Jan 23, 2024

Owyn commented Jan 23, 2024

RetroAbstract commented Aug 2, 2024

Owyn commented Aug 3, 2024 • edited Loading

RetroAbstract commented Aug 4, 2024 • edited Loading

Owyn commented Aug 4, 2024 • edited Loading

RetroAbstract commented Aug 4, 2024

Owyn commented Aug 4, 2024 • edited Loading

RetroAbstract commented Aug 4, 2024 • edited Loading

martinrotter commented Aug 5, 2024

Owyn commented Aug 5, 2024

martinrotter commented Aug 5, 2024

Owyn commented Jan 19, 2024 •

edited

Loading

martinrotter commented Jan 19, 2024 •

edited

Loading

martinrotter commented Jan 23, 2024 •

edited

Loading

Owyn commented Jan 23, 2024 •

edited

Loading

Owyn commented Aug 3, 2024 •

edited

Loading

RetroAbstract commented Aug 4, 2024 •

edited

Loading

Owyn commented Aug 4, 2024 •

edited

Loading

Owyn commented Aug 4, 2024 •

edited

Loading

RetroAbstract commented Aug 4, 2024 •

edited

Loading