feat: max price filtering #383

xninjax · 2020-09-29T20:58:29Z

Description

This feature will address issues such as: Let us set price ranges to avoid 3rd party sellers/price gouging #310

Improves filtering by allowing a supplied MAX_PRICE in .env. This can help filter out price gouging sellers and is an alternate to having to specifically list banned sellers.

It would be better to have an array of prices (one for each card type) - but this should suffice as a framework to build upon.

Testing

test:series card was used against all stores (updated some test card links for euro stores, to ensure euro price formatting was taken care of properly).

Ensured that euro price formatting i.e 1.234,99 was converted to 'standard' formatting i.e 1234.99.

Tested against multiple amazon-de/nl products with varying number of digits in the pricetag, to ensure conversion works over a wide range of supplied prices
store model must include optional euroFormat: true for conversion to take place (see example below)

Ensured that Max Price was obeyed

MAX_PRICE="5" was set in .env and all test cards were confirmed to be skipped over. A new logger.info message is produced anytime the store price is greater than the threshold price.
MAX_PRICE="" and MAX_PRICE="2000" was set in .env, and all test cards were again able to trigger in-stock alerts.

Note: Manufacturer direct sales sites (ASUS, NVIDIA, etc) do not have max_price selectors currently set - and is likely not required.

Reproducing the test is pretty easy, set a MAX_PRICE in .env to a low enough value. Enable test:series card. Output will display a logger.info message including the store card price vs threshold price, if test card price exceeds max price .

Optional Max price container must be set in store model, under 'labels', for price filtering to function for that store.

maxPrice: {
		container: 'span[class*="PriceString"]',
		euroFormat: true
	  }

euroFormat is optional, and only required when store provides euro price formatting - as described above.

Functional demonstration

Test 1
MAX_PRICE="0.1"

✅ amazon-de and amazon-nl demonstrate proper currency format conversion (converts to dollars) and price limit filtering activated
✅ newegg-ca demonstrates proper currency format checking (remains as dollars) and price limit filtering activated
✅ ASUS demonstrates ability to trigger alert due to no price container set in asus store model

Test 2
MAX_PRICE="325"

✅ newegg-ca and amazon-de price remains above MAX_PRICE, and price limit filtering is activated
✅ amazon-nl is now below MAX_PRICE and thus allowed to trigger alert
✅ ASUS same as before

- currently one price for all cards per store (should implement price per card) - currently hardcoded into model (should implement in .env). - See newegg-ca for example.

- todo: MAX_PRICE per card? - fight with linter, linter won

added amazon and amazon-ca container tags

…ies to all cards/stores)

… should read "The time in milliseconds" ...

xninjax · 2020-09-29T21:27:22Z

@jef sorry for all the commits in this branch... I'm definitely not a git-guru by any means ¯\_(ツ)_/¯

Hope it wasn't too early for a PR, as testing on other pages hadn't been done (aside from amazon/-ca and newegg/-ca), but the framework is there now. My intent of max price should really only apply to sites with 3rd party marketplace sellers anyhow... I wouldn't imagine a need for max price on the manufactures sites?

EDIT: Changed PR to draft: found issue with price parsing for locales that use comma ',' as decimal place i.e € 832,38.
Will apply conditional logic to hopefully handle this, then re-submit PR.

UPDATE: All stores with exception of manufacturers retail sites have a Max price filter selector applied and tested. During the testing found some selectors on amazons and bestbuys to be quite elusive! However, I believe I have found some reliable selectors to use.

karmanbadhesha · 2020-09-29T22:50:41Z

I had accomplished something similar by looking at the "Sold and shipped by Amazon.ca" but it can result in cases where a cheap 3rd party reseller listing might get missed! Price is definitely the way to go

- all stores with exception of manufactures direct sales sites - euro price format conversion implemented - new type ‘Pricing’ created - euro test cards links updated for more realistic testing (amazons)

jef

Overall, I really like it! I appreciate the contribution! I know a lot of people will like this. Especially with Amazon third party sellers.

Just a few comments. Thanks for all the help!

jef · 2020-09-30T04:27:19Z

src/store/lookup.ts

@@ -145,6 +145,35 @@ async function lookupCardInStock(store: Store, page: Page, link: Link) {
 }
 }

+ if (store.labels.maxPrice && Config.store.maxPrice) {


Do you mind abstracting this logic in this if block just like the if statements above and below using the pageIncludesLabels?

It would help with complexity.

That being said, take a look at that function. I believe what you're wanting is something very similar. As I think your approach is great, but I think there is some extra work you're doing here that may not be needed since we have those helper functions.

Yes no worries, I'll take a look, appreciate the feedback! 😎

Thanks! Look forward to see what's to come.

Abstraction injection sequence complete 😄

Could not utilize pageIncludesLabels directly - as a mathematical comparison was needed, but it became a good jumping off point as reference for a new helper function...
Open to feedback 👍

src/store/model/adorama.ts

simplified code: - utilizing extractPageContents helper function - utilized ternary operators where possible

- cardPriceLimit returns 'null' if no price limit or card price ok, otherwise returns actual card price for use in logger.info message

…price-check-testing # Conflicts: # .env-example # src/store/model/amazon-de.ts # src/store/model/amazon-nl.ts

…ck-testing

updated documentation

jef · 2020-10-01T17:27:35Z

I think this is ready to ship! I just had a question on the containers. How come they seem more "complex"? Why not just grab the main selector that holds the price? Was it not working accordingly?

Looks good, just some comments now

xninjax · 2020-10-01T21:20:24Z

Good question about the containers, what I was finding was that the original/simple selectors were sometimes returning a completely different value... It appeared some js would modify prices and containers after load, likely bot protection 🤖 The more complex container matching helped ensure to grab the final result container with the actual price 🏆 😁

DishBrain · 2020-10-01T22:48:01Z

Hey there, I was just testing this out against amazon.ca where the TUF OC has a scalper trying to sell it for $1,879.99 CDN and it wasn't working.
This is what a price span looks like on amazon.ca:
<span id="priceblock_ourprice" class="a-size-medium a-color-price priceBlockBuyingPriceString">CDN$ 1,879.99</span>

I looked into it, and in the cardPriceLimit function, at the return statement, the cardpriceNumber is showing as 1.879 instead of 1879

xninjax · 2020-10-02T01:35:26Z

Hey there, I was just testing this out against amazon.ca where the TUF OC has a scalper trying to sell it for $1,879.99 CDN and it wasn't working.
This is what a price span looks like on amazon.ca:
<span id="priceblock_ourprice" class="a-size-medium a-color-price priceBlockBuyingPriceString">CDN$ 1,879.99</span>

I looked into it, and in the cardPriceLimit function, at the return statement, the cardpriceNumber is showing as 1.879 instead of 1879

Thanks for letting me know about this! - I'll take a look. Perhaps the regex parsing needs some tweaking. Do you have a link to the item still? EDIT: I found the card on amazon. will test.

As for the price container - amazon currently using span[class*="PriceString"]' as a selector. It has wildcard matching, as sometimes the id changes to something like span id="priceblock_saleprice" while other times it's like what you posted here. The selector seems to have picked the correct container, but regex is not parsing the decimal place correctly now.

DishBrain · 2020-10-02T01:46:59Z

Thanks for letting me know about this! - I'll take a look. Perhaps the regex parsing needs some tweaking. Do you have a link to the item still? EDIT: I found the card on amazon. will test.

As for the price container - amazon currently using span[class*="PriceString"]' as a selector. It has wildcard matching, as sometimes the id changes to something like span id="priceblock_saleprice" while other times it's like what you posted here. The selector seems to have picked the correct container, but regex is not parsing the decimal place correctly now.

Glad you found the card, the scalper has been putting it up and taking it down repeatedly, so it’s not always there with that price.

And yes, I was using the selector as you had it with the wildcard matching. I threw in some log statements and saw that it was grabbing that whole string of CDN$ 1,879.99. I didn’t break apart your one line that did the regex matching and joining after to see exactly where this was occurring.

It’s possible I messed something up, as I more or less manually copied your changes into my locally cloned master, for some reason git wouldn’t let me pull in your forked branch. So maybe I missed something when I was doing that, not sure.

xninjax · 2020-10-02T02:44:14Z

Glad you found the card, the scalper has been putting it up and taking it down repeatedly, so it’s not always there with that price.

And yes, I was using the selector as you had it with the wildcard matching. I threw in some log statements and saw that it was grabbing that whole string of CDN$ 1,879.99. I didn’t break apart your one line that did the regex matching and joining after to see exactly where this was occurring.

It’s possible I messed something up, as I more or less manually copied your changes into my locally cloned master, for some reason git wouldn’t let me pull in your forked branch. So maybe I missed something when I was doing that, not sure.

Ok, found the issue.
After I changed to passing the regex as a variable into the parser (instead of hard-coded), I must have missed re-testing with prices that contain commas (or contains periods in the case of euro format). 🤦‍♂️

By removing the quotations around the regex stored into the priceSeperator variable, everything began to work again.

Should be: const priceSeperator = query.euroFormat ? /\./g : /,/g; <note no quotes around regex

Thank-you @HickTrick for pointing this out - big save!

(incoming commit soon)

Test on the amazon-ca scalper card

Test on Euro laptop EDIT: test on Euro laptop with cents (can never do too much thorough testing! 😄 )

DishBrain · 2020-10-02T02:53:56Z

Ok, found the issue.

(incoming commit soon)

Awesome, glad you found it and figured out the solution! Glad to help!

…ase of euro format)

jef

Very thorough work done here. I really appreciate the time spent and the considerations you made. Thanks for making this project better!

xninjax added 9 commits September 28, 2020 23:00

simple max price checking

b9cf429

- currently one price for all cards per store (should implement price per card) - currently hardcoded into model (should implement in .env). - See newegg-ca for example.

created global MAX_PRICE in .env

0f215c5

- todo: MAX_PRICE per card? - fight with linter, linter won

no message

a0a3b04

Merge branch 'main' into xninjax/price-check

6e5ede8

improved price parsing (can now include text)

87fbb9f

added amazon and amazon-ca container tags

updated .env-example to include MAX_PRICE (currently global, aka appl…

5deaff9

…ies to all cards/stores)

cleanup comments for pr

6121079

updated documentation

82293de

addresses issue #376, documentation, NVIDIA_SESSION_TTL | description…

3f92e81

… should read "The time in milliseconds" ...

xninjax requested a review from jef as a code owner September 29, 2020 20:58

xninjax marked this pull request as draft September 29, 2020 22:04

MAX Price filtering thoroughly implemented

29aee05

- all stores with exception of manufactures direct sales sites - euro price format conversion implemented - new type ‘Pricing’ created - euro test cards links updated for more realistic testing (amazons)

xninjax marked this pull request as ready for review September 30, 2020 03:42

Merge branch 'main' into xninjax/price-check-testing

e88e0fd

jef previously requested changes Sep 30, 2020

View reviewed changes

g1forfun mentioned this pull request Sep 30, 2020

Add method to filter out scalpers #391

Closed

xninjax added 5 commits September 30, 2020 18:47

pre-helper function implementation commit

d4ce8c0

simplified code: - utilizing extractPageContents helper function - utilized ternary operators where possible

max price checking abstracted to new cardPriceLimit helper function

703e9cd

- cardPriceLimit returns 'null' if no price limit or card price ok, otherwise returns actual card price for use in logger.info message

linting

b73f182

Merge remote-tracking branch 'jef/nvidia-snatcher/main' into xninjax/…

76deec2

…price-check-testing # Conflicts: # .env-example # src/store/model/amazon-de.ts # src/store/model/amazon-nl.ts

Merge branch 'xninjax/price-check-abstraction' into xninjax/price-che…

9973437

…ck-testing

xninjax requested a review from jef October 1, 2020 02:24

xninjax added 4 commits September 30, 2020 20:27

re-insterting MAX_PRICE to .env

a451ff1

restaging price selectors on amazon-nl and amazon-de

5a09a2f

updated documentation

documentation cleanup

1a39eb2

moved MAX_PRICE to follow alphabetical convention in .env-example

7d5c138

xninjax added 2 commits October 1, 2020 21:11

fixed regex parse issue on pricing with commas (with periods in the c…

122dde1

…ase of euro format)

Merge branch 'main' into xninjax/price-check-testing

664110d

jef approved these changes Oct 2, 2020

View reviewed changes

jef merged commit fd294d2 into jef:main Oct 2, 2020

jef mentioned this pull request Oct 2, 2020

feat: enhanced lookup behaviour #270

Merged

xninjax deleted the xninjax/price-check-testing branch October 2, 2020 17:46

This was referenced Oct 2, 2020

doc: NVIDIA_SESSION_TTL | description should read "The time in milliseconds" ... #376

Closed

fix(config): MAX_PRICE quotations #426

Merged

psa: MAX_PRICE default in .env-example uses incorrect quotation marks #428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: max price filtering #383

feat: max price filtering #383

xninjax commented Sep 29, 2020 •

edited

Loading

xninjax commented Sep 29, 2020 •

edited

Loading

karmanbadhesha commented Sep 29, 2020

jef left a comment

jef Sep 30, 2020

xninjax Sep 30, 2020

jef Sep 30, 2020

xninjax Oct 1, 2020

jef commented Oct 1, 2020

xninjax commented Oct 1, 2020 •

edited

Loading

DishBrain commented Oct 1, 2020

xninjax commented Oct 2, 2020 •

edited

Loading

DishBrain commented Oct 2, 2020 •

edited

Loading

xninjax commented Oct 2, 2020 •

edited

Loading

DishBrain commented Oct 2, 2020

jef left a comment

feat: max price filtering #383

feat: max price filtering #383

Conversation

xninjax commented Sep 29, 2020 • edited Loading

Description

Testing

Functional demonstration

xninjax commented Sep 29, 2020 • edited Loading

karmanbadhesha commented Sep 29, 2020

jef left a comment

Choose a reason for hiding this comment

jef Sep 30, 2020

Choose a reason for hiding this comment

xninjax Sep 30, 2020

Choose a reason for hiding this comment

jef Sep 30, 2020

Choose a reason for hiding this comment

xninjax Oct 1, 2020

Choose a reason for hiding this comment

jef commented Oct 1, 2020

xninjax commented Oct 1, 2020 • edited Loading

DishBrain commented Oct 1, 2020

xninjax commented Oct 2, 2020 • edited Loading

DishBrain commented Oct 2, 2020 • edited Loading

xninjax commented Oct 2, 2020 • edited Loading

DishBrain commented Oct 2, 2020

jef left a comment

Choose a reason for hiding this comment

xninjax commented Sep 29, 2020 •

edited

Loading

xninjax commented Sep 29, 2020 •

edited

Loading

xninjax commented Oct 1, 2020 •

edited

Loading

xninjax commented Oct 2, 2020 •

edited

Loading

DishBrain commented Oct 2, 2020 •

edited

Loading

xninjax commented Oct 2, 2020 •

edited

Loading