Enhanced utils.py and webdriver.py #274
Conversation
…mitigating 403 errors. Revamped webdriver.py: Incorporated random timing during login to prevent 429 errors.
|
@dmytrokoren This replaces #268 right? And what's different about it? |
@Bubba8291 No difference, just minor tweaks. |
I haven't pushed the latest changes yet. These changes don't fix 403 yet. I will push latest tomorrow. Please check latest tag not develop6 (it's outdated) thanks @jdholtz |
@jdholtz - please try :latest tag |
|
I've tried this on my machine, a Pi, and an Ubuntu server. They all work well. The only issue I'm running into is I'm occasionally getting the following error. It seems to happen more consistently on the Ubuntu server using Google Chrome instead of Chromium, but I did get it as well on the Pi for one run I saw through screenshots that the popup did not appear, so I tried ignoring it. However, then the headers never were loaded. This PR is very close to fixing the issue, but I'm not exactly sure why this particular issue exists. |
That is awesome news! I will fix this issue tonight and push changes. |
|
@jdholtz check this tag :develop |
I just pulled this tag. However, I'm getting the following error when running it through Docker on my laptop. I think the last strategy worked well (the MY_ACCOUNT_URL), but we just need to figure why it isn't always loading. However, it may be because it is being detected which would mean a new strategy is needed. There is another strategy that may be worth experimenting with which is running SeleniumBase with a virtual display: seleniumbase/SeleniumBase#2390 (comment). I believe I tried this a few months ago and it didn't work, but it's worth trying again in case I didn't do it correctly or a SB update improved it. In my opinion, the best way to fix 403s is to find out why they are happening on headless environments (docker, ubuntu server, etc.), for the most part, but not happening on devices with displays. I've spent a lot of time since the issue started to try and find out why but have not been successful yet. |
That's so weird. It worked on my docker Mac m1. I will need to use js_click that will avoid this error. I'll fix it again tonight :) |
|
I will check to see if we can utilize the virtual desktop but I did start to poke at this but like you said unsuccessful :/ |
I have tried the virtual display (sbvirtualdisplay) it's not working for me, I received 429 error on login. The real issue here is user-agent it's detecting somehow. If we set is_mobile=False in virtual display it will fail fare check every time and login session with 429 error intermittently. Right now is_mobile=True it's working, with proper workarounds. No need virtual display. |
|
@dmytrokoren How often are the errors now with your branch? I told my cousin I would get her a good boarding position on Friday. But I assured her that it's not guaranteed. I am also suitable to run it on a stable residential internet, so no VPS. |
@Bubba8291 try this tag (develop) https://hub.docker.com/r/dmytrokoren/auto-southwest-check-in/ I'm running it without any issues on fare check or scheduled check in. It's been stable for me at least. Give it a go and report back here. |
@dmytrokoren I do not use Docker. If I fetch and pull dmytrokoren/auto-southwest-check-in:master, it's the same commits as your latest Docker develop tag, right? |
If using desktop, I'm not sure if it works on Windows platform. On Mac it works but make sure it does not go to sleep. First, download the script onto your computer git clone https://github.com/dmytrokoren/auto-southwest-check-in.git |
Yep. Thank you. I already have it setup on a headless machine. I just wanted to make sure your docker tag was the same as your latest GitHub branch. |
Yup it is. Keep us posted how it checks in ) |
|
The only issue I was getting was a 403 when it was fetching the reservation information within the checkin scheduler. What fixed it for me (I'm not 100% confident because I don't think I got another failed attempt) is to add I'll probably change it to default to making random delays between requests and turn it off within the checkin handler, but you don't need to worry about doing that for this PR. |
Reverted to original state
I'm glad to help. This is a great project and it is fun to work on. |
|
@dmytrokoren I would like to double check since it's not in the README yet. Which browser is better for your fork using headless Ubuntu Focal (20.04)? Google chrome or Chromium? |
|
@dmytrokoren Good news kind of. My cousin got A46. But it took 10 attempts to be successful. I think she might've gotten lucky since it took 14 extra seconds. |
|
@Bubba8291 A46 is good, considering people pay for A15-A30 early bird. Regarding GET request was attempting multiple times, I think SW api was not responsive at that time as there is no 403 error. At this point there is very little that can be done related to this. My environment just like yours DSM, is working good right now with everything one attempt. I'm yet to try check-in, I have a trip coming up in mid July, I can report back how my check-in went. |
I'm using Chromium headless. How it is setup in docker file, nothing changed. |
Added random_sleep into two areas to avoid bot detection
Added the random_sleep (prev aka is_fare_checker) to _get_reservation_info in checkin_scheduler.
@jdholtz also added random_sleep to make_request in [checkin_handler: line 188] the reason is that @Bubba8291 had some api request issue and it only went through after 10 attempts. btw random_sleep I have made a change instead of (1, 6) it is now (1, 3) so it does not have a high delay time. |
|
@dmytrokoren the linting/formatting workflow is failing. Could you fix those issues before I merge? As a tip, I recommend using pre-commit (details are in the developers doc) so the formatters automatically run and fix it for you. |
Done with pre-commit black....................................................................Passed |
|
@jdholtz @dmytrokoren I've started using this version and it does seem pretty good. One thing I did notice is when trying to log in or check fares for many accounts and/or reservations at once, I get errors again after the first couple. Can we stagger the log ins and fare checks when running the script against many accounts and/or reservations? I understand we can't do that with the actual check ins. |
Yeah, it may work better if there was a stagger between each account/reservation. Because the script sleeps for exactly 24 hours (or whatever retrieval interval is set), this only needs to be implemented when each process is started as the times between each will stay consistent throughout the script's running time. @dmytrokoren I have also been starting to see 403s with just one account and 2 reservations again. Very confused what could've changed... |
@jdholtz I've been running 4 accounts / 0 reservations for 4 days now in docker (synology). Not one single 403/429 error (fingers crossed) - fare checks and header refresh working flawlessly. That's why im confused why its happening to other people?! |
| @@ -87,8 +87,8 @@ def set_up_check_in(arguments: List[str]) -> None: | |||
| logger.error("Invalid arguments. For more information, try '--help'") | |||
| sys.exit(2) | |||
|
|
|||
| logger.debug( | |||
| "Monitoring %d accounts and %d reservations", len(config.accounts), len(config.reservations) | |||
| print( | |||
There was a problem hiding this comment.
Did you mean to use print instead of logger.debug?
There was a problem hiding this comment.
I think it was meant to be print. It's helpful for debugging though so I'll make it a logger.info.
|
@jdholtz is the PR going to be merged? |
|
Yes, just did @dmytrokoren. Thanks! |
|
Tested on:
NOTE: You have to be on your residential IP address (including docker), VPN will throw 403 on fare check!