-
Notifications
You must be signed in to change notification settings - Fork 381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Fix issues with the latest API used by Investing.com to pull data #602
base: master
Are you sure you want to change the base?
Conversation
@alvarobartt nice work with the fix! Any ETA on when this may be merged to master and updated in pip? |
I also still get the 403 error. Are you still on it @alvarobartt ? Do you think you can fix it? – Is it working with you, @jmizgajski ? |
When will you fix "get_currency_cross_historical_data"? File "D:\Anaconda3\lib\site-packages\investpy\currency_crosses.py", line 674, in get_currency_cross_historical_data |
Hi everyone! So it was working but it lasted just some requests, then it got blocked, not due to any sort of new Investing.com protection against |
Thanks for trying, @alvarobartt ! |
@alvarobartt |
Have you tested a sleep of 1 to 5 seconds between calls? I have been using that with investpy. Maybe it will not trigger the Cloudflare's WAF. If not possible some quotes are still better than none. Thanks for the work BTW |
So, will investpy become a dead project? |
Thank you for taking a look @alvarobartt - am using investpy for my own analysis and found it awesome! |
I was using this to get data for my thesis, thankfully I had the whole data backed up but now I'll have to rewrite a lot of functions. Thanks for trying to solve this problem and for creating this amazing library in the first place. Hope this somehow gets solved. |
You have created something wonderful. It was soo useful for me to carryout various analysis using investing.com. ( investing.com was the only site provided the data I needed) . I am sincerely hoping that you will be able to find a way around this issue. Hoping for the best. |
@alvarobartt have you tried allowing requests through a paid proxy provider? this could be a workaround for those willing to pay a proxy provider. Many scraping libs do it. Also if you need to simulate the browser maybe something like https://github.com/pyppeteer/pyppeteer could come in handy. This is a puppeteer clone but pure python, installable from pip. |
If this was an option I would be interested.. |
Interesting that this seems to have appeared on Investing Com 6 days ago about not having an API.. https://www.investing-support.com/hc/en-gb/articles/115005473825-Do-you-provide-an-API- |
This guy is using Puppeteer – apparently with some success: |
I just want to thank you for your application. It really feels like Cloudflare will stop at nothing to ruin all the fun on the internet. |
sound like it works using https://github.com/pyppeteer/pyppeteer, need to change api call a bit, but works |
Yeah, it seems to work using Puppeteer, here the changes between the previous version if you need. Also remember to set a user agent or it keeps not working (Cloudflare makes a good job on blocking some traffic). Idk Python much so I can't help much more than this. |
Thanks, i used it to make it work in python, here is the code:
Do you know how the endpoint to pass |
This code fails for me and returns False. trying to return json.loads(content) alone gives the error: |
Hi @sahelanthropussy, I've released P.S. If you have any doubts don't hesitate asking me, and open all the related feature requests and bug reports at https://github.com/alvarobartt/investiny/issues or open a discussion at https;//github.com/alvarobartt/investiny/discussions |
Ah wow thank you alvarobartt... will try it out soon.. |
Sure! Feel free to open an issue at https://github.com/alvarobartt/investiny/issues and I'll try to add it ASAP 👍🏻 |
Cool @RPDev2002, thank you too! |
Good morning @alvarobartt, thank you so much for the investpy code it is a really helpful tool. I has been used the following Investing.com url for a personal finance project inspired by your code:
I have tried to change my code to the:
But I keep getting the ERR:403 as you have mentioned in the different posts. Have you achieved to make it work with the new investing.com API? Thank you so much in advance. PD: In case you haven´t figure out yet I will start trying with the code from investiny |
Hi @alesegura96, as you point out, both Investing.com APIs https://www.investing.com/instruments/HistoricalDataAjax and https://api.investing.com/api/financialdata/historical are not working fine as those block all the incoming requests with HTTP 403... So I'd suggest you and everyone to use Let's see if I can get to contact Investing.com, and get a response from them so as to continue the development of |
Did you try using Puppeteer as we mentioned before? @alvarobartt I think it's the smallest change to make it work, at least it was for my project |
Hi, |
Hi, Thanks for your work on this library. Seems like these issues are popping up because of the differences between the scraping requests and their "legitimate" web requests. Its seems hardcoding a connection on our is part is the problem, since its easy to block a whole group of people using the same connection behavior. Would the most flexible strategy here be to allow for a definable user connection so they can modify their headers, ect to seem passible to the investing.com servers? |
So I don't know how strict is Investing.com protection right now, but even with custom requests objects, you may be blocked at some point, since some people tried to connect from VPNs that never sent any request to Investing.com and those were also blocked... |
Have you ever tried https://github.com/yifeikong/curl_cffi, @alvarobartt ? |
🐛 Bug Fixes
get_stock_recent_data
andget_stock_historical_data
have been fixed and using https://api.investing.com/api/financialdata/historical instead of https://www.investing.com/instruments/HistoricalDataAjax