🤏🏻 Use `investiny` while we fix `investpy`! #611

alvarobartt · 2022-09-30T15:42:11Z

Hi everyone, I'm Alvaro Bartolome, ML Engineer, open source developer, and the creator of investpy. As many of you may already know, Investing.com which is the data source that investpy uses has recently changed its APIs as well as the protection of those with Cloudflare, which means that the data is not accessible the same way as before. I and other community members are working actively on solving this issue and looking for alternatives to pull the data from Investing.com.

Most of the solutions proposed by the community worked at some point, but are now discontinued or no longer working, but one solution proposed by @ramakrishnamekala129 at #610 seems to be working fine and consistently.

So on, I've decided to develop investiny in the meantime, so that everyone can pull the data from Investing.com but in a simpler and faster way. Also as Investing.com is changing so many things lately, I decided to create a new repository named investiny so as to be easily adaptative to changes. Note that investiny is not a replacement for investpy, just an intermediate solution in the meantime while we solve the current issues with investpy.

I'd like all of you to star investiny and contribute to it, as the code is much simpler and easy to understand, but it obviously has less functionality, but we'll get there at some point. Also showing me support by starring, liking, retweeting, etc. is much appreciated.

Thanks, everyone! 🤗

The text was updated successfully, but these errors were encountered:

alvarobartt · 2022-09-30T15:48:46Z

❓ How to use it?

from investiny import historical)data

data = historical_data(investing_id=6408, from_date="09/01/2022", to_date="10/01/2022")
print(data)

🤔 Q&A

Where does the investing_id parameter come from? -> It's the Investing.com unique identifier for each asset that they have listed in their platform, you can look for it either by inspecting the code of Investing.com, or just checking the ID field in the investpy/resources which are a collection of CSV files scraped from Investing.com and contain some of that inner information.

For example, the value investing_id=6408 used in the example above, is retrieved from united states,Apple,Apple Inc,apple-computer-inc,US0378331005,6408,USD,AAPL in https://raw.githubusercontent.com/alvarobartt/investpy/master/investpy/resources/stocks.csv from the column id.

Why investiny instead of publishing a patch to investpy? -> Since Investing.com is constantly changing things, and currently the investpy code is outdated and not well formatted, I've decided to create investiny to easily develop new features there and let everyone contribute as the code is much simpler, while also improving the size of the package, the requests' latency, etc.

investiny available at https://github.com/alvarobartt/investiny

andresrogers · 2022-10-05T14:26:36Z

Can it be used to get currency crosses ?

alvarobartt · 2022-10-06T07:48:04Z

Hi, @andresrogers yes, it can be used for that, if you want to know how, please open a new Q&A discussion at https://github.com/alvarobartt/investiny/discussions/categories/q-a Thanks 🤗

yanghoonkim · 2022-10-19T08:03:38Z

I found investiny doesn't work too

alvarobartt · 2022-10-19T08:39:22Z

Hi @yanghoonkim it seems that the Investing.com APIs that investiny was using are blocked by Cloudflare too... Let's see if I get a response from Investing.com as I contacted them to get to an agreement / collaboration.

webdevzilla · 2022-10-19T21:03:22Z

@alvarobartt Not quite sure why there's hesitation to use my solution for now.

Here's the API I built specifically for this community: http://api.scraperlink.com/investpy/

Using it in your scripts is just a matter of prefixing your current fetch requests with this: http://api.scraperlink.com/investpy/?email=your@email.com&url=

Example:

http://api.scraperlink.com/investpy/?email=your@email.com&url=https%3A%2F%2Fapi.investing.com%2Fapi%2Ffinancialdata%2Fhistorical%2F7%3Fstart-date%3D2022-09-29%26end-date%3D2022-09-29%26time-frame%3DDaily%26add-missing-rows%3Dfalse

I have also added another option that supports historical data for stocks, cryptos, currencies and bonds:

Basics:
* email= So we can send important updates
* type= historical_data
* products=cryptos,currency_crosses,stocks,bonds
* from_date=m/d/Y
* to_date=m/d/Y
- time_frame=Daily,Monthly,Weekly

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

* - Required

If anyone needs anything else added, it won't take me more than a few minutes to do it.

It works perfectly fine and is better than nothing.

varshasahasrabuddhe · 2022-10-19T21:21:22Z

Hi,
we are using investpy.economic_calendar and that is breaking too

anarchy89 · 2022-10-27T09:08:31Z

@webdevzilla could you add all the categories under the markets button?

Indices
Stocks
Commodities
Cryptocurrency
Currencies
ETFs
Funds
Bonds
Certificates

patrikot · 2022-10-30T06:57:01Z

@webdevzilla yes, please open up for indices, currencies and commodities, as well.

superjunk1 · 2022-11-07T18:21:40Z

This works fine for me with currencies and stocks. But I can't figure out how to make it work for funds. Is this category available too?

alvarobartt · 2022-11-08T08:15:12Z

@webdevzilla can you migrate all those discussions/issues related to your project to a repository under your GitHub? Thanks

webdevzilla · 2022-11-10T02:09:22Z

@webdevzilla could you add all the categories under the markets button?

Indices Stocks Commodities Cryptocurrency Currencies ETFs Funds Bonds Certificates

@webdevzilla yes, please open up for indices, currencies and commodities, as well.

This works fine for me with currencies and stocks. But I can't figure out how to make it work for funds. Is this category available too?

Yep, it's all added: http://api.scraperlink.com/investpy/

Here's the sample APIs:

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

commodities:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=commodities&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&name=gold

etfs:
- country=
* id=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=etfs&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&country=united%20states&symbol=VTI

funds:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=funds&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=VTSAX

indices:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=indices&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=SPX

certificates:
- country=
* id=, symbol=, or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=certificates&from_date=2022-11-08&to_date=2022-11-09&time_frame=Daily&symbol=DEAA01WG

* - Required

@webdevzilla can you migrate all those discussions/issues related to your project to a repository under your GitHub? Thanks

This was only meant to serve as a temporary solution. It wasn't my intention to create a competing repository.

If you feel that's appropriate, I can put one together.

elricStormBringer · 2022-11-17T03:30:45Z

@alvarobartt Not quite sure why there's hesitation to use my solution for now.

Here's the API I built specifically for this community: http://api.scraperlink.com/investpy/

Using it in your scripts is just a matter of prefixing your current fetch requests with this: http://api.scraperlink.com/investpy/?email=your@email.com&url=

Example:

http://api.scraperlink.com/investpy/?email=your@email.com&url=https%3A%2F%2Fapi.investing.com%2Fapi%2Ffinancialdata%2Fhistorical%2F7%3Fstart-date%3D2022-09-29%26end-date%3D2022-09-29%26time-frame%3DDaily%26add-missing-rows%3Dfalse

I have also added another option that supports historical data for stocks, cryptos, currencies and bonds:
Basics:
* email= So we can send important updates
* type= historical_data
* products=cryptos,currency_crosses,stocks,bonds
* from_date=m/d/Y
* to_date=m/d/Y
- time_frame=Daily,Monthly,Weekly

stocks:
- country= (if not using 'id')
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=stocks&country=united%20states&symbol=TSLA&from_date=09/27/2022&to_date=09/28/2022

cryptos:
* id=, symbol= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=cryptos&symbol=BTC&from_date=09/27/2022&to_date=09/28/2022

currency_crosses:
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=currency_crosses&name=USD/CAD&from_date=09/27/2022&to_date=09/28/2022

bonds:
- country= (if not using 'id')
* id= or name=
Example: http://api.scraperlink.com/investpy/?email=your@email.com&type=historical_data&product=bonds&from_date=01/01/2022&to_date=09/28/2022&time_frame=Monthly&country=india&name=India%2010Y

* - Required 
If anyone needs anything else added, it won't take me more than a few minutes to do it.

It works perfectly fine and is better than nothing.

hi @alvarobartt / @webdevzilla , I came to know of this great python module 'investiny' and 'investpy' today, from StackOverflow.com.

I tried using it, and keeps getting the dreaded '403' error even though I am a new user. Is this APIscraperLink the only current fix?

I have been using python for more than 4 years, but my programming is more in modelling, rather than IT infrastructure coding...

mscharstrom · 2022-11-28T17:14:12Z

@alvarobartt any updates from investing.com? I'm mostly interested in the economic calendar. Does your workaround fix that @webdevzilla ?

Thanks in advance.

vincentli2023 · 2022-12-06T03:39:32Z

@webdevzilla I appreciate it! Do you think your solution will still function for another few months in the future? Cause I am worried that it will be blocked again by investing.com and I have to change my code infrastructure once again....

webdevzilla · 2022-12-06T06:51:41Z

@alvarobartt any updates from investing.com? I'm mostly interested in the economic calendar. Does your workaround fix that @webdevzilla ?

Thanks in advance.

Sure, I'll add it. Please email me so that I can update you once added.

@webdevzilla I appreciate it! Do you think your solution will still function for another few months in the future? Cause I am worried that it will be blocked again by investing.com and I have to change my code infrastructure once again....

Shouldn't have any problems.

I am processing millions of queries without issues.

vincentli2023 · 2022-12-06T08:17:10Z

@webdevzilla I think there might be an issue with bond API endpoints, as the response can't be converted to json. The response text of bond API is in html format instead of dictionary/json format

webdevzilla · 2022-12-06T18:20:57Z

Yes, ScraperLink only captures the raw response. It doesn't parse the data. This repository probably has the code for parsing the HTML response into JSON. I can also add parsing to JSON directly, but I'm super bogged down right now. I might need a few days to get to it. If you email me, I can update you once it's added.

anarchy89 · 2022-12-07T07:31:16Z

Yes, ScraperLink only captures the raw response. It doesn't parse the data. This repository probably has the code for parsing the HTML response into JSON. I can also add parsing to JSON directly, but I'm super bogged down right now. I might need a few days to get to it. If you email me, I can update you once it's added.

Hi @webdevzilla could you explain how your solution works and share the code for your website and how you implemented it im really curious as to how you did it. It's a really good solution.

webdevzilla · 2022-12-07T07:51:33Z

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

vincentli2023 · 2022-12-07T10:48:23Z

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

@webdevzilla thank you for providing this alternative solution! Really appreciate it!!

anarchy89 · 2022-12-07T10:50:19Z

It requires a massive pool of IP addresses. Not sure if I should say more. 🤐

I understand but how do you get around cloudflare haha

webdevzilla · 2022-12-07T20:53:45Z

I don't get Cloudflare blocks when using my proxies.

ramakrishnamekala129 · 2022-12-08T04:06:53Z

if u send us your proxies solution may be we can investiny or investpy

…

On Thu, Dec 8, 2022 at 2:23 AM webdevzilla ***@***.***> wrote: I don't get Cloudflare blocks when using my proxies. — Reply to this email directly, view it on GitHub <#611 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHEE5RBA35QIWAWC4JAHU3LWMD2OHANCNFSM6AAAAAAQZ4NRQY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

webdevzilla · 2022-12-08T04:14:06Z

There wouldn't be a way for me to control where they're used. My proxy pool would get abused to oblivion.

The solution I've provided is 100% free and I can presumably make it work forever. In fact, I will probably add backups to it as well, so that even if Investing.com no longer exists, the API will continue working seamlessly.

I need this for my own purposes so there's no way I would let it die.

anarchy89 · 2022-12-08T09:11:52Z

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

webdevzilla · 2022-12-08T09:27:33Z

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

Yes.

anarchy89 · 2022-12-08T10:18:12Z

@webdevzilla so you funnel the requests through your proxies and that's why cloudflare allows the requests? I don't want access to your pools, just wanna understand how it's technically done.

Yes.

Is it something like this? https://www.bestproxyreviews.com/how-to-bypass-cloudflare/

webdevzilla · 2022-12-08T17:15:14Z

I have my own pool of proxies. If you want access it, we can work something out.

vincentli2023 · 2022-12-09T01:57:48Z

@webdevzilla I can't run bonds API endpoint this morning, I also tried the example you gave but no luck, the error message is below:

<title>Access denied | www.investing.com used Cloudflare to restrict access</title> <script>

webdevzilla · 2022-12-09T02:07:50Z

@webdevzilla I can't run bonds API endpoint this morning, I also tried the example you gave but no luck, the error message is below:
<title>Access denied | www.investing.com used Cloudflare to restrict access</title> <script>

Fixed. :)

mscharstrom · 2022-12-09T08:02:56Z

I was able to get what I needed from investing.com webtools since I pushed my data from investpy to my own local website (flask application). Maybe it helps someone else. It isn't as clean as my original solution, but it gives the data I need.

https://www.investing.com/webmaster-tools/

anarchy89 · 2022-12-11T06:11:00Z

@webdevzilla I can't run bonds API endpoint this morning, I also tried the example you gave but no luck, the error message is below:

<title>Access denied | www.investing.com used Cloudflare to restrict access</title> <script>

Fixed. :)

Is there anyway you can create an api for scraping the news headlines and content?

webdevzilla · 2022-12-11T06:49:56Z

@webdevzilla I can't run bonds API endpoint this morning, I also tried the example you gave but no luck, the error message is below:

<title>Access denied | www.investing.com used Cloudflare to restrict access</title> <script>

Fixed. :)

Is there anyway you can create an api for scraping the news headlines and content?

You can use Option #1 to scrape anything: http://api.scraperlink.com/investpy/

alvarobartt added the information label Sep 30, 2022

alvarobartt self-assigned this Sep 30, 2022

alvarobartt pinned this issue Sep 30, 2022

Repository owner deleted a comment from webdevzilla Oct 1, 2022

This was referenced Oct 2, 2022

ConnectionError: ERR#0015: error 403, try again later. #600

Open

ConnectionError: ERR#0015: error 403, try again later. #613

Open

ConnectionError: ERR#0015: error 503, try again later. #440

Open

alvarobartt unpinned this issue Oct 20, 2022

hjoaquim mentioned this issue Oct 21, 2022

[Bug] Integration test failing : test_economy.openbb OpenBB-finance/OpenBB#2971

Closed

gbonariva mentioned this issue Oct 24, 2022

Possible way to fix 403 Error alvarobartt/investiny#62

Open

Repository owner locked and limited conversation to collaborators Dec 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🤏🏻 Use `investiny` while we fix `investpy`! #611

🤏🏻 Use `investiny` while we fix `investpy`! #611

alvarobartt commented Sep 30, 2022

alvarobartt commented Sep 30, 2022 •

edited

Loading

andresrogers commented Oct 5, 2022

alvarobartt commented Oct 6, 2022 •

edited

Loading

yanghoonkim commented Oct 19, 2022

alvarobartt commented Oct 19, 2022

webdevzilla commented Oct 19, 2022

varshasahasrabuddhe commented Oct 19, 2022

anarchy89 commented Oct 27, 2022

patrikot commented Oct 30, 2022

superjunk1 commented Nov 7, 2022

alvarobartt commented Nov 8, 2022

webdevzilla commented Nov 10, 2022

elricStormBringer commented Nov 17, 2022

mscharstrom commented Nov 28, 2022

vincentli2023 commented Dec 6, 2022

webdevzilla commented Dec 6, 2022 •

edited

Loading

vincentli2023 commented Dec 6, 2022 •

edited

Loading

webdevzilla commented Dec 6, 2022 •

edited

Loading

anarchy89 commented Dec 7, 2022

webdevzilla commented Dec 7, 2022

vincentli2023 commented Dec 7, 2022

anarchy89 commented Dec 7, 2022

webdevzilla commented Dec 7, 2022

ramakrishnamekala129 commented Dec 8, 2022 via email

webdevzilla commented Dec 8, 2022

anarchy89 commented Dec 8, 2022 •

edited

Loading

webdevzilla commented Dec 8, 2022

anarchy89 commented Dec 8, 2022

webdevzilla commented Dec 8, 2022

vincentli2023 commented Dec 9, 2022

webdevzilla commented Dec 9, 2022

mscharstrom commented Dec 9, 2022

anarchy89 commented Dec 11, 2022

webdevzilla commented Dec 11, 2022

🤏🏻 Use investiny while we fix investpy! #611

🤏🏻 Use investiny while we fix investpy! #611

Comments

alvarobartt commented Sep 30, 2022

alvarobartt commented Sep 30, 2022 • edited Loading

❓ How to use it?

🤔 Q&A

andresrogers commented Oct 5, 2022

alvarobartt commented Oct 6, 2022 • edited Loading

yanghoonkim commented Oct 19, 2022

alvarobartt commented Oct 19, 2022

webdevzilla commented Oct 19, 2022

varshasahasrabuddhe commented Oct 19, 2022

anarchy89 commented Oct 27, 2022

patrikot commented Oct 30, 2022

superjunk1 commented Nov 7, 2022

alvarobartt commented Nov 8, 2022

webdevzilla commented Nov 10, 2022

elricStormBringer commented Nov 17, 2022

mscharstrom commented Nov 28, 2022

vincentli2023 commented Dec 6, 2022

webdevzilla commented Dec 6, 2022 • edited Loading

vincentli2023 commented Dec 6, 2022 • edited Loading

webdevzilla commented Dec 6, 2022 • edited Loading

anarchy89 commented Dec 7, 2022

webdevzilla commented Dec 7, 2022

vincentli2023 commented Dec 7, 2022

anarchy89 commented Dec 7, 2022

webdevzilla commented Dec 7, 2022

ramakrishnamekala129 commented Dec 8, 2022 via email

webdevzilla commented Dec 8, 2022

anarchy89 commented Dec 8, 2022 • edited Loading

webdevzilla commented Dec 8, 2022

anarchy89 commented Dec 8, 2022

webdevzilla commented Dec 8, 2022

vincentli2023 commented Dec 9, 2022

webdevzilla commented Dec 9, 2022

mscharstrom commented Dec 9, 2022

anarchy89 commented Dec 11, 2022

webdevzilla commented Dec 11, 2022

🤏🏻 Use `investiny` while we fix `investpy`! #611

🤏🏻 Use `investiny` while we fix `investpy`! #611

alvarobartt commented Sep 30, 2022 •

edited

Loading

alvarobartt commented Oct 6, 2022 •

edited

Loading

webdevzilla commented Dec 6, 2022 •

edited

Loading

vincentli2023 commented Dec 6, 2022 •

edited

Loading

webdevzilla commented Dec 6, 2022 •

edited

Loading

anarchy89 commented Dec 8, 2022 •

edited

Loading