[Enhancement] /stocks/ins Return more than 100 results

[jczernia]

Describe the bug
regardless of limit parameter the command returns max ~50 items

To Reproduce
OpenBB Terminal v1.9.0
/stocks/ins/lip -l 500 - returns max ~50 items.
OpenInsider web page returns full 500 rows table

Desktop (please complete the following information):

• OS: Windows
• Python version 3.11.0

[jczernia]

Same issue in Terminal v2.0.0

[jmaslek]

Okay I have located the issue.

The weblink we scrape defaults to 100 results:

If I change the screener to 500 rows, the link changes to http://openinsider.com/screener?s=&o=&pl=&ph=&ll=&lh=&fd=730&fdr=&td=0&tdr=&fdlyl=&fdlyh=&daysago=&xp=1&vl=&vh=&ocl=&och=&sic1=-1&sicl=100&sich=9999&grp=0&nfl=&nfh=&nil=&nih=&nol=&noh=&v2l=&v2h=&oc2l=&oc2h=&sortcol=0&cnt=500&page=1

So it looks like we can get 100. So why do we only get 40?

This is because we are treating the empty columns in "X" with nans and then dropping them :/

I think I can patch this shortly, but will still be limited to 100 based on the hardcoded link.

[jczernia]

The problem with limit 100 is that it barely cover 1 day. have a look at 28th of Nov almost 90 entries. Assuming you wish to build some history you should run the script few times a day to have high confidence you don't miss something. 500 gives you about buffer about one week

[jmaslek]

So if you are interesred in changing the urls to get more than the 100 on the default page, one would need to replace the dictionary with the edited links that gives more than 100 results (and change the request url),

d_open_insider = {
    "lcb": "latest-cluster-buys",
    "lpsb": "latest-penny-stock-buys",
    "lit": "latest-insider-trading",
    "lip": "insider-purchases",
    "blip": "latest-insider-purchases-25k",
    "blop": "latest-officer-purchases-25k",
    "blcp": "latest-ceo-cfo-purchases-25k",
    "lis": "insider-sales",
    "blis": "latest-insider-sales-100k",
    "blos": "latest-officer-sales-100k",
    "blcs": "latest-ceo-cfo-sales-100k",
    "topt": "top-officer-purchases-of-the-day",
    "toppw": "top-officer-purchases-of-the-week",
    "toppm": "top-officer-purchases-of-the-month",
    "tipt": "top-insider-purchases-of-the-day",
    "tippw": "top-insider-purchases-of-the-week",
    "tippm": "top-insider-purchases-of-the-month",
    "tist": "top-insider-sales-of-the-day",
    "tispw": "top-insider-sales-of-the-week",
    "tispm": "top-insider-sales-of-the-month",
}

If you are interested in doing so, I can assign this to you.

[jczernia]

Python is not my thing, but what I see this line is responsible for query link
link += f"cnt={d_others['MaxResults']}&"

and in theory following max results should be supported
possible_maxresults = [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]

but can't find where the 100 limit comes from

[jmaslek]

Adding ?cnt=500 to the request does not change the number returned. My guess is that the /latest-insider-purchases (and all other links) just links to a specific screener with a predefined cnt=100

[jczernia]

This is the url which returns 500 via browser
http://openinsider.com/screener?s=&o=&pl=&ph=&ll=&lh=&fd=730&fdr=&td=0&tdr=&fdlyl=&fdlyh=&daysago=&xp=1&vl=&vh=&ocl=&och=&sic1=-1&sicl=100&sich=9999&grp=0&nfl=&nfh=&nil=&nih=&nol=&noh=&v2l=&v2h=&oc2l=&oc2h=&sortcol=0&cnt=500&page=1

[jmaslek]

Yup. One would need to get that link for all possible options on that model (dictionary linked above)