[API] Stopped getting new updates since last month from getGamesAmerica

[bvanzile]

Hey all,

I've noticed that my getGamesAmerica calls have stopped returning any lastModified values after Sept 14, 2020 (unix timestamp 1600099408951 specifically).

For example, the code below gets all games from getGamesAmerica and pulls out the most recent lastModified value and prints it.

const { getGamesAmerica } = require('nintendo-switch-eshop');

(async () => {
    try {
        const games = await getGamesAmerica();
      
        console.log(games.length);

        var latest = 0;

        for (let i = 0; i < games.length; ++i) {
            if (games[i].lastModified > latest) {
                latest = games[i].lastModified;
            }
        }

        console.log(latest);
    } catch (error) {
        console.error(error);
    }
})();

For me, that returns:

2858                                      // all games I get from the api call
1600099408951                  // latest lastModified timestamp found

I see 24 games with this timestamp for lastModified. Anyone run into an issue like this? I've updated to the latest and, in the process of trying to figure this out, lost some custom changes that was returning ~3800 games, but the issue remains. The 2858 on a fresh install is kind of problematic since the eshop shows 4792 for me, but that's another issue.

Anyone experiencing similar? What does the above code give you guys?

Regards,
Bryan

[favna]

I think I may have found the problem, when comparing the request made by this library to the request that shows up in the network tab under devtools when searching on https://nintendo.com I noticed they use a different indexName as well as a different Algolia Search API key. I'll be updating these things tomorrow and see what I get from your quest afterwards.

I can already say though that trying a simple query with the old indexName and key didn't give me "GONNER 2" (a game released 4 days ago: https://www.youtube.com/watch?v=S3f72l_MIG0) while searching with the new key does.

[favna]

Well I've doing some further digging and it's not necessarily what I thought in my previous comment. There are (at least) 3 indices, ncom_game_en_us and ncom_all_en_us, and noa_aem_game_en_us. We have to use the latter to get even remotely close to all games but despite that index saying there are 6105 results, it doesn't seem to allow fetching past the 998th game (an interesting number because they also do not allow settings a hitsPerPage higher than 1000).

Honestly at this point I just don't know anymore. It's impossibly hard to fetch all games, while it's super easy to fetch one or some games (by adding a query= to the request, something this lib doesn't do because it was originally designed to fetch all games). Personally I would say, think about why you want to fetch all games at all times, and if you can get away with only showing some games (i.e. a limit of 10 as based on some query).

I might just have to remove getGamesAmerica in its current state and instead rework it to return top X (default 200) games matching query Y then release a major version for that.... it really does seem that the days of fetching all games are over.

[bvanzile]

I took a look at the Algolia calls today that Nintendo is making on the US eshop to try to understand it better and I came up with a request that I think does a good job of getting 99+% of the games.

Apparently 1000 is the default limit for amount of records that Algolia will ever return in a query, which seems like is set on db creation. So that is always gonna be limited.
https://www.algolia.com/doc/api-reference/api-parameters/paginationLimitedTo/

But looking at the way the eshop was using Algolia and indexName made it seem like we can search for games in a single query from both ends. Specifically with these two indexNames, ncom_game_en_us_title_asc and ncom_game_en_us_title_des, we can make two requests to get all games by ascending title and all games by descending title, giving an upper limit of 2000 games possible to retrieve in a single query.

If we use the ESRB facetFilters for Everyone (1850 nbHits), E10+ (1125 nbHits), Teen (nbHits 1312), and Mature (468 nbHits), I think we can guarantee retrieving all games in each facetFilter if we make a call for the 1000 games by title in ascending order and the 1000 games by title in descending order (culling the duplicates from overlapping). Plus one extra call for the "Coming Soon" Availability facet for completion (since this probably contains most or all of the games without an ESRB rating).

So,
2 requests for Everyone
2 for Everyone10+
2 for Teen
1 for Mature (since we get all 468 in one ascending title call)
1 for Coming Soon

The request pretty much boils down to the below parameters (hitsPerPage is just at 1000 for now):

var options = {
        method: 'POST',
        uri: 'https://u3b6gr4ua3-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(3.33.0)%3B%20Browser%20(lite)%3B%20JS%20Helper%202.20.1&x-algolia-application-id=U3B6GR4UA3&x-algolia-api-key=c4da8be7fd29f0f5bfa42920b0a99dc7',
        body: {
            requests: [
                {
                    indexName: 'ncom_game_en_us_title_asc',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.everyone}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_des',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.everyone}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_asc',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.everyone10}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_des',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.everyone10}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_asc',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.teen}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_des',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.teen}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_asc',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["esrbRating:${Config.esrb.mature}"],["platform:Nintendo Switch"]]`
                },
                {
                    indexName: 'ncom_game_en_us_title_asc',
                    params: `query=&hitsPerPage=${Config.hitsPerPage}&maxValuesPerFacet=30&page=0&analytics=false&facets=["generalFilters","platform","availability","genres","howToShop","virtualConsole","franchises","priceRange","esrbRating","playerFilters"]&tagFilters=&facetFilters=[["availability:Coming soon"],["platform:Nintendo Switch"]]`
                }
            ]
        },
        json: true
    };

The compiled response:

1000 games from query: Everyone, Title Asc
1000 games from query: Everyone, Title Des
1000 games from query: Everyone10+, Title Asc
1000 games from query: Everyone10+, Title Des
1000 games from query: Teen, Title Asc
1000 games from query: Teen, Title Des
468 games from query: Mature, Title Asc
125 games from query: Availability:Coming Soon + Title Asc
Total found games after removing duplicates:  4619

4619 is pretty close to the 4778 results on the eshop website and from spot checking, some of these games are straight up duplicated in the Algolia call. You can even see duplicated games scrolling down the eshop website.

Anyway, I just wanted to share what I was up to today before I go to bed and hopefully someone finds something useful here. Let me know what you think.

[favna]

If you're still interested in getting all games this way, could you make a pull request with these changes?

[favna]

@bvanzile I have just release v4.0.0 to NPM which includes your fix, as well as a function to get games based on a query as I mentioned in an earlier comment.

[bvanzile]

Thanks for the invite to contribute. Happy that I was helpful.