[gatsby-source-contentful] Gatsby unaware of change in Contentful unless delete cache folder

[hrisng]

Hi, I'm facing a lot of issues that relate directly to the cache folder.

Description

1. If I query a reverse item in one of my query, for example, content type Brand is linked to Category, and I have the following query:

contentfulCategory {
  id
  brand {
    id
  }
}

Gatsby would not aware if I publish a new Brand, and the Category page remain the same. If I query for brand directly instead in brand template, the brand page is generated successfully. When I delete the cache folder, and run again, all query and pages work as expected.

2. If I add a new field to content type Category (that Brand link to) and is queried as above, then run gatsby develop, I would get the error unknown field brand in contentfulCategory. (query for brand from Category also won't work in Graphiql IDE). Again, manually delete the cache folder and run again solve the problem.

3. When I delete a field successfully in Contentful, that field will still be queryable in Gatsby. Delete cache folder also solve this.

Step to reproduce

1. Create content type Category and Brand (or any A and B)
2. In Brand, add a reference to Category
3. Run gatsby develop, it compiles successfully
4. Try each case in the description
5. Run gastby develop, and see the result
6. Delete the cache folder and run again, it work correctly.

Environment

Gatsby version: 1.9.145
Node.js version: 6.9.1
Operating System: Windows 6.1.7601

More Info

I'm using gatsby-plugin-react-next.
Before I found out deleting the cache folder help with the issues. Running gatsby develop multiple times would occasionally solve the problem.
Edit: It worked because I had some change in gatsby-node.js, which triggers the deletion of cache folder.

Request

Also, it would be great if there's a way to tell Gatsby to completely delete the cache folder on new build in the meantime. I didn't find anything in the documentation. Appreciate if you can show me the direction.

[KyleAMathews]

Sorry you're running into troubles and thanks for creating a detailed bug report!

A cache that you delete all the time isn't very helpful :-)

This sounds like a bug perhaps in gatsby-source-contentful where it's not correctly incrementally updating data in Gatsby so Gatsby can't correctly make the new schema.

[hrisng]

I'd love to dig deeper in the package code when I have time to see if there's anything suspicious. Deleting the cache is not desirable for me also. But for now, it's a temporary solution.

The point is we're planning to use Contentful webhook to tell Gastby to automatically rebuild on new content, so that my non-tech teammate can push new content on their own. So yeah, having to manually delete the cache kinda defeat the purpose.

Thanks for quick response and for Gatsby!

[kpollich]

Running into this today as well. Removing .cache isn't resolving for us though (or at least not for me).

Scenario:

1. Create content type in Contentful with Link field
2. Create entry for content type
3. Add single link to entry
4. Run gatsby develop
5. Gatsby generates a GraphQL schema where it assumes since only one Content type appears in the Link field that the Link field is limited to one type
6. Add another entry to the entry's Link field of a different type
7. Contentful GraphQL schema doesn't update on cache removal, rebuild, etc.

In the example here, we have a fullSlot field that contains a single Entry (could be many types), and it's "stuck" as ContentfulPlainTextModuleTest even though we have an entry whose fullSlot contains a different type of Entry.

Full error output

"errors": [
    {
      "message": "Expected value of type \"ContentfulPlainTextModuleTest\" but got: [object Object].",
      "locations": [
        {
          "line": 28,
          "column": 5
        }
      ],
      "path": [
        "allContentfulHomePage",
        "edges",
        0,
        "node",
        "grid",
        0,
        "fullSlot"
      ]
    }
  ],

[KyleAMathews]

Hmmm Gatsby should be creating a Union Type when there's multiple types being linked to:

gatsby/packages/gatsby/src/schema/infer-graphql-type.js

Line 316 in 3945d01

// If there's more than one type, we'll create a union type.

Perhaps something is wrong with the reference maps gatsby-source-contentful creates

gatsby/packages/gatsby-source-contentful/src/normalize.js

Line 89 in 3945d01

exports.buildForeignReferenceMap = ({

[hrisng]

@KyleAMathews I think I find out what went wrong. I'm super new to the project and how it actually works under the hood, so I might be wrong though. So please bear with me.

1. The first problem occured I think because of this:
   

   gatsby/packages/gatsby-source-contentful/src/gatsby-node.js

   Line 134 in 3945d01

   .filter(n => _.includes(newOrUpdatedEntries, n.id))

   
   We are checking the id of existing node against the ids of entries that changed, and in case of creating a new entry it will result in an empty array. So new reverse linkage for old nodes was not getting added.

2. We are using the sync endpoint to get the entryList (which only contain entries that were changed or deleted after the initial build).
   

   gatsby/packages/gatsby-source-contentful/src/fetch.js

   Line 38 in 3945d01

   let currentSyncData

   
   And we are also building foreignReferenceMap base on that list.
   

   gatsby/packages/gatsby-source-contentful/src/gatsby-node.js

   Line 115 in 3945d01

   const foreignReferenceMap = normalize.buildForeignReferenceMap({

   
   So what happened in the second case is that when I add a new field and populate it in Category, and do nothing with the other entries in Brand, only the updated category entries get returned in the entryList. It then attempts to build the ref map, which only contain ref in Category entries. If it has ref in itself to other content type, there's no problem because we do check it against a resolvable list. However, if something else was pointing to it, like in this case, it has no way to know.
   I did a small test to confirm this. If I also update the brand entries along with the categories, the reverse mapping work.

3. The third case I believe also happened because of the sync endpoint. That endpoint only response with changes in entries, no? Old nodes are not aware about the change in data model.

(Sorry for typos and wrong use of words. Not a native speaker, but I'm trying to communicate better)

[KyleAMathews]

Cool! What you're suggesting sounds reasonable — would you like to put together a PR with your fixes?

[hrisng]

@KyleAMathews I'd love to. It might take some time though cause I'm still kinda new to JS. In the meantime, if anyone need to fix this issue asap, please feel free to pick it up.

[ro-savage]

Running into a similar issue. I delete my export const query = graphql and yet I still see the data, I had to manually delete the cache.

[hrisng]

Sorry, I've been quite busy lately and this issue was on low priority for our own project. Gonna put together a fix in the next few days.

[calcsam]

Great, thanks for tackling this!

…

On Sat, Feb 3, 2018 at 10:30 PM, Harris Nguyen ***@***.***> wrote: Sorry, I've been quite busy lately and this issue was on low priority for our own project. Gonna put together a fix in the next few days. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3495 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEL3W66L9SUV9XTsUIv2oLxTfNwtKkaoks5tRU6NgaJpZM4Rb4OQ> .

[alex-saunders]

How's the fix coming along @harrisnguyen94 ? I've ran into the same issue and would be interested to hear how close we are to fixing this

[hrisng]

@alex-saunders hi, updating the reverse mapping for existing nodes (which solve issue 1 is simple and pretty much done. Issue 2 needs some thoughts though, but we have located what went wrong and kinda figure out how to fix it.
Issue 3 however, is a little bit different (making existing nodes aware of change in data model), and maybe will be tackled later.
So yeah, it depends on what issues you're facing. If 1 and 2, I would say we're pretty close. But if it's 3, it may take longer; if this is the case, please feel free to pick it up.

It was Lunar New Year so I was off for the last 2 weeks. Sorry for the delay. Looking forward to seeing these problems solved asap.

[alex-saunders]

Hi @harrisnguyen94 , no worries about the delay, sounds like it's pretty much there anyway! I'm only really needing (1) so happy to hear that the fix should be coming asap.

[hrisng]

@alex-saunders Hi, I need to confirm this. Are you by any chance also using multiple locales on the Contentful side? This info would help me deal with a new found problem.

[rubengarciam]

Hi all. Same issue here but in my case deleting ".cache" doesn't solve it (i.e., moving a content from published to draft)

[hrisng]

@rubengarciam Hmmm. So the contents moved to Draft still show up? Normally deleting the cache should force Gatsby to fetch fresh data. Could you please do a quick test? After moving content to draft, slightly edit that content, and see if the status change is reflected correctly.

[rubengarciam]

Test 1 - post still showing up

1. Post unpublished and saved as draft
2. Restart gatsby dev server

Test 2 - post still showing up

1. Post unpublished and saved as draft
2. rm -rf .cache/
3. Restart gatsby dev server

Test 3 - post still showing up and edit is reflected

1. Post unpublished and saved as draft
2. Edited post in draft mode
3. Restart gatsby dev server

Test 4 - post still showing up and edit is reflected

1. Post unpublished and saved as draft
2. Edited post in draft mode
3. rm -rf .cache/
4. Restart gatsby dev server

---

@harrisnguyen94 So it seems it's fetching the updates to the content correctly, no matter if it has been moved to draft or it's still published. Once the item has been published for the first time looks like any status change is not reflected anymore (unless archived)

[hrisng]

Hmmm. That's new. Thanks for the info @rubengarciam. Just one more thing. Make sure you're using the Content Delivery API instead of Preview API. Other than that, I'll give it a check. I remember hitting a rock last time and almost forget about this.

[rubengarciam]

Yup, all that config is ok. Looking forward to the update! Thanks

[KyleAMathews]

@Khaledgarbaya mentioned there's a contentful api to get all entities that reference a now deleted entity — anyone want to add support for that? (Or I think we can just iterate through Contentful nodes locally and remove references from there which should be faster than hitting the external API) #3054 (comment)

[hrisng]

I think iteration is the way to go. Because we also have problem with removed field still showing up, not just reference. With fresh ContentTypeItems (which is already available), I imagine we can figure out which field need to be removed locally.

I'm afraid I cannot tackle this that (make nodes aware of change in data model) in a timely manner. So yeah, it's great if someone can pick it up.

[nickclicksco]

Having exactly the same issue here.

[audiolion]

I believe I am running into the same issue, I have to delete the cache for related links to appear in gatsby-source-contentful, if I query the field that the link relationship is on it is up-to-date, but the reverse relationship doesn't update.

[janosh]

I'm having a weird issue that might be related where sometimes all my posts fail to load from Contentful. They do show up in GraphiQL but equal null when Gatsby tries to compile. I have yet to indentify which actions exactly trigger the load to fail. But once it has, the only way to recover is to delete the cache and compile from scratch.

[KyleAMathews]

Due to the high volume of issues, we're closing out older ones without recent activity. Please open a new issue if you need help!

[nandorojo]

I'm still having this issue, did anyone figure it out? Publishing edited content on contentful doesn't update locally when running gatsby develop. If I quit out of gatsby develop and run it again, it works with updated contentful data. Is there any way to have the local server listen for changes and auto-update?

[axe312ger]

I'm still having this issue, did anyone figure it out? Publishing edited content on contentful doesn't update locally when running gatsby develop. If I quit out of gatsby develop and run it again, it works with updated contentful data. Is there any way to have the local server listen for changes and auto-update?

No this is not possible, as Contentful somehow would need to contact your local dev environment via web hook.

[nbangba]

I am also experiencing this issue. Gatsby uses the old data even when I make changes in contentful. I am stuck on the project I am
working on

[axe312ger]

@nbangba

I am also experiencing this issue. Gatsby uses the old data even when I make changes in contentful. I am stuck on the project I am
working on

I see possible mistakes you do:

1. You do changes on another environment as your Gatsby site connects to
2. You are using the deliver api, which will only contain your changes when you hit publish

Hope that helps

[nbangba]

The changes are shown in the graphql query but not used in the component. Thanks for the suggestions.

[nbangba]

I had to do Gatsby clean for changes to reflect.

[david-a]

Exporting ENABLE_GATSBY_REFRESH_ENDPOINT=true as environment variable (e.g using dotenv)
AND
Sending curl -X POST http://localhost:8000/__refresh

Usually works for me without cleaning/restarting the dev server.
(the refresh button in the GraphiQL not working for me)