Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gatsby with graphql won't load new data #2847

Closed
Necromant1k opened this issue Nov 8, 2017 · 20 comments
Closed

gatsby with graphql won't load new data #2847

Necromant1k opened this issue Nov 8, 2017 · 20 comments

Comments

@Necromant1k
Copy link

Necromant1k commented Nov 8, 2017

I have an issue - gatsby with graphql won't update data from database ( firebase ).

  1. Server started ( npm run develop ) in my gatsby-node.js data fetched and dynamic pages created.
  2. I'm going to add some rows into my database - added
  3. Reloading page (where i should see new items ) => I see there only old items (no new appear). Same even if i will go to localhost:8000/___graphql There is only old data
  4. Going to close connection and start server again
  5. New data appear
    And i have to do it always to see new data.

Any solutions on this?
Thanks!

gatsby-node.js

const _ = require(`lodash`);
const Promise = require(`bluebird`);
const path = require(`path`);
const slug = require(`slug`);
const slash = require(`slash`);

exports.createPages = ({ boundActionCreators, graphql }) => {
  const { createPage } = boundActionCreators

  return new Promise((resolve, reject) => {
    resolve(
        graphql(
            `
            {
              allImports {
                edges {
                  node {
                    id,
                    opened,
                    unsubscribed,
                    uid,
                    totalCsvLength,
                    clicked,
                    complained,
                    delivered,
                    duplicate,
                    failed,
                    importStartedAt
                  }
                }
              }
            }
        `
        ).then(result => {
          if (result.errors) {
            reject(result.errors)
          }
          const importTemplate = path.resolve(`src/templates/import-detail.js`);
          result.data.allImports.edges.map(edge => {
            createPage({
              path: `/${slug(edge.node.id)}/`,
              component: slash(importTemplate),
              context: {
                id: edge.node.id,
              },
            })
          })
        })
    )
  })
};

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `example`,
  },
  plugins: [
      `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-firebase`,
      options: {
        // point to the firebase private key downloaded
        credential: require("./config/firebase.json"),

        // your firebase database root url
        databaseURL: "https://url.firebaseio.com",

        // you can have multiple "types" that point to different paths
        types: [
          {
            type: "Imports",
            path: "imports",
          }
          ]
      }
    }
  ],
}

@dardub
Copy link
Contributor

dardub commented Nov 8, 2017

While in development mode, the server will watch for changes to files and rebuild/update accordingly.

In production, this is a static site and content is only updated when you do a fresh build and deploy to your host.

There is no way for Gatsby to know that you made changes to your database. Firebase would need to trigger an event to your CI server that you need a new build.

Another approach is to not use Firebase as a source and have it update content dynamically in your app. But then you are losing some of advantages of having static compiled content.

@dardub
Copy link
Contributor

dardub commented Nov 8, 2017

Just to add, I would make sure that using Firebase is necessary. Can you not achieve the same outcome using markdown or yaml? This would make things simple and easy to maintain.

@jtschulz
Copy link
Contributor

jtschulz commented Nov 9, 2017

What if there was an optional flag when restarting the server that allows you to add a query parameter that refetches from your data source? Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks. Use case: our content editors want to preview their entries before publishing.

I would be interested in working on something like this, but could use a little guidance through the code, since I'm new to the project.

@jtschulz
Copy link
Contributor

Any thoughts on this proposed solution? Would gatsby be amenable to one of these or some variation thereof?

@KyleAMathews
Copy link
Contributor

My preferred solution is the last idea here #2271 (comment)

We'd expose a way to run commands while gatsby develop one of which would be to refresh data from sources.

@KyleAMathews
Copy link
Contributor

Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting 🤔

@jtschulz
Copy link
Contributor

Admittedly not the greatest option, but it would be behind an oauth proxy so only users in our organization can access the site. How are other people serving up previews of their site?

@jtschulz
Copy link
Contributor

I like the version of using a source plugin to add new nodes. Was not aware that was a supported use case. Perhaps I could add this as an option for gatsby-source-contentful

@bjrn
Copy link

bjrn commented Nov 16, 2017

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-contentful matches my use case exactly, @PepperTeasdale — can I help out making it happen in any way?

@jtschulz
Copy link
Contributor

Sending you an email @bjrn

@tlvenn
Copy link

tlvenn commented Nov 23, 2017

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-xxxx

This is also my use case with prismic instead of contentful. I dont really understand why we could not start gatsby develop with an additional flag that would force gatsby to refetch its data for the requested page.

Pushing that concern for this particular use case (preview mode) to each individual source plugin does not seem like the right approach.

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting

I dont really see any other way to let authors preview their contents before it's getting published. And for many working with a headless CMS, this is really mandatory for adoption.

There is definitely a legitimate use case to run gatsby on a server in order to provide previews imho and it seems gatsby develop already provide 95% of what is needed.

It could probably be a good idea to introduce gatsby preview rather than passing an additional flag to gatsby develop.

@jtschulz
Copy link
Contributor

I agree with @tlvenn , a gatsby preview server sounds like the ideal solution. It could offer a basic version to allow re-bootstraping before handling requests, taking the responsibility off the plugins. Even better would be to expose an API to receive webhooks from data sources and allow source plugins to update accordingly.

I took a look at the filesystem source plugin as @KyleAMathews recommended and noticed it was using chokidar to watch filesystem changes. At first I thought I could hook into onClientEntry and use EventEmitter to trigger a re-running of sourceNodes, but then realized that browser api and node api wouldn't work together that way. Any suggestions on which parts of the API I need to be looking at to make this work?

@jtschulz
Copy link
Contributor

I guess ignore the latter part of the above comment, I came up with some options. @tlvenn , maybe have a look at #3054 if you'd like to join that conversation

@jtschulz
Copy link
Contributor

jtschulz commented Dec 28, 2017

Well, my solution was working when i was running gatsby locally and using yarn link to include it in my project. Unfortunately the pages are not actually updating now that the changes were merged int. Trying to hunt down why still, but I'm thinking some data must be cached and not updating.

When the endpoint is hit, sourceNodes is called, and the CREATE_PAGES action is dispatched, the updated data is fetched from my source (contentful) and the writePages() function in src/internal-plugins/query-runner/pages-writer.js is being called. Looking at the .cache directory, the .cache/redux-state.json file has been updated, but the data .cache/json/${page-title} has not. If I manually change the latter, the page updates I've noticed. I'm going to keep digging, but if anyone has any ideas on why something would be different when running a local version of gatsby vs. the npm published version or if there have been any code changes in this area in the past couple of weeks that might affect this behavior, please let me know. Thanks!

@jtschulz
Copy link
Contributor

jtschulz commented Jan 6, 2018

I did a git bisect and found that the sourceNodes solution stopped working with 3e2d852e64861956b9c10e46b447782aac9588c0 . @KyleAMathews might be able to give a little context, but it looks like gatsby is running source nodes, but the page query runner has kept the ID of the page in memory in seenIdsWithoutDataDependencies and therefore does not run the query for the page. In order for the existing pages to load new data, they'll need some way to bust the cache here. I would appreciate any advice on how to do this!

@KyleAMathews
Copy link
Contributor

Due to the high volume of issues, we're closing out older ones without recent activity. Please open a new issue if you need help!

@osartun
Copy link
Contributor

osartun commented Oct 30, 2018

Hey guys

This conversation is already quite old, but I'm wondering what the outcome was.

We have setup right now where the editors trigger a webhook whenever they publish a page which starts a build on Netlify.
Problem is: That can take up to several minutes until the site is live. Editors waiting for several minutes before seeing the change they did significantly slows down their workflow.

So, I'm looking into solutions to that problem and also had the idea of running a development server on a web server somewhere and then make use of hot reloading. However, I need to refetch the data from our CMS (neither prismic nor contentful) ideally without stopping and starting the server.

That's how I ended up reading this thread which is probably the closest to this idea. Was the problem solved in Gatsby?
Or if not: How did you guys resolve the issue?

@rahul-nath
Copy link

I sourced my data with gatsby-graphql-source which has a refreshInterval you can input in (value is in seconds). Setting this value will pull from your datasource at that interval and run createPages. Look at the Gatsby docs on the strategies around that -- you can batch requests to make the refresh faster. Better than setting up a build endpoint imo.

@osartun
Copy link
Contributor

osartun commented May 18, 2020

I just randomly found my comment from 1.5 years ago (#2847 (comment)) and given the number of reactions on that comment I want to leave this here: The Gatsby Cloud service is the very idea that I'm proposing in the comment.
So, if anyone else is trying to fix this issue – save the engineering time and get yourself a subscription: https://www.gatsbyjs.com/ (I'm not getting paid to say that, unfortunately)

@AjayDhimanELS
Copy link

Hi guys, Any update on this "refreshing of graphQL content/data"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants