Provide API to integrate with custom servers

[jlongster]

The main problem I have so far is that I have my own node server for the backend. I could run both that and the create-react-app server, but I think we've learned from webpack's dev server that it's not ideal to have to manage 2 processes.

Would you be open to exposing an API for this project in addition to the CLI? One option is for it to take an express server and add all the url handlers, but then you're binding yourself down to express which you probably don't want. Of course, if you moved away from express it'd be impossible to integrate with existing express servers anyway.

Isn't there a way for node's low-level http servers to be composed? If so, you could make sure that requests go through the dev server first, and then through the user's server. Theoretically that means the types of server's don't have to match (the dev server could be using express, the user's hapi).

[lacker]

What exactly is the problem with managing 2 processes? If you aren't using node on the backend, you'll have to have the dev server and your backend server as two processes anyway. And your database is probably a separate process already so you likely already are managing multiple processes for your development environment.

If it's just annoying to run multiple processes when you want to start development, I would suggest writing some sort of script around both of them rather than integrating the dev server into your backend node server directly - it just seems much less bug prone.

[gaearon]

Most issues I saw with two separate processes were related to server rendering. Since this project doesn't support it I'm not sure I fully understand the problem but I'd like to learn more.

That said we should definitely make it as easy as possible to talk to another local or remote server. If there's something we're doing wrong (like CORS issues on our side? Somebody mentioned that in eject survey) we should fix that.

[jlongster]

If you aren't using node on the backend, you'll have to have the dev server and your backend server as two processes anyway.

Sure, but using node is a hugely common scenario and I think it's worth at least thinking about.

And your database is probably a separate process already so you likely already are managing multiple processes for your development environment.

That's extremely different, if you're using an existing database it's not just a separate process but a whole different system that is not stopped when you shutdown or started when you start the server.

There are several small pain points: the URL of the bundle must include the full URL (to point to a different port), and if I want to look at the bundle that's loading it's more annoying that just going to "/bundle.js". More problematic is that multiple processes are always harder over time: deciding what to do when one crashes, making sure they all die when you quit, etc.

I just looked at the code, and I didn't realize all this was doing was using WebpackDevServer. It would be a significant change to get this working; you'd have a small express server and use webpack-{dev/hot}-middleware. But the integration point would be pretty simple:

if(DEV) {
  createReactApp.wrap(http.createServer(myApp)).listen(4000);
}

It takes a standard node server and returns another one that will intercept requests. It also allows some configuration like listening on a different port without actual configuration files.

[gaearon]

@jlongster

I still don’t quite understand the pain points. Would it be too much for me to ask you to create a sample mini-project using a separate Express instance to guide me through the pain points on a specific example?

[jlongster]

That's a perfectly reasonable request, and I'll try to do that soon (will be traveling for the next 3 weeks so not sure how much time I'll have). I think some of this is just from experience though and doesn't easily boil down to specific examples. Having the app be an atomic process just makes the experience "nicer".

I don't see why we should make it harder to integrate this project for various setups. What if I am doing server-side rendering? With the above API, I could still take advantage of all of the goodness from this project. The API does not violate the philosophy of it, I think, and avoid kludges like #85. I really want to adapt this and just not care about how it works, but without adding more complexity of handling multiple processes and making sure to load the bundle from http://localhost:3000/bundle.js instead of just /bundle.js in dev mode (and not in prod; right now my index.html it static, doesn't event use a templating engine, so it's just annoying).

EDIT: After typing that out, there are 2 pretty good pain points right there (let me control the port and don't worry about port clashing, and forcing me to dynamically switch out the bundle.js URL for dev/prod which means I can't use a static index.html).

[lacker]

In a typical setup with a backend I think you would still want both http://localhost:3000/ and http://localhost:3000/bundle.js served from npm start. You would want some API to hit your backend rather than the dev server, though. The way I would want to set this up is to have some environment variable like "API_SERVER" that is different between development and production. In development the API_SERVER gets set to localhost:4000 and you run your node or rails or whatever process on localhost:4000. In production the API_SERVER gets set to api.yourserver.com, or you could have a nginx/apache layer that decides whether to serve a static file or not. I think currently you can use process.env.NODE_ENV for this; if that's a bit janky maybe we will need a better way to do dev vs prod environment variables.

That's how I'd configure a mini-project that used a backend. If you're doing server-side rendering then you are probably just going to want to eject though. I think the right solutions for server-side rendering and the right solutions for general backends are going to be different, so it's useful to distinguish those. IMO this project should be able to handle deployment alongside a backend server without too much trouble; handling server-side rendering is going to be trickier.

[gaearon]

What if I am doing server-side rendering?

Then it's going to be quite more complicated because you'd have to compile everything with Webpack for the server. I know it's doable but it's not super straightforward, and quite error prone in my experience. This is why we don't support this use case in this setup in principle. (Some other similar projects do, so maybe they will fit your use case better.)

I think the right solutions for server-side rendering and the right solutions for general backends are going to be different, so it's useful to distinguish those.

Agreed.

[jlongster]

Server-side rendering is a distracting feature to talk about. Especially since you don't usually use it in dev anyway. I should have used a better example.

The way I would want to set this up is to have some environment variable like "API_SERVER" that is different...

Sure, just like I would setup my webpack config, tweak the entry points, etc... My point is that this project has the chance to also give a lot of people that want this integrated a very smooth experience for setting up React. I don't actually need any of the stuff you mentioned, I would have to do all of that just to work with this.

(Some other similar projects do, so maybe they will fit your use case better.)

I definitely agree about the server rendering part. But I hope that this project grows to be a little more adaptable. The draw of ember-cli is that's it's what most people use, and I know that this scope is far smaller, but I bet it'll grow over time.

On the contrary for me giving small pain points, I haven't heard any drawbacks to the API I proposed above. What specifically is bad about it, or how will is constrain you? As far as I can see, it's future-proof because you'll always have a node dev server.

I don't feel like dragging much longer though.

[tomkis]

Discussion might be relevant to #172

[jlongster]

Discussion might be relevant to #172

I think it's not that relevant because that's mostly a speedup for production and is probably part of the build step for production, but if it helps, good.

[gaearon]

I think the reason this discussion is not productive so far is because it is not obvious to us why having a separate process is painful if you don’t support server rendering. Is it just because making fetch() calls at /api is simpler than [host:port]/api? Are there other problems?

[tomkis]

@jlongster well my point rather was that I do not mind having two separate processes for API server & static assets server (might be potentially nginx / express / http-server) where it would make sense to have some lightweight development proxy to keep everything transparent (so that user does not need to hassle with CORS).

[jlongster]

@gaearon I've mentioned several so I don't think that's the only reason this hasn't been productive. What you mentioned is one, and having index.html dynamically figure out where to serve the bundle, etc. are all configuration annoyances that could just go away. And all with the simplest integration point imaginable, which also allows advanced users to potentially do other things themselves. You may not hit these pain points, which makes it hard to relate, but that doesn't mean they don't exist.

[jlongster]

I don't think is going to go anywhere, so I'll pre-emptively close it. I was hoping to be able to integrate this and contribute back but not sure how to integrate it with my current project yet. Maybe sometime in the future.

[gaearon]

Sorry, I think I might have not made my points clear.

I’m not saying they don’t exist, I’m just asking you to be more descriptive 😉 . For example, I don’t really understand what

having index.html dynamically figure out where to serve the bundle

means.

I’m asking you to help me understand it by providing a step-by-step scenario that explains what is the pain point. The descriptions I saw so far seemed related to server rendering. I want to understand the problems you encounter without it, but I need your help.

I’m reopening because those problems are probably valid. I’m asking you to help me understand them by being as descriptive as possible; not shutting you down.

[gaearon]

Maybe a step-by-step description would help. For example:

1. I have an API server at <...> port
2. I’m running npm start
3. Now I have this problem: <...>
4. I can work it around by <...>, but this poses another problem: <...>

[gaearon]

(And, to be clear, I don’t have experience with building Node apps. Single-sentence descriptions of problems may be obvious to you, but I really ask you to go into details. They’re not obvious to me at all.)

[jlongster]

Ok, sorry. I subconsciously assumed that others would relate to these problems as well. But note that some preferences come from general experience and hard to convey like this. I've enjoyed working with simpler dev servers in ways that I can't really explain if others don't have the same experiences.

One problem I can detail is this:

1. I have a server running on port 4000, and it provides an API and serves an index.html.
2. I run react-scripts start which listens on port 3000
3. Going to http://localhost:4000/index.html can't find the bundle because it's trying to load like this: <script src="/bundle.js"></script>, which resolves to http://localhost:4000/bundle.js.
4. The bundle is served from http://localhost:3000/bundle.js in dev, so it needs to load it from there. But I can't hardcode that URL, because it needs to be /bundle.js in prod. So it needs to dynamically check for the dev environment and use a templating system, like {% if DEV %}http://localhost:3000/bundle.js{% else %}/bundle.js{% endif %}.

Now, I could just load index.html from the webpack dev server. Then it would go like this:

1. API server is on port 4000
2. dev server listening on port 3000
3. Now I go to http://localhost:3000/index.html. The relative reference to the bundle /bundle.js works fine.
4. However, my code posts to the API with relative URLs like /get-items, which is now resolving to http://localhost:3000/get-items. So all my API calls aren't working.
5. I need to provide a constant API_SERVER that I need to require everywhere I make an API call, and do API_SERVER + /get-items. Or I could wrap all my calls in a function that automatically prefix that URL.

But I don't need that complexity. My site will never be big enough to require the ability to separate out the API server, so I just don't need to do that. I also don't need the complexity of using a templating system to make index.html dynamic.

So, sure, I could solve this either way. But it's additional complexity that I don't really feel like dealing with.

I'm fine if you all choose not to do this. It's your project, and you should execute your vision, and I fully respect that. If it's important enough to me to reap the benefits of it, eventually I'll try to figure this out. I would prefer the ability to simply integrate it though.

[gaearon]

Thanks, this is super helpful!

Now, I could just load index.html from the webpack dev server.

This is certainly the workflow I had in mind, not the other way around.

Would the ability to proxy some requests to an arbitrary server satisfy you? For example you could always use /api/whatever in code, you could always use http://localhost:3000 in dev, but you would set up http://localhost:300/api/* to go to your Node app. In production, this wouldn’t have any effect, so they’d run on the same server/port.

[jlongster]

@gaearon That certainly solves my biggest complaint. Might be a larger surface area for errors, but it would work.

Does webpack dev server already support a proxy? What is this? https://github.com/webpack/webpack-dev-server/blob/master/lib/Server.js#L121 If that's already implemented and been tested, that does seem like a good solution.

[gaearon]

Yea. Unfortunately it seems to be the usual variety of options: http://webpack.github.io/docs/webpack-dev-server.html#proxy. If we could figure out a good subset of this configuration that would work for 95% of use cases, we might consider adding support for it.

[jlongster]

proxy: {
  '/api/*': {
    target: 'http://localhost:4000',
    secure: false
  }
}

That's good enough for me.

[acco]

I like the proxy idea! In addition to the pain point that @jlongster highlighted, it removes the need for developers to both identify and fix an inevitable CORS issue in development.

If it's helpful for those following this thread to visualize an implementation, I whipped one up. I still need to add switches for dev vs prod, but it paints the picture.

[gaearon]

Got it. Thanks for comments. We won't add this yet, but we'll consider this use case together with the others, and maybe add something like this in the future.

[lacker]

Another suggestion: you could try having a separate proxy server, which runs on say localhost:2000, proxies /index.html and /bundle.js to localhost:3000 and proxies everything else to localhost:4000. (And also proxies whatever websocket thing is happening for the reloading to localhost:3000.) That might sound like a pain to write, but, it could be a totally separate module without changing anything about create-react-app. If someone built such a react-app-proxy then anyone could use it by invoking a one-liner in your node server. It has no CORS problems. It's similar to how you'd be deploying with nginx or apache in production. And then it wouldn't require putting any extra dependencies in create-react-app itself, which lets us keep this minimal.

[jlongster]

Managing my own wepack config sounds simpler than that. The amount of changes needed for either my proposed solution or the proxy pales in comparison to some of the other features this project is taking on. There's no reason this shouldn't just solve it.

[gaearon]

If I understand correctly some developers want to use their own custom server, wouldn't using this help out with the problem as well as make way for server-side rendering since they get to use their own server?

IMO this would make it too hard for us to change things. I think we want to be in control of the dev server.

make way for server-side rendering since they get to use their own server?

Using own server is the least of the problems related to server rendering. Ensuring bundling mechanism is the same, language features are transpiled (but in an efficient way), suggesting a good data fetching solution, etc, are all much harder, so this is currently a non-goal of this project.

[kennetpostigo]

@gaearon thanks for clearing things up

[insin]

How about also providing an Express middleware people can plug into their own server? I've used this approach to hook nwb into my projects at work, which has worked pretty well.

• You would need a separate Webpack config for use with webpack-hot-middleware (which would also need to polyfill EventSource for IE)
• Build status reporting would have to be different - is there a standard way to hook into whatever logging the user has set up? I don't know.

var assert = require('assert');
var webpack = require('webpack');
var config = require('./config/webpack.config.middleware');

module.exports = function(express) {
  assert(
    express && typeof express.Router === 'function',
    'The express module must be passed to react-scripts middleware'
  );

  var compiler = webpack(config);
  var router = express.Router();

  router.use(require('webpack-dev-middleware')(compiler, {
    noInfo: true,
    publicPath: webpackConfig.output.publicPath,
    quiet: true,
    watchOptions: {
      ignored: /node_modules/
    }
  }));

  router.use(require('webpack-hot-middleware')(compiler, {
    log: false
  }));

  return router;
};

[jlongster]

IMO this would make it too hard for us to change things. I think we want to be in control of the dev server.

My original proposal allows you to be in control. The dev server would sit "on top" of the backend server by composing the 2 servers into one. I should probably make a proof of concept of this. Thank you for making one with the proxy, I will take a look soon.

[gaearon]

@jlongster Please let me know if the proxy solution works well for you. I’m inclined to get it in because I don’t see any major issues and this seems like the easiest least powerful way to support your use case.

[cloudmu]

My backend is Java-based rest services (yes I am one of those poor souls who juggles between java and javascript on a daily-basis). My webpack dev server would proxy api calls to the Java back-end:

// proxy for restful services
const proxy = httpProxy.createProxyServer();
app.all('/api/*', function (req, res) {
    proxy.web(req, res, {
        target: 'http://localhost:8080/api/'
    });
});

So the proxy solution discussed here would work great for my use case. Thanks.

[jlongster]

@gaearon I can't seem to get that PR to work, it's something about the mayProxy regex. It's failing for URLs that should work. If trim it down to something simple like /(?!(\/index\.html$))/ it works, so I'm not sure why it's failing to match all other URLs. I looked at it for a few minutes and couldn't figure it out.

However, this approach works for me. I'm fine if you want to merge that in.

[gaearon]

It's failing for URLs that should work.

Can you give any examples?
I wanted to release this tomorrow but I can’t guess which URLs were failing. 😄

[jlongster]

Can you give any examples?

Sorry, I've been on PTO! I couldn't get any URL to proxy, it would always be served by the webpack dev server. I don't know why, it must have been my setup, and I would go ahead and release it if it works for you. I'll try it out again soon and file new bugs as I find them!

[gaearon]

@jlongster Maybe you were requesting with text/html in accept header? That's the heuristic we used.

[jlongster]

@gaearon Oh, that's definitely it, because I set up a test case that wasn't even using ajax but was hitting various URLs directly in the browser. The downside of not allowing that is I can't "copy URL" from devtools and view the requests directly in the browser. Why not just proxy all URLs not in the whitelist, or all under /api/*, etc?

[gaearon]

Why not just proxy all URLs not in the whitelist

How do we know if something is an API route or a client-side pushState route? We want /todos/42 to fall back to index.html because that’s what most apps want.

or all under /api/*

We didn’t want to be prescriptive about prefixes outside the React app (which your backend is). However /api/ might be a nice stronger heuristic for disabling the HTML5 fallback, so we could use that.

[jlongster]

I don't feel too opinionated at this point, but I see that you still want control over the static files. If using the proxy, I'd say that it's unlikely I'll be dealing much with static files or care about the index.html resolution because the routing will be client-side and using an API server. But I don't have a strong case.

/api/ could work but I'd worry that it get confusing to have multiple ways of working (i.e. the way it currently works but also if you use /api/ you can view GET requests in your browser, otherwise not). Not sure. Maybe just keep doing what it's doing now and see how it feels.

[kkwezard]

I got the similar problem. I have a REST service that serves at 8001 port and open the dev server at port 3000. The ajax request sent from my react app, like /api/posts/1 can't reach the back-end API server since it will be route to 3000 port.
One solution I used is compile the app into production release and serve it under nginx proxy and then also make the REST server behind the proxy. Therefore I can access the app and REST server via the same origin, like 80 port. But this solution loses the capability of debugging and hot reloading and is inconvenience for development.
Another solution is to rewrite all cross-domain ajax requests with JSONP tricks. However, this hard-coding solution doesn't meet the requirement of production release.
I think enabling the proxy feature in dev-server is the best solution here. I tried to modify the webpack.config.dev.js in react-scripts to enable proxy but failed.
I wonder if there is any way I can let dev-server proxying for the current release.

[gaearon]

You can try 0.3.0-alpha which already includes support for proxy field in package.json. The template/README.md in this repo shows how to use it.

[gaearon]

Fixed by #282, available in 0.2.3.

[curran]

@taion You mentioned the approach of "standing up an Nginx server that sits in front of both my webpack dev server and my backend server or servers". Also, as @lacker pointed out, "it's just a pain because you need to configure nginx".

Would anyone be willing to share an example NGINX configuration that does this proxying?

I would like to adopt this approach, as the heuristic is not working for my use case, but I'm getting tripped up in all the configuration options (having never configured NGINX before). Thank you.

[curran]

I ended up solving the NGINX configuration. Here's what worked for me in case others are also struggling:

server {

    location / {
        proxy_pass http://localhost:3000;
    }

    location /api {
        proxy_pass http://localhost:8080;
    }

    # Handle WebSockets.
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

The above is the complete contents of the file /etc/nginx/sites-enabled/default (in Ubuntu).

This configuration:

• serves on port 80,
• proxies requests under /api to the custom server running on port 8080,
• proxies all other requests to the Webpack dev server running on port 3000,
• supports WebSockets for the dev server live-reloading, and
• supports WebSockets API connections under /api.

Draws from:

• NGINX Beginners Guide: Setting Up a Simple Proxy Server - For proxy configuration.
• StackOverflow: Node.js + Nginx - What now? - for WebSocket support.

[newcl]

With the proxy config in my package.json "proxy": "http://localhost:5000", I can make the GET requests work, but for POST it does not work, the POST requests still get routed to "http://localhost:3000".

create-react-app --version = 1.0.3

[gaearon]

@newcl File a new issue and describe the problem there please, with a reproducing example on GitHub. Thanks!

[ptica]

as I have a bit different use-case (sneaking one react component in otherwise a full LAMP site)
however I've managed to adapt the @curran 's forestanding nginx solution to apache .htaccess for a dev server as

<IfModule mod_rewrite.c>
    RewriteEngine On

    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
    RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
    RewriteRule .* ws://localhost:3000%{REQUEST_URI} [P]

    RewriteRule hot-update.json$ http://localhost:3000%{REQUEST_URI} [P]
    RewriteRule hot-update.js$ http://localhost:3000%{REQUEST_URI} [P]
    RewriteRule sockjs-node http://localhost:3000%{REQUEST_URI} [P]
    RewriteRule static http://localhost:3000%{REQUEST_URI} [P]
</IfModule>

so a big 👍 for this thread!

[curran]

@ptica Happy to hear it!