Support proxies for HTTP requests used by PeerTube

[jonghyo]

Describe the current behavior
After updating to 3.2.x, it fails to retrieve the plugin list.

I use peertube behind corporate proxy, and in 3.1.x, I was able to search and install plugins by setting the environment variables
http_proxy and https_proxy .

I guess it because of http library.
Since v3.2.x, http library is changed from request to Got.

request support http_proxy, https_proxy with environment variables, but Got needs to use tunnel or hpagent library for supporing proxy.

Additional information

• PeerTube instance:

  • URL: private
  • version: 3.2.1
  • NodeJS version: 12.22.1
  • Ffmpeg version: 4.1.6-1~deb10u1

• console log:

error[2021/6/21 17:36:29] Cannot list available plugins from index https://packages.joinpeertube.org/api/v1/plugins.
{
  "err": {
    "stack": "RequestError: connect ETIMEDOUT 178.63.240.150:443\n    at ClientRequest.<anonymous> (/app/node_modules/got/dist/source/core/index.js:956:111)\n    at Object.onceWrapper (events.js:421:26)\n    at ClientRequest.emit (events.js:326:22)\n    at ClientRequest.EventEmitter.emit (domain.js:483:12)\n    at ClientRequest.origin.emit (/app/node_modules/@szmarczak/http-timer/dist/source/index.js:39:20)\n    at TLSSocket.socketErrorListener (_http_client.js:427:9)\n    at TLSSocket.emit (events.js:314:20)\n    at TLSSocket.EventEmitter.emit (domain.js:483:12)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)",
    "message": "connect ETIMEDOUT 178.63.240.150:443",
    "name": "RequestError"
  }
}

[AnonymousWebHacker]

Transparent proxy or with user and pass authentication?

[jonghyo]

I don't use transparent proxy or with user and pass authentication.
But I think supporting transparent proxy and with user and pass authentication is useful for proxy users.

[smilekison]

Problem: PeerTube 3.2.x does not support proxy

Packages used:
env-smart: 2.2.1
hpagent: 0.1.2

Proposed Solution:

1. Check if the use of proxy is enabled in default.yaml file
2. If yes => we create a Middleware inside server/middlewares/proxy.ts to fetch proxy address from
   • OS environment
   • Config file
   • .env file
     The logic for middleware is as below:

import { CONFIG } from "../initializers/config"
const envs = require("env-smart")
// read environment variable
const OS_HTTP_PROXY = envs.load().HTTP_PROXY
const OS_HTTPS_PROXY = envs.load().HTTPS_PROXY
// read config.yaml
const CONFIG_HTTP_PROXY = CONFIG.CORPORATE_PROXY.HTTP_PROXY
const CONFIG_HTTPS_PROXY = CONFIG.CORPORATE_PROXY.HTTPS_PROXY
// read env file
const ENV_HTTP_PROXY = process.env.PEERTUBE_HTTP_PROXY
const ENV_HTTPS_PROXY = process.env.PEERTUBE_HTTPS_PROXY
const proxy = () => {
  if (typeof OS_HTTPS_PROXY !== "undefined" && OS_HTTPS_PROXY) {
    console.log("OS HTTPS_PROXY Environment variable :", `${OS_HTTPS_PROXY}`)
    return OS_HTTPS_PROXY
  } else if (typeof OS_HTTP_PROXY !== "undefined" && OS_HTTP_PROXY) {
    console.log("OS HTTP_PROXY Environment variable :", `${OS_HTTP_PROXY}`)
    return OS_HTTP_PROXY
  } else if (typeof CONFIG_HTTPS_PROXY !== "undefined" && CONFIG_HTTPS_PROXY) {
    console.log("Reading https_proxy from Config File:", `${CONFIG_HTTPS_PROXY}`)
    return CONFIG_HTTPS_PROXY
  } else if (typeof CONFIG_HTTP_PROXY !== "undefined" && CONFIG_HTTP_PROXY) {
    console.log("Reading http_proxy from Config File:", `${CONFIG_HTTP_PROXY}`)
    return CONFIG_HTTP_PROXY
  } else if (typeof ENV_HTTP_PROXY !== "undefined" && ENV_HTTP_PROXY) {
    console.log("Reading https_proxy from  env file:", `${ENV_HTTP_PROXY}`)
    return ENV_HTTP_PROXY
  } else if (typeof ENV_HTTPS_PROXY !== "undefined" && ENV_HTTPS_PROXY) {
    console.log("Reading http_proxy from  env file:", `${ENV_HTTPS_PROXY}`)
    return ENV_HTTPS_PROXY
  } 
}
module.exports = proxy

We create custom defination inside default.yml file as below:

corporate_proxy:
  enabled: true
  https_proxy: ''
  http_proxy: ''

We use hpagent HttpsProxyAgent to resolve the proxy inside server/helpers/requests.ts file inside peertubeGot=got.extend({ ..... }).

agent: {
    https: new HttpsProxyAgent({
      keepAlive: true,
      keepAliveMsecs: 1000,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: "lifo",
      proxy: proxy()
    })
  },

3. If No, we don't use HttpsProxyAgent and let the code run with default settings.
4. The architecture of the above solution is as below:

#4202

Should we proceed to create a pull request with the mentioned solution? @Chocobozzz
We updated the architecture image.

[Chocobozzz]

Seems good @smilekison, please create a PR

Some things:

• We don't use .env in peertube (except for docker but it's not directly used by peertube) so just check process.env.HTTP_PROXY
• I don't think we should also use expose proxy in the configuration because it could be confusing for admins (I disabled the proxy in the config but peertube still uses implicitly a proxy because of env var)

[smilekison]

@Chocobozzz we have changed our Architecture a little. We have removed the flag (Enabled:true) in the production.yml file. And also, we changed middleware to only accept proxy from OS_ENV. We have prepared the following two architectures and we would like to know which approach is better?

1. Architecture with Enabled flag

2. Architecture without Enabled flag

[Chocobozzz]

I would use the second one: it's the same behaviour than before where we did not have any complain :)