Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy and minify = true #3956

skramsa · 2020-04-28T07:40:14Z

The toolbar Icons are not showing and I get missing font family errors, but they're all in their folders, nothing was changed. Can you help me fix this issue?

GET /static/font/Roboto-Regular.ttf net::ERR_ABORTED 404 (Not Found)

I get this for all font families.

JohnMcLear · 2020-04-28T08:43:19Z

Please provide details.

Are you running behind reverse proxy?
Are you running in development or production mode.
Minified?
Secured copy or your settings.json
Etc.

https://mclear.co.uk/2015/01/15/how-to-write-a-bug-report/

skramsa · 2020-04-28T08:52:57Z

It's running in development mode, although the systemd service has "Environment=NODE_ENV=production" in it, no reverse proxy, the css is minified, settings.json is barely touched (only mysql and admin pw is changed).

It looks like etherpad can't load all fonts because it looks at /static/font/*.ttf instead of /etherpad/src/static/font/

JohnMcLear · 2020-04-28T08:56:12Z

Try turn minify to false and maxage to 0 in settings.json

skramsa · 2020-04-28T08:59:49Z

I did, nothing changed

JohnMcLear · 2020-04-28T09:10:25Z

At my laptop now, might be able to help more. Most useful thing is to get your settings.json (but remove any passwords). Can you provide that please?

FWIW I tested 1.6.* > 1.8.3 and didn't experience this but it could be related to your settings.json having some legacy cruft in :!

JohnMcLear · 2020-04-28T09:13:13Z

If you don't want to provide settings one thing to do is:

cp settings.json settings.backup.json
cp settings.json.template settings.json
Then copy your database host/user/pass from settings.backup.json into settings.json

Restart and see if it's fixed.

skramsa · 2020-04-28T09:19:45Z

/*
  This file must be valid JSON. But comments are allowed

  Please edit settings.json, not settings.json.template

  To still commit settings without credentials you can
  store any credential settings in credentials.json
*/
{
  // Name your instance!
  "title": "HelpMe-Etherpad",

  // favicon default name
  // alternatively, set up a fully specified Url to your own favicon
  "favicon": "favicon.ico",

  //IP and port which etherpad should bind at
  "ip": "0.0.0.0",
  "port" : 9001,

  // Option to hide/show the settings.json in admin page, default option is set to true
  "showSettingsInAdminPage" : true,

  /*
  // Node native SSL support
  // this is disabled by default
  //
  // make sure to have the minimum and correct file access permissions set
  // so that the Etherpad server can access them

  "ssl" : {
            "key"  : "/path-to-your/epl-server.key",
            "cert" : "/path-to-your/epl-server.crt",
            "ca": ["/path-to-your/epl-intermediate-cert1.crt", "/path-to-your/epl-intermediate-cert2.crt"]
          },

  */

  //The Type of the database. You can choose between dirty, postgres, sqlite and mysql
  //You shouldn't use "dirty" for for anything else than testing or development
  /*"dbType" : "dirty",
  //the database specific settings
  "dbSettings" : {
                   "filename" : "var/dirty.db"
                 },

   An Example of MySQL Configuration*/
   "dbType" : "mysql",
   "dbSettings" : {
                    "user"    : "etherpad",
                    "host"    : "localhost",
                    "password": "",
                    "database": "etherpad",
                    "charset" : "utf8mb4"
                  },
   "users": {
	"admin": {
		"password": "",
		"is_admin": true
		}
            },

  //the default text of a pad
  "defaultPadText" : "Willkommen zum CN-Etherpad!\n\nDieses Dokument wird beim Tippen laufend aktualisiert, sodass jeder immer denselben Text sieht. So kann man nahtlos zusammen an Dokumenten arbeiten.\n",

  /* Default Pad behavior, users can override by changing */
  "padOptions": {
    "noColors": false,
    "showControls": true,
    "showChat": true,
    "showLineNumbers": true,
    "useMonospaceFont": false,
    "userName": false,
    "userColor": false,
    "rtl": false,
    "alwaysShowChat": false,
    "chatAndUsers": false,
    "lang": "de-de"
  },

  /* Should we suppress errors from being visible in the default Pad Text? */
  "suppressErrorsInPadText" : false,

  /* Users must have a session to access pads. This effectively allows only group pads to be accessed. */
  "requireSession" : false,

  /* Users may edit pads but not create new ones. Pad creation is only via the API. This applies both to group pads and regular pads. */
  "editOnly" : false,

  /* Users, who have a valid session, automatically get granted access to password protected pads */
  "sessionNoPassword" : false,

  /* if true, all css & js will be minified before sending to the client. This will improve the loading performance massivly,
     but makes it impossible to debug the javascript/css */
  "minify" : false,

  /* How long may clients use served javascript code (in seconds)? Without versioning this
     may cause problems during deployment. Set to 0 to disable caching */
  "maxAge" : 21600, // 60 * 60 * 6 = 6 hours

  /* This is the absolute path to the Abiword executable. Setting it to null, disables abiword.
     Abiword is needed to advanced import/export features of pads*/
  "abiword" : "/usr/local/bin/abiword",

  /* This is the absolute path to the soffice executable. Setting it to null, disables LibreOffice exporting.
     LibreOffice can be used in lieu of Abiword to export pads */
  "soffice" : null,

  /* This is the path to the Tidy executable. Setting it to null, disables Tidy.
     Tidy is used to improve the quality of exported pads*/
  "tidyHtml" : "/usr/bin/tidy",

  /* Allow import of file types other than the supported types: txt, doc, docx, rtf, odt, html & htm */
  "allowUnknownFileEnds" : true,

  /* This setting is used if you require authentication of all users.
     Note: /admin always requires authentication. */
  "requireAuthentication" : false,

  /* Require authorization by a module, or a user with is_admin set, see below. */
  "requireAuthorization" : false,

  /*when you use NginX or another proxy/ load-balancer set this to true*/
  "trustProxy" : false,

  /* Privacy: disable IP logging */
  "disableIPlogging" : false,

  /* Users for basic authentication. is_admin = true gives access to /admin.
     If you do not uncomment this, /admin will not be available! */
  

  // restrict socket.io transport methods
  "socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"],

  // Allow Load Testing tools to hit the Etherpad Instance.  Warning this will disable security on the instance.
  "loadTest": false,

  // Disable indentation on new line when previous line ends with some special chars (':', '[', '(', '{')
  /*
  "indentationOnNewLine": false,
  */

  /* The toolbar buttons configuration.
  "toolbar": {
    "left": [
      ["bold", "italic", "underline", "strikethrough"],
      ["orderedlist", "unorderedlist", "indent", "outdent"],
      ["undo", "redo"],
      ["clearauthorship"]
    ],
    "right": [
      ["importexport", "timeslider", "savedrevision"],
      ["settings", "embed"],
      ["showusers"]
    ],
    "timeslider": [
      ["timeslider_export", "timeslider_returnToPad"]
    ]
  },
*/
  /* The log level we are using, can be: DEBUG, INFO, WARN, ERROR */
  "loglevel": "INFO",

  //Logging configuration. See log4js documentation for further information
  // https://github.com/nomiddlename/log4js-node
  // You can add as many appenders as you want here:
  "logconfig" :
    { "appenders": [
        { "type": "console"
        //, "category": "access"// only logs pad access
        }
    /*
      , { "type": "file"
      , "filename": "your-log-file-here.log"
      , "maxLogSize": 1024
      , "backups": 3 // how many log files there're gonna be at max
      //, "category": "test" // only log a specific category
        }*/
    /*
      , { "type": "logLevelFilter"
        , "level": "warn" // filters out all log messages that have a lower level than "error"
        , "appender":
          {  Use whatever appender you want here  }
        }*/
    /*
      , { "type": "logLevelFilter"
        , "level": "error" // filters out all log messages that have a lower level than "error"
        , "appender":
          { "type": "smtp"
          , "subject": "An error occurred in your EPL instance!"
          , "recipients": "bar@blurdybloop.com, baz@blurdybloop.com"
          , "sendInterval": 300 // 60 * 5 = 5 minutes -- will buffer log messages; set to 0 to send a mail for every message
          , "transport": "SMTP", "SMTP": { // see https://github.com/andris9/Nodemailer#possible-transport-methods
              "host": "smtp.example.com", "port": 465,
              "secureConnection": true,
              "auth": {
                  "user": "foo@example.com",
                  "pass": "bar_foo"
              }
            }
          }
        }*/
      ]
    }
}

JohnMcLear · 2020-04-28T09:24:18Z

Okay cool, gimme 2 mins.

JohnMcLear · 2020-04-28T09:26:46Z

Nope, that didn't replicate, toolbar buttons exist for me..

Is this instance public that I can access? feel free to email me URL contact@etherpad.org

Stupid question.. Did you

Clear browser cache?
Attempt to run using bin/run.sh

Thanks!

skramsa · 2020-04-28T09:35:09Z

Sadly it's not a public instance.
I replaced the settings.json with the new template, also changed skin to no-skin. stilll no success..
And yes, I've run run.sh, restarted the service and cleared the cash :)

skramsa · 2020-04-28T09:38:51Z

This is the systemd service btw:
[Unit]
Description=Etherpad
After=syslog.target network.target

[Service]
Type=simple
User=etherpad
Group=etherpad
WorkingDirectory=/opt/etherpad
Environment=NODE_ENV=production
ExecStart=/opt/etherpad/bin/run.sh
Restart=always

[Install]
WantedBy=multi-user.target

JohnMcLear · 2020-04-28T09:45:05Z

How did you install? Did you git clone or did you grab the released .zip and unpack?

JohnMcLear · 2020-04-28T09:47:46Z

Request URL: https://dev.com:9001/static/font/RobotoMono-Regular.ttf

jose@server:~/develop$ locate RobotoMono | grep develop | grep Reg
/home/jose/develop/src/static/font/RobotoMono-Regular.ttf

All looks fine to me.

skramsa · 2020-04-28T09:57:25Z

I installed it via git clone, or rather updated it. I believe it was installed a few years ago (by another admin) through zip unpack. I used rsync to update everything

JohnMcLear · 2020-04-28T10:20:53Z

Ah, I wonder if the last admin put some hacks in? if you git status do you see any modified files?

JohnMcLear · 2020-04-28T10:21:30Z

What happens if you git clone a fresh repo then copy your settings.json over? That will work but obviously isn't fixing the problem that something is weird..

skramsa · 2020-04-28T12:16:34Z

I'm going to do that now. yes, it's very weird.. as if the root folder for the fonts is not the same as etherpad

JohnMcLear · 2020-04-28T12:40:09Z

The code is right btw. src/ is the root folder..

I think the previous admin hacked something, I can't say what without seeing a git diff. Can we close this now?

skramsa · 2020-04-28T12:44:29Z

previous admin didn't use git (was not installed), still doesn't work after clean clone..

JohnMcLear · 2020-04-28T12:54:58Z

@skramsa lulwut. Completely clean clone and run and it doesn't work? :! I have a feeling something is weird with your setup.. Any errors or warnings in log?

What if you do a completely clean clone and don't change any settings? Does it still fail then?

Any chance you can make it public temporarily for me to debug?

skramsa · 2020-04-28T13:14:29Z

no it's on a secured intranet.. could the upgrade from node 6 to 12 have something to do with it?

JohnMcLear · 2020-04-28T13:21:43Z

Not sure.. I'm not really sure other debug steps here..

The only clue I have is..
https://github.com/ether/etherpad-lite/blob/develop/src/node/utils/Minify.js#L429

But you said you cleaned all cache and do a clean run..

You disabled minify, did rm -Rf src/node_modules && rm src/package-lock.json then ran bin/run.sh yeah? I mean you did a clean clone so ;\

JohnMcLear · 2020-04-28T13:22:28Z

What's your git sha, npm --version and nodejs version values please?

skramsa · 2020-04-28T13:25:39Z

npm is 6.14.4 and node -v is 12.16.2. stupid question, but what exactly do you mean with "my" git sha?

skramsa · 2020-04-28T13:34:46Z

I rerolled to 1.6.2 and set minify to false and suddenly it works again (before that not even the old version's icons did function). That's a relief.. I will try to make another clean setup next week with this settings, maybe this time it will work :D Thank you so much for your help!!

JohnMcLear · 2020-04-28T14:05:40Z

In your etherpad folder type git rev-parse HEAD

I wanna see which commit you are on.

julpec · 2020-04-29T01:43:40Z

Hello
I have exactly the same problem except I'm behind a proxy.
I made a test by saving my installation and cloning the repository. Then I simply copied my configuration file and restarted the service.
I still have the same problem despite the fresh installation.

But :
On the other hand, interesting thing, when I access the service without using the proxy and without https, no problem, the icons are displayed correctly !!

The apache proxy configuration is coming from the documentation, nothing exotic on my side.

lucaguindani · 2020-04-29T06:31:11Z

Had the exact same problem.

I use a reverse proxy and the Etherpad instance is in a subdirectory (/etherpad). I managed to make it right adding the following to my apache configuration.

<Location /static/font>
    ProxyPass http://localhost:9001/static/font
    ProxyPassReverse http://localhost:9001/static/font
</Location>

I did not change anything in settings.json.

JohnMcLear · 2020-04-29T06:31:47Z

@skramsa hasn't mentioned any reverse proxy so I don't think it's related? Skramsa please confirm there is no reverse proxy in your setup.

ashishhari · 2020-05-14T03:49:41Z

Thank your for acknowledging. Setting minify to false did not solve the problem for me. I am using this on a jitsi installation on an AWS EC2 instance, and there is a nginx server being used to host jitsi and also to start the node server for etherpad service on same machine on port 9001. Does this mean I may need to tweak some other nginx setting? (I dont understand reverse proxy properly but could this be that case?), Or do I just need to wait for the fix to be available.

…

On Thu, May 14, 2020 at 6:07 AM muxator ***@***.***> wrote: We have various issues going on here. 1. the test Nginx config file had the wrong redirect for /static. The whole location /static section has to be deleted from here <#3956 (comment)>. In the email storm received during these days, I think to remember that *a user complained - indeed - that /static was going to clash with a lot of other web applications*. Probably he was misled by an example similar to this one; This has to be fixed going through the documentation (the wiki? Somehwere else on the internet?) and *improving the quality of the examples*. 2. 4177b3f <4177b3f> moved the font-face declarations from src/static/css/pad.css to two imported files (src/static/css/pad/fonts.css and src/static/css/pad/toolbar.css) in a different directory, and URLs in src are relative to the stylesheet path ***@***.***/src>; 3. on top of that, after merging 95fd5ce <95fd5ce> for the forthcoming *1.8.4*, we would have a cache staleness problem that also needs to be addressed. I'm doing some tests and preparing a PR. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3956 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH7VXAS65ZUAR33CPKSI6OLRRM4KZANCNFSM4MSSRTRQ> .

julpec · 2020-05-14T08:00:39Z

@skramsa, @julpec, @lucaguindani, @ashishhari (and John, obviously): is someone of you in the position of testing that PR to see if it works for them? I would like an independent validation, if possible.

Ok for me to test. I don't have special configuration for static.

Thank your for acknowledging. Setting minify to false did not solve the problem for me.

But it works for me. Under apache, I switched to false and everything started working fine.

ashishhari · 2020-05-14T08:17:02Z

I can test it.

…

On Thu, May 14, 2020 at 1:30 PM julpec ***@***.***> wrote: @skramsa <https://github.com/skramsa>, @julpec <https://github.com/julpec>, @lucaguindani <https://github.com/lucaguindani>, @ashishhari <https://github.com/ashishhari> (and John, obviously): is someone of you in the position of testing that PR to see if it works for them? I would like an independent validation, if possible. Ok for me to test. I don't have special configuration for static. Thank your for acknowledging. Setting minify to false did not solve the problem for me. But it works for me. Under apache, I switched to false and everything started working fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3956 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH7VXAXKTJKZCAZVRXZIJX3RROQLPANCNFSM4MSSRTRQ> .

ashishhari · 2020-05-14T08:22:19Z

Also confirming -> setting minify to false did work correctly on my system. (i probably did not restart the service correctly last time).

…

On Thu, May 14, 2020 at 1:30 PM julpec ***@***.***> wrote: @skramsa <https://github.com/skramsa>, @julpec <https://github.com/julpec>, @lucaguindani <https://github.com/lucaguindani>, @ashishhari <https://github.com/ashishhari> (and John, obviously): is someone of you in the position of testing that PR to see if it works for them? I would like an independent validation, if possible. Ok for me to test. I don't have special configuration for static. Thank your for acknowledging. Setting minify to false did not solve the problem for me. But it works for me. Under apache, I switched to false and everything started working fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3956 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH7VXAXKTJKZCAZVRXZIJX3RROQLPANCNFSM4MSSRTRQ> .

muxator · 2020-05-14T08:31:40Z

But it works for me. Under apache, I switched to false and everything started working fine.
Also confirming -> setting minify to false did work correctly on my system.

@julpec, @ashishhari: setting minify = false is a known workaround.
The hope now is to see if #4002 fixes the bug when minify = true.

The procedure would be the following, do this only if all the steps are clear for you:

Start replicating the bug:

ensure minify = true in settings.json
restart Etherapd
visit Etherpad page and verify that the Toolbar icons are not shown

Try the fix:

keep minify = true in settings.json
apply Fix/fonts behind reverse proxy when minify = true #4002 (do it only if you are comfortable with it)
restart Etherpad
visit Etherpad page and verify if the Toolbar icons are now correctly shown

julpec · 2020-05-14T08:36:56Z

I would like but I don't know the procedure to apply the PR :(

JohnMcLear · 2020-05-14T09:55:08Z

Thanks @muxator I will test within the hour.

JohnMcLear · 2020-05-14T09:56:48Z

Oh wow I would have never come up with that fix! That's bonkers.

JohnMcLear · 2020-05-14T10:00:00Z

git remote add mux https://github.com/muxator/etherpad-lite/
git fetch mux
git checkout mux/fix/fonts-behind-reverse-proxy

For those also wanting to test..

after testing remember to git checkout develop else you will in muxs' stale branch

julpec · 2020-05-14T10:01:50Z

thanks, I will test this afternoon

JohnMcLear · 2020-05-14T10:07:31Z

Hrm I think something is not quite performant @muxator.. I'm seeing ~54 requests, vs ~49 (on current develop) -- Note that I have plugins enabled giving extra requests. So this PR would be a page load performance regression to satisfy an undocumented/unsupported use case which is not acceptable IMHO.

Note the multiple requests for the same file... :( It's defo progress but multiple requests for the same file is not the solution we're looking for.

seballot · 2020-05-14T10:46:02Z

the multiple request come from the different iframe I think. Both main page and editor iframe need the font, it has always been like this no?

JohnMcLear · 2020-05-14T10:47:56Z

afaik cleanCSS was used to fix that issue @seballot

muxator · 2020-05-14T11:03:27Z

it has always been like this no?

Yes, it was always this way.

seballot · 2020-05-14T11:15:36Z

I think cleanCss was more to avoid loading each css partial (pad/popups.css, pad/editbar.css, pad/chat.css ...), and instead load only one pad.css

But using iframe, the browser is obliged to request twice each file share with the iframe. But the second call should use local cache ("loaded from cache")

julpec · 2020-05-14T12:07:56Z

I applied the pull request and it works without any problem (with apache)

muxator · 2020-05-14T12:30:15Z

Thanks @julpec.

This evening I'll try to address the # of requests concern @JohnMcLear brought up. If there is nothing going on, I'll merge the PR.

JohnMcLear · 2020-05-14T14:04:30Z

Awesome tnx mux

muxator · 2020-05-14T22:12:29Z

Is there a way to upload HAR files somewhere in Github? I have tested #4002 and I am willing to share the recordings, but I do not know where to upload them.

Common elements:

visited http://localhost:9001/p/ciao
no reverse proxy
production mode: false
start with empty browser cache

Variables:

version: 4816785 vs Fix/fonts behind reverse proxy when minify = true #4002
cache: cold (first load) vs hot (>= 2nd load)
minify: true vs false
browser: chromium 81 vs firefox 76

These are four binary variables -> 2^4 = 16 possibilities.
This took a lot of time!

Here's the more relevant cases:

Description	current develop (`4816785`)	PR #4002
Firefox, minified, first load	34 requests 1,61 MB 1.61 MB transferred	36 requests 1,61 MB 1.53 MB transferred
Firefox, minified, hot cache	29 requests 1,12 MB 216 KB transferred	31 requests 1,18 MB 216 KB transferred
Chromium, minified, first load	44 requests 2,0 MB 1,5 MB transferred	109 richieste 3,0 MB 1,1 MB transferred
Chromium, minified, hot cache	45 requests 2,2 MB 5,7 KB transferred	129 requests 4,7 MB 4,7 KB transferred

The bottom line is:

Chrome and Firefox have a completely different way of measuring their traffic
PR Fix/fonts behind reverse proxy when minify = true #4002 introduces more requests to the server
The cache hit ratio improves: the transferred column decreases!
Load times stay the same, within limits of measurement noise

JohnMcLear · 2020-05-14T22:19:53Z

These were my findings too, I also took the time to research page load speed performance profiling tools (beyond developer tools) and decided that if developer tools weren't giving me significant deviation from a mean then the results were suitable enough for our application.

Total transfer size decreasing is clearly a good thing, but http request overhead can be expensive due to latency. We can merge this but be mindful that when we rewrite the front end (which feels inevitable due to all of the inline JS that will ultimately need to be shifted around) then this is an area we should strive to improve.

All things said, Etherpad is great compared to most others in htis field, the exception being Google docs which is the gold standard for absolutely shockingly fast "time to edit" which in this world is king, not end of page 'load'.

JohnMcLear · 2020-05-14T22:21:24Z

So TLDR; Merge away @muxator - this thread exists as a resource for those looking to optimize for Etherpad instances NOT in subfolders.

4177b3f moved the font-face declarations from src/static/css/pad.css to two imported files (src/static/css/pad/fonts.css, src/static/css/pad/toolbar.css) in a different directory. This results in the font files being invoked from CSSes residing in different directories in the minified and un-minified case. URLs in the src attribute are relative to the stylesheet path [0], and so we have to start requiring clean-css to rebase them. Before this change, the non minified casse worked by chance, because there were a lot of "..", which ended up resolving to the root of the site anyways. Fixes #3956 [0] https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/src

seballot · 2020-05-15T06:03:12Z

great work !

JohnMcLear · 2020-05-15T07:20:52Z

+1 tnx!

julpec · 2020-05-15T07:28:07Z

👍

JohnMcLear added the Needs confirmation label Apr 28, 2020

muxator changed the title ~~Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy~~ Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy and minify = true May 14, 2020

muxator closed this as completed in #4002 May 14, 2020

Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy and minify = true #3956

Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy and minify = true #3956

Comments

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020 • edited Loading

skramsa commented Apr 28, 2020 • edited by JohnMcLear Loading

JohnMcLear commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020 • edited Loading

skramsa commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

skramsa commented Apr 28, 2020

skramsa commented Apr 28, 2020

JohnMcLear commented Apr 28, 2020

julpec commented Apr 29, 2020

lucaguindani commented Apr 29, 2020 • edited Loading

JohnMcLear commented Apr 29, 2020

ashishhari commented May 14, 2020 via email

julpec commented May 14, 2020

ashishhari commented May 14, 2020 via email

ashishhari commented May 14, 2020 via email

muxator commented May 14, 2020

julpec commented May 14, 2020

JohnMcLear commented May 14, 2020

JohnMcLear commented May 14, 2020

JohnMcLear commented May 14, 2020 • edited Loading

julpec commented May 14, 2020

JohnMcLear commented May 14, 2020

seballot commented May 14, 2020

JohnMcLear commented May 14, 2020

muxator commented May 14, 2020

seballot commented May 14, 2020

julpec commented May 14, 2020

muxator commented May 14, 2020

JohnMcLear commented May 14, 2020

muxator commented May 14, 2020

JohnMcLear commented May 14, 2020

JohnMcLear commented May 14, 2020

seballot commented May 15, 2020

JohnMcLear commented May 15, 2020

julpec commented May 15, 2020

JohnMcLear commented Apr 28, 2020 •

edited

Loading

skramsa commented Apr 28, 2020 •

edited by JohnMcLear

Loading

JohnMcLear commented Apr 28, 2020 •

edited

Loading

lucaguindani commented Apr 29, 2020 •

edited

Loading

JohnMcLear commented May 14, 2020 •

edited

Loading