No outbound FTP access from datahub

[cboettig]

Scripts that require access to ftp resources e.g. the famous Mauna Loa CO2 dataset, https://climate.nasa.gov/vital-signs/carbon-dioxide/, ftp://aftp.cmdl.noaa.gov/products/trends/co2/co2_mm_mlo.txt, times out if we attempt to download it on r.datahub.berkeley.edu. This and many other resources from NOAA and NASA datasets still rely on ftp. Is it really blocked? Is this unavoidable or can it be fixed?

Thanks!

[yuvipanda]

We had disabled it mostly since nobody has asked for it, and I was trying to make sure we only have outbound ports we really needed open.

I'll open it up. Ideally I'd like to allow only readonly anonymous FTP access, but alas I don't wanna maintain an FTP proxy...

[cboettig]

cool, totally makes sense. thanks for opening this up.

(ironically we used to use Travis-CI as our CI testing, which also blocks ftp. We switched to GH-Actions this year and I though 'yay, we can use ftp again!)

[yuvipanda]

:D I think it won't be a bad idea for them to move off FTP, but I bet they know that already...

[cboettig]

@yuvipanda Seems we still cannot access ftp resources from data hub?

Ironically/thankfully that famous CO2 data is now on https, but loads of stuff is still on FTP. e.g. I cannot seem to access this classic sea ice data, ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv on r.datahub.berkeley.edu, though works fine from my machine.

[yuvipanda]

@cboettig yeah, I think due to the way FTP requires arbitrary ports to be open, it's difficult to allow in a setup like ours. Most protocols (HTTPS, SFTP, etc) require one outbound port to be open. FTP however, doesn't know which port needs to be open until the ftp command is run, so it's hard for us to allow them explicitly in config here. So it'll be difficult to enable FTP without opening a lot of ports.

https://www.techrepublic.com/article/how-ftp-port-requests-challenge-firewall-security/ has a little bit more information.

So I guess the short version is that outbound FTP is difficult in our setup, and it is currently not enabled. I'm not sure how to enable it without opening up all outbound ports. I'd love to find an alternate solution that doesn't involve opening up all outbound ports, and right now I've not had a lot of time to research them :(

An outbound FTP proxy is perhaps the solution, but looking around I didn't find any modern FTP proxies we can use.

[cboettig]

Thanks @yuvipanda. at least good to confirm I'm not crazy here. I'd just mirror copies of these data, but despite the ancient technology involved these are mostly streaming datasets, so we need to keep hitting the ftp to get the latest stuff. We might just throw a github action into the student repos that grabs the data... (at least unlike travis, github seems to enable FTP).

[yuvipanda]

@cboettig FTP is now available! Thank you for your patience as we worked through this :)