Restrict file types #370

harminius · 2025-02-06T07:59:04Z

Resolves https://github.com/MerginMaps/server-private/issues/2732

Problem:
We allow users to upload any file, even the (potentially) malicious ones 🏴‍☠️

See tickets for more details and the Pentest report

Solution:
We are more confident with using blacklists. 🛡️

Validate extension during project push - aka push_start. Abort the upload if it includes files with blacklisted extensions.
Do not rely solely on the extension check. Determine the file type from its header and block blacklisted types - once we have the file on our filesystem (in push_finish).

Zipping the unsupported makes it possible to upload it.

E.g. block .exe

Extension renaming of the unsupported file doesn't help ⛔

Potential whitelists are in the comment

~~Do not allow to sync files with extensions unless whitelisted~~

~~Whitelist approach is suggested in this PR as it is considered to be safer. However, using blacklists is also a way, see this comment for suggested blacklists.~~

server/mergin/sync/storages/disk.py

server/mergin/sync/utils.py

server/mergin/sync/public_api_controller.py

MarcelGeo · 2025-02-06T13:53:06Z

server/mergin/sync/public_api_controller.py

+    validate_filename(filename)
+
+
+def validate_file(head, filename):


I am not sure why we do not have one function for validating mime types and extensions?

server/mergin/sync/utils.py

server/mergin/sync/public_api_controller.py

coveralls · 2025-02-06T15:54:12Z

Pull Request Test Coverage Report for Build 13240989483

Details

50 of 50 (100.0%) changed or added relevant lines in 3 files are covered.
49 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.9%) to 93.275%

Files with Coverage Reduction	New Missed Lines	%
server/mergin/sync/utils.py	1	84.13%
server/mergin/commands.py	12	31.71%
server/mergin/app.py	36	74.52%

Totals
Change from base Build 13240347753:	0.9%
Covered Lines:	6755
Relevant Lines:	7242

💛 - Coveralls

harminius · 2025-02-07T06:46:17Z

Current approach is (in web security recommended) whitelist.
If we are afraid to be too restrictive we could use blackilists.
Blocking logic would stay the same.

Here's the suggestion:

FORBIDDEN_EXTENSIONS = {
    ".ade", ".adp", ".app", ".appcontent-ms", ".application", ".appref-ms",  
    ".asp", ".aspx", ".asx", ".bas", ".bat", ".bgi", ".cab", ".cdxml", ".cer",  
    ".chm", ".cmd", ".cnt", ".com", ".cpl", ".crt", ".csh", ".der", ".diagcab",  
    ".dll", ".drv", ".exe", ".fxp", ".gadget", ".grp", ".hlp", ".hpj", ".hta",  
    ".htc", ".htaccess", ".htpasswd", ".inf", ".ins", ".iso", ".isp",  
    ".its", ".jar", ".jnlp", ".js", ".jse", ".jsp", ".ksh", ".lnk", ".mad",  
    ".maf", ".mag", ".mam", ".maq", ".mar", ".mas", ".mat", ".mau", ".mav",  
    ".maw", ".mcf", ".mda", ".mdb", ".mde", ".mdt", ".mdw", ".mdz", ".msc",  
    ".mht", ".mhtml", ".msh", ".msh1", ".msh2", ".mshxml", ".msh1xml", ".msh2xml",  
    ".msi", ".msp", ".mst", ".msu", ".ops", ".osd", ".pcd", ".pif", ".pl",  
    ".plg", ".prf", ".prg", ".printerexport", ".ps1", ".ps1xml", ".ps2",  
    ".ps2xml", ".psc1", ".psc2", ".psd1", ".psdm1", ".pssc", ".pst", ".py",  
    ".pyc", ".pyo", ".pyw", ".pyz", ".pyzw", ".reg", ".scf", ".scr", ".sct",  
    ".settingcontent-ms", ".sh", ".shb", ".shs", ".sys", ".theme", ".tmp",  
    ".torrent", ".url", ".vb", ".vbe", ".vbp", ".vbs", ".vhd", ".vhdx",  
    ".vsmacros", ".vsw", ".webpnp", ".website", ".ws", ".wsb", ".wsc",  
    ".wsf", ".wsh", ".xbap", ".xll", ".xnk"
}

FORBIDDEN_MIME_TYPES = {
    "application/x-msdownload",
    "application/x-sh",
    "application/x-bat",
    "application/x-msdos-program",
    "application/x-dosexec",
    "application/x-csh",
    "application/x-perl",
    "application/javascript",
    "application/x-python-code",
    "application/x-ruby",
    "application/java-archive",
    "application/vnd.ms-cab-compressed",
    "application/x-ms-shortcut",
    "application/vnd.microsoft.portable-executable",
    "application/x-ms-installer",
    "application/x-ms-application",
    "application/x-ms-wim",
    "text/x-shellscript",
}

MarcelGeo · 2025-02-07T08:44:05Z

Current approach is (in web security recommended) whitelist. If we are afraid to be too restrictive we could use blackilists. Blocking logic would stay the same.

Here's the suggestion:

FORBIDDEN_EXTENSIONS = {
    ".ade", ".adp", ".app", ".appcontent-ms", ".application", ".appref-ms",  
    ".asp", ".aspx", ".asx", ".bas", ".bat", ".bgi", ".cab", ".cdxml", ".cer",  
    ".chm", ".cmd", ".cnt", ".com", ".cpl", ".crt", ".csh", ".der", ".diagcab",  
    ".dll", ".drv", ".exe", ".fxp", ".gadget", ".grp", ".hlp", ".hpj", ".hta",  
    ".htc", ".htaccess", ".htpasswd", ".img", ".inf", ".ins", ".iso", ".isp",  
    ".its", ".jar", ".jnlp", ".js", ".jse", ".jsp", ".ksh", ".lnk", ".mad",  
    ".maf", ".mag", ".mam", ".maq", ".mar", ".mas", ".mat", ".mau", ".mav",  
    ".maw", ".mcf", ".mda", ".mdb", ".mde", ".mdt", ".mdw", ".mdz", ".msc",  
    ".mht", ".mhtml", ".msh", ".msh1", ".msh2", ".mshxml", ".msh1xml", ".msh2xml",  
    ".msi", ".msp", ".mst", ".msu", ".ops", ".osd", ".pcd", ".pif", ".pl",  
    ".plg", ".prf", ".prg", ".printerexport", ".ps1", ".ps1xml", ".ps2",  
    ".ps2xml", ".psc1", ".psc2", ".psd1", ".psdm1", ".pssc", ".pst", ".py",  
    ".pyc", ".pyo", ".pyw", ".pyz", ".pyzw", ".reg", ".scf", ".scr", ".sct",  
    ".settingcontent-ms", ".sh", ".shb", ".shs", ".sys", ".theme", ".tmp",  
    ".torrent", ".url", ".vb", ".vbe", ".vbp", ".vbs", ".vhd", ".vhdx",  
    ".vsmacros", ".vsw", ".webpnp", ".website", ".ws", ".wsb", ".wsc",  
    ".wsf", ".wsh", ".xbap", ".xll", ".xnk"
}

FORBIDDEN_MIME_TYPES = {
    "application/x-msdownload",
    "application/x-sh",
    "application/x-bat",
    "application/x-msdos-program",
    "application/x-dosexec",
    "application/x-csh",
    "application/x-perl",
    "application/javascript",
    "application/x-python-code",
    "application/x-ruby",
    "application/java-archive",
    "application/vnd.ms-cab-compressed",
    "application/octet-stream",
    "application/x-ms-shortcut",
    "application/vnd.microsoft.portable-executable",
    "application/x-ms-installer",
    "application/x-ms-application",
    "application/x-ms-wim",
    "text/x-shellscript",
}

This looks more sensible for me now.

harminius · 2025-02-10T08:52:55Z

whitelist:
ALLOWED_EXTENSIONS = {
# Geospatial
# Shapefile components
".shp",
".shx",
".dbf",
".prj",
".cpg",
".qix",
".sbn",
".sbx",
# Vector data formats
".geojson",
".kml",
".kmz",
".gpx",
".dxf",
".svg",
".gpkg",
".json",
# Raster data formats
".tif",
".tiff",
".geotiff",
".asc",
".vrt",
".grd",
".img",
".adf",
# Point cloud data formats
".las",
".laz",
".ply",
".xyz",
".e57",
".pcd",
# Database and container formats
".mbtiles",
".sqlite",
".gpkg",
# Geospatial metadata and styles
".sld",
".qml",
".lyr",
".qgz",
".qgs",
# Other specialized formats
".hdf",
".hdf5",
".netcdf",
".nc",
".dem",
".dt2",
".dt0",
".map",
".tab",
".mif",
".mid",
# Images
".jpg",
".jpeg",
".png",
".bmp",
".gif",
".heic",
".webp",
".tif",
".tiff",
# Text documents
".pdf",
".doc",
".docx",
".odt",
".rtf",
".txt",
# Others
".zip",
}

ALLOWED_MIME_TYPES = {
"application/x-shapefile",
"application/x-dbf",
"text/plain",
"application/octet-stream",
"application/geo+json",
"application/vnd.google-earth.kml+xml",
"application/vnd.google-earth.kmz",
"application/gpx+xml",
"application/geopackage+sqlite3",
"application/vnd.sqlite3",
"application/json",
"text/xml",
"text/html",
"application/vnd.mapbox-vector-tile",
"application/x-sqlite3",
"application/vnd.ogc.sld+xml",
"application/xml",
"application/x-qgis",
"application/x-hdf",
"application/x-hdf5",
"application/x-netcdf",
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.oasis.opendocument.text",
"application/rtf",
"text/plain",
"application/zip",
}

server/mergin/sync/utils.py

varmar05 · 2025-02-10T14:15:14Z

server/mergin/sync/utils.py

+def is_supported_extension(filepath) -> bool:
+    """Check whether file's extension is supported."""
+    ext = os.path.splitext(filepath)[1].lower()
+    return ext and ext not in FORBIDDEN_EXTENSIONS


Do not we have some input specific file that would not give extension? Just to make sure we do not break something with forbidden files with no ext.

there could be empty extension.

>>> import os >>> os.path.splitext("Makefile") ('Makefile', '')

harminius requested a review from MarcelGeo February 6, 2025 08:00

MarcelGeo reviewed Feb 6, 2025

View reviewed changes

server/mergin/sync/public_api_controller.py Outdated Show resolved Hide resolved

harminius changed the base branch from develop to path_validation_in_upload February 7, 2025 14:47

harminius changed the title ~~Restrict file types and filepaths~~ Restrict file types Feb 10, 2025

harminius mentioned this pull request Feb 10, 2025

Path validation in upload #372

Merged

varmar05 requested changes Feb 10, 2025

View reviewed changes

server/mergin/sync/utils.py Outdated Show resolved Hide resolved

Base automatically changed from path_validation_in_upload to develop February 10, 2025 11:56

harminius added 4 commits February 10, 2025 13:19

Split path manipulation and file type validation

fb2b9d6

Switch to blacklists

6d2c22b

reformat

189c952

Move filetype check

67ac1ee

harminius force-pushed the file_upload_restriction branch from edbca50 to 67ac1ee Compare February 10, 2025 12:19

harminius added 2 commits February 10, 2025 13:21

residue

05189b1

Add magic lib

4c3c647

varmar05 reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restrict file types #370

Restrict file types #370

harminius commented Feb 6, 2025 •

edited

Loading

MarcelGeo Feb 6, 2025

coveralls commented Feb 6, 2025 •

edited

Loading

harminius commented Feb 7, 2025 •

edited

Loading

MarcelGeo commented Feb 7, 2025

harminius commented Feb 10, 2025

varmar05 Feb 10, 2025

MarcelGeo Feb 11, 2025

		validate_filename(filename)


		def validate_file(head, filename):

Restrict file types #370

Are you sure you want to change the base?

Restrict file types #370

Conversation

harminius commented Feb 6, 2025 • edited Loading

MarcelGeo Feb 6, 2025

Choose a reason for hiding this comment

coveralls commented Feb 6, 2025 • edited Loading

Pull Request Test Coverage Report for Build 13240989483

Details

💛 - Coveralls

harminius commented Feb 7, 2025 • edited Loading

MarcelGeo commented Feb 7, 2025

harminius commented Feb 10, 2025

varmar05 Feb 10, 2025

Choose a reason for hiding this comment

MarcelGeo Feb 11, 2025

Choose a reason for hiding this comment

harminius commented Feb 6, 2025 •

edited

Loading

coveralls commented Feb 6, 2025 •

edited

Loading

harminius commented Feb 7, 2025 •

edited

Loading