Bash script to format a block device (hard drive or Flash drive) in UDF. The output is a drive that can be used for reading/writing across multiple operating system families: Windows, macOS, and Linux. This script should be capable of running in macOS or in Linux.
- Formats a block device (hard drive or Flash drive) in Universal Disk Format (UDF)
- UDF revision 2.01 used for maximal compatibility (see note on Linux support below)
- Resulting file system can be read/written across multiple operating system families (Windows, macOS, and Linux)
- Runs on any OS having a Bash environment
- Optionally wipes device before formatting
- Ability to override detected device block size
- Option to force non-interactive mode (useful for scripting)
- Writes a fake MBR for added compatibility on Windows (optionally disabled)
For the advanced user, format-udf is also capable of formatting a single existing partition, without modifying the partition table. Beware that using this method will render the newly formatted UDF partition unusable on macOS (but still usable on Linux and Windows). (See #24 for caveats.) Because of this limitation, the recommendation is to format the entire device.
format-udf was created to address some OS-specific quirks that prevent a naively-formatted UDF device from working across various operating systems. Here are some of the complicating factors, which format-udf aims to abstract away:
- Different operating systems support different versions of the UDF specification. Some OS versions only offer read-only support.
- Windows seems to only mount UDF devices if the file system block size matches the device's logical block size
- Different operating systems (like Windows XP) only attempt mounting UDF file systems with a hard-coded block size
- Windows does not support hard disks without a partition table. (This is strange because Windows does not apply the same limitation to flash drives.)
- macOS seems to only mount UDF file systems that utilize the full disk (not just a partition)
At first glance, these constraints appear to be in partial conflict. The solution, as suggested by Pieter, is to place a fake partition table (via MBR) in the first block of the drive, which lists a single entire-disk partition. This works because UDF (perhaps intentionally) doesn't utilize the first block. Unfortunately, there has been no easy way to do this, while juggling all of the other variables (such as device logical block size). format-udf writes such a fake MBR for added compatibility on Windows. If this is not what you desire, you can disable the MBR with -p none.
The goal of this project is to provide access to a cross-platform file system with modern features, in such a way that is:
- Easy to use for the average user
- Maximally compatible across operating systems
- As compliant as is reasonable with the UDF specification
- Maximally flexible to help users with uncommon needs
Not all operating systems support UDF. The following tables detail operating system support for UDF. Data was adapted from https://en.wikipedia.org/wiki/Universal_Disk_Format#Compatibility (as retrieved on 2017-06-16).
Both read/write are supported unless otherwise listed below.
| Operating System | Read-only | Note |
|---|---|---|
| Windows XP, Server 2003 | Read-only | Write support available with third party utilities |
| Windows Vista, 7, 8, 10 | Referred to by Microsoft as "Live File System"; Requires fake full-disk partition | |
| Mac OS 9 | ||
| Mac OS X 10.5 through 10.11 | ||
| macOS 10.12+ | ||
| Linux 2.6+, 3.x | UDF revisions 2.01 and before have read/write. After UDF revision 2.01, read-only. | |
| AIX 5.2, 5.3, 6.1 | ||
| BeOS, magnussoft ZETA, Haiku | ||
| DosBox | ||
| eComStation, OS/2 | Additional-fee drivers on OS/2 | |
| NetBSD 5.0 | ||
| Solaris 8, 9, 10 |
| Operating System | Note |
|---|---|
| Windows 95 OSR2+, 98 | Utilities include DLA and InCD |
| Windows 2000, ME |
| Operating System | Note |
|---|---|
| DOS, FreeDOS, Windows 3.11 or older | File systems that have an ISO9660 backward compatibility structure can be read |
Not all operating systems support 4K drives (Advanced Format). If you operating system supports UDF, but not your 4K drive, you still may encounter issues using format-udf.
The following tables detail Windows support for 4K drives. Data was adapted from the Microsoft support policy for 4K sector hard drives in Windows (as retrieved on 2017-06-16). Overlaid into this table are testing results from the format-udf community. (Special thanks to @pali for his testing on XP.)
| Size / OS | 512-byte native | 512 emulation (AKA "512e") |
4K native (AKA "4Kn") |
|---|---|---|---|
| Logical block size | 512 bytes | 512 bytes | 4096 bytes |
| Physical block size | 512 bytes | 4096 bytes | 4096 bytes |
| Maximum UDF file system capacity |
2 TiB | 2 TiB | 16 TiB |
| Windows XP | Supported; Works |
Unsupported; Doesn't work |
Unsupported; Doesn't work |
| Windows XP Pro x64, Server 2003, Server 2003 R2 |
Supported; Likely works but untested |
Unsupported | Unsupported |
| Windows Vista, Server 2008 |
Supported; Likely works but untested |
Supported; Likely works but untested |
Unsupported |
| Windows 7, Server 2008 R2 |
Supported; Likely works but untested |
Supported; Likely works but untested |
Unsupported |
| Windows 8, Server 2012 |
Supported; Likely works but untested |
Supported; Likely works but untested |
Supported; Likely works but untested |
| Windows 8.1, Server 2012 R2 |
Supported; Likely works but untested |
Unsupported | Unsupported |
| Windows 10, Server 2016 |
Supported; Likely works but untested |
Unsupported | Supported; Likely works but untested |
If you have conducted testing and would like to update this table to benefit future users of format-udf, please send a pull request. Please include a link to your raw data or testing results.
- Any OS having a Bash environment
- The following tools must be installed, executable, and in the PATH:
printfxxd- One of the following:
blockdev,ioreg - One of the following:
blockdev,diskutil - One of the following:
lsblk,diskutil - One of the following:
umount,diskutil - One of the following:
mkudffs,newfs_udf
To install necessary prerequisites on Ubuntu:
sudo apt-get install udftools coreutils vim-common
format-udf is a self-contained script. Simply copy format-udf.sh to a directory of your choosing. Don't forget to make it executable:
chmod +x format-udf.sh
Bash script to format a block device (hard drive or Flash drive) in UDF.
The output is a drive that can be used for reading/writing across multiple
operating system families: Windows, macOS, and Linux.
This script should be capable of running in macOS or in Linux.
Usage: ./format-udf.sh [-b BLOCK_SIZE] [-f] [-p PARTITION_TYPE] [-w WIPE_METHOD] device label
./format-udf.sh -v
./format-udf.sh -h
-b BLOCK_SIZE
Block size to be used during format operation.
If absent, defaults to value reported by blockdev/diskutil.
This is an expert-only option. Please consult the README for details.
-f
Forces non-interactive mode. Useful for scripting.
Please use with caution, as no user confirmation is given.
-h
Display help information and exit.
-p PARTITION_TYPE
Partition type to set during format operation.
Currently supported types include: mbr, none
mbr - Master boot record (default)
none - Do not modify partitions
If absent, defaults to 'mbr'.
See also:
https://github.com/JElchison/format-udf#why
-v
Display version information and exit.
-w WIPE_METHOD
Wipe method to be used before format operation.
Currently supported types include: quick, zero, scrub
quick - Quick method (default)
zero - Write zeros to the entire device
scrub - Iteratively writes patterns on device
to make retrieving the data more difficult.
Requires 'scrub' to be executable and in the PATH.
See also http://linux.die.net/man/1/scrub
If absent, defaults to 'quick'.
Note: 'zero' and 'scrub' methods will take a long time.
device
Device to format. Examples:
* /dev/sdx (Linux, where 'x' is a letter) or
* /dev/diskN (macOS, where 'N' is a number)
label
Label to apply to formatted device.
Example: ./format-udf.sh /dev/sdg "My UDF External Drive"
On Ubuntu:
user@computer:~$ ./format-udf.sh /dev/sdg "My UDF External Drive"
[+] Validating arguments...
[+] Testing dependencies...
[+] Looking for drive detail tool... using /sbin/blockdev
[+] Looking for drive listing tool... using /sbin/blockdev
[+] Looking for drive info tool... using /bin/lsblk
[+] Looking for drive summary tool... using /sbin/blkid
[+] Looking for unmount tool... using /bin/umount
[+] Looking for UDF tool... using /usr/sbin/mkudffs
[+] Detecting logical block size...
[sudo] password for user:
[*] Detected logical block size of 512
[+] Validating detected logical block size...
[+] Detecting physical block size...
[*] Detected physical block size of 512
[+] Validating detected physical block size...
[+] Validating file system block size...
[*] Using file system block size of 512
[+] Detecting total size...
[*] Detected total size of 8019509248
[+] Validating detected total size...
[+] Gathering drive information...
/dev/sdg: LABEL="Old Drive" UUID="4843-D1BD" TYPE="vfat"
RO RA SSZ BSZ StartSec Size Device
rw 256 512 512 0 8019509248 /dev/sdg
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend: yes
[+] Unmounting device...
umount: /dev/sdg: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.450716 s, 4.7 MB/s
[+] Formatting /dev/sdg ...
filename=/dev/sdg
label=My UDF External Drive
uuid=5e4924cc17b50769
blocksize=512
blocks=15663104
udfrev=2.01
start=0, blocks=64, type=ERASE
start=64, blocks=13, type=VRS
start=77, blocks=19, type=ERASE
start=96, blocks=16, type=MVDS
start=112, blocks=16, type=ERASE
start=128, blocks=16, type=LVID
start=144, blocks=112, type=ERASE
start=256, blocks=1, type=ANCHOR
start=257, blocks=15662590, type=PSPACE
start=15662847, blocks=1, type=ANCHOR
start=15662848, blocks=96, type=ERASE
start=15662944, blocks=16, type=RVDS
start=15662960, blocks=143, type=ERASE
start=15663103, blocks=1, type=ANCHOR
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.0037039 s, 4.3 kB/s
2+0 records in
2+0 records out
2 bytes copied, 3.7193e-05 s, 53.8 kB/s
[+] Successfully formatted /dev/sdg: UUID="5e4924cc17b50769" LABEL="My UDF External Drive" TYPE="udf" PTTYPE="dos"
Please disconnect/reconnect your drive now.
On macOS:
computer:~ user$ ./format-udf.sh /dev/disk4 "My UDF External Drive"
[+] Validating arguments...
[+] Testing dependencies...
[+] Looking for drive detail tool... using /usr/sbin/ioreg
[+] Looking for drive listing tool... using /usr/sbin/diskutil
[+] Looking for drive info tool... using /usr/sbin/diskutil
[+] Looking for drive summary tool... using (none)
[+] Looking for unmount tool... using /usr/sbin/diskutil
[+] Looking for UDF tool... using /sbin/newfs_udf
[+] Detecting logical block size...
[*] Detected logical block size of 512
[+] Validating detected logical block size...
[+] Detecting physical block size...
[+] Validating file system block size...
[*] Using file system block size of 512
[+] Detecting total size...
[*] Detected total size of 8019509248
[+] Validating detected total size...
[+] Gathering drive information...
/dev/disk4 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *8.0 GB disk4
1: Windows_FAT_32 Old Drive 8.0 GB disk4s1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend: yes
[+] Unmounting device...
Password:
Unmount of all volumes on disk4 was successful
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
2097152 bytes transferred in 3.113956 secs (673469 bytes/sec)
[+] Formatting /dev/disk4 ...
write to block device: /dev/disk4 last written block address: 15663103
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes transferred in 0.000615 secs (26021 bytes/sec)
2+0 records in
2+0 records out
2 bytes transferred in 0.000644 secs (3106 bytes/sec)
[+] Successfully formatted
Please disconnect/reconnect your drive now.
If's extremely important that format-udf use the correct block size when formatting your drive. format-udf will attempt to detect and use the correct (logical) block size. If you know what you're doing, the format-udf -b BLOCK_SIZE option can be used to explicitly override the detected block size value.
If the wrong block size is used while formatting (i.e. one that doesn't match the logical block size of your drive), the resultant drive will likely have OS compatibility issues and suffer from non-optimal performance issues.
In the same way, it's just as important that the resultant drive be mounted using the correct block size. Many operating systems will only attempt one block size (usually whatever the mount utility defaults to). For example, In order to mount a UDF device, Windows seems to require that the UDF file system use a block size equal to the logical block size. If your block size isn't the OS's default, then auto-mounting likely will not work on your OS. While a small nuisance, manual mounting attempts should still succeed for nonstandard block sizes.
Example of how to manually mount on Linux:
$ mount -t udf -o bs=4096 /dev/sdX /mnt/mount-point
Example of how to manually mount on macOS:
$ sudo mount_udf -b 4096 /dev/diskN /Volumes/MountPoint
Sadly, anything with block size different than 512 doesn't seem to mount on Windows XP.
For more info, see #12, #13, #16, and #31.
The UDF format has a maximum of 2^32 blocks. With format-udf, these blocks equate to logical blocks.
- If your drive's logical block size is 512 bytes, then your maximum UDF file system capacity will be 2 TiB
- If your drive's logical block size is 4096 bytes, then your maximum UDF file system capacity will be 16 TiB
If your drive has capacity in excess of this maximum size, the extra capacity will not be used. This is a limitation of UDF itself.
For maximal OS compatibility, use format-udf on a device having a logical block size of 512 bytes. This will limit your total capacity to 2 TiB, but the resultant device should work on the most operating systems.
For maximal resultant UDF file system capacity, use use format-udf on a device having a logical block size of 4096 bytes. This will increase your total capacity (from 2 TiB) to 16 TiB, but will limit the number/types of operating systems that will be able to mount/read/write the resultant device. See compatibility tables above for more detail.
For a human-readable device label, use format-udf in one of the following configurations:
- Run format-udf on Linux
- Run format-udf on macOS, but modify the drive label using Linux or Windows
- After installing GRUB2 on a partitionless drive, you can use
fdiskto set the partition as active if your BIOS can't boot from partitionless drives. (Thanks to @tome- for the tip.)
- Thanks to @walterav1984 for his contribution to add NVMe support