qsfs is a FUSE based filesystem that allows you to mount a qingstor bucket in Linux.
- Large subset of POSIX including reading/writing files, directories, symlinks, .etc, (chmod/chown/utimens will be supported later).
- File renames via server-side move.
- File Transfer:
- Large files uploads via multipart parallel uploads.
- Large files downloads via parallel byte-range downloads.
- Large files transfer in chunks (10 MB chunks by default). If you are uploading large files (e.g. larger than 1 GB), you can increase the transfer buffer size and the max parallel transfers.
- Data integrity via MD5 hashes.
- Cache:
- In-memory metadata caching.
- In-memory file data caching. For a big file, partial file data may been stored in a local disk file when the im-memory file cache is not available.
- Logging/Debugging:
- Support to log messages to console or to a directory.
- Support turn on debug message to log by specifying debug option -d, this option will also enable FUSE debug mode.
- You can turn on debug message from curl by specifying option -U.
See the INSTALL for installation instructions.
Enter your QingCloud API access keys in a file /path/to/cred, go to QingCloud Console to create them if you do not have one:
$ echo YourAccessKeyId:YourSecretKey > /path/to/credMake sure the file /path/to/cred has proper permissions (if you get 'permissions' error when mounting):
$ chmod 600 /path/to/credRun qsfs with an existing bucket mybucket and directory /path/to/mountpoint:
$ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/credIf you encounter any errors, enable debug output:
$ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -dYou can also mount on boot by entering the following line to /etc/fstab:
qsfs#mybucket /path/to/mountpoint fuse _netdev,-z=pek3a,-c=/path/to/cred,allow_other 0 0Notice: in /etc/fstab, you need to use short options (for example, use short option
-cinstead of long option--credentials) in order to pass them to qsfs.
You can log messages to console:
$ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -fOr you can log messages to log file by specifying a directory /path/to/logdir/:
$ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -l=/path/to/logdir/Specify log level (INFO,WARN,ERROR and FATAL):
$ [sudo] qsfs mybucket /path/to/mountpoint -c=/path/to/cred -L=INFO -dTo umount:
$ [sudo] fusermount -uqz /path/to/mountpointor
$ [sudo] umount -l /path/to/mountpointFor help:
$ qsfs -hSupported general options are listed as following,
| short | full | type | required | usage |
|---|---|---|---|---|
| -z | --zone | string | Y | Specify zone or region |
| -c | --credentials | string | N | Specify credentials file, default path is /etc/qsfs.cred |
| -l | --logdir | string | N | Specify log directory, default path is /var/log/qsfs/ |
| -L | --loglevel | string | N | Specify min log level (INFO,WARN,ERROR or FATAL), message lower than this level don't logged; default value is INFO. |
| -F | --filemode | octal | N | Specify the permission bits in st_mode for file objects without x-qs-meta-mode header. The value is given in octal representation; default value is 0644 |
| -D | --dirmode | octal | N | Specify the permission bits in st_mode for directory objects without x-qs-meta-mode header. The value is given in octal representation; default value is 0755 |
| -u | --umaskmp | octal | N | Specify the permission bits in st_mode for the mount point directory. This option only works when you set with the fuse allow_other option. The resulting permission bits are the ones missing from the given umask value. The value is given in octal representation; default value is 0000 |
| -r | --retries | integer | N | Specify number of times to retry a failed transaction, default value is 3 times |
| -R | --reqtimeout | integer | N | Specify time (seconds) to wait before giving up connection, default value is 300 seconds |
| -Z | --maxcache | integer | N | Specify max in-memory cache size (MB) for files, default value is 200 MB |
| -k | --diskdir | string | N | Specify the directory to store file data when in-memory cache is not availabe, default path is /tmp/qsfs_cache/ |
| -t | --maxstat | integer | N | Specify max count (K) of cached stat entrys, default value is 2 K |
| -e | --statexpire | integer | N | Specify expire time (minutes) for stat entries, negative value will disable stat expire, default is no expire |
| -i | --maxlist | integer | N | Specify max count of files of ls operation. A value of zero will list all files, default is to list all files |
| -y | --fscap | integer | N | Specify filesystem capacity (GB), default value is 1PB |
| -n | --numtransfer | integer | N | Specify max number file tranfers to run in parallel, you can increase the value when transfer large files, default value is 5 |
| -b | --bufsize | integer | N | Specify file transfer buffer size (MB), this should be larger than 8MB, default value is 10 MB |
| -j | --prefetchsize | integer | N | Specify file read max prefetch size (MB), default value is 20 MB |
| -H | --host | string | N | Specify host name, default value is qingstor.com |
| -p | --protocol | string | N | Specify protocol (https or http) default value is https |
| -P | --port | integer | N | Specify port, default is 443 for https and 80 for http |
Supported miscellaneous options are list as following,
| short | full | type | required | usage |
|---|---|---|---|---|
| -J | --prefetch | bool | N | Enable read prefetch |
| -m | --contentMD5 | bool | N | Enable writes with MD5 hashs to ensure data integrity |
| -K | --keeplogdir | bool | N | Do not clear log directory at beginning |
| -C | --nodatacache | bool | N | Clear the file data cache |
| -f | --forground | bool | N | Turn on log to STDERR and enable FUSE foreground mode |
| -s | --single | bool | N | Turn on FUSE single threaded option - disable multi-threaded |
| -d | --debug | bool | N | Turn on debug messages to log |
| -g | --fusedbg | bool | N | Enable FUSE debug option |
| -U | --curldbg | bool | N | Turn on debug message from curl |
| -h | --help | bool | N | Print qsfs help |
| -V | --version | bool | N | Print qsfs version |
You can also specify FUSE specific mount options with -o opt [,opt...] e.g. nonempty, allow_other, etc. See the FUSE's README for the full set.
Generally qingstor cannot offer the same performance or semantics as a local file system. More specifically:
- Random writes or appends to files require rewriting the entire file
- Metadata operations such as listing directories have poor performance due to network latency
- Eventual consistency can temporarily yield stale data
- No atomic renames of files or directories
- No coordination between multiple clients mounting the same bucket
- No hard links
If you notice any issue, please open an issue on GitHub. Please search the existing issues and see if others are also experiencing the issue before opening a new issue. Please include the version of qsfs, Compiler version, CMake version, and OS you’re using.
See the LICENSE for details. In summary, qsfs is licensed under the Apache License (Version 2.0, January 2004).