|
| 1 | +--- |
| 2 | +created_at: '2025-10-30T17:00:00Z' |
| 3 | +tags: [] |
| 4 | +title: GUFI |
| 5 | +vote_count: 0 |
| 6 | +vote_sum: 0 |
| 7 | +--- |
| 8 | + |
| 9 | +test |
| 10 | + |
| 11 | +GUFI (Grand Unified File Index) is a file system metadata indexing tool designed for large-scale data centers to enable fast, secure, and comprehensive searches of files and directories. It works by creating a hierarchical index that preserves file access permissions, allowing users to efficiently find and characterize data across multiple, potentially disparate file systems. This results in significantly faster search times and lessens impact/load on parallel filesystems compared to traditional methods. |
| 12 | + |
| 13 | +There are two commands that GUFI provides: |
| 14 | + |
| 15 | +* `gufi_find`: For finding files and subfolders in a directory |
| 16 | +* `gufi_du`: For obtaining the size of files and folders |
| 17 | + |
| 18 | +!!! warning |
| 19 | + This method uses a database that is updated on a weekly basis. It may not find or measure the size of files that were created or moved around Mahuika within the last week. |
| 20 | + |
| 21 | +!!! note |
| 22 | + The filesystems that `gufi_find` and `gufi_du` work on are: |
| 23 | + |
| 24 | + * `\home` |
| 25 | + * `\projects` |
| 26 | + * `\nobackup` |
| 27 | + |
| 28 | + For `gufi_find` and `gufi_du` to work, you must give the path name beginning from `project` or `nobackup`. Local addresses starting with `.` will also not work. For example: |
| 29 | + |
| 30 | + * `nobackup/nesi12345/a_folder` and `/nobackup/nesi12345/a_folder` are acceptible, but |
| 31 | + * `/nesi/nobackup/nesi12345/a_folder`, `nesi12345/a_folder`, or `./nesi12345/a_folder` will not work. |
| 32 | + |
| 33 | + |
| 34 | +## Prerequisite: Must Load `gufi` Module |
| 35 | + |
| 36 | +To use `gufi_find` and `gufi_du`, you must load them by entering in the following command in Mahuika: |
| 37 | + |
| 38 | +```sh |
| 39 | +module load .gufi |
| 40 | +``` |
| 41 | + |
| 42 | + |
| 43 | +## Finding Files and Folders using GUFI |
| 44 | + |
| 45 | +The usual method for searching for a file using the terminal is: |
| 46 | + |
| 47 | +```sh |
| 48 | +find path/to/folder/to/search -name <your_file> |
| 49 | +``` |
| 50 | + |
| 51 | +In GUFI, you provide the same arguments to `gufi_find` as you do with `find` (but starting with either `home`, `projects`, or `nobackup`): |
| 52 | + |
| 53 | +```sh |
| 54 | +gufi_find path/to/folder/to/search -name <your_file> |
| 55 | +``` |
| 56 | + |
| 57 | +!!! example |
| 58 | + If you want to find `.bashrc` in your home directory: |
| 59 | + |
| 60 | + ```sh |
| 61 | + gufi_find home/$USER -name .bashrc |
| 62 | + ``` |
| 63 | + |
| 64 | + If you want to find the largest file in your folder: |
| 65 | + |
| 66 | + ```sh |
| 67 | + gufi_find home/$USER -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1 |
| 68 | + gufi_find project/nesi12345 -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1 |
| 69 | + gufi_find nobackup/nesi12345 -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1 |
| 70 | + ``` |
| 71 | + |
| 72 | +## Obtaining the Size of Files and Folders using GUFI |
| 73 | + |
| 74 | +The usual method for obtaining the size of a file or folder using the terminal is: |
| 75 | + |
| 76 | +```sh |
| 77 | +du -s path/to/file/or/folder |
| 78 | +``` |
| 79 | + |
| 80 | +In GUFI, you provide the same arguments to `gufi_du` as you do with `du` (but starting with either `home`, `projects`, or `nobackup`): |
| 81 | + |
| 82 | +```sh |
| 83 | +gufi_du -s path/to/file/or/folder |
| 84 | +``` |
| 85 | + |
| 86 | +For more options, see `gufi_du --help` |
| 87 | + |
| 88 | +!!! example |
| 89 | + If you want to find the size of your in your home directory: |
| 90 | + |
| 91 | + ```sh |
| 92 | + gufi_du -s home/$USER |
| 93 | + ``` |
| 94 | + |
| 95 | + If you want to find the size of your `.bashrc` in your home directory: |
| 96 | + |
| 97 | + ```sh |
| 98 | + gufi_du -s home/$USER/.bashrc |
| 99 | + ``` |
| 100 | + |
| 101 | + If you want to find the size of your project folder in `projects` or `nobackup`: |
| 102 | + |
| 103 | + ```sh |
| 104 | + gufi_du -s project/nesi12345 |
| 105 | + gufi_du -s nobackup/nesi12345 |
| 106 | + ``` |
| 107 | + |
| 108 | + If you want to obtain the number of files in a folder, such as `nobackup/nesi12345/a_folder`, you would do the following: |
| 109 | + |
| 110 | + ```sh |
| 111 | + gufi_du --inodes -s nobackup/nesi12345/a_folder |
| 112 | + ``` |
| 113 | + |
| 114 | +## Troubleshooting |
| 115 | + |
| 116 | +### My file or folder exists, but `GUFI` does not include it during its search |
| 117 | + |
| 118 | +If your file or folder exists but `GUFI` does not find it, it is likely that your folder has not been indexed by `GUFI` yet. You will need to wait until the end of the week for those files and folders to be indexed by `GUFI`. |
| 119 | + |
| 120 | +* In the meantime, you can either use `find` or `du` instead of `gufi_find` or `gufi_du` as alternatives for these functions. |
| 121 | + |
| 122 | +!!! example |
| 123 | + |
| 124 | + ```sh |
| 125 | + john.doe@login03:~$ gufi_find home/new_folder |
| 126 | + Could not get realpath of "/search/home/new_folder": No such file or directory (2) |
| 127 | + ``` |
| 128 | + |
| 129 | +### I get the message: `Does "XYZ" have treesummary data?` |
| 130 | + |
| 131 | +If you get a message like this: |
| 132 | + |
| 133 | +```sh |
| 134 | +gufi_du --inodes -s nobackup/nesi99991 |
| 135 | +Error: Skipping directory "/search/nobackup/nesi99991": Permission denied (13) |
| 136 | +0 nobackup/nesi99991 |
| 137 | +Warning: Did not get any results from gufi_query. |
| 138 | +Does "nobackup/nesi99991" have treesummary data? |
| 139 | +``` |
| 140 | + |
| 141 | +This means that `gufi_find` or `gufi_du` was not able to find any information about the path you gave it to search. This could be because: |
| 142 | + |
| 143 | +1. The folder you are search in doesn't exist, |
| 144 | +2. You don't have permissions to look at the files and folders that you were trying to search in, or |
| 145 | +3. Your files and folders were created before they were included in the `GUFI` database. |
| 146 | + |
| 147 | +If 3 applies to you, you will need to wait until the end of the week for those files and folders to be indexed by `GUFI`. |
| 148 | + |
| 149 | +### I get the message: `Error: Skipping directory "nobackup/XYZ": Permission denied (13)` |
| 150 | + |
| 151 | +If you see this error, this is because you do not have the correct permissions to view this directory. |
| 152 | + |
| 153 | +* If you want to use `GUFI` on this directory, you will need to get read permissions from the person who created this directory. |
| 154 | + |
| 155 | +!!! example |
| 156 | + |
| 157 | + ```sh |
| 158 | + john.doe@login03:/nesi/nobackup/nesi12345$ gufi_du nobackup/nesi12345/test.txt |
| 159 | + Error: Skipping directory "nobackup/nesi12345/test.txt": Permission denied (13) |
| 160 | + ``` |
| 161 | + |
| 162 | +### I can not use tab to autocomplete, and sometimes autocompleting using tab logs me out of my Mahuika login |
| 163 | + |
| 164 | +This is a known problem. We are currently looking for a fix for this. |
0 commit comments