Skip to content

Commit 4cd6e44

Browse files
Add gufi to modules (#837)
Previewing GUFI added to Modules section of Support Page. * May not be the best place for it at the moment, but available for users to preview --------- Co-authored-by: jen-reeve <31419037+jen-reeve@users.noreply.github.com>
1 parent b58f487 commit 4cd6e44

File tree

3 files changed

+170
-1
lines changed

3 files changed

+170
-1
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@ production/*
33
**.pyc
44
.venv/*
55
dictionary.dic
6+
.DS_Store

README.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,11 @@ pip-compile --allow-unsafe > requirements.txt
5656
pip install -r requirements.txt
5757
```
5858

59-
Make sure to test it on a GitHub runner (not just locally), as this is the actual build environment.
59+
Make sure to test it on a GitHub runner (not just locally), as this is the actual build environment. To run locally:
60+
61+
```sh
62+
mkdocs serve -c
63+
```
6064

6165
## Migration
6266

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
---
2+
created_at: '2025-10-30T17:00:00Z'
3+
tags: []
4+
title: GUFI
5+
vote_count: 0
6+
vote_sum: 0
7+
---
8+
9+
test
10+
11+
GUFI (Grand Unified File Index) is a file system metadata indexing tool designed for large-scale data centers to enable fast, secure, and comprehensive searches of files and directories. It works by creating a hierarchical index that preserves file access permissions, allowing users to efficiently find and characterize data across multiple, potentially disparate file systems. This results in significantly faster search times and lessens impact/load on parallel filesystems compared to traditional methods.
12+
13+
There are two commands that GUFI provides:
14+
15+
* `gufi_find`: For finding files and subfolders in a directory
16+
* `gufi_du`: For obtaining the size of files and folders
17+
18+
!!! warning
19+
This method uses a database that is updated on a weekly basis. It may not find or measure the size of files that were created or moved around Mahuika within the last week.
20+
21+
!!! note
22+
The filesystems that `gufi_find` and `gufi_du` work on are:
23+
24+
* `\home`
25+
* `\projects`
26+
* `\nobackup`
27+
28+
For `gufi_find` and `gufi_du` to work, you must give the path name beginning from `project` or `nobackup`. Local addresses starting with `.` will also not work. For example:
29+
30+
* `nobackup/nesi12345/a_folder` and `/nobackup/nesi12345/a_folder` are acceptible, but
31+
* `/nesi/nobackup/nesi12345/a_folder`, `nesi12345/a_folder`, or `./nesi12345/a_folder` will not work.
32+
33+
34+
## Prerequisite: Must Load `gufi` Module
35+
36+
To use `gufi_find` and `gufi_du`, you must load them by entering in the following command in Mahuika:
37+
38+
```sh
39+
module load .gufi
40+
```
41+
42+
43+
## Finding Files and Folders using GUFI
44+
45+
The usual method for searching for a file using the terminal is:
46+
47+
```sh
48+
find path/to/folder/to/search -name <your_file>
49+
```
50+
51+
In GUFI, you provide the same arguments to `gufi_find` as you do with `find` (but starting with either `home`, `projects`, or `nobackup`):
52+
53+
```sh
54+
gufi_find path/to/folder/to/search -name <your_file>
55+
```
56+
57+
!!! example
58+
If you want to find `.bashrc` in your home directory:
59+
60+
```sh
61+
gufi_find home/$USER -name .bashrc
62+
```
63+
64+
If you want to find the largest file in your folder:
65+
66+
```sh
67+
gufi_find home/$USER -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1
68+
gufi_find project/nesi12345 -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1
69+
gufi_find nobackup/nesi12345 -type f -printf '%s %p\n' 2>/dev/null | sort -nr | head -n 1
70+
```
71+
72+
## Obtaining the Size of Files and Folders using GUFI
73+
74+
The usual method for obtaining the size of a file or folder using the terminal is:
75+
76+
```sh
77+
du -s path/to/file/or/folder
78+
```
79+
80+
In GUFI, you provide the same arguments to `gufi_du` as you do with `du` (but starting with either `home`, `projects`, or `nobackup`):
81+
82+
```sh
83+
gufi_du -s path/to/file/or/folder
84+
```
85+
86+
For more options, see `gufi_du --help`
87+
88+
!!! example
89+
If you want to find the size of your in your home directory:
90+
91+
```sh
92+
gufi_du -s home/$USER
93+
```
94+
95+
If you want to find the size of your `.bashrc` in your home directory:
96+
97+
```sh
98+
gufi_du -s home/$USER/.bashrc
99+
```
100+
101+
If you want to find the size of your project folder in `projects` or `nobackup`:
102+
103+
```sh
104+
gufi_du -s project/nesi12345
105+
gufi_du -s nobackup/nesi12345
106+
```
107+
108+
If you want to obtain the number of files in a folder, such as `nobackup/nesi12345/a_folder`, you would do the following:
109+
110+
```sh
111+
gufi_du --inodes -s nobackup/nesi12345/a_folder
112+
```
113+
114+
## Troubleshooting
115+
116+
### My file or folder exists, but `GUFI` does not include it during its search
117+
118+
If your file or folder exists but `GUFI` does not find it, it is likely that your folder has not been indexed by `GUFI` yet. You will need to wait until the end of the week for those files and folders to be indexed by `GUFI`.
119+
120+
* In the meantime, you can either use `find` or `du` instead of `gufi_find` or `gufi_du` as alternatives for these functions.
121+
122+
!!! example
123+
124+
```sh
125+
john.doe@login03:~$ gufi_find home/new_folder
126+
Could not get realpath of "/search/home/new_folder": No such file or directory (2)
127+
```
128+
129+
### I get the message: `Does "XYZ" have treesummary data?`
130+
131+
If you get a message like this:
132+
133+
```sh
134+
gufi_du --inodes -s nobackup/nesi99991
135+
Error: Skipping directory "/search/nobackup/nesi99991": Permission denied (13)
136+
0 nobackup/nesi99991
137+
Warning: Did not get any results from gufi_query.
138+
Does "nobackup/nesi99991" have treesummary data?
139+
```
140+
141+
This means that `gufi_find` or `gufi_du` was not able to find any information about the path you gave it to search. This could be because:
142+
143+
1. The folder you are search in doesn't exist,
144+
2. You don't have permissions to look at the files and folders that you were trying to search in, or
145+
3. Your files and folders were created before they were included in the `GUFI` database.
146+
147+
If 3 applies to you, you will need to wait until the end of the week for those files and folders to be indexed by `GUFI`.
148+
149+
### I get the message: `Error: Skipping directory "nobackup/XYZ": Permission denied (13)`
150+
151+
If you see this error, this is because you do not have the correct permissions to view this directory.
152+
153+
* If you want to use `GUFI` on this directory, you will need to get read permissions from the person who created this directory.
154+
155+
!!! example
156+
157+
```sh
158+
john.doe@login03:/nesi/nobackup/nesi12345$ gufi_du nobackup/nesi12345/test.txt
159+
Error: Skipping directory "nobackup/nesi12345/test.txt": Permission denied (13)
160+
```
161+
162+
### I can not use tab to autocomplete, and sometimes autocompleting using tab logs me out of my Mahuika login
163+
164+
This is a known problem. We are currently looking for a fix for this.

0 commit comments

Comments
 (0)