gfid
----

This repository contains tools that assist in identifying
files/directories with mismatched gfid's on a GlusterFS backend.

Sample session
--------------

Let's say that you have a 4-node distribute-replicate cluster that has
inodes with mismatched gfid's. We'll assume the bricks are called:

/export/brick1
/export/brick2
/export/brick3
/export/brick4

It doesn't matter that these are on different machines. The tools in
this repository only work on file lists gathered (manually) from these
bricks.

The first step is to construct the master list of all files:

# cd /export/brick1
# find . > brick1.txt

# cd /export/brick2
# find . > brick2.txt

# cd /export/brick3
# find . > brick3.txt

# cd /export/brick4
# find . > brick4.txt

# cat brick1.txt brick2.txt brick3.txt brick4.txt | sort -u > master_list.txt

Then we need to get the gfid's of all the inodes from these bricks:

# cd /export/brick1
# gfid-list /path/to/master_list.txt > brick1.gfid

# cd /export/brick2
# gfid-list /path/to/master_list.txt > brick2.gfid

# cd /export/brick3
# gfid-list /path/to/master_list.txt > brick3.gfid

# cd /export/brick4
# gfid-list /path/to/master_list.txt > brick4.gfid

Now we're ready to identify the mismatched inodes:

# gfid-mismatch brick1.gfid brick2.gfid brick3.gfid brick4.gfid > mismatched.txt

Another script named gfid-mismatch2 allows to detect the inode that have
missing gfid's or multiple gfid's for a same inode, in a context where bricks
are really broken, with different set of files, in replicated or distributed
(or both) mode:

# gfid-mismatch2 brick1.gfid brick2.gfid brick3.gfid brick4.gfid > mismatched.txt

This will generate the list of files with mismatched gfid's, one filename per line.

We can delete the gfid's now by doing:

# gluster volume stop <affected volume>

# cd /export/brick1
# gfid-delete /path/to/mismatched.txt

Repeat for the other bricks. `gfid-delete' will also produce a log
with one entry for each file, which is either:

usr/bin/factor: removed OK
        OR
usr/bin/vim: No such file or directory

(or another error).

IMPORTANT NOTE: The deletion of gfid's must be done ONLY ON A STOPPED
VOLUME. Deleting the gfid's on a running volume with mounted clients
will cause more problems instead of solving them.

Verbose mode
------------

If you want to know details about the mismatch you can run:

# gfid-mismatch -v brick1.gfid brick2.gfid brick3.gfid brick4.gfid > mismatched.txt

This will produce output something like this:

usr/bin/factor: brick1.gfid brick2.gfid | brick3.gfid brick4.gfid
usr/bin/seq: brick1.gfid | brick3.gfid
[...]

The first line means that usr/bin/factor on brick1 and brick2 has the
same gfid which is different from the gfid on brick3 and brick4.

The second line means that the gfid of usr/bin/seq on brick1 is
different from that on brick3. brick2 and brick4 either don't have the
attribute or the inode doesn't exist on them.

Note that if you use the verbose mode you can't directly pass its
output to gfid-delete. You'll have to strip out everything except the
filename by doing:

# cat mismatched.txt | cut -f1 -d: > mismatched2.txt

However, beware that this will not work properly if the filenames
themselves contain the ':' character.

Deleting .glusterfs links
-------------------------

In Gluster versions 3.3 and above, there is a hidden .glusterfs directory that contains links to files and directories. The names of these links are based on the gfids. You will want to delete these links as well.

Create a gfid list for only mismatched files on each brick:

# cd /export/brick1
# gfid-list /path/to/mismatched.txt > brick1_mismatched.gfid

# cd /export/brick2
# gfid-list /path/to/mismatched.txt > brick2_mismatched.gfid

# cd /export/brick3
# gfid-list /path/to/mismatched.txt > brick3_mismatched.gfid

# cd /export/brick4
# gfid-list /path/to/mismatched.txt > brick4_mismatched.gfid


Delete the links for the defunct gfids:

# cd /export/brick1/.glusterfs
# gfid-unlink /path/to/brick1_mismatched.gfid

PLEASE NOTE: You must cd into the hidden .glusterfs directory on each brick before running gfid-unlink


----
Vikas Gorur
http://github/vikasgorur