nfs-Ganesha integration

[aravindavk]

• Add new API/CLI to enable and export Kadalu Storage Volumes via NFS ganesha

# Export a Volume
kadalu nfs-ganesha export <mgr-url:port>:/<pool>/<volume>

# Unexport a Volume
kadalu nfs-ganesha unexport <mgr-url:port>:/<pool>/<volume>

# List Local exports
kadalu nfs-ganesha list-local

[aravindavk]

@amarts, @sac the issue is available for nfs-ganesha export. Any suggestions to make export and unexport command to look like local command. That is, it exports the Volume from the current node where we run the command.

[aravindavk]

With this gluster/glusterfs#3745 enhancement in Gluster. Vol file server is available with every Storage Unit processes. No changes required from Kadalu Storage to export with NFS Ganesha. We need to package nfs-ganesha + gluster integration as nfs-ganesha-kadalu package to avoid mixing with Gluster versions.

• Document manual way - Create the export config file and export
• Implement the above mentioned utility commands that takes care of generating config files.

[vatsa287]

@aravindavk @amarts

Intial Observations on nfs-ganesha + moana

• NFS ganesha requires the volume to be run 24007/glusterfs default port, tried with no port option with moana volume and failed.
  NFS-ganesha should be able to process other ports as well. This needs to fixed at nfs-ganesha repo.

• The client volfile in /var/lib/kadalu/volfiles/client-dev-vol1.vol need to be renamed to <volname>.vol during storage_unit init at
  https://github.com/kadalu/moana/blob/main/mgr/src/server/plugins/volume_utils.cr#L287 & in accessing these volfiles in mount_script.cr at https://github.com/kadalu/moana/blob/main/mgr/src/cmds/mount_script.cr#L181

• nfs-ganesha has a reliance on glusterd for getting volume uuid by one of the gf-api. So this is a blocker which restarts the nfs-ganesha process after remote volume is found.

https://github.com/gluster/glusterfs/blob/devel/api/src/glfs-mgmt.c#L363

    /* Need to fetch volume_uuid */
    glfs_get_volume_info(fs);

[2022-11-24 06:18:51.043320 +0000] E [MSGID: 104066] [glfs-mgmt.c:275:mgmt_get_volinfo_cbk] 0-gfapi: GET_VOLUME_INFO RPC call is not successful [{errno=22}, {error=Invalid argument}] 
[2022-11-24 06:18:51.043336 +0000] E [MSGID: 104015] [glfs-mgmt.c:340:mgmt_get_volinfo_cbk] 0-gfapi: In GET_VOLUME_INFO cbk, received error [{error=Invalid argument}, {errno=22}, {error=Invalid argument}] 
[2022-11-24 06:18:51.043350 +0000] E [MSGID: 104016] [glfs-mgmt.c:385:pub_glfs_get_volumeid] 0-gfapi: Unable to fetch volume UUID [{errno=22}, {error=Invalid argument}] 

nfs-ganesha setup

• Create moana volume with port 24007
• Add export conf at /etc/ganesha/ganesha.conf

EXPORT
{
       # Export Id (mandatory, each EXPORT must have a unique Export_Id)
       Export_Id = 1;

       # Exported path (mandatory)
       Path = "/vol1";

       # Pseudo Path (required for NFS v4)
       Pseudo = "/vol1";

       # Required for access (default is None)
       # Could use CLIENT blocks instead
       Access_Type = RW;

       # Allow root access
       Squash = No_Root_Squash;

       # Security flavor supported
       SecType = "sys";

       # Exporting FSAL
       FSAL {
               Name = "GLUSTER";
               Hostname = kadalu-dev;
               Volume = "vol1";
               enable_upcall = true;
               Transport = tcp; # tcp or rdma
       }
}

• Trigger nfs-ganesha exporter cmd ganesha.nfsd -f nfs-ganesha.conf -L nfs-ganesha.log -N NIV_FULL_DEBUG
• cmd to check for successful mount showmount -e kadalu-dev. This cmd currently outputs showmount -e kadalu-dev clnt_create: RPC: Unknown host
• Logs of ganesha at /var/log/ganesha/ganesha-gfapi.log & /var/log/ganesha/ganesha.log

[aravindavk]

The client volfile in /var/lib/kadalu/volfiles/client-dev-vol1.vol need to be renamed to .vol during storage_unit init at
https://github.com/kadalu/moana/blob/main/mgr/src/server/plugins/volume_utils.cr#L287 & in accessing these volfiles in mount_script.cr at https://github.com/kadalu/moana/blob/main/mgr/src/cmds/mount_script.cr#L181

Fix this issue for 1.0 release itself since changing this lator may need extra effort to handle in backward compatible way.

nfs-ganesha has a reliance on glusterd for getting volume uuid by one of the gf-api. So this is a blocker which restarts the nfs-ganesha process after remote volume is found.

@amarts is it possible to add volume-id in the generated Client Volfile and change gfapi to use that to get Volume ID. If not exists in Volfile then call the existing method so that it will be backward compatible.

[amarts]

Should be possible to add volume-id in client volfile.

[aravindavk]

NFS Ganesha integration worked! Little changes to the nfs-ganesha build steps(GLUSTER_PREFIX) if glusterfs is installed as source installation(github.com/kadalu/glusterfs kadalu_1 branch)

cmake -DUSE_FSAL_GLUSTER=ON -DGLUSTER_PREFIX=/usr/local ../src/

One more dependency required was rpcbind

[aravindavk]

CLIs:

kadalu nfs enable <pool>/<volume> [--nodes=node1,node2]
kadalu nfs disable <pool>/<volume> [--nodes=node1,node2]

Above commands enable or disable the NFS Ganesha exports in any of the managed nodes. If --nodes not given then creates the NFS ganesha export files from all the Volume nodes.

Note: Do not start/restart nfs-ganesha service, only handle export and unexport of volumes.

Another option to enable the NFS export locally (Outside the Storage cluster).

kadalu nfs enable/disable <mgr-url:port>:/<pool>/<volume> --local

To enable:

• Create a export config file using template with a export ID generated
• Add the export config file in the main ganesha config
• Send dbus signal to notify running nfs-ganesha server (Refer: How to add and remove exports using dbus-send? nfs-ganesha/nfs-ganesha#94 (comment))

To disable:

• Remove the export file
• Remove the entry from main ganesha config file
• Send dbus signal

[aravindavk]

Ref: Gluster Hook script that enables the NFS ganesha service

https://github.com/gluster/glusterfs/blob/1d534b57d279a59b8bded3a5fe57801c20a3e32c/extras/hook-scripts/start/post/S31ganesha-start.sh