I am wondering what the best way to set up plotman archiving would be.

[DendyDarzartar]

Hello All,

I have my destination drives and my temporary drives both set up within the root of the /mnt/ folder. However, I see that it is not recommended to use /mnt/ as the site root for archiving purposes.

What should I have for the archiving section if I just want plotman to write across specific sub folders within some of the drives I have mounted?

My lsblk output (I only want plots being written as a final destination to sub-folders within the /mnt/storageX folders):

 sdb                     8:16   0 894.3G  0 disk  
└─sdb1                  8:17   0 894.3G  0 part  
  └─md127               9:127  0   1.8T  0 raid0 /mnt/plottingraid1
sdc                     8:32   0 894.3G  0 disk  
└─sdc1                  8:33   0 894.3G  0 part  
  └─md127               9:127  0   1.8T  0 raid0 /mnt/plottingraid1
sdd                     8:48   0   1.8T  0 disk  
└─sdd1                  8:49   0   1.8T  0 part  /mnt/plottingdisk1
sde                     8:64   0  10.9T  0 disk  
└─sde1                  8:65   0  10.9T  0 part  /mnt/storage1
sdf                     8:80   0  14.6T  0 disk  
├─sdf1                  8:81   0   128M  0 part  
└─sdf2                  8:82   0  14.6T  0 part  /mnt/storage2
sdg                     8:96   0  14.6T  0 disk  
└─sdg1                  8:97   0  14.6T  0 part  /mnt/storage3
sdh                     8:112  0  14.6T  0 disk  
└─sdh1                  8:113  0  14.6T  0 part  /mnt/storage4
sdi                     8:128  0  16.4T  0 disk  
└─sdi1                  8:129  0  16.4T  0 part  /mnt/storage5
sdj                     8:144  0  10.9T  0 disk  
└─sdj1                  8:145  0  10.9T  0 part  /mnt/storage6
sdk                     8:160  0  14.6T  0 disk  
└─sdk1                  8:161  0  14.6T  0 part  /mnt/storage7
sdl                     8:176  0   9.1T  0 disk  
└─sdl1                  8:177  0   9.1T  0 part  /mnt/storage8
sdm                     8:192  0  14.6T  0 disk  
└─sdm1                  8:193  0  14.6T  0 part  /mnt/storage9
sdn                     8:208  0  14.6T  0 disk  
└─sdn1                  8:209  0  14.6T  0 part  /mnt/storage10
sr0                    11:0    1  1024M  0 rom   
nvme0n1               259:0    0   1.8T  0 disk  
└─nvme0n1p1           259:1    0   1.8T  0 part  /mnt/plottingdisk2

Theses are the aforementioned specific sub-folders I would want the plots to write to and are posted here as written in the plotman.yaml's destination drive section

            - /mnt/storage3/pool plots
            - /mnt/storage4/pool plots
            - /mnt/storage5/pool_plots
            - /mnt/storage7/pool_plots
            - /mnt/storage9/pool plots

So in order to have the archive split the storage of plots evenly amongst the above but not have it write to my plotting drives such as /mnt/plottingraid for example, what would I need to do?

Would I want to have it look something like the below with all of the entries listed or is it going to mess things up with multiple entries?

 archiving:
  target: local_rsync
  env:
    site_root: /mnt/storage3/pool plots
                 /mnt/storage4/pool plots
                 /mnt/storage5/pool_plots
                 /mnt/storage7/pool_plots
                 /mnt/storage9/pool plots

I can make a separate folder called "/mnt/storage" directory and modify the fstab to point to that and remount them so that the individual "storageX" folders show under "/mnt/storage" if need be and just have the one site root value of "/mnt/storage" but then I wonder how would I be able to ensure only the destination sub-folders listed above get written to as I have other sub-folders on the same drive specifically for solo plots that I would not be plotting to with plotman and I would have no way of differentiating pool and solo plots if they started plotting to locations outside of the explicitly named folders.

Any advice you can give is greatly appreciated!

[altendky]

We might need a couple back and forths but yes, I would recommend putting the drives you want archived to into /mnt/storage or such. Note that plotman archiving doesn't "distribute evenly" it "fills all drives". As to having a pool_plots directory, a feature was added for that recently. It is the path_suffix: entry.

https://github.com/altendky/farm/blob/cd1e8be061df328e1b66bb9145f3569010f29d2c/config.server.yaml#L60-L68

     path_suffix: pool.xch1ph3flzr053eswzjc72gjvxjes6fcdamvl9kq5yqw6dnmwtpmlzkq8lzjmn

A archival target definition could certainly be written that configures the specific paths you want to archive to, but so far it hasn't been a priority for me and I haven't seen anyone else do it. Basically, you would just rewrite the disk_space_script: script to return the directories you wanted and their free space based on a new parameter you would add with the list of directories in it.

plotman/src/plotman/resources/target_definitions.yaml

Lines 18 to 24 in 77f85e3

	disk_space_script: |
	#!/bin/bash
	set -evx
	site_root_stripped=$(echo "${site_root}" | sed 's;/\+$;;')
	# printf with %.0f used to handle mawk such as in Ubuntu Docker images
	# otherwise it saturates and you get saturated sizes like 2147483647
	df -aBK | grep " ${site_root_stripped}/" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'

Let me know how close this gets you.

[DendyDarzartar]

Ahh ok, I think I have a bit more reading to do before I understand all of what goes into setting up archiving.
Thank you for getting me headed in the right direction!

[altendky]

If you want to stop by the #plotman channel on Chia's Keybase team we could try to have a more interactive discussion if that would help.