-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vdev_id and multiple disk enclosures #2074
Comments
Hi @ramonfernandez, we've only tested vdev_id with a limited range of hardware and software configurations, so it's not surprising to find cases where it doesn't work as expected. Can you post the output from this command in a gist:
where block_device is one of the disk names (just the name, not the full path, i.e. sda) that isn't getting a by-vdev entry? |
On Wed, Jan 22, 2014 at 11:20:41AM -0800, Ned Bass wrote:
Attached. This is for "sdy", which is on slot-1 of the second enclosure in Cheers,
|
I would think it would be more useful to have the new enclosure show up as another "channel" (i.e. C) rather than extending the slot numbers of A. After all, the idea is to simplify physically locating a disk, and it may not be obvious which enclosure has slots 1-12 and which has 13-24. The "sas_direct" topology is based on the notion of one enclosure per HBA port. Since both of your enclosures appear on the same port their disks are getting conflicting names. To define channels in daisy-chained topologies we need an additional piece of location information beside PCI slot and port number; we need to know the position in the chain. To preserve backward compatibility with the existing "channel" syntax I think we'd need to define a new topology to support this, i.e. "sas_daisychain". I don't have access to a system like this, so I'm not sure how the daisy-chain architecture gets reflected under /sys. But please do also post the vdev_id trace output for sde, so I can see where the paths under /sys start to diverge. |
I attached a third enclosure and I think I have it. All enclosures show up /sys/devices/pci0000:00/0000:00:04.0/0000:0c:00.0/host9/port-9:0/expander-9:0 After that each disk in the first enclosure is on a port number, and the udevadm info -q path -p /sys/block/sde.../port-9:0:0/end_device-9:0:0/target9:0:0/9:0:0:0/block/sde (where "..." is the common path above). This is disk 1 on enclosure 1, udevadm info -q path -p /sys/block/sdm.../port-9:0:9/expander-9:1/port-9:1:0/end_device-9:1:0/target9:0:9/9:0:9:0/block/sdm This is disk 1 on enclosure 2. Same thing for enclosure 3: udevadm info -q path -p /sys/block/sdy.../port-9:0:9/expander-9:1/port-9:1:13/expander-9:2/port-9:2:0/end_device-9:2:0/target9:0:22/9:0:22:0/block/sdy It seems that the names expander-X:Y, end_device-X:Y:Z, and port-X:Y:W all I fully agree that disk identification is the main goal, and increasing Hope this helps. I'll be happy to send further information if needed, and Cheers, set -e while [ $path != "/" ]; do case $name in echo |
When parsing the patah provided by udevadm in reverse, that is :) |
Thanks, that helps a lot. In hindsight, the config file format I chose isn't very easily extensible 😃 But, I'm thinking we could add a new optional parameter for the "channel" keyword in
This would address the current case without overly-complicating the parsing logic. But I can see that vdev_id will need to move to a more flexible config file format if it is to be extended much further. |
Hi Ned -- yes, I think that would work. You wrote:
Wouldn't it be easier for backwards compatibility to swap enclosure number BTW, another thing I found out is that looking at the bay_identifier file Anway, thanks a lot for your help, much appreciated. Cheers, |
Yes, it would be easier that way, but it seems more intuitive to me to keep the location "coordinates" grouped together, followed by the channel name. In hindsight, I should have made the channel name be the first parameter. (Well, in hindsight the order should have been insignificant, but this was a bit quick and dirty too.)
You're welcome. I'm happy to get the feedback. I'll post a patched vdev_id for testing shortly. |
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same HBA and port. The channel keyword now supports an optional numberic enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Signed-off-by: Ned Bass <bass6@llnl.gov> Fixes openzfs#2074
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same physical port. The channel keyword now supports an optional numeric enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Signed-off-by: Ned Bass <bass6@llnl.gov> Fixes openzfs#2074
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same physical port. The channel keyword now supports an optional numeric enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Signed-off-by: Ned Bass <bass6@llnl.gov> Fixes openzfs#2074
@ramonfernandez please give 62f2d9e a try. |
btw, you can run vdev_id by hand to test it out before installing the patched version. e.g.
It should hopefully print something like
|
As for cXtYdZ naming, you could always simulate that by encoding the cXtYd part in your channel names. |
Ned, in 62f2d9e, line 132 ("sas_switch" in case statement) of the original multipath no topology sas_direct phys_per_port 4 channel 0c:00.0 0 0 c0e1d channel 0c:00.0 0 1 c0e2d channel 0c:00.0 0 2 c0e3d So far so good. I then connected the last enclosure of this chain to the multipath no topology sas_direct phys_per_port 4 channel 0c:00.0 0 0 c0e1d channel 0c:00.0 0 1 c0e2d channel 0c:00.0 1 2 c1e3d This did not work well, because the number assigned to the enclosures is channel 0c:00.0 0 0 c0e1d channel 0c:00.0 0 1 c0e2d channel 0c:00.0 1 3 c1e3d to channel 0c:00.0 0 0 c0e1d channel 0c:00.0 0 2 c0e2d channel 0c:00.0 1 1 c1e3d Looks like the controller number from vdev_id and the one in the config A more flexible option would be to add a new "enclosure" keyword to [9:0:8:0] enclosu sas:0x50014380004fa7a5 - [9:0:21:0] enclosu sas:0x50014380004db4a5 - [9:0:34:0] enclosu sas:0x50014380002d1ca5 - so a vdev_id.conf like this: channel 0c:00.0 0 A- channel 0c:00.0 1 B- enclosure E1- 0x50014380004fa7a5 enclosure E2- 0x50014380004db4a5 enclosure E3- 0x50014380002d1ca5 could produce the right devices (eg: A-E1-1, A-E3-1, B-E2-1, etc.). I can Cheers, |
Thanks for testing it out.
Actually I suspect this would also be a problem for things like power supply failure that can cause an enclosure to disappear and come back. This is not an uncommon situation for large scale production environments, so we'd need to come up with a more reliable means of identifying the enclosure before this change can land.
That's an interesting idea. I prefer solutions that use generic location information as opposed to unique hardware identifiers. This is because we run clusters of identical servers in my environment, and it's much easier to manage a single common configuration file. However, I'm not opposed to providing the functionality you suggest if you want to propose a patch. My suggestion would be to use a name=value syntax for the parameters, i.e.
That would allow for new parameters to be added in the future to support other identification methods. |
I am having this issue as well with a supermicro storage server. The second enclosure in the system is not being recognized by vdev as a second set of disks on the same channel and does not show up. I am going to try the patch above (out of the box it is crashing on line 144 as s drop in vdev replacement) with the current 6.4 release. |
As Ramon said earlier, with this code, you need to add: |
Just to add my $0.02, this fixes my chained enclosure problems as well. Supermicro's "front and back" enclosures (eg: this beast) uses distinct, chained expanders for the front and back sides so this fix is needed. For quick reference 62f2d9e plus the syntax fix for the |
@DeHackEd please do, let's get a PR open with the proposed fix. |
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same physical port. The channel keyword now supports an optional numeric enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Original-version-by: Ned Bass <bass6@llnl.gov> Signed-off-by: DHE <git@dehacked.net> Fixes openzfs#2074
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same physical port. The channel keyword now supports an optional numeric enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Original-version-by: Ned Bass <bass6@llnl.gov> Signed-off-by: DHE <git@dehacked.net> Fixes openzfs#2074
Disks in enclosures in daisy-chained configurations will currently get conflicting names when using the sas_direct and sas_switch topologies. This is because the "channel" keyword syntax lacks sufficient location information to distinguish between enclosures connected the same physical port. The channel keyword now supports an optional numeric enclosure_id parameter to identify the position of an enclosure in a daisy-chained configuration. Daisy-chained enclosures are numbered starting from 0. Original-version-by: Ned Bass <bass6@llnl.gov> Signed-off-by: DHE <git@dehacked.net> Fixes openzfs#2074 Requires-builders: style build
The vdev_id support was extend some time ago to support multiple enclosures in various arrangements. Closing. |
How was this issue resolved? I'm looking at the current documentation for vdev_id to try and set up a daisy-chained enclosure configuration but I don't see any of the discussed solutions in there. I've been unable to find a proper configuration for this type of topology. |
Thanks! I didn't see the |
Greetings!
I've daisy-chained two MSA60 12-disk enclosures to a box running ZFS via a SAS1068E controller (the second port has a third MSA60 attached to it, so I've come to the point where I need the daisy-chaining). I use the vdev_id.conf displayed below. Using one enclosure per port I do get devices A1..A12 and B1..B12 in /dev/disk/by-vdev, but when the second enclosure is connected I don't get any more devices (ie: A13..A24).
The disks are recognized by the system (ie: I get /dev/sd?? nodes) and are accessible. I'm using 0.6.2-1~precise from the Ubuntu ppa on a 3.11.0-15-generic kernel (but this is not yet a production box so I'm open to trying other versions of ZFS).
Is this expected? Or am I doing something wrong? I can set up aliases for all new disks, but having vdev IDs is such a nice feature that I'm sad to not have it :-)
Thanks in advance fro any help you can provide!
Cheers,
Ramón.
/etc/zfs/vdev_id.conf
multipath no
topology sas_direct
phys_per_port 4
channel 0c:00.0 0 A
channel 0c:00.0 1 B
The text was updated successfully, but these errors were encountered: