One client on two Xsans

01 Nov 2022

We have mentioned that we have several environments with multiple Xsans and clients that are connected to both of them. In all cases there are 2 Xsans.  One hosts the production volume with fast storage and many clients for video, audio, and graphics production.  The other Xsan hosts a nearline volume for synced backups of production and often another volume for a file share.  These volumes can be made with larger, slower drives and generally only have a few fibre connected clients.  

In the past we have dedicated clients to talking to each other over a 10Gb network connection for syncing.  This would require 2 clients for each connection; one on production and one on nearline.  Now we can have a dedicated bridge client with fibre connections to both Xsans and our syncs can run over 16Gb or even 32Gb fibre connections.

There are three things a client needs to be able to connect to an Xsan. First it needs to know where the MDCs are. Secondly, it needs to be authorized to talk to the MDCs. Lastly, the client needs to be able to access the LUNs that make up the additional volumes.

Authorization to talk to an MDC

When an Xsan is created, a string called the auth_secret is generated and stored in /Library/Preferences/Xsan/.auth_secret and in Open Directory.  If the auth_secret is removed from all MDCs and Open Directory, any client that can reach the MDC can connect (but we don’t recommend this option).  The client needs the auth_secret to show it is supposed to be part of the Xsan.

Each client can only have one auth_secret.  So for a client to be able to communicate with 2 (or more) sets of MDCs, the MDCs must all use the same auth_secret.  We have shown how to rebuild an Xsan and how to manually configure a profile with a new auth_secret.  This can work, but a cleaner option would be to rebuild the Open Directory of one of the Xsans with the matching auth_secret. We will rebuild the nearline Xsan as there are fewer clients to reconnect.

Contacting additional MDCs

Now that the clients are authorized to talk to all the MDCs, they need to know where they are.  The “primary” MDCs are still defined by the profile installed and written to /Library/Preferences/Xsan/fsnameservers.  Additional MDCs should be defined in /Library/Preferences/Xsan/fsforeignservers.  This file needs to contain the metadata IPs and/or hostnames of the additional MDCs, one per line. i.e.

192.168.0.1
192.168.0.2
mdc.domain.priv
bmdc.domain.priv

See man fsforeignservers for more information about the file.

Zoning the LUNs

Configuring various fibre switch zoning is beyond the scope of this blog, but presumably if you are at this point you have some clients zoned to see LUNs for the production volume and some clients zoned to see the nearline volume.  We need this bridge client to see all the data LUNs for all the necessary volumes (excluding the metadata and journal LUNs; only MDCs need those). 

Conclusion

Now that the client knows about both sets of MDCs (fsnameservers for the primary and fsforeignservers for the additional), is authorized to talk to all of them, and can see all the necessary LUNs, it likely needs a restart.  After that xsanctl list should show all the volumes from those MDCs and xsanctl mount <volumename> should mount them appropriately.