Erase and reconfigure your Xsan MDCs

by Eric Hemmeter

in Xsan

14 Apr 2022

We recently had a situation where an Xsan was misbehaving. Both MDCs were regularly freezing up and becoming unresponsive. We found that the systems were regularly creating core dumps and filling the /cores directory until the drive filled up. We tried to diagnose the cause of this, but were not able to figure it out from the core dump files. So we erased and reinstalled both MDCs at the same time!
We have also used the method to add a separate metadata network to an Xsan built without one and to do major OS upgrades on an Xsan that has been upgraded in place for many years and needed fresh start.

Backup

The critical part to restore an existing Xsan is to save the /Library/Preferences/Xsan directory. On each MDC open the Terminal and run the following commands to create a tar bundle of the directory.

$ cd /Library/Preferences
$ sudo tar -cvf ~/Desktop/[b]mdc_$(date "+%F").tar Xsan

Take these tar files and save them somewhere off the systems. If there is anything else special on these Macs, back it up as well.

Erase and Reinstall macOS

To erase and reinstall the Macs in place, there are several options. If you are already on Monterey (on an M1 or T2 based Mac), System Preferences has an Erase All Content and Settings option in the System Preferences menu. Another option is startosinstall from a macOS installer application. These steps can be used on several recent versions of macOS:

$ softwareupdate --fetch-full-installer #this will download the current Install macOS to /Applications/
$ "/Applications/Install macOS Monterey.app/Contents/Resources/startosinstall" --eraseinstall --newvolumename [b]mdc-HD # This will prompt you to confirm the erase and then agree to a license.

Then the OS will prepare, restart, and install.
There are also tools to help automate this process, such as Erase-Install

Once the MDCs are reinstalled, configure your networking (making sure DNS is configured properly), install Server.app (we’ve seen problems with OpenDirectory if Server.app is not installed and allowed to setup the system), and make sure your hostname is set properly. See Create a new Xsan on macOS Monterey for help with that.

Build new Xsan

Once all the prerequisites are in place we can create our new san. On the primary MDC, use

sudo xsanctl createSan <sanName> createMaster --cert-auth-name <CertAuthName> --cert-admin-email <CertAdminEmail> --user <AdminFullName> --account <AdminAccountShortName> --pass <AdminPassword>

The OD master can also be made in Server.app if you’d rather. Then you only need sudo xsanctl createSan <sanName>.

Restore our volumes

After the san is created the directory /Library/Preferences/Xsan will be populated. Now we can bring back our old configuration. We need to copy back any volume configuration file (volumeName.cfgp) and the fsmlist file. The cfgp files tell the san what the volumes are made up of (mapping LUNs to metadata/journal/data) and the fsmlist tells it what volumes to care about. We could also restore the .auth_secret file, but that adds some complications that we will visit in a separate article. So now we want copy [b]mdc_$date.tar back and expand it by double clicking (we will assume to the Desktop in our commands). Then

sudo cp ~/Desktop/Xsan/<volumeName1>.cfgp ~/Desktop/Xsan/<volumeName2>.cfgp ~/Desktop/Xsan/fsmlist /Library/Preferences/Xsan/

(you can type sudo cp and then drag the files in to add the sources. Make sure to add the destination). Now that the files on the system have our full configuration, we can push that to ldap so the join process for the backup MDC and clients can pull it all easily. Run sudo xsanctl pushConfigUpdate. As the manpage says “This subcommand adjusts the SAN configuration in ldap to match the configuration status in /Library/Preferences/Xsan.” We could restart to activate the Xsan volume, or we can load it with sudo launchctl bootstrap system /System/Library/LaunchDaemons/com.apple.xsan.plist.

Create the backup MDC

Now we can connect our backup MDC. On that Mac, run

sudo xsanctl joinSan <sanName> --controller-name <mdcHostName> --controller-user <adminAccount> --controller-pass '<adminPassword>' createReplica --master <odMasterHostName> --account <directoryAdminName> --pass '<directoryAdminPassword>'

Again, you can use Server.app to join the OpenDirectory as a Replica. At this point the volumes should be available on the backup MDC. If they don’t show on the Desktop, check with xsanctl list.

Reconnect the clients

For clients, we need a new profile. From either MDC, run sudo xsanctl exportClientProfile --path ~/Desktop/ and the new <sanName>.mobileconfig profile will be generated on the desktop. Copy this to any clients. On each client, open System Preferences -> Profiles. Select the already installed Xsan Configuration Profile and click the minus sign at the bottom of the list of profiles. Then double click the new <sanName>.mobileconfig. System Preferences will prompt you to accept the profile. Return to System Preferences, select the new Xsan Configuration Profile, and click to accept. The volume(s) should now mount on your clients.

Troubleshooting

If the createSan step fails with

errorCode = "-319";
errorDescription = "Duplicate Xsan profiles are not supported.";

go to System Preferences -> Profiles and remove the old profile. Try the command again and it should succeed.

Monterey