Xsan Ventura & Sonoma upgrade failures and a workaround using profile “Magic”

26 Sep 2023

We have recently been upgrading our clients’ Xsans from macOS Monterey to macOS Ventura.  We had been holding off since, in our testing over the last year, we have never had a successful in-place upgrade.  When trying an in-place upgrade on an MDC, the install progresses until the first restart.  At that point, the Mac boots to recovery with an error message: “Your computer started up in Recovery because a failure occurred during installation.  An error occurred migrating user data during an install.  Reinstall macOS to resolve the issue.” (Feedback number FB11519623)  Reinstalling macOS does not resolve the issue.  The systems will boot in safe mode, but I haven’t been able to resolve the issue from there. I would love to hear that others have had better experiences.

Because of these problems, our upgrades were really erase and reinstalls.  We know this works, but it does take more time than it should.  It solves the above boot-to-recovery problem as well as an issue with Open Directory data disappearing.  This problem occurs with just Open Directory running.  Even minor updates like 13.3 to 13.4 would cause the Mac to revert to a directory client.  It thinks it is bound to itself, but /var/db/openldap would be reverted to defaults. (Feedback number FB12227824 – amazingly marked as “works as currently designed”)

We just discovered a way to get these updates and upgrades to work.  Before starting, definitely still back up /Library/Preferences/Xsan with cd /Library/Preferences/; sudo tar cvzf ~/Desktop/Xsan.tar.gz Xsan.  With that, you can always rebuild from scratch and get your volume(s) back.  Now, on your MDC, we need to do three new things: 

1. Create a backup of our Open Directory database with sudo slapconfig -backupdb ~/Desktop/OD. This will create OD.sparseimage on your user’s Desktop folder.  Supply an archive password when prompted.
2. Then make sure you have a copy of your Xsan mobile config profile with xsanctl exportClientProfile --path ~/Desktop/.  This will create a config profile and put it on your user’s Desktop.
3. Then, in System Settings -> Privacy & Security -> Profiles, select the Xsan Configuration Profile and click the minus sign.  This removes the Xsan configuration from your Mac.  It unmounts any volumes, stops the Xsan processes (fsm, fsmpm, etc), and removes the files in /Library/Preferences/Xsan.  This was the key. Just deleting the files wouldn’t let upgrades complete, but removing the profile does.

Now, perform your upgrade.  We’ve tested this with minor updates (13.4 -> 13.6) and major upgrades (13.5 -> 14.0).

After the upgrade is complete, we need to rebuild and restore Open Directory, and then re-activate the Xsan configuration.  

1. Confirm Open Directory is not running with sudo slapconfig -getstyle.  We expect this to say 2 - Directory Client.
2. Create a new Open Directory with sudo slapconfig -createldapmasterandadmin 'Directory Administrator' diradmin 1000, where the first name is the display name, the second the account name, and the third is the uid, all for the directory administrator.  This will prompt for a password during the setup.
3. Then, restore the Open Directory archive with sudo slapconfig -restoredb ~/Desktop/OD.sparseimage.  Provide the archive password when prompted.
4. Now double-click the Xsan Configuration Profile on your Desktop.  This will notify you to open System Settings to install.
5. Open System Settings -> Privacy & Security -> Profiles and double-click the Xsan Configuration Profile listed at the top.  Enter the local admin name and password when prompted and click install.
6. Confirm that this Mac is now taking part in hosting the volume with sudo cvadmin.

Now, we can upgrade the backup metadata controller.  

1. To get this to work cleanly, run sudo xsanctl leaveSan on the bmdc.
2. Run the update/upgrade.
3. When that is complete, run sudo xsanctl joinSan <san-name> --controller-name <mdc-fqdn> --controller-user <local admin> --controller-pass '<local admin pass>' createReplica --master <mdc-fqdn> --account <diradmin name> --pass '<diradmin pass>'
4. Confirm that both systems are taking part in hosting the volume with sudo cvadmin.  Confirm that both are listed in the Cluster output.

The reason that we need the backup metadata controller to leaveSan is that if it is still listed in Open Directory when we try to joinSan, the system will mount the volume but not join in hosting.  It will be missing the fsmlist and volume configuration file.  By leaving the san and then re-joining, it properly gets the full configuration.

For completeness, we have not had any problems with client updates or upgrades.

If you run into problems with what is in Open Directory, it is possible to edit the Xsan data with Directory Utility.  On the MDC, open /System/Library/CoreServices/Applications/Directory Utility.app, select Directory Editor in the toolbar, select /LDAPv3/127.0.0.1 from the node pop-up menu, click the lock, and authenticate as the directory administrator.  Then select Groups from the container list (next to Viewing) and select com.apple.xsan.conf.<sanName>.  On the right, select the XMLPlist entry, and you can view/edit the data in the area on the lower right.  Be careful here, but if anything bad does happen, you can always restore your archive again.