Create a new Xsan on macOS Monterey

10 Nov 2021

In macOS Monterey the Server.app interface for setting up and managing Xsan is gone.  Apple has produced an Xsan Management Guide (as of 2021-11-2 available on the Apple developer site).  In this post I will walk through creating a new Xsan from scratch on macOS Monterey.

Our test environment is made up of 2 Intel Mac Minis with ATTO Thunderlink fibre adapters connected to a fibre switch and one 12 drive RAID unit connected to the same switch.  We have installed the ATTO driver and allowed it in System Preferences -> Security & Privacy -> General.  The fibre switch is already zoned so that the Macs can see the storage.  We also have a dedicated metadata network. Both Minis have a secondary USB-C to ethernet adapter and a static IP on that port.

Planning

• Our Xsan will have a metadata LUN, a dedicated journal LUN, and 2 user data LUNs.  
• We need to provide DNS for our metadata network.  
  • We will install and configure dnsmasq on our MDC
  • Each MDC (and client) needs a second ethernet port with static IP
• Check the speeds that the fibre switch, client HBAs, and storage support.  
  • Our Brocade Fibre Switch only supports 16 or 32 Gb/s SFPs, so we needed to upgrade our test RAID to also support 16Gb/s.

DNS

To provide DNS services for the metadata network, you could use any number of solutions (a Raspberry Pi, a smart switch with that service, etc).  For our solution we installed dnsmasq on our MDC.  Running additional services on an MDC isn’t recommended, but works for our test system.  dnsmasq is available in Homebrew or MacPorts, but for our Xsan, I didn’t want to install all the extra that comes with a package manager.

1. Download the current dnsmasq tar.gz from their site on a Mac that matches the architecture of your MDCs (Intel vs M1) and has Xcode or the Xcode Command-line tools installed.
2. Expand the tar.gz file in your Downloads location
3. Open a Terminal window to the expanded folder (i.e. ~/Downloads/dnsmasq-2.86)
4. Type make and press enter
5. The binary will be in the src folder.
6. Copy src/dnsmasq, contrib/MacOSX-launchd/uk.org.thekelleys.dnsmasq.plist, and dnsmasq.conf.example to your MDC
7. On the Mac that will host your dns, copy/move these items, creating the folders as necessary:
   • dnsmasq -> /usr/local/sbin/dnsmasq
   • dnsmasq.conf.example -> /usr/local/etc/dnsmasq/dnsmasq.conf
   • uk.org.thekelleys.dnsmasq.plist -> /Library/LaunchDaemons/uk.org.thekelleys.dnsmasq.plist
8. dnsmasq can be very powerful.  We just need the most basic functionality.  So we will leave the conf file as is.
9. Edit the /etc/hosts file to add both MDCs (and any fibre connected clients)
   1. In a Terminal window run sudo nano /etc/hosts (or use your plain text editor of choice)
   2. At the bottom of the file add a line for each system
   3. The lines should be formatted as ip (tab) hostname, for example:

      #
      # Host Database
      #
      # localhost is used to configure the loopback interface
      # when the system is booting.  Do not change this entry.
      ##
      127.0.0.1       localhost
      255.255.255.255 broadcasthost
      ::1             localhost
      192.168.139.10  xsantest-mdc.testsan.priv
      192.168.139.11  xsantest-bmdc.testsan.priv

10.  Now start the dnsmasq process by bootstrapping the launchdaemon: sudo launchctl bootstrap system /Library/LaunchDaemons/uk.org.thekelleys.dnsmasq.plist
11. On both MDCs (and eventually any clients) add this server’s IP to the DNS servers list for the primary (non-metadata service) in System Preferences.  Make sure your other DNS server(s) are still listed and are below the MDC ip.
12. Test your DNS entries with host <mdchostname> and host <bmdchostname>
13. If you haven’t, assign the appropriate IPs to the metadata ethernet ports in System Preferences
14. Assign the private hostnames to each of the systems with sudo scutil --set HostName <fqdn>

Creating and labeling LUNs

Different storage vendors have different interfaces for creating LUNs on their storage.  Please see your vendors manual on the exact method for your storage.  This section will discuss some strategies and then see how to label the LUNs for use in Xsan.  Apple provides some discussion and guidance on drives, LUNs, and Storage Pools, in the Xsan Management Guide, which we won’t be discussing here.

In this example we want to end up with a 2 drive, RAID 1 (mirror) LUN for metadata, a second 2 drive, RAID 1 LUN for the journal, and then two RAID 5 data LUNs. However, don’t necessarily create them all at once.  To use the LUNs in Xsan, they need to be labeled.  If we create them all at once and then try to label them, there isn’t really a way to know which is which (except size differences, but all data LUNs should be the same size).  So ideally we will create the LUNs one at a time and label them as we go.  This isn’t crucial, and your Xsan will work, but it can be nice to know the mapping of LUN label to specific physical drives.

1. Create the metadata LUN (2 drive RAID 1) on your storage
2. On your MDC run sudo cvlabel -c
   That will output something like CvfsDisk_UNKNOWN /dev/rdisk2 # host 2 lun 0 sectors 1639102431 sector_size 512 inquiry [IFT GS 3000 Series 152A] serial 600D023100094D1E204148BA6955E5DD
3. Eventually we will get an entry like that for each LUN.  To use it, run the command again, and redirect the output to a file: sudo cvlabel -c > Desktop/labels.txt
4. Open the label file with a plain text editor like BBEdit or nano
5. Change CvfsDisk_UNKNOWN to be a useful name for the LUN (metadataLUN, dataLUN1, dataLUN2, etc)
   metadataLUN /dev/rdisk2 1639102431 EFI 0 # host 2 lun 0 sectors 1639102431 sector_size 512 inquiry [IFT GS 3000 Series 152A] serial 600D023100094D1E204148BA6955E5DD
6. Save the file
7. Now use the file to label the LUN with sudo cvlabel Desktop/labels.txt
8. The process will ask you to confirm the changes.  Type Y at each prompt.  Once you are comfortable with what it is doing and how it works, you can use sudo cvlabel -f Desktop/labels.txt to force the labeling.

% sudo cvlabel Desktop/labels.txt

*WARNING* This program will over-write volume labels on the
devices specified in the file "Desktop/labels.txt".

After execution, the devices will only be usable by the
Xsan. You will have to re-partition the
devices to use them on a different file system.


Do you want to proceed? (Y / N) -> Y

/dev/rdisk5 [IFT GS 3000 Series 152A] unknown Controller 'default', Serial '600D023100094D1E7F2118043CE4C4FC', Sector Size 512, Sectors Max 1639102431 (839.2GB)

Do you want to label it acfs-EFI - Name: JournalLUN Sectors: 1639102431 (Y / N) -> Y
New Volume Label -Device: /dev/rdisk5 acfs Label: JournalLUN Sectors: 1639102431.

Done. 1 source lines. 1 labels.
Requesting disk rescan .

Now repeat the process for the rest of the LUNs.

1. 1. Create a LUN on the storage
   2. Create the label file with sudo cvlabel -c > Desktop/labels.txt
   3. Edit the file (you should remove the already labeled lines, or you will end up getting asked about each one)
   4. Apply the label with sudo cvlabel Desktop/labels.txt
   5. Continue until all LUNs are created and labeled

Create the Xsan

On the Mac that will be the primary MDC, we will use xsanctl to create the Xsan.  Xsan relies on the Open Directory service and will create an Open Directory Master.  For that to succeed, we need to supply

• san name
• certificate name
• email address
• directory administrator’s “full name”
• directory administrators’s account name
• a password for the directory administrator.  

This is the point of the process where having the correct hostnames and DNS becomes important.

Create the Xsan with: sudo xsanctl createSan <sanName> createMaster --cert-auth-name <certName> --cert-admin-email <adminEmailAddress> --user <diradminFullName> --account <diradminAccountName> --pass '<password>'

% sudo xsanctl createSan TestSan createMaster --cert-auth-name testsan_cert --cert-admin-email me@domain.com --user diradmin --account diradmin --pass 'P@ssW0Rd*!'
2021-11-01 11:21:53.521 xsanctl[8657:68456] create OD master succeed
/System/Library/LaunchDaemons/com.apple.xsan.plist: Could not find specified service
Unload failed: 113: Could not find specified service
2021-11-01 11:21:56.613 xsanctl[8657:68456] buildSanConfig started
2021-11-01 11:21:56.613 xsanctl[8657:68456] buildSanConfig about to check LDAP
2021-11-01 11:21:56.614 xsanctl[8657:68456] buildSanConfig: getXsanConfig said nothing
2021-11-01 11:21:57.251 xsanctl[8657:68456] buildSanConfig step 4 with {
}
2021-11-01 11:21:57.251 xsanctl[8657:68456] New config time
2021-11-01 11:21:57.251 xsanctl[8657:68456] We are a new controller inside of buildGlobalConfig
2021-11-01 11:21:57.252 xsanctl[8657:68456] Hosted set is {(
)}
2021-11-01 11:21:57.252 xsanctl[8657:68456] needStart is {(
)}
2021-11-01 11:21:57.252 xsanctl[8657:68456] buildSanConfig saving config {
globals = {
certSetRevision = "49FD7DD9-B5B7-441E-9487-80E13BABCC9F";
controllers = {
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "17.229.239.203";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
}
);
notifications = {
FreeSpaceThreshold = 20;
};
revision = "00000000-0000-0000-0000-000000000000";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "C1CEA8F0-C54A-4EA0-8E30-CB94778BD9DD";
sharedSecret = "********";
};
volumes = {
};
}
SAN successfully created

You can confirm that the Xsan was created with xsanctl listSan which should output TestSan (xsantest-mdc.testsan.priv).  Also make sure that the hostName in the output is your private (metadata) network hostname.  If the output doesn’t look right, check the hostname with sudo scutil --get HostName, and fix it with sudo scutil --set HostName <fqdn>. Then remove the Xsan with sudo xsanctl destroySan and try the creation again.

Add the backup MDC

On the Mac that is going to be the backup MDC, we will use xsanctl again to join the Xsan we just created.  We will need:

• name of the san
• the MDCs hostname
• an admin account and password on the MDC
• the Open Directory Master’s hostname
• the directory administrator’s name and password

The command looks like xsanctl joinSan <sanName> --controller-name <mdcHostName> --controller-user <adminAccount> --controller-pass '<adminPassword>' createReplica --master <odMasterHostName> --account <directoryAdminName> --pass '<directoryAdminPassword>'

% sudo xsanctl joinSan TestSan --controller-name xsantest-mdc.testsan.priv --controller-user ladmin --controller-pass 'P@ssW0Rd*!' createReplica --master xsantest-mdc.testsan.priv --account diradmin --pass 'P@ssW0Rd*!'
Password:
2021-11-02 11:59:30.049 xsanctl[1074:7095] create OD replica succeed
/System/Library/LaunchDaemons/com.apple.xsan.plist: Could not find specified service
Unload failed: 113: Could not find specified service
2021-11-02 11:59:33.371 xsanctl[1074:7095] buildSanConfig started
2021-11-02 11:59:33.371 xsanctl[1074:7095] buildSanConfig about to check LDAP
2021-11-02 11:59:33.372 xsanctl[1074:7095] buildSanConfig step 4 with {
globals = {
certSetRevision = "66DBDF66-5134-4161-918B-0802B24ED0ED";
controllers = {
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "192.168.139.10";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
}
);
notifications = {
FreeSpaceThreshold = 20;
};
revision = "AF691D1B-D9A0-4F53-96C7-C53C9CC955C9";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "8998B227-807B-4884-8F53-021C8E2E9B0B";
sharedSecret = "********";
};
volumes = {
};
}
2021-11-02 11:59:33.372 xsanctl[1074:7095] We are a new controller inside of buildGlobalConfig
2021-11-02 11:59:33.372 xsanctl[1074:7095] Hosted set is {(
)}
2021-11-02 11:59:33.373 xsanctl[1074:7095] needStart is {(
)}
2021-11-02 11:59:33.373 xsanctl[1074:7095] buildSanConfig saving config {
globals = {
certSetRevision = "F2B06A76-4CBF-4FA3-81BC-DA5FF4614277";
controllers = {
"744262BB-D13F-5BFF-B892-974963AAC4F8" = {
IPAddress = "192.168.139.11";
hostName = "xsantest-bmdc.testsan.priv";
};
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "192.168.139.10";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
},
{
addr = "192.168.139.11";
uuid = "744262BB-D13F-5BFF-B892-974963AAC4F8";
}
);
revision = "AF691D1B-D9A0-4F53-96C7-C53C9CC955C9";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389",
"ldaps://xsantest-bmdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "8998B227-807B-4884-8F53-021C8E2E9B0B";
sharedSecret = "********";
};
volumes = {
};
}
SAN successfully joined

At this point we have 2 metadata controllers talking to each other over the metadata network and our LUNs are ready to be added to a volume.

Create a Volume

To create our volume we will again use xsanctl. We will need:

• a volume name (this will be visible on all the Xsan clients)
• any options we want for this volume (we will be using the enableACLs option). See man xsanctl for all the options in the VOLUME DESCRIPTION area
• the LUN labels for metadata, journal, and all the data LUNs

There are some considerations for how many data LUNs per Storage Pool in our Volume.  Again see the Xsan Management Guide for more details.  In our example, both data LUNs will be in the same Storage Pool.

We build the volume by naming the storage pools and which LUNs are a member.  There must be one storage pool that contains metadata.  This pool can also contain the journal, in which case we would use –defaultFirstPool –addLUN mdjLUN.  In our example, we are separating the metadata and journal storage pools to increase our volume performance.  Then we name at least one data storage pool and assign LUNs.  This is traditionally done on the primary MDC, but could be completed on either.

Our command then looks like:

% sudo xsanctl addVolume TestSanVolume --enableACLs --storagePool Metadata --metadata --addLUN MetadataLUN --storagePool Journal --journal --addLUN JournalLUN --storagePool DataPool1 --data --addLUN DataLUN1 --addLUN DataLUN2
xsanctl: Volume creation successful

The TestSanVolume should now mount on both MDCs.  Depending on your Finder Preferences it may show on the Desktop and/or in the sidebar of a Finder window, or just in the Computer area of the Finder.

Connect clients

Now we have a working Xsan with 2 MDCs, but no one can actually use our volume.  We can now add clients, either actual clients that people will use, or file servers so that people can connect less directly.  These clients will all need:

• fibre connections to the switch
• the switch zoned so they can see all the storage
• a wired connection to the metadata network
• to use the DNS server we setup on the MDC (or where ever the metadata DNS is running)
• the profile to understand the Xsan volume

The first 4 are up to you!  To create and install the profile:

On either of the MDCs, run xsanctl exportClientProfile

2021-11-03 13:59:49.374 xsanctl[9679:330794] Client profile is successfully saved at /Users/ladmin/TestSan.mobileconfig
If we look at the contents of the file we will see something like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>PayloadContent</key>
<array>
<dict>
<key>PayloadDisplayName</key>
<string>Xsan Configuration</string>
<key>PayloadEnabled</key>
<true/>
<key>PayloadIdentifier</key>
<string>com.apple.xsan.6A28E74E-D0CC-4C4E-8CF3-A08AE4B27757</string>
<key>PayloadType</key>
<string>com.apple.xsan</string>
<key>PayloadUUID</key>
<string>6A28E74E-D0CC-4C4E-8CF3-A08AE4B27757</string>
<key>PayloadVersion</key>
<integer>1</integer>
<key>sanAuthMethod</key>
<string>auth_secret</string>
<key>sanConfigURLs</key>
<array>
<string>ldaps://xsantest-mdc.testsan.priv:389</string>
<string>ldaps://xsantest-bmdc.testsan.priv:389</string>
</array>
<key>sanName</key>
<string>TestSan</string>
</dict>
</array>
<key>PayloadDisplayName</key>
<string>Xsan Configuration Profile</string>
<key>PayloadIdentifier</key>
<string>com.apple.xsan.5C7DE0D7-9333-43B3-882E-A6FF57F91D72</string>
<key>PayloadRemovalDisallowed</key>
<false/>
<key>PayloadScope</key>
<string>System</string>
<key>PayloadType</key>
<string>Configuration</string>
<key>PayloadUUID</key>
<string>5C7DE0D7-9333-43B3-882E-A6FF57F91D72</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
</plist>

Now we could use MDM to deploy this profile, or copy it manually to our clients and double click it.  If you double click, you will be prompted to manually open System Preferences -> Profiles and complete the installation there.  This requires a local administrator name and password.

Troubleshooting

If a mistake is made at some point while creating the Xsan or volume, most things can be undone with xsanctl commands like leaveSan, destroySan, or dropVolume.  If however, the system gets into an inconstant state where it thinks there should be an Open Directory but there isn’t, sudo xsanctl wipeConfig can get you back to zero. For Open Directory, sudo slapconfig -getstyle  will show what the system thinks is configured.  If something is wrong, sudo slapconfig -destroyldapserver will remove everything and let you start over.  For example, if you run sudo xsanctl leaveSan on the backup MDC, the Open Directory Replica is still running.  These commands are destructive, so once there is data on the volume, make sure things are backed up!

Now that we don’t have the Server.app interface for view our Xsan setup, the next best thing is the output of sudo xsanctl dumpLdapConfig.  I recommend redirecting it to a file like sudo xsanctl dumpLdapConfig > Desktop/san.txt.  Note: there is a --json option for that command, but in my experience it crashes with no output in macOS 12.0.1 (BuildVersion: 21A559).  Feedback #FB9735952 if anyone wants to duplicate or reference it.