Create a new Xsan on macOS Monterey
10 Nov 2021
In macOS Monterey the Server.app interface for setting up and managing Xsan is gone. Apple has produced an Xsan Management Guide (as of 2021-11-2 available on the Apple developer site). In this post I will walk through creating a new Xsan from scratch on macOS Monterey.
Our test environment is made up of 2 Intel Mac Minis with ATTO Thunderlink fibre adapters connected to a fibre switch and one 12 drive RAID unit connected to the same switch. We have installed the ATTO driver and allowed it in System Preferences -> Security & Privacy -> General. The fibre switch is already zoned so that the Macs can see the storage. We also have a dedicated metadata network. Both Minis have a secondary USB-C to ethernet adapter and a static IP on that port.
Planning
- Our Xsan will have a metadata LUN, a dedicated journal LUN, and 2 user data LUNs.
- We need to provide DNS for our metadata network.
- We will install and configure dnsmasq on our MDC
- Each MDC (and client) needs a second ethernet port with static IP
- Check the speeds that the fibre switch, client HBAs, and storage support.
- Our Brocade Fibre Switch only supports 16 or 32 Gb/s SFPs, so we needed to upgrade our test RAID to also support 16Gb/s.
DNS
To provide DNS services for the metadata network, you could use any number of solutions (a Raspberry Pi, a smart switch with that service, etc). For our solution we installed dnsmasq on our MDC. Running additional services on an MDC isn’t recommended, but works for our test system. dnsmasq is available in Homebrew or MacPorts, but for our Xsan, I didn’t want to install all the extra that comes with a package manager.
- Download the current dnsmasq tar.gz from their site on a Mac that matches the architecture of your MDCs (Intel vs M1) and has Xcode or the Xcode Command-line tools installed.
- Expand the tar.gz file in your Downloads location
- Open a Terminal window to the expanded folder (i.e. ~/Downloads/dnsmasq-2.86)
- Type
make
and press enter - The binary will be in the src folder.
- Copy
src/dnsmasq
,contrib/MacOSX-launchd/uk.org.thekelleys.dnsmasq.plist
, anddnsmasq.conf.example
to your MDC - On the Mac that will host your dns, copy/move these items, creating the folders as necessary:
- dnsmasq -> /usr/local/sbin/dnsmasq
- dnsmasq.conf.example -> /usr/local/etc/dnsmasq/dnsmasq.conf
- uk.org.thekelleys.dnsmasq.plist -> /Library/LaunchDaemons/uk.org.thekelleys.dnsmasq.plist
- dnsmasq can be very powerful. We just need the most basic functionality. So we will leave the conf file as is.
- Edit the /etc/hosts file to add both MDCs (and any fibre connected clients)
- In a Terminal window run
sudo nano /etc/hosts
(or use your plain text editor of choice) - At the bottom of the file add a line for each system
- The lines should be formatted as ip (tab) hostname, for example:
#
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
192.168.139.10 xsantest-mdc.testsan.priv
192.168.139.11 xsantest-bmdc.testsan.priv
- In a Terminal window run
- Now start the dnsmasq process by bootstrapping the launchdaemon:
sudo launchctl bootstrap system /Library/LaunchDaemons/uk.org.thekelleys.dnsmasq.plist
- On both MDCs (and eventually any clients) add this server’s IP to the DNS servers list for the primary (non-metadata service) in System Preferences. Make sure your other DNS server(s) are still listed and are below the MDC ip.
- Test your DNS entries with
host <mdchostname>
andhost <bmdchostname>
- If you haven’t, assign the appropriate IPs to the metadata ethernet ports in System Preferences
- Assign the private hostnames to each of the systems with
sudo scutil --set HostName <fqdn>
Creating and labeling LUNs
Different storage vendors have different interfaces for creating LUNs on their storage. Please see your vendors manual on the exact method for your storage. This section will discuss some strategies and then see how to label the LUNs for use in Xsan. Apple provides some discussion and guidance on drives, LUNs, and Storage Pools, in the Xsan Management Guide, which we won’t be discussing here.
In this example we want to end up with a 2 drive, RAID 1 (mirror) LUN for metadata, a second 2 drive, RAID 1 LUN for the journal, and then two RAID 5 data LUNs. However, don’t necessarily create them all at once. To use the LUNs in Xsan, they need to be labeled. If we create them all at once and then try to label them, there isn’t really a way to know which is which (except size differences, but all data LUNs should be the same size). So ideally we will create the LUNs one at a time and label them as we go. This isn’t crucial, and your Xsan will work, but it can be nice to know the mapping of LUN label to specific physical drives.
- Create the metadata LUN (2 drive RAID 1) on your storage
- On your MDC run
sudo cvlabel -c
That will output something likeCvfsDisk_UNKNOWN /dev/rdisk2 # host 2 lun 0 sectors 1639102431 sector_size 512 inquiry [IFT GS 3000 Series 152A] serial 600D023100094D1E204148BA6955E5DD
- Eventually we will get an entry like that for each LUN. To use it, run the command again, and redirect the output to a file:
sudo cvlabel -c > Desktop/labels.txt
- Open the label file with a plain text editor like BBEdit or nano
- Change
CvfsDisk_UNKNOWN
to be a useful name for the LUN (metadataLUN, dataLUN1, dataLUN2, etc)metadataLUN /dev/rdisk2 1639102431 EFI 0 # host 2 lun 0 sectors 1639102431 sector_size 512 inquiry [IFT GS 3000 Series 152A] serial 600D023100094D1E204148BA6955E5DD
- Save the file
- Now use the file to label the LUN with
sudo cvlabel Desktop/labels.txt
- The process will ask you to confirm the changes. Type
Y
at each prompt. Once you are comfortable with what it is doing and how it works, you can usesudo cvlabel -f Desktop/labels.txt
to force the labeling.
% sudo cvlabel Desktop/labels.txt
*WARNING* This program will over-write volume labels on the
devices specified in the file "Desktop/labels.txt".
After execution, the devices will only be usable by the
Xsan. You will have to re-partition the
devices to use them on a different file system.
Do you want to proceed? (Y / N) -> Y
/dev/rdisk5 [IFT GS 3000 Series 152A] unknown Controller 'default', Serial '600D023100094D1E7F2118043CE4C4FC', Sector Size 512, Sectors Max 1639102431 (839.2GB)
Do you want to label it acfs-EFI - Name: JournalLUN Sectors: 1639102431 (Y / N) -> Y
New Volume Label -Device: /dev/rdisk5 acfs Label: JournalLUN Sectors: 1639102431.
Done. 1 source lines. 1 labels.
Requesting disk rescan .
Now repeat the process for the rest of the LUNs.
- Create a LUN on the storage
- Create the label file with
sudo cvlabel -c > Desktop/labels.txt
- Edit the file (you should remove the already labeled lines, or you will end up getting asked about each one)
- Apply the label with
sudo cvlabel Desktop/labels.txt
- Continue until all LUNs are created and labeled
Create the Xsan
On the Mac that will be the primary MDC, we will use xsanctl
to create the Xsan. Xsan relies on the Open Directory service and will create an Open Directory Master. For that to succeed, we need to supply
- san name
- certificate name
- email address
- directory administrator’s “full name”
- directory administrators’s account name
- a password for the directory administrator.
This is the point of the process where having the correct hostnames and DNS becomes important.
Create the Xsan with: sudo xsanctl createSan <sanName> createMaster --cert-auth-name <certName> --cert-admin-email <adminEmailAddress> --user <diradminFullName> --account <diradminAccountName> --pass '<password>'
% sudo xsanctl createSan TestSan createMaster --cert-auth-name testsan_cert --cert-admin-email me@domain.com --user diradmin --account diradmin --pass 'P@ssW0Rd*!'
2021-11-01 11:21:53.521 xsanctl[8657:68456] create OD master succeed
/System/Library/LaunchDaemons/com.apple.xsan.plist: Could not find specified service
Unload failed: 113: Could not find specified service
2021-11-01 11:21:56.613 xsanctl[8657:68456] buildSanConfig started
2021-11-01 11:21:56.613 xsanctl[8657:68456] buildSanConfig about to check LDAP
2021-11-01 11:21:56.614 xsanctl[8657:68456] buildSanConfig: getXsanConfig said nothing
2021-11-01 11:21:57.251 xsanctl[8657:68456] buildSanConfig step 4 with {
}
2021-11-01 11:21:57.251 xsanctl[8657:68456] New config time
2021-11-01 11:21:57.251 xsanctl[8657:68456] We are a new controller inside of buildGlobalConfig
2021-11-01 11:21:57.252 xsanctl[8657:68456] Hosted set is {(
)}
2021-11-01 11:21:57.252 xsanctl[8657:68456] needStart is {(
)}
2021-11-01 11:21:57.252 xsanctl[8657:68456] buildSanConfig saving config {
globals = {
certSetRevision = "49FD7DD9-B5B7-441E-9487-80E13BABCC9F";
controllers = {
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "17.229.239.203";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
}
);
notifications = {
FreeSpaceThreshold = 20;
};
revision = "00000000-0000-0000-0000-000000000000";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "C1CEA8F0-C54A-4EA0-8E30-CB94778BD9DD";
sharedSecret = "********";
};
volumes = {
};
}
SAN successfully created
You can confirm that the Xsan was created with xsanctl listSan
which should output TestSan (xsantest-mdc.testsan.priv)
. Also make sure that the hostName in the output is your private (metadata) network hostname. If the output doesn’t look right, check the hostname with sudo scutil --get HostName
, and fix it with sudo scutil --set HostName <fqdn>
. Then remove the Xsan with sudo xsanctl destroySan
and try the creation again.
Add the backup MDC
On the Mac that is going to be the backup MDC, we will use xsanctl
again to join the Xsan we just created. We will need:
- name of the san
- the MDCs hostname
- an admin account and password on the MDC
- the Open Directory Master’s hostname
- the directory administrator’s name and password
The command looks like xsanctl joinSan <sanName> --controller-name <mdcHostName> --controller-user <adminAccount> --controller-pass '<adminPassword>' createReplica --master <odMasterHostName> --account <directoryAdminName> --pass '<directoryAdminPassword>'
% sudo xsanctl joinSan TestSan --controller-name xsantest-mdc.testsan.priv --controller-user ladmin --controller-pass 'P@ssW0Rd*!' createReplica --master xsantest-mdc.testsan.priv --account diradmin --pass 'P@ssW0Rd*!'
Password:
2021-11-02 11:59:30.049 xsanctl[1074:7095] create OD replica succeed
/System/Library/LaunchDaemons/com.apple.xsan.plist: Could not find specified service
Unload failed: 113: Could not find specified service
2021-11-02 11:59:33.371 xsanctl[1074:7095] buildSanConfig started
2021-11-02 11:59:33.371 xsanctl[1074:7095] buildSanConfig about to check LDAP
2021-11-02 11:59:33.372 xsanctl[1074:7095] buildSanConfig step 4 with {
globals = {
certSetRevision = "66DBDF66-5134-4161-918B-0802B24ED0ED";
controllers = {
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "192.168.139.10";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
}
);
notifications = {
FreeSpaceThreshold = 20;
};
revision = "AF691D1B-D9A0-4F53-96C7-C53C9CC955C9";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "8998B227-807B-4884-8F53-021C8E2E9B0B";
sharedSecret = "********";
};
volumes = {
};
}
2021-11-02 11:59:33.372 xsanctl[1074:7095] We are a new controller inside of buildGlobalConfig
2021-11-02 11:59:33.372 xsanctl[1074:7095] Hosted set is {(
)}
2021-11-02 11:59:33.373 xsanctl[1074:7095] needStart is {(
)}
2021-11-02 11:59:33.373 xsanctl[1074:7095] buildSanConfig saving config {
globals = {
certSetRevision = "F2B06A76-4CBF-4FA3-81BC-DA5FF4614277";
controllers = {
"744262BB-D13F-5BFF-B892-974963AAC4F8" = {
IPAddress = "192.168.139.11";
hostName = "xsantest-bmdc.testsan.priv";
};
"C2B087D3-B1B5-5E75-977B-B7CFC28569B3" = {
IPAddress = "192.168.139.10";
hostName = "xsantest-mdc.testsan.priv";
};
};
fsnameservers = (
{
addr = "192.168.139.10";
uuid = "C2B087D3-B1B5-5E75-977B-B7CFC28569B3";
},
{
addr = "192.168.139.11";
uuid = "744262BB-D13F-5BFF-B892-974963AAC4F8";
}
);
revision = "AF691D1B-D9A0-4F53-96C7-C53C9CC955C9";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://xsantest-mdc.testsan.priv:389",
"ldaps://xsantest-bmdc.testsan.priv:389"
);
sanConfigVersion = 100;
sanName = TestSan;
sanState = active;
sanUUID = "8998B227-807B-4884-8F53-021C8E2E9B0B";
sharedSecret = "********";
};
volumes = {
};
}
SAN successfully joined
At this point we have 2 metadata controllers talking to each other over the metadata network and our LUNs are ready to be added to a volume.
Create a Volume
To create our volume we will again use xsanctl
. We will need:
- a volume name (this will be visible on all the Xsan clients)
- any options we want for this volume (we will be using the
enableACLs
option). Seeman xsanctl
for all the options in the VOLUME DESCRIPTION area - the LUN labels for metadata, journal, and all the data LUNs
There are some considerations for how many data LUNs per Storage Pool in our Volume. Again see the Xsan Management Guide for more details. In our example, both data LUNs will be in the same Storage Pool.
We build the volume by naming the storage pools and which LUNs are a member. There must be one storage pool that contains metadata. This pool can also contain the journal, in which case we would use –defaultFirstPool –addLUN mdjLUN. In our example, we are separating the metadata and journal storage pools to increase our volume performance. Then we name at least one data storage pool and assign LUNs. This is traditionally done on the primary MDC, but could be completed on either.
Our command then looks like:
% sudo xsanctl addVolume TestSanVolume --enableACLs --storagePool Metadata --metadata --addLUN MetadataLUN --storagePool Journal --journal --addLUN JournalLUN --storagePool DataPool1 --data --addLUN DataLUN1 --addLUN DataLUN2
xsanctl: Volume creation successful
The TestSanVolume should now mount on both MDCs. Depending on your Finder Preferences it may show on the Desktop and/or in the sidebar of a Finder window, or just in the Computer area of the Finder.
Connect clients
Now we have a working Xsan with 2 MDCs, but no one can actually use our volume. We can now add clients, either actual clients that people will use, or file servers so that people can connect less directly. These clients will all need:
- fibre connections to the switch
- the switch zoned so they can see all the storage
- a wired connection to the metadata network
- to use the DNS server we setup on the MDC (or where ever the metadata DNS is running)
- the profile to understand the Xsan volume
The first 4 are up to you! To create and install the profile:
On either of the MDCs, run xsanctl exportClientProfile
2021-11-03 13:59:49.374 xsanctl[9679:330794] Client profile is successfully saved at /Users/ladmin/TestSan.mobileconfig
If we look at the contents of the file we will see something like
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>PayloadContent</key>
<array>
<dict>
<key>PayloadDisplayName</key>
<string>Xsan Configuration</string>
<key>PayloadEnabled</key>
<true/>
<key>PayloadIdentifier</key>
<string>com.apple.xsan.6A28E74E-D0CC-4C4E-8CF3-A08AE4B27757</string>
<key>PayloadType</key>
<string>com.apple.xsan</string>
<key>PayloadUUID</key>
<string>6A28E74E-D0CC-4C4E-8CF3-A08AE4B27757</string>
<key>PayloadVersion</key>
<integer>1</integer>
<key>sanAuthMethod</key>
<string>auth_secret</string>
<key>sanConfigURLs</key>
<array>
<string>ldaps://xsantest-mdc.testsan.priv:389</string>
<string>ldaps://xsantest-bmdc.testsan.priv:389</string>
</array>
<key>sanName</key>
<string>TestSan</string>
</dict>
</array>
<key>PayloadDisplayName</key>
<string>Xsan Configuration Profile</string>
<key>PayloadIdentifier</key>
<string>com.apple.xsan.5C7DE0D7-9333-43B3-882E-A6FF57F91D72</string>
<key>PayloadRemovalDisallowed</key>
<false/>
<key>PayloadScope</key>
<string>System</string>
<key>PayloadType</key>
<string>Configuration</string>
<key>PayloadUUID</key>
<string>5C7DE0D7-9333-43B3-882E-A6FF57F91D72</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
</plist>
Now we could use MDM to deploy this profile, or copy it manually to our clients and double click it. If you double click, you will be prompted to manually open System Preferences -> Profiles and complete the installation there. This requires a local administrator name and password.
Troubleshooting
If a mistake is made at some point while creating the Xsan or volume, most things can be undone with xsanctl
commands like leaveSan
, destroySan
, or dropVolume
. If however, the system gets into an inconstant state where it thinks there should be an Open Directory but there isn’t, sudo xsanctl wipeConfig
can get you back to zero. For Open Directory, sudo slapconfig -getstyle
will show what the system thinks is configured. If something is wrong, sudo slapconfig -destroyldapserver
will remove everything and let you start over. For example, if you run sudo xsanctl leaveSan
on the backup MDC, the Open Directory Replica is still running. These commands are destructive, so once there is data on the volume, make sure things are backed up!
Now that we don’t have the Server.app interface for view our Xsan setup, the next best thing is the output of sudo xsanctl dumpLdapConfig
. I recommend redirecting it to a file like sudo xsanctl dumpLdapConfig > Desktop/san.txt
. Note: there is a --json
option for that command, but in my experience it crashes with no output in macOS 12.0.1 (BuildVersion: 21A559). Feedback #FB9735952 if anyone wants to duplicate or reference it.