Moving to M1/arm native rsync

by Eric Hemmeter

in macOS

20 Jan 2023

We are in the process of replacing Intel Mac minis with M1 Mac minis in our customers’ server rooms. For the MDCs we are using migration assistant and rebuilding the Open Directory master first, then migrating the BMDC and cleaning up the Xsan configuration and re-joining the Xsan. For other systems (file servers, sync servers, etc) one of main tools for data movement is rsync (along with Archiware’s P5 and cvcpSync, which uses rsync). The migration brings the Intel version of rsync over, but we want to get any speed benefits the native version can bring. This article will discuss the process we used to build an installer package for the M1 native rsync v3.2.7 and then share the package we created.rsync is distributed by the Samba organization. They have a good page on installing for different operating systems including macOS. They list the following libraries/packages are needed for macOS:

automake, which requires autoconf
xxhash
zstd
lz4
openssl

The recommendation is to use brew to install these, but at that point, we would use brew to install rsync also. Some of our customers want to limit brew installs so we are going to install these ourselves.

The important first step is a clean M1 system meaning we haven’t installed brew or other additional command line tools. Specifically we want an empty /usr/local/ directory. This will make it much easier to create a clean package once we get everything installed.

We need to download the various packages from their repositories. Here are the current links as of 2022-12-6:

The other thing that we need is the software to build these packages. You can download Xcode.app from Apple or install the Xcode command line tools with xcode-select --install.

We need to extract the contents of each of the repositories. The simplest thing is to select them all and double click one (or hit command-O) and allow Archive Utility to expand them all into their folders.

First we want to build Autoconf, so in a Terminal window we will cd into the autoconf-2.71 directory. This isn’t a post about all the variations in building command line tools, but a basic rule of thumb is if there is a configure file, make sure you are in the directory and then we want to run ./configure, then make, then sudo make install. That is what we do for autoconf.

autoconf-2.71 % ./configure
autoconf-2.71 % make
autoconf-2.71 % sudo make install

Then we can move to the others (saving rsync for last). For lz4, notice there isn’t a configure command, so we only run make and sudo make install.

These builds require the ./configure step:

autoconf
openssl

These builds only need make and sudo make install:

lz4
xxHash
zstd

After building the 5 prerequisites, we are now ready to build rsync. Again, cd into the rsync directory (rsync-3.2.7). Then run ./configure, then make, then sudo make install. If you see something like

Configure found the following issues:
- Failed to find zstd.h for zstd compression support

Go back to the referenced folder and try the steps again.

After a successful install, close the terminal window and open a new one. To confirm that we are looking to the new copy of rsync, run which rsync and it should report /usr/local/bin/rsync. Then run rsync -V and it should report version 3.2.7 and the various additions that we installed.

We are now ready to package this all up. I use Packages from Stéphane Sudre. We create a new Raw Package, give it a name like rsync-3.2.7_arm and choose where to save the package project. In the Settings tab, you can set the package identifier to reference your company or brand using reverse dns notation. Then we build out the payload tab. To create folders that don’t show in the interface, right click on the parent folder and choose new folder. For example, to start, right click on the hard drive icon, choose New Folder, and name it usr. Now we could grab the entire /usr/local/ directory and drag it into the usr folder we created, but we don’t necessarily need everything. In particular, Autoconf installs a lot of extra stuff that we only needed to build some of the other tools. In our package, I created /usr/local/bin, include, lib, and share. Within share, I created /man/man1 and man5. Then I populated these folders with the contents I knew weren’t from autoconf. It is possible that I got some extra, but in testing the package on another M1 Mac, rsync worked as needed.

Our package of rsync v3.2.7 is available on our Github.

M1 Monterey Xsan