Speeding up many small transfers to a unifi nas

Moving some of my old local ML projects to a NAS was taking an unbearably long time. A folder with 25k small files was estimating a whole day to transfer. The whole thing was syncing kilobytes at a time.

There's a lot of overhead to moving small files. Especially over the network. Each file has separate headers, checksum verification, etc. Not to mention it seems there are some Mac specific gotchas with the default SMB config that might exacerbate the issue.

When bundled into a single tarfile the transfer finished in 2 minutes flat. First, build a new uncompressed tarball. If you're not limited by your connection and have enough storage overhead on your system to duplicate your folder size, this will be much faster than having to do the round-trip of compression -> transfer -> decompression.

tar -cvf myfolder.tar  myfolder/

Then ssh into your admin account configured in the panel. I have my local ssh key added as a remote trusted key, which keeps me from digging up the password every time.

ssh root@<unas-ip>

When you drop into the UNAS shell you’re inside the root filesystem of UniFi OS, not the data pool. The disks you created in UniFi Drive are mounted under the volume1 ZFS dataset, inside a hidden service directory.

cd /volume1/.srv/.unifi-drive
ls -1          # every shared drive you made is listed here

Each drive has two sub-directories:

• .data – the actual payload if the drive is not encrypted
• .unencrypted / .encrypted – shown instead of .data when you enabled encryption for that drive

So a plain, un-encrypted drive called Media lives at:

cd /volume1/.srv/.unifi-drive/Media/.data

If you encrypted it:

cd /volume1/.srv/.unifi-drive/Media/.unencrypted   # filenames are still clear

(Those same paths are what NFS exports. showmount -e <unas-ip> will list them exactly.)

When you're in the right place, unzip on the remote:

tar -xvf <archive_name>.tar

And just like that. The full 25GB safely stored on the network in a few minutes instead of an entire day. Small files, man. They'll get you.