Rsync to OSX with foreign letters in filenames - delete copy loop

Submitted by Janak on Sun, 05/08/2011 - 14:12

So, you have a few files with foreign letters in filenames and you back them up using rsync. All is well unless you rsync from a linux machine to OSX machine. It seems OSX handles UTF-8 characters in a custom way - using 3 bytes.. hmm.. so:

Linux -rsync-> OSX

  • First pass: rsync from linux to OSX and all files appear to be copied
  • Second pass: rsync deletes the files on remote OSX server and copies them again
  • Third pass: same as above

So, clearly, somewhere along the line rsync doesnt see the remote file the same as the local version - due to OSX filesystem using funky 3 byte for UTF-8..

Rsync Problem with OSX

quote from

Something else that can trip up rsync is a filesystem changeing the filename behind the scenes. This can happen when a filesystem changes an all-uppercase name into lowercase, or when it decomposes UTF-8 behind your back.

An example of the latter can occur with HFS+ on Mac OS X: if you copy a directory with a file that has a UTF-8 character sequence in it, say a 2-byte umlaut-u (\0303\0274), the file will get that character stored by the filesystem using 3 bytes (\0165\0314\0210), and rsync will not know that these differing filenames are the same file (it will, in fact, remove a prior copy of the file if --delete is enabled, and then recreate it).

You can avoid a charset problem by passing an appropriate --iconv option to rsync that tells it what character-set the source files are, and what character-set the destination files get stored in. For instance, the above Mac OS X problem would be dealt with by using --iconv=UTF-8,UTF8-MAC (UTF8-MAC is a pseudo-charset recognized by Mac OS X iconv in which all characters are decomposed).

Rsync UTF-8 solution

Rsync example command from linux

rsync -avh --progress --del  --iconv=UTF-8,UTF8-MAC  --rsync-path=/usr/local/bin/rsync --exclude-from=/rsyncexcludes.txt /music/ -e 'ssh -i /rsync_key/key -c arcfour' user@remote_server:/{remote_path}/music/