rsync -a -v -e "ssh -c arcfour" user@server:/there ./here
Simply switching the encryption to arcfour makes a huge difference.
Faster SSH by Using Blowfish
Speed up scp file transfer
Rsync Basics