You can make SSH much faster with the ControlMaster configuration directive. When enabled, only the first SSH connection to a server will incur slow connection overhead. Each additional connection will simply reuse the first connection via a tunnel.
This is especially useful when using command-line completion over SSH (using zsh, for example). Using tab completion on a predicate like scp gibson:www/ht will result in a multi-second delay each time, but with SSH ControlMaster connections, the delay is sub-second.
Just add this to your ~/.ssh/config:
Host * ControlPath /tmp/%r@%h:%p ControlMaster auto
That's it. Now your first SSH connection will automatically act as a ControlMaster. Each subsequent connection will be almost instantaneous. In my extremely scientific sample size of one host-server pair, this made subsequent connections about 16 times as quick (0.1 seconds to connect instead of 1.6).