Keep it loose. Basically when you pull the martingale slip it should tighten up behind the ears just tightly enough that it isn't going to choke the dog. So when the dog is not pulling it should rest very loosely at the bottom of the neck.
"All changes, even the most longed for, have their melancholy; for what we leave behind us is a part of ourselves; we must die to one life before we can enter another." -Anatole France