I hate to say it but, they probably left from being messed with too much. You basically moved them twice. A swarm is a lot like a package you want to get them settled as soon as you can and leave them a lone until they get established some. Once they settle in and start comb they are unlikely to leave. If you add a frame of brood from one of your other hives to future swarms it will also keep them from leaving.