Mine usually swarms 3-5 times (once was 7) and they look really empty when done. Two months later you couldn't even tell. Note that you may actually cause the problem this time if you remove frames. It will be a week or two before the new queen starts laying so you won't be able to confirm queen presence unless you remove every frame until you find her. Even then, if she is out mating, you won't find her. Whomever gave you the advice to ignore the drone layer has done you a disservice.
Remember, swarming is instinct and bees are well adapted to re-queening in the process. However, the new queen could be eaten by a bird or something while mating. The worst time to disturb a hive is right after the swarming process when the new queen is getting mated and settled. This is when poking around can cause problems. If you want to make sure you are queenright, offer a new queen (in a cage) a week or so after the last swarm. If they attack her you are good, if not, introduce the queen and you are good. Look up videos on queenless hive behavior when offered a caged queen--there are good ones here somewhere.
Good luck (I think you will be fine--90% probability anyway).