Actually, after observing this several times in the observation hive I would say this:
I don't think they can refuse to feed the queen. She asks, and they feed. What they do is back fill the brood nest so she has no where to lay. When she lays less, she eats less. When she eats less, she shrinks. When she shrinks she can fly. If she can't fly, the swarm can't go. This happens over a period of a couple of weeks. As johathan says, she lays a few eggs up until she leaves, but not very many.