With no information about the hive you put them in, other than you put Lemon Grass Oil? in it, hard to make any informed comment why they might have left.
However the lemon grass oil may have been too powerful and driven them out, or a common reason newly hived swarms leave is because the sun shines on the hive and makes it too hot in there, before they have had time to work out their ventilation system.