It is very likely that you have missed the main portion of the colony. It will most likely be in the soffit. You can pull down one or two pieces of the vinyl soffit to see if that is so. It is also likely that you do not have the queen as she will usually retreat when you begin to disturb the colony.