When my very O/C neighbour got upset about a swarm in a tree in his yard, I did not say they were my bees (could have been from anywhere) but obviously he was convinced the nearest hive, mine, was the origin (and it might have been...no one saw where they came from). I was able to quickly call one of our two local Master Beekeepers (if it had been a weekday I could also have given him the local state apiculturalist's number), who came to retrieve the swarm, which within the hour it took for her to arrive had dissipated. I left him with a contact list for any future incidents.
There is also a list of local beekeepers who will come retrieve swarms.
I too will have to find an outyard to expand my hobby. I don't mind that but I do want to keep one hive here at home as I love to watch the bees.
Which is all by way of saying....in these incidents never admit it is or may be your bees unless there is concrete evidence it WAS your bees, but offer all the arm's length help possible as a gesture of good will. Neutrality is important...suggest remedies that the homeowner can take if possible, and offer to find them help for swarm retrieval if you can. Sympathize but do not apologize.