This is the perfect situation for 3-part messaging.

When you said this....state the problem

I felt...what did you feel?

And that means....in your case, that you wont be joining the group.

I taught my students to use this approach. It never fails to work. The bully has lost all power.