set up chat for certain groups, however can start chat with people not in said group

[obi-two]

As above really,

Just installed chat,
when I tick "Enable only for specific groups " and then set two groups up I am able to start a chat with users not in those groups,

Also when starting a group chat I can't seem to be able to remove user that I've added.