That's a normal behavior for XenForo. Even if you have the category to not show on the index, once a user knows the path, if they have permissions to access that area, they can simply use the URL to access it.
The only way to get around that would be by revoking the view
permissions.... and that requires at a user group or actual users level, which could get rather "fun" in keeping up with programatically. I am not particularly crazy about the add-on creating new ACP groups for each "social group" that is created, then trying to keep up with adding users in/removing them as time progresses.