Limit member access to repos in an organization

[beschloo]

I am trying to invite a client to Github so that they can create tickets for future features and issues.

Things that I have done.

I created a team under the organization for the project and connected the related repos to the project to the team.

What I am finding is that when I invite the client to the organization so they can join the team, they are gaining access to all repos in the organization.

How can I add a user to the org but limit the repos they have access to?