Credentials are stored in ~/.cargo/config, which defaults to chmod 644.
It might be worth splitting them out into a separate file which is something like chmod 600, e.g. RubyGems uses ~/.gem/credentials for this.
Encrypting them under a passphrase couldn't hurt either 😉