Right. Its more important to dress right for one set of skills to maximize how they work.. so if you go tough, ditch the charm gear, its worthless.
There is an option to get a skill called judge of character that lets you tell whether a suspect will respond better to charm or tough. But if you are wearing really high tough gear, even a suspect that would prefer charm will buckle under for you.
If you pick charming, you can do a little better with members of the opposite sex by adding flirting. I don't think that works if you are tough - but someone else might know better. If you go 'tough' a tommy gun will do you a lot more good than flirting.