Excellent question, because it touches on a sensitive but necessary topic.
I think it's very important to not only match people up with my campaign style, but also the other folks at the table. While some tension can be a very good thing, too much of it between players can be a very bad thing.
Before I ever bring a new gamer into my house, I give them a list of things that I require of anyone who visits. These are actually not all related to gaming, and are very simple; but they are also very important pass/fail points.
- You have to be clean. I tell people up front to wear clean clothes and shower or bathe appropriately.
- I have a wife and kids. You have to be nice to them.
- I have a dog and cat. If you have strong allergies, skip on it.
- Smoking is for outside. Butts are for the ash can. Do not fail on either count.
- Drinking is welcome, casually. Getting drunk, or doing any form of drugs is not.
- Rules are debated after the gaming session, unless it's a "live or die" ruling or I state otherwise in advance.
- Everyone will create characters that can play as a team, unless the campaign dictates otherwise.
I know the first one sounds especially mean, and I don't enjoy saying it -- but it has been quite useful and many people sound relieved to hear it when I say it. I have run into more than a few stinky gamers in my time (an extreme minority, but still) and I just don't want to deal with it in my house.
Which is the point of most of my rules up there. They set the stage for what I expect at the table: people who are clean, affable, and ready to have fun in the game with each other.
The question about what to do if it's not working out is an important one too. Not every campaign is for every player; and not every mix of personalities works out. I have not had to do this much, but I have told more than one person that they are no longer welcome in my home. The key to it is to be polite. You are already kicking them in the nuts by telling them they aren't welcome to play; you don't have to beat them into a pulp over it. That's not the goal. The goal is to just admit that you don't run a game they can enjoy, and that it's best that they look for one that they can.
I can't say that this makes people smile and thank me, but by sticking with a simple message that does not say "We all decided that you are a jerk and can't imagine anyone liking you." it makes the conversation quick enough and direct enough to get it done. I don't advise waiting until they show up for the next session to do this either. That's rude.
The minute you know you don't want them around, you need to contact them and tell them. It does not have to be in person, but it has to be clearly communicated by the best means you have -- and you can't beat around the bush.
For the games I run, that's about all the advice I can share because it's the fullness of what I use. The rules above filter out enough people who see it and decide that I'm not for them, and while I have not done it in more than a decade, my willingness to politely remove people who don't match after a few sessions takes care of the rest.
The end result is what I think is the most important: A table full of people who can have a great time gaming, and who can become friends in the process. I've moved enough times and restarted from nothing to see that this works and hope it's helpful to others.
Now I have to go face the firing squad...