Having trouble making your protagonist sympathetic? Torture the hell out of ‘em like Vonnegut says! If they’re a kid, kill their dog or a beloved parent. If they’re an adult, kill their kid or significant other. Give ‘em some sort of physical or mental deformity that makes socializing painfully difficult. Torment them with the memory of a tragic mistake that they’re still paying for. Or try a number of other terrible things on them.
3) Regarding Antagonists/Villains: Everyone is the hero of their own story, so don’t make your villain “evil for the sake of evil.” Examples of these sorts of villains include the Joker, a maniac who IRL would never ever have a criminal career because no real criminal would ever trust or rely upon him; any Captain Planet villain, who pollutes the environment and cackles evilly about it; and any “destroy the world and rebuild it with me in charge” villain, whom would never be followed by even the craziest of cultists. Properly-written bad guys do what they do for a reason, and they justify their actions. This actually applies to all characters, but especially to villains since they tend to be the most ridiculous and illogical characters in bad fiction. Read any Silver Age comic book and you’ll know what I mean.
Also keep in mind that “antagonist” and “villain” aren’t synonyms: the antagonist of the story can be a “heroic” character who is misguided or undeserving of the title. Likewise your protagonist can be a villainous sort, as long as he/she is still sympathetic to the viewer or reader. Look at the film Despicable Me as an example: the main character is a villain, yet he’s sympathetic and very human. The antagonist is anyone who opposes the protagonist.
4) Regarding Antiheroes: antihero and villain are not the same thing. The antihero is either a bad person who (often reluctantly) dons the role of hero when it becomes necessary, or a bad person who is also the protagonist.
If the former, the antihero has to actually be heroic* at some point; AND the antihero has to be the sort of person who would logically do something heroic. If your cold-blooded killer decides that he cares about a long-lost daughter, or risks his own life to save someone else’s, there’d better be a logical reason for him to do so, and that reason better be foreshadowed early in the story.
If the latter — like Tony Montana in Scarface — he has to show more humanity that his peers. Tony was a brute and a drug lord, but he was a man of his word, he had lines that he wouldn’t cross, and he was a poor working-class hood trying to make it in a world full of priveleged ones. He’s still bad, but he’s not as bad as most everyone else in the movie: the coward who won’t do his own dirty work, the madman who doesn’t think twice about murdering children, the cop who abuses his position in society, even the creepy back-shooter who does him in at the end.
In either case, the antihero still has to be sympathetic. He/she has to show more humanity than the villain does: he can’t go around raping women and murdering children and expect the reader/viewer to root for him; but he can have done those terrible things in his past, hate himself for it, and want to redeem himself somehow, like William Munney. See what I mean?
Deckard in Blade Runner is a good example of an effective antihero. He’s a scumbag, a killer, and a bigot. His job is to kill replicants who don’t want to live like slaves (the ingrates), and in the beginning he views replicants as nothing more than machines. But we see sparks of decency in him: we see that he doesn’t like killing, and we see that he’s (somewhat) capable of sympathy. As the story progresses and Deckard gets to know the replicant Rachael, we can see his condescending view of replicants changing, though not without a fight; and we keep watching in the hope that he does change for the better and do the right thing, which he finally does.
* Let me be clear about “heroism”: I’m not talking about badass feats that make the character awesome, like the daily lives of pro athletes or some dude who goes around getting into fights with large groups of enemies. Heroes commit acts of self-sacrifice for the sake of other people, like the man who dives into oncoming traffic to save a child from getting run over. The antihero Conan the Cimmerian (as played by Arnold) completed his self-centered quest for revenge when he killed Thulsa Doom; he wasn’t a hero until he decided to also burn down Thulsa’s temple, ensuring it would never rise again.
5) The most important thing: if your characters aren’t complex, they should at least be consistent. Character consistency is actually more important than complexity, when you come right down to it. The Star Wars universe is filled with one-dimensional characters who serve their purpose: they carry the plot. Here and there they show sparks of complexity, but for the most part they’re archetypes, and they work. You don’t have to have volumes and volumes of backstory for each character. Just make sure that, when important decisions must be made in the course of the story, that the goody-goody does what a goody-goody would do, the scoundrel does what a scoundrel would do, and the bigot does what a bigot would do. Just make sure when one of them defies our expectations (selfish bitch risks life to save protagonist), there’s a logical and well-established reason for it (protagonist was selfish bitch’s first love, and she still hasn’t gotten over it).
As a bonus tip, if you’re planning on making a horror story with an ensemble cast of shallow assholes whose only purpose is monster fodder, DON’T. Seriously, stop it. Those characters are not interesting or fun: they’re an absolute poison to have to hang out with for however long it takes them to die, and it prevents the reader/viewer from becoming invested in anything. Yes, it works, if you only want to make disposable entertainment; if that’s all you want, your standards are too low. Make the characters interesting and sympathetic. Then when they get scared, we get scared, too, and the story is way more fun.
Give Unforgiven a watch or two for a great example of how to play with character. It’s about a gunman renowned for murdering men, women, and children, and how he butts heads with a sheriff who likes to keep his town clean.
The sheriff is the villain.