Case A1: A PC wants to convince a guard to let the PC into the castle
Case A2: The PC wants to seduce a woman in a bar
Case A3: The PC wants to compel a witness to admit to lying with his testimony
Case A4: The PC wants to determine if an NPC is lying
Case A5: The PC wants the NPC to agree to going to work the next day naked and singing a song to his boss.
Case A6: Same as case #5, but wants the NPC to actually do it (not just agree to doing it) (Note that this is similar to case A1, but with more extreme stakes that are delayed).
In some RPGs, all of the above cases are handled with role-playing only. The dice nor rules would ever be consulted. The GM would just decide the effects based on the merits of the player's words, the situation, and the NPC's personality. In case A4, the GM may or may not intentionally give some "tells" to hint to the players that he is lying.
My problem with this is that a player who has trouble thinking on his feet will never be able to convince someone of anything (and therefore could never successfully play that type of character). For example, I could never be able to portray someone who could pick up a woman in a bar (case A2) because I don't have that skill as a player. Also, the GM might not be a good judge of the qualities of a pick-up line.
My preferences is to use role-playing combined with rules. Several RPGs use this method. A player's role-playing serves as a modifier (normally set by the GM) to a skill roll of some sort. Other modifiers are based on the situation and NPC personality. For example, case A5 and A6 would have huge negative modifiers in most systems.
Case A4 can still be problematic. If a PC fails his "detect lie" roll but the player still feels like the NPC is lying, should the player portray his character as if he believed the NPC even though the player does not? Perhaps the "detect lie" skill only gives an additional way a player can detect a lie (if the player was unable). This is connected to the next topic - social actions against a PC.
Situation B: The role of the rules vs. role-playing when an NPC makes a social action against a PC
Case B1: An NPC wants to convince a PC guard to let the NPC into the castle
Case B2: An NPC wants to seduce a PC in a bar
Case B3: An NPC wants to compel a PC to admit to lying with his testimony
Case B4: An NPC wants to determine if a PC is lying
Case B5: An NPC wants the PC to agree to going to work the next day naked and singing a song to his boss.
Case B6: Same as case #5, but wants the PC to actually do it (not just agree to doing it) (Note that this is similar to case B1, but with more extreme stakes that are delayed).
In many games, the GM will argue for the NPC and the player will decide his own actions based on what he thinks his character would do. Hopefully, he will portray his character's personality if the rules have a mechanism for this. Many games have a reward system (such as experience points, Fate points, etc.) for when a player chooses to disadvantage themselves when in-line which they character's flaws.
One of the big questions I have is:
Should an NPC be able to make a roll to compel a PC to take a certain action? If so, how do you enforce this and what are the consequences?
Many players have a problem with this because it limits their agency. But how is this different that a character being knocked out in a physical conflict? Of course, being unconscious limits a character's agency as well.
Let's take case B1 as an example:
A PC is a guard and an NPC with a high persuade skill wants to get in the castle. Can the GM role-play the situation then make a roll for the NPC (something like a Persuade skill vs. the PC's willpower) and say, "He convinces you to let him in."? Would it matter if the NPC used magic (either to boost his own persuasion skill or to reduce the PC's willpower)? Why or why not? A player may choose to portray a weak-willed character and may role-play it when he wants, but should the rules force this? If so, should the rules use a carrot or a stick?
Perhaps case B4 is the most straight forward. Most systems would allow the NPC to make a detect lie roll or would force the PC to make a bluff roll.
Fear and morale and mind control effects often all under Situation B as well.
CLASSIFICATIONS OF GAMES
It would be interesting to classify how specific games handle social conflict.
1. Have no social combat system
2. Only have social combat for PCs affecting NPCs (sometimes this is explicit but often the autors never mention it one way or the other)
3. Have social combat both directions (including NPCs affecting PCs)
Sometimes, it is not a clear distinction. For example, many games only have Fear and Morale as social combat against PCs. For example, a character may suffer a Fear penalty if he fails a Willpower roll.
PC vs PC sometimes is a special case.
HOW DO VARIOUS GAMES HANDLE SITUATION B?
It seems like games handle social combat against PCs in three ways:
1. Ignore it. Many games do not have any rules for social actions against PCs and only deal with it via role-playing. A subset of these games also lack rules for PC social actions against NPCs.
2. Carrot. Some games reward players for allowing NPCs to affect them socially, especially when it causes problems for the PCs. This is sometimes called "good role-playing" as the player accurately portrays his character's quirks and disadvantages. Example rewards include additional experience points, Fate points, Destiny points, etc. The reward could be immediate (like Fate Points) or delayed (like experience points).
3. Stick. Some games punish players for not allowing NPCs to affect them socially. Again, this could be an immediate cost (like costing Willpower [Exalted] or Fate points [FATE]), an immediate penalty (like CoC???), or a delayed penalty (like reduced experience points)
Some games also have a mixture of these.
Note that some games include social combat, yet still do not address Situation B. For example, Burning Wheel's Duel of Wits.
A few examples:
Song of Ice and Fire
Call of Cthuhlu