The weight of words can be weird, sometimes. When you have a game that is hundreds of pages of rules upon rules upon rules, but then a paragraph in the beginning (or end, or wherever) states "hey, these are just tools, the GM has the authority to run the game however he/she wants," which one do you take more seriously? The D20 Dynasty that started in the early 2000s is where the scales got tipped, to where the rules became so big and bold and clear that the "these are just tools" print looked limp and faded, by comparison.
Part of what makes a GM's job so damn hard is in recognizing that aforementioned truth. A game can be 500 pages of solid rules, but when that game hits the table, it's the GM who actually manages what rules are used, how they are used, and why they should be used. The players have three choices: they can accept that and enjoy the game as the GM runs it, they can negotiate with the GM to adjust the adherence level to the rules, or they can find another GM.
So when should a GM use a rule? Here are two of my general guidelines:
1. When using the rule would add to the fun. Every rookie GM knows to throw a rule out if it's bogging down a table, but sometimes rules can add tension, excitement, and drama to a game. If that's the case, then be sure to play with them. Here's a mistake I still struggle with sometimes: when a combat starts going long and I just want to get on with the game, I'll suddenly turn the PCs into killing machines and let them rip apart their opponents. Or their opponents suddenly become cowards and run away, screaming. Inevitably, at least one or two players will grumble that the combat felt cheap. And they're right. I endeavor to push my shit aside, and be able to play the game as it's written. The players are expecting a big, tactical fight? Then I should be ready to give them one, with all the rules that fight entails.
2. When the rules help reinforce tone or theme. Picture a group of PCs making an ardous journey across a harsh land. Should food, water, and encumbrance matter? You bet your dice sack they should! The scarcity of those supplies and the capabilities of the travelers are part of the reason the journey is so ardous. If you're doing a game where survival in the face of grim adversity is a theme, then any rule that helps reinforce that grim adversity, no matter how far out of your wheelhouse, should be considered, if not flat-out used. Doing anything less is a disservice to yourself and the game.
3. When not using a rule would screw over a player. It's generally understood that a GM hand-waving his way through an adventure is all fun and games until suddenly someone gets shafted by it. Take the awesome ace starfighter pilot who suddenly doesn't get to use any of his skillset because you didn't read the space combat rules carefully enough and just decide that he automatically blows up the bad guys. See there? Even if your hand-waving benefits the player, you've denied them a chance to shine, because of your "artistic freedoms," and that ain't cool. If you tell a player that making a space ace is a fine and valid choice, then you, not the player, need to be ready for the consequences of that decision.
And, for clarity, here are some guidelines on when they should not be used. As I alluded to above, "when rules prevent the game from being fun" is generic, cliche advice that I shall ignore, though I do technically agree with it:
1. When the rule moves outside your game's focus. This is kind of the opposite of my second point, above. Say a big action scene takes place at a pier. You expect dead bodies and disabled parties to be regularly thrown in the water. But what you didn't expect was a fleeing thug to dive into the water, and one of your bloodthirsty PCs diving in after her! So do you start looking up the rules for underwater combat? Hell no! That's not what you signed on for. That's not what the scene was about. The water was supposed to be a detail, not the arena itself, so in this situation, you do whatever feels right. Whether that's automatically letting the PC kill her, whether it's ruling that the thug is a freakishly good swimmer and pulls way too far away from a PC before he can even get close, whether you just want to throw some light penalties on the situation and play it out....whatever. But the point is, you never let rules steal away the focus of your game.
2. When a player is meta-gaming you. This sounds like it's intentional, but it's often not. I once had a player who would load himself up like a walking gun store with every character he made, even if they weren't gunmen (though they were typically gunmen). He did this because he knew I didn't care about encumbrance or weapon rarities at the time. It became a joke; sometimes for a laugh, me and the group would just ask him to read off his equipment list. It was hilarious. Now, of course, I didn't mind it too much, becuase my adventures back then seldom had much combat, and he was aware of that going in, so no harm, no foul. But nowadays, I tend to run more balanced games, and therefore I expect my players to be more balanced, too. I still hand-wave encumbrance, but you better believe if someone starts using that to their advantage I will crack down on them. I don't mean that you suddenly have to start playing by a rule you're not comfortable with just to keep your players in line, but you do need to make sure, as I've said above, that your actions (or non-actions) don't lead to unintended consequences.
3. When the rule would contradict previous rulings. This also is pretty GMing 101 advice, but unlike the bit about fun, I'll go ahead and repeat this one. If you decided you're not going to make a big deal about people grappling each other in combat, and your PCs are using that against you and choking out your every monster, you can't suddenly break out the grappling rules during the next fight and expect everyone to be cool with it. It's not their fault you got lazy. In this situation, you're going to have to just drag yourself to the finish line for that night's session, go over those rules next time, and let the player knows well in advance that that shit ain't going to fly next time.
One last thing I wanted to say. A lot of new games these days have backlashed against the rules-first approach ushered in by the D20 Dynasty and simply offer thin, waif-like corebooks that just don't have that many rules in there to worry about. And some people...notably, gamers who came into the hobby during the Dynasty's hayday...have rejoiced over this renaissance. It's absolutely fantastic that there are some sharp, story-first games out there who are helping gamers realize that little "rules are tools" paragraph should be the biggest, not the smallest, of the text in most games. It's great that there are whole games out there embracing this concept from the jump. But, gamers, remember you can in fact have it both ways. You can take a heavy game, flip through it, say "hey, I don't want to use most of this shit, but I do want to use some of it," and play it (provided your players are cool with your interpretation). You don't necessarily need to convert it, and you definitely don't need to just ignore it, or worse, hate on it for being something you're not into. So play whatever you want, however you want to play it, and let the people who matter the most to you...your players...tell you if you're doing it right.