In my years of running adventures, I've discovered that the best adventures are the ones where I, as GM, have just the right amount of grasp on the story as it unfolds. Not so much that I'm railroading the PCs (to those unfamiliar with the terminology: "railroading" means you don't let the PCs guide the story; you more or less make them play the adventure the way you want them to play it), but not so loose that I have no idea what's going on. It's a very Zen-like thing; to have a definite direction, but also allow yourself to be swayed by the winds of your players' whims. Put another way, I've also thought of adventure design like building a house of cards. Yeah, you might build something beautiful and elaborate, but you better be ready for that thing to come tumbling to the ground on a moment's notice!
In a lot of new-school RPGs, the idea of "turning the questions back on the players" has become all the rage. Used to be, back in my day, when a player wanted to know what an elf looked like, there was a specific page in the book that would answer that question. Nowadays, the GM is recommended to say: "I don't know. What do you think an elf looks like?" That's a great little storytelling judo move, but I've discovered one very important caveat: if your player has no idea what an elf looks like and is looking to you for guidance, you better have something for them! If they don't know what an elf looks like, and you don't know what an elf looks like, the game can go south very quickly from there. This whole "turn the question back on the players thing" is fantastic if your players are creative, or if they're playing a game expecting to do that kind of thing. But if they aren't, then you're in trouble.
But I definitely understand the guiding principle behind it. There's nothing more infuriating to me than when I hear stories about players with legitimately great ideas for a character or a background or something and the GM simply vetoes it because it doesn't jive with the game they're playing, or the prep the GM made. To me, that's not good role-playing. It's not enough to be creative. You have to be flexible, too.
So, here are a couple of tips and techniques I've picked up over the years to help hit that sweet-spot between too prepared and not prepared enough. If you, dear reader, have any adendums or other tips and tricks you'd like to share, please let me know. I'm always interested in opportunities to hone my craft.
1. Write up your adventure, plot-point by plot-point. For almost every adventure I write, I have a little list, like a laundry list, of plot points for the PCs to pick up before the adventure ends. They don't need to hit them in order, but when the PCs don't know what to do or where to go, I look at that list and I guide them to the next item on it.
2. Always use what the players give you. If a player writes up a six-page backstory on their character, work that stuff into the adventure! The more of that player's backstory you put into the game, the less work you have to do yourself. If you're trying to figure out why the players are going to explore this dank, dark dungeon and one of them wrote about how his sister mysteriously disappeared when he was young, put that sister in the dungeon! Or, at the very least, let the player believe his sister is in the dungeon...
3. Be mindful of every player at the table. Another new-school piece of advice handed out these days is to make sure every player at the table has a chance to be the star of the show. I agree with this, on roughly the same caveat: it's a great idea except when it's not. I have had players in my game who essentially want to do little else than hang back and watch everything unfold. Maybe they're shy. Maybe they're new to RPGs and learn by watching. These people often don't want the spotlight; they may even be uncomfortable in it. So I say "look out for each player." If someone keeps trying to say something and someone else keeps cutting him off, direct some action towards that player so he has a chance to do whatever it is he or she is trying to do. Likewise, if someone is staying quiet but watching everything with a big smile on his or her face, ask that player how they're doing, if there's anything they would like their character to do, then back off.
4. Leave a few blanks on the page. This is some new-school advice I can get completely behind. You should have a good idea of everything that's going on throughout an adventure, but maybe leave one or two spots blank, so you can fill them in with exactly what the group needs at the right time. This is an espeically handy trick for switching up the pacing in an adventure; if there's too much fighting going on, then maybe one of those blanks is a role-playing scene, or a puzzle or something. Likewise, if the players are getting bored, maybe one of those rooms has a few dozen orcs in it, ripe for the slaughter!
5. Put the "role-playing" in role-playing games. What makes RPGs different from every other game out there is the chance for the player to not just act like someone else, but to actually be someone else. Give players that opportunity, whenever you see it. I haven't met a player yet who doesn't enjoy hamming it up in-character every once and awhile. Either let them interact with NPCs, or just set up a scene between two players and tell them to role-play it out. When it's done (or when it starts to drag), then give each player a fate point or some experience points or whatever and move on.