For me, as somebody who GMs far more than he plays, roleplaying is all about rolling with the punches. You obviously can’t predict 100% what your players will do, and you can’t predict die rolls, or whatever random element is important in your system. Basically, shit happens, and you’ll have to deal with it. Here are a few tips I’ve found to make that easier:

#1 – Practice Lying

• Like with any skill, you’ll only improve by doing it a lot. Most of quick thinking is a combination of creative thought, staying calm, and lying.
• Lying can be a lot of things; it can be lying about dice, lying in character to cover an NPC’s tracks, or even telling the truth in a way that hides information.
• It’s not really ABOUT lying. It’s about being able to think quickly and say something without the players reading your whole game from how you introduce the villain. I actually like mostly open games, but you still need to be able to keep secrets from your players.

#2 – Co-opt the players’ ideas. AKA “give the players what they want”

• I’ve had at least 3 campaigns where somebody said “I’ll bet I’ve got it! X did Y and that’s what caused Z!” What should you do in those circumstances? Well, obviously, continue with your current plot. But maybe drop subtle hints that whatever they suggested also might be true. Now you have an extra option, and if the characters don’t catch onto the one you thought, you have one to fallback on. Plus side? The player who guessed that gets to feel really cool for cracking the case early. And while he might not have been satisfied with your explanation, of course he’ll find HIS cool, because he thought of it!
• Law and Order shows do something like this all the time: random evidence will pile up against a whole number of people; somebody has motive but you later find out they have a solid alibi, somebody else was at the crime scene 30 minutes in advance but couldn’t have committed the murder due to crime scene evidence suggesting that the murderer was taller, and so on.
• Don’t overuse this one, the players can’t realize that whatever they say comes true.

#3 – Never reveal post-game

• A lot of GMs have a tendency to just spill their whole campaign post-game: “yeah, and if you guys had done this, this would have happened!” or “you didn’t find this out, but this character was actually responsible for that murder!” I can guarantee that half of your players just don’t care. You know what would make them interested in that idea? If you tweaked it and used it in a future campaign.

#4 – Stay Calm and Carry On

• The most important thing to do is to keep the game moving. If you have to hem and haw over your notes to figure out what happens, you’re doing it wrong. Pick a course, even a broad one, and go with it; you can work out the details as you go along.
• This means you have to be able to relax. You can’t get nervous at the first sign of a penny on the tracks because any given game could shoot off the rails into something you didn’t predict.

#5 – When in doubt, pick the road of conflict.

• At some point, I was playing a thief/assassin character who was trying to break into a mansion and kill somebody who had ordered my execution. Of course, there were two windows, and I didn’t know which was the right one. I picked one at random, the GM rolled some dice, I snuck in the window, and lo and behold, I found myself in the room of the massive orc bodyguard. Was it random? I don’t know. But what I know is that as a GM, I’d probably have rolled some dice for show, then sent the character to the bodyguard because that’s what’s INTERESTING.

  Which makes a better story? “… So I snuck into his room and killed him and snuck out and the game was over” or “… So I snuck into his room, only it wasn’t his room! I heard the click of a pistol, so I hurled myself to the ground just to survive. I look up and this massive orc is barreling down at me, so I roll out of the way, and dodge desperately, not even able to get an attack in on my own. So there I am, parrying every one of his blows, feeling the shock travel up my arms and just hoping my sword won’t break. Eventually, I land a hit, but it’s obvious he doesn’t feel anything! I roll behind this table, and he PICKS THE TABLE UP! And I realize I can’t block a table with my sword, so I just start dodging around the room, trying to find the door, but it’s so dark that I still can’t see anything, and I’m just waiting for my friends to show up from outside when — WHAM! — he catches me with the table, and now I know I’m fucked…but wait, I hear running footsteps from down the hallway; either those are my friends, or I’m seriously screwed…”

Basically, improvising is simple. Cast a broad net as far as campaign hooks, because that gives you options (and it mirrors reality; it’s rarely going to be the case that EVERYTHING points in one direction). But when that fails (or if you’re running a more directed campaign), learn to feel comfortable with improvising by doing it a lot. When improvising, prioritize conflict and intrigue; you want the choices that make for the most twists to be the ones available to the players, and don’t be afraid to lie and manipulate the game to get the players there; it’s your job to give them an interesting game!