The absolute best handgun for self defense is the .357 Magnum with 125 grain jacketed hollow points. Numerous studies have shown this round to have a one shot stop rate of over 90%. Of course, this is primarily available in revolvers. But once again, for reliability, you can't beat a revolver. In the event of a misfire or primer failure, you simply pull the trigger again.
Avoid the 158 grain loads as they have a tendency to over-penetrate, are harder to control, and their muzzle flash can cause flash blindness in low light conditions. But you can't go wrong with something like the Remington Golden Saber, or the Federal Personal Defense Hydra-Shok.
The .357 Magnum also has the advantage that if you find you cannot handle the magnum loads at first, you can shoot lighter / less recoiling .38 Special ammo in the .357 Magnum. Once you feel comfortable with that, you can move up to the .357 Magnim loads.
If, for whatever reason, you don't want to go with a revolver and insist on a semi-auto, than I would look for something in the .45 ACP caliber. It's not quite as effective as the .357 Magnum when it comes to one shot stops. But it's very close.
@Newell Yeah. They dropped the .357 Magnum in favor of the vastly inferior 9mm. At least a lot of agencies have wised up now though and dumped the 9mm in favor of the .40 S&W.
There are two main reasons that most agencies don't use the .357 Magnum. One is that they want higher capacity and faster reloads. But that's rarely an issue with self-defense situations. 6 shots are more than enough in almost all self defense situations. And if you really think you will need more, you can become quite proficient with a revolver speed loader.
The other reason is that rookie cops had trouble handling the .357 Magnum because of its substantial recoil. That's why I suggest starting out with .38 Special ammo in the .357 Magnum until you are comfortable with it, and then switching to the more powerful Magnum loads.