HSS drill bits won't break. They don't cut as fast, but if you are snapping carbide bits left and right, maybe that's the place to start.
With carbide bits-- be careful not to exert any side pressure, always let the tool do the cutting (only very gentle pressure), keep the flutes clean by backing off every few turns (we're talking plastic here; only a quarter turn in brass) to clear the swarf. If you are having trouble holding the drill and workpiece steady, try holding your arm tight against your body so that only your wrist is free to move. If you still can't control the shakes, you need to devise some kind of fixture.