I play solo rounds almost exclusively, so run into this regularly. And I share the same sense of "anxiety" about joining a group for a single hole and shanking my drive off into never-never land. That being said, short of skipping the hole or staying behind a much slower group, you've got to join them for a bit and take your best shot.
I've found that some groups will actually let you tee off and then run up ahead to finish the hole before the tee, and others will let you tee along with them, and hurry ahead to finish the hole. Either works fine for me and I always thank the group for letting me through. I recognize that I may not play my best on that hole, but I'm not that good anyway...
There have been times when I'm near the end of the round and I just don't want to bother with playing through, so I'll do some of the things suggested by Trifocal. My favorite is to go back and repeat the hole. If there is no one crowding you from behind, why not??
The other thing that I've found is that if you catch up with one or two players, or they catch up with you, joining them on their round is a good way to meet other golfer and maybe enjoy something other than a solo round for a change.