You let the players play thru if the situation permits. There is no automatic play thru card. If you're a foursome holding up a twosome occasionally then keep playing at the flow you're at. If they've caught you at the tee multiple times, offer to let them play thru.
We tend to evaluate the skill of the group behind us and how fast they're playing. Our group usually ranges from above average players to pros so a group of rec players probably won't get the invitation to play thru unless they're hustling and a twosome. Nothing's worse than letting the play thru happen and then having to wait on every hole.
We all start at hole one and it's ridiculous that people behind you are "entitled" to play thru. That is one of the worst things about this sport (the few downfalls there are)...lack of etiquette. This goes both ways. People feeling they have a right to play thru even though they've only caught you a couple times during the round (catching meaning on the tee). People not letting you play thru even though you've been on their heels the hole round. People cutting, jumping holes, throwing in on you, etc. This is supposed to be a (modified) gentlemen's game but now it seems like it's more of a boys game with no manners. So what you have to wait for a few minutes. That's the way it goes when you're second in line on a tee. Deal with it.
Either way, being thrown on is inexcusable!!! You showed great restraint. I'd thrown their disc into the woods and had a few choice words for them. This coming from some one who's been hit in the face twice and multiple times else where. No excuse!