honestly man, the plastic molding for disc golf is not something the manufacturers have perfected. i wouldn't be surprised if it's just easier for the manufacturer to spitball weight ranges when they're loading up the machines. there are plenty of discs, for instance, listed as 175g that are actually 176/177/178, etc. your destroyer might not even weigh quite what it says on the back.
to answer your question, though - no. even up to 7-8 grams won't really make an appreciable difference, depending on the disc. for instance, i like to throw my drivers in the 167-170 range when possible. i can throw over 400'. it's not big distance by professional standards, but i have no problems driving big into headwinds so weight within 7-8 grams for a good, overstable disc like a destroyer is definitely not a concern. the real question is the stability inherent to the disc itself. once you start going into the lower 160s, then things start to get different.
it depends on the mold, too. a firebird from 175-160 will still be a firebird. the 160g will fly farther but still be HSS. you won't penetrate headwinds as much and it might get bounced around a little more, but on the whole the difference is not much. a polecat or a comet will act vastly different: since they start very neutral to begin with, losing that weight makes them more touchy.
you'll need to drop down to 150 class to start seeing major differences in flights with destroyers, but beware - some discs mold up differently at lower weights and act just as overstable, sometimes more overstable, than their heavyweight variants.
you'd be better served by changing plastics, IMO. use a pro destroyer to match your star and beat it up, or get a dx destroyer. that will make much more of a difference than just dropping weight. the reason being that these types of plastics have much lower PLH on average than a star destroyer.
has anyone told you, or do you know, what PLH stands for and what it means for discs?