Weighted Reviews

[wellsbranch250]

I've seen the topic brought up before on weighted reviews, most recently by BrotherDave on the vendetta purge thread. https://www.dgcoursereview.com/forums/showthread.php?t=131131

I think it's been brought up before but what if TR's ratings were more heavily weighted than non-TR's? So if a TR comes through and gives course X a 3.5 it will take a lot more drive bys giving it a 1 to drag the average down. Higher the TR rating, higher the weighting? You've got to play a lot of courses to become a TR and usually just by osmosis that tends to make better reviewers. It's not perfect b/c you can become a TR easily enough if you whore around for up thumbs but it seems like it might work. I'd rather have the solution be more reviewer based rather than nebulous, moderator or algorithm based. Seems like that would encourage more reviews and better yet make reviewers better at reviewing.

This is the current top 25 using weighted reviews.
the formula for this graphic is (.1 for less than 5 reviews, .2 for non TRs, .4 for bronze, .6 for silver, .8 for gold and 1 for diamond.)
the top ten would have 3 new comers and the top 25 would have 4.

 

[Cgkdisc]

Looks like Langley is both 14 and 25?

 

[wellsbranch250]

whoops, i set the the tyler start east overlay to 0% fill. or perhaps Langley is just that good

Looks like Langley is both 14 and 25?

 

[Three Putt]

I mean the .1 for <5 reviews would take the bite out of the impact of the driveby reviews, and people who got a lot of thumbs down (indicating their reviews are not good) would get stuck at .2 and minimize their impact as well.

On the other hand if Paul McBeth decided to review courses, he would start out at .1. :\ Although I'd assume if he did he would be a thumbs up magnet and hit TR status quickly.

I'm not sure what I think. I don't love it enough to advocate for it or hate it enough to try to kill it. I'm interested in seeing where this goes.

 

[wellsbranch250]

yes the .1 was an attempt to lessen the impact of the driveby vendettas and homer reviews. As for Paul McBeth. just because he's one the greatest players in the world doesn't automatically make him a great reviewer. he'd half to earn it, just like he earned becoming a top player by dominating on the golf course.

I mean the .1 for <5 reviews would take the bite out of the impact of the driveby reviews, and people who got a lot of thumbs down (indicating their reviews are not good) would get stuck at .2 and minimize their impact as well.

On the other hand if Paul McBeth decided to review courses, he would start out at .1. :\ Although I'd assume if he did he would be a thumbs up magnet and hit TR status quickly.

I'm not sure what I think. I don't love it enough to advocate for it or hate it enough to try to kill it. I'm interested in seeing where this goes.

 

[Three Putt]

yes the .1 was an attempt to lessen the impact of the driveby vendettas and homer reviews. As for Paul McBeth. just because he's one the greatest players in the world doesn't automatically make him a great reviewer. he'd half to earn it, just like he earned becoming a top player by dominating on the golf course.

I'm not disagreeing with that, but it kinda goes back to what disc golf is. Disc golf is a sport that is run by players for players. I know a lot of people who have a problem with this site because of the idea that we earn our respect as TRs at a keyboard when where you earn respect in disc golf is on the course. The idea that somebody who is a past World Champion would need to earn anything from a bunch of "keyboard warriors" would drive them nutso.

The flip side of that is that people who have taken the site seriously, written a bunch of reviews that have been given approval (thumbs up) and reach the TR status probably have a better handle on the rating system and why shouldn't their opinion have more weight?

I can see both sides of it.

I guess it comes down to how much of a problem we think the drive by 5's and vendetta 1's are, and does the fix for that do more harm than good. Which is at least worth discussing.

I kinda lean toward the idea that it probably wouldn't make that much difference but I've been wrong before. Maybe we could get some specific courses like Black Falls that has a 1.5 review that would be rated at a .2 and see what it does to those courses ratings. Black Falls has two drive by 5's and two drive by 4.5's along with that possible mistaken identity 1.5, so it may come out in the wash.

The Woodshed obviously was mentioned as a course to look at. I'm sure people will chime in with other courses where there is a weird dud review and we could get some idea of how much of an impact it will have for those courses. That might help us get an idea of if this will even help address what we perceive as a problem.

 

[DiscGolfCraig]

Cue the piano. A tradition unlike any other...the annual drivel over giving more importance to TR's ratings. If any other TRs think his voice is more important than others on the site, well, that's why Tim created the ignore button.

 

[DavidSauls]

Interesting, thanks for compiling it. A skeptic's question:

(1) Does it really make enough difference to be worth the effort? (Other than to the private courses that get a boost, of course).

(2) Does it have unintended consequences, changing the way people review courses to get more thumbs, or diminishing the desire of new members to review courses? (I doubt it)

(3) Does it open the door to a debate that, though weighting may be good, different values for the weights would be even better? (It would certainly open up the debate that some other factors, like weighting by age of review, should be done).

 

[biscoe]

I think if i were going to try to "fix" the "problem" I would simply weight reviews by first time reviewers .5 and leave the rest alone.

 

[biscoe]

yes the .1 was an attempt to lessen the impact of the driveby vendettas and homer reviews. As for Paul McBeth. just because he's one the greatest players in the world doesn't automatically make him a great reviewer. he'd half to earn it, just like he earned becoming a top player by dominating on the golf course.

Nor does doing a bunch of reviews make one a great reviewer.

 

[wellsbranch250]

I guess it comes down to how much of a problem we think the drive by 5's and vendetta 1's are, and does the fix for that do more harm than good. Which is at least worth discussing.

I kinda lean toward the idea that it probably wouldn't make that much difference but I've been wrong before. Maybe we could get some specific courses like Black Falls that has a 1.5 review that would be rated at a .2 and see what it does to those courses ratings. Black Falls has two drive by 5's and two drive by 4.5's along with that possible mistaken identity 1.5, so it may come out in the wash.

The Woodshed obviously was mentioned as a course to look at. I'm sure people will chime in with other courses where there is a weird dud review and we could get some idea of how much of an impact it will have for those courses. That might help us get an idea of if this will even help address what we perceive as a problem.

there are some differences, 3 different top 10s and 4 different top 25s. I'm not suggesting replace the current number for the average, but rather presenting a different list. from looking at the data i now now which courses had vendetta hit impact (maple Hill) and which one have a lot of drive by 5s (vision quest, phantom, smugglers)

I also ran the weighed number on the woodshed and black falls, but they are probably both outside the top 40. I ran scores for 44 courses
of the 44 i ran, 4.43 blackfalls was 40th and 4.30 woodshed was 44th.

Cue the piano. A tradition unlike any other...the annual drivel over giving more importance to TR's ratings. If any other TRs think his voice is more important than others on the site, well, that's why Tim created the ignore button.

I'm just compiling statistics to see what the data says. I have not made an the argument that TRs ratings are worth more, but rather attempting to see what that type of evaluation looks like. If you look at the top of graphic for the orange list, its a separate list from the main list. Some people may be interested in what the veteran reviewer thinks. It would be kind of like critics choice for movies.

Interesting, thanks for compiling it. A skeptic's question:

(1) Does it really make enough difference to be worth the effort? (Other than to the private courses that get a boost, of course).

(2) Does it have unintended consequences, changing the way people review courses to get more thumbs, or diminishing the desire of new members to review courses? (I doubt it)

(3) Does it open the door to a debate that, though weighting may be good, different values for the weights would be even better? (It would certainly open up the debate that some other factors, like weighting by age of review, should be done).

1. that's for the programmer to decide. I have no idea what it would take to write this formula

2. I though about that. I think so. I think it has the potential to encourage people to write more reviews and be stick to the forum rules.

3. I also pondered that, using timing the reviews out as a method. This is similar to the pga tour were a golfer win 2 years ago isnt worth as much as yesterdays win in his world ranking. ill see reviews about a course in 2009, complaining about natural tees and no signage and giving it a 1. but now the same layout in 2018 has all these items and is getting 4s. I like to see the history of a course and that shouldn't be erased, but perhaps there could be a way to preserve the writing but eliminate the now unjustified rating.

Nor does doing a bunch of reviews make one a great reviewer.

very much agreed. on your other point, the formula can be compiled any way. this was just one scenario

 

[Steve West]

...i set the the tyler start east overlay to 0% fill....

If I had a nickel for every time I've done that...

 

[scarpfish]

The problem with this is the same problem with the TR system in general. The fact that it is geographically discriminatory, particularly at the bronze level. People who live in urban areas or review top ten courses can get thumbs and to TR status much faster than those who do not.

 

[blake833]

3.[....] ill see reviews about a course in 2009, complaining about natural tees and no signage and giving it a 1. but now the same layout in 2018 has all these items and is getting 4s. I like to see the history of a course and that shouldn't be erased, but perhaps there could be a way to preserve the writing but eliminate the now unjustified rating.

What if reviewers got a notification on reviews that turned 2 or 3 years old to update them?

 

[timg]

I've never seen that graphic before... I usually look at the threads in the suggestions area so if it was in this forum the first time around, it's probably why I missed it.


I think it's interesting although timing out reviews wouldn't really work unless I had a mechanism that said "x course was improved on this date". Some courses just stay the same so those old reviews are still pretty valid.


The idea of reducing weight for people that have less than 5 reviews seems the like the best balance of avoiding homers / vendetta reviews. Increasing the weights of TR reviews I feel could be gamed to some extent. And someone that's bronze maybe just lives in an area where there aren't many courses so they don't have an opportunity to write more reviews. I don't think they should be penalized for where they live or their lack of ability/desire to travel.

 

[timg]

The problem with this is the same problem with the TR system in general. The fact that it is geographically discriminatory, particularly at the bronze level. People who live in urban areas or review top ten courses can get thumbs and to TR status much faster than those who do not.

That can be difficult if you played some courses while you were in say, California and you live in NY.

 

[wellsbranch250]

I've never seen that graphic before... I usually look at the threads in the suggestions area so if it was in this forum the first time around, it's probably why I missed it.


I think it's interesting although timing out reviews wouldn't really work unless I had a mechanism that said "x course was improved on this date". Some courses just stay the same so those old reviews are still pretty valid.


The idea of reducing weight for people that have less than 5 reviews seems the like the best balance of avoiding homers / vendetta reviews. Increasing the weights of TR reviews I feel could be gamed to some extent. And someone that's bronze maybe just lives in an area where there aren't many courses so they don't have an opportunity to write more reviews. I don't think they should be penalized for where they live or their lack of ability/desire to travel.

You didn't miss anything timg, I photoshopped it yesterday afternoon. I can adjust the formula any way desired, This was just one method. Do you have a suggested formula you'd like to see?

 

[timg]

Oh, I meant the added categories, New Courses, Outside the US. As far as formula, you could try it with just the < 5 review crowd losing some weight and seeing if it makes any significant difference.

 

[BrotherDave]

I'd try weighing all the TRs more or less the same. Once you get to Bronze you're pretty well vetted as a decent reviewer. If Paul McBeth can't get to bronze TR than he doesn't deserve to be called McBeast. :|

Weighing the <5 reviews people is probably going to make the most impact (though I'd keep the actual number a secret so drive bys don't just hammer out x number of crappy reviews to qualify).

 

[Three Putt]

I also ran the weighed number on the woodshed and black falls, but they are probably both outside the top 40. I ran scores for 44 courses
of the 44 i ran, 4.43 blackfalls was 40th and 4.30 woodshed was 44th.

I wasn't so much concerned where on the list they fall but what it does to the rating. I think the assumption was that if you minimize that one weird review of Black Falls it would bump up the rating, but in this case where you minimize both the weird review and the drive by reviews it actually drops the rating from 4.53 to 4.43. The Woodshed has a nominal bump from 4.28 to 4.30. I don't think that is the kind of change a lot of people were expecting.