Gear Reviews request

[sAFETY]

Just a suggestion, but I looked over a tonne of recent NSMB reviews and ALL but one of them were rated 4+ out of five overall (the Norco 4x got a 3.5).

I think it would be a lot more helpful if the reviews were a little more critical, or the rating system were somewhat more specific. This would make the review much more valuable for the riders.

I've got experience with a few of the reviewed products and I'd have to say that their quality and value can differ a lot more than a half a rating point.

Just a suggestion...

[corey@nsmb.com]

You raise a good point, and I think we discussed this before a year or so ago.

A review is based on the individual's experience with the product. We are as objective as objective gets when it comes time to grade them. I think a misconception is that there is something "in it for us" by giving good reviews, when the reality is, there isn't.

Manufacturers are firm in that they want honesty. They know the product you are reviewing better than you do, and will know if you are blowing smoke up their ass by praising something where it really needs improvement.

The reviews we do are often used as feedback for design refinements and changes, they are worthless to a manufacturer if we just give everything a 4 or 5 when it should really be a 2 or 3.

There will always be times where a consumer like yourself may read a review and buy the product based on that feedback, but may find their experience different from ours. Everyone has different expectations, different riding styles, different bodies with different proportions, etc.


Not sure if that helps. Feel free to ask more questions! :)

[corey@nsmb.com]

Question for you: What sorts of changes would you like to see on our rating system? :)

[stuart@nsmb.com]

I'll answer this a little more succintly than Corey, if I can.

Roger Ebert recently wrote the following:

"The star rating system is the bane of my existence. Here's a message from Jason Eaken of Warrensburg, Mo., who points out that I gave negative reviews to "The Usual Suspects," "Fight Club" and "The Ladykillers," but positive reviews to "The Fast and the Furious," "Ghosts of Mars," "Taking Lives" and "High Crimes."

"True, I did. But in the reviews I tried to explain how I felt; just counting stars makes me seem simpleminded. I have always awarded stars in a relative, not an absolute way, based to some degree on a movie's success in doing what it wants to do, and what the audience expects from it. Of course, if you think stars are limited, you ought to try working with thumbs."

In other words, put value in the words and use the stars as a guideline.

Thanks for the input, and keep it coming.

Stuart Kernaghan
Associate Editor
nsmb.com

[corey@nsmb.com]

Thank's Stu.

I was a tad long winded... *keyboard is still smouldering*

[corey@nsmb.com]

Safety, here's an example of a review that didn't do so hot in regards to the rating system. :)

http://www.nsmb.com/gear/yeti_asx_review_09_03.php

[sleeper72]

Maybe it's not feasible, but I'd personally like to see more comparison reviews or mutiple item "shootouts". From other mags and other industries, they seem to work best when there are as many reviewers (perspectives) as products. For example, if NSMB collected some Roach leg armour, Beast, 661, and Dianese and had four people try each of them out and give relative impressions, that would be pretty good info for readers. When people have a selection of five manufacturers, and they are all pretty good, they're just looking for some relative information to tip the scales one way or another.

Obviously, that takes considerable planning and resources so maybe it's just not practical. But for something like grips it wouldn't be too hard to organize, especially if they were all lock-on types.

Still, all the NSMB reviews are still WAY better than the user reviews in the "Product Reviews" section of the website. Those are practically useless. I prefer the opinions of people with some demostrated qualifications, rather than random "dude, it sucks" comments.

[corey@nsmb.com]

Good ideas Sleeper.

I enjoy the "shoot outs" in some of the magazines. Grips would be cool to do.

[Nelson]

Speaking of star ratings, can we resurrect the littles bikes each user used to have? Those were a real self esteem booster....









:lol:

[sAFETY]

Yes, the shootout idea is a very good one. If someone was looking for a product, leg armour for example, and they read the recent Roach review they may be convinced that the Rally pads are close to 5 star perfection. In reality/IMHO (I've had them and other modern comparable ones) they are middle of the road compared to other contemporary designs. A shootout style review would help a user decide what to buy over what, or give them a perspective on the gear if they have experience with the other.

I've got ideas for other systems that involve 3d mapping and multiple lines of ratings (what can I say, I'm a geek) but they'd be just as difficult to publish and understand as they would be informative.

An easier way to improve the reviews would be to have more than one person review them in each article. That would give an alternative view, have a better chance of matching a readers size/skill/style, and would allow the writers to play off each other.

One person might LOVE the product, so the other person would be able to point out any shortcomings.

[DangerousDean]

maybe make the rating system out of 10 too

[corey@nsmb.com]

Keep in mind, (this is what Stu was saying) is that the end rating is only a rough guide. It isn't the tell-all portion of the review, and doesn't really evaluate the product to the degree the actual article itself does.

I'd rather not get too hung on the physical rating of the product, instead I think everyone benefits more from the time put into the article and the text itself. That's where the real evaluation lies.

It's easy to slap a rating on something and move on to the next piece, but we put more time into a detailed review. The Pro's / Con's serves to highlight the main points. Looking at the whole package gives the most accurate answer in regards to the product.

[brentomatic]

I'd like to see the Denny's Grand Slam reviewed. Who's up for the task?

Jerry-Rig? :fu:

[corey@nsmb.com]

Sign me up !

When Kits New West was open (I was in Grade 11 and 12 when we used to go, every Wednesday for cheap high-balls...) we'd always hit Denny's at North Road and Austin for fuel. The Grand Slam was amazing after a handful of bevy's!

[sanrensho]

All user reviews need to be viewed with a grain of salt, especially for products that are priced at the higher end of their product segment.

There's a very human tendency to overrate and perceive the superiority of a product that Joe Blow paid big bucks for, versus a product that costs less and performs the same.

The other tendency is for users to confirm the negative (or positive) comments in previous reviews, leading to a wave of negative reviews.

However, I guess there's nothing you can do about the fact that we consumers are deluded sheep.;)

[sAFETY]

what year was that?

I wasted a lot of time there. I mean a lot of time wasted.

[corey@nsmb.com]

Originally posted by sanrensho
All user reviews need to be viewed with a grain of salt, especially for products that are priced at the higher end of their product segment.

There's a very human tendency to overrate and perceive the superiority of a product that Joe Blow paid big bucks for, versus a product that costs less and performs the same.

The other tendency is for users to confirm the negative (or positive) comments in previous reviews, leading to a wave of negative reviews.

However, I guess there's nothing you can do about the fact that we consumers are deluded sheep.;)


Good point.

With our reviews, the gear isn't ours, so we are not tied to is economically in any way. We don't have an investment to justify, unlike some of the feedback you may see elsewhere.

[stuart@nsmb.com]

There are a couple of other things to bear in mind when you look at these reviews...

One of those is the length. This particular review was over 1,500 words long.

To put that in context, most major features in bike mags are that length. Product reviews in the same mags are 300 to 500 words long - or less. So you're getting a review that goes into a lot more detail that you'll find in most other places.

And that give us a chance to talk about the finer points of a product - which is why the stars at the end kind of lose their real significance. A star won't tell you that I put the brake lever through the pad, but the article itself does.

The other thing is that these products are being tested on the trails - and in the conditions - our riders are familiar with. A review of Product X based on tests conducted in SoCal is going to read quite differently than for something tested on Seymour and Fromme.

Which review is going to be more useful to you when you're shopping for gear to use on the Shore?

All of us reviewing products do our best to write down the things we think are important. At the end of the day, this is just one person's perspective on a product. Mind you, it's usually one person who has tested their fair share of products...

Keep the comments coming.

Stuart

[sanrensho]

Originally posted by corey@nsmb.com
Good point.

With our reviews, the gear isn't ours, so we are not tied to is economically in any way. We don't have an investment to justify, unlike some of the feedback you may see elsewhere.

I definitely trust the NSMB reviews and some magazine reviews (particularly "blind" tests) more than user reviews.

If anything, user reviews are more useful for gaining feedback about reliability and problems with long term use.

This is my long-winded way of saying that I enjoy the NSMB product reviews, as well as the shorter Gear Shots, which there could be more of.

[brentomatic]

Originally posted by corey@nsmb.com
Sign me up !

When Kits New West was open (I was in Grade 11 and 12 when we used to go, every Wednesday for cheap high-balls...) we'd always hit Denny's at North Road and Austin for fuel. The Grand Slam was amazing after a handful of bevy's!

Corey, if you review the Grand Slam, I'll buy it for you! I may even let you take one home.

I'm a giving human and I like to give.

[sanrensho]

Originally posted by brentomatic
Corey, if you review the Grand Slam, I'll buy it for you! I may even let you take one home.


This should definitely be a "He said, she said" type review. Diggler can be the counterpoint reviewer.:)

[corey@nsmb.com]

Originally posted by sanrensho
If anything, user reviews are more useful for gaining feedback about reliability and problems with long term use.


Long term is key for durability tests.

You will find in the "user reviews" that the typical review comes in the first few weeks, when the new owner is still in the "ecitement" phase. Few of those reviews some from people who have logged 2 months or more on the product.

Not many magazines are able to do that. We typically spend this kind of time on products prior to reviewing them/

[sleeper72]

hmm, thinking about it a bit more, I guess the things that makes me think "that's a well written review" would be:
a) A gauge of how close the reviewer's tastes/ability/dimensions are to my own. Since completely objective and clinical reviews can be incredibly boring to read, I don't mind if there's some opinion and personal preference thrown in, but I want to have some concept of how their ideas of what's important compare to mine. It's good when the reviewer mentions what they used to or normally use, since it gives a reference point. i.e. Saying a tire has "great" grip is useless if you don't have a reference point.

Like movie reviews, you'll always be able to find someone who loves it and someone who hates it, it's just a matter of finding someone who has preferences/needs like your own and hearing what they have to say.

b)A reasonable concept of engineering trade-offs. This is what seems to be lacking most in the User Review section, but NSMB does a pretty good job of it. Every product is a trade-off in some way, and a good review will talk about any compromises that have been made and realize you can't have everything.

Reading back on what I just typed, I guess it's pretty obvious stuff. But I'm going to post it anyway!

[sanrensho]

Originally posted by corey@nsmb.com
We typically spend this kind of time on products prior to reviewing them/

That's good to know, and is something that NSMB should strongly emphasize in its reviews.

[corey@nsmb.com]

Originally posted by sanrensho
That's good to know, and is something that NSMB should strongly emphasize in its reviews.

We typically do.

Stu's Roach review indicates the test duration, and the stuff I've done indicates the durations too.

It's important. Something can work great for a week....it takes time to wear it down to see how it survives.

[bokinator]

read the review on the 4x. Don't mean to be a dick, but I thought that was just a bad review. The bike was tested in a cross country environment and there was only one sentence to describe how the bike felt down aline.

[switch]

Originally posted by bokinator
read the review on the 4x. Don't mean to be a dick, but I thought that was just a bad review. The bike was tested in a cross country environment and there was only one sentence to describe how the bike felt down aline.

That's an issue that makes some reviews hard to gather needed information. If an MTB magazine does a shootout of few 6" dualies, and they ride the sands of Utah with 2.1" tires, the feedback is going to be different than if they ride Fromme.

If Corey reviews a bike, and talks about how the bike performed on Cypress, that is meaningful to me.

One slight peeve I have is with regards to pros/cons. If I see a $3K bike reviewed, and a con is a creaky seatpost, then I'm not getting what I want to know. I don't care about the stock $20 seatpost; it's going to be replaced anyways.

[apeshape]

Reading mulitple reviews by multiple reviewers of the same product would help me find commonalities. The RMX review was a good example of an initial impression revisited after time spent with the product. Durability is one of the more important characteristics I look for and it would be nice to know a product stands the test of time.

I also find that many products are reviewed by smaller riders who may not be as hard on gear as I am. Being over 6 feet and weighing about a deuce I'd like to see a club tall and tubby review in there somewhere.

[Trini-dad72]

i have a dumb question...

When will the 04 Devinci spots be up? I have had my bike for a while now, and I'd like to post a review of it. I may not be the best reviewer, but I'd like to post anyway.

Thanks Folks.

[FatHack]

ok, a little commentary here.

For the most part I find the homepage reviews to be pretty spot on with my impressions of products. Maybe a little less technical than I would like at times, but you can't really expect too much when the majority of ppl just want to know how the bike rides.

My main complaints about ppl's gear opinions come from here on the board, now I know this is a little off topic, but here goes any ways. It drives me nuts to see someone ask a legitimate question about a product, thinking that they will get an informed answer from ppl that have actually tried it, worked on it or even heaven forbid owned it. So much of the stigma that a product gets around here is hear say from one or two bad experiences that some hack buddy who knows someone, that knows someone with a computer. If you have actually owned something and have an opinion on it, great, shout it from the roof tops. If not, stop slinging crap about it. All too often good products get bad raps, or the other way around.

Also, has anyone noticed that there has be a significant decrease in the number of industry folks on the board as of late?

ok, that was my 2 cents.

[sAFETY]

I think the lack of industry folks has a lot to do with the time of year. They're all working 25 hour days right not.

As for the reviews, I think it's gotten better as far as badmouthing goes. If anything the converstations have just gotten more useless lately. Things Like:

Q: Has anyone tried product A, or gotten it to work with product B. If so, any insight into it's long-term durability?

A1: Product A is :thepimp:

A2: I have a buddy that has product B lol

A3: Product C is Awesome I wanna get one when my Product D comes back from warranty :mad: :D

[Joe Dick]

Originally posted by bokinator
read the review on the 4x. Don't mean to be a dick, but I thought that was just a bad review. The bike was tested in a cross country environment and there was only one sentence to describe how the bike felt down aline.

I agree the bike was tested out of it's enviroment, which was some what adressed in the review itself. wrong horse for the course but maybe a good review in an over all riding sence.
have you ridden in S.A.? sure it's not Fromm, but it's not true XC, it's B.C. XC. it has a little bit of every thing. I can't compare it to where you ride cause I don't know it. Switch makes a good point as well. if I where to read a review of some one riding in your enviroment and use that as the bacis for buying a bike for my self it's not going to work unless I read between the lines. reading is the key here.

NSMB does a good job as far as an honest review goes, but it's all a mater of prespective.

some thing I don't get about bike reviews, why spend so much time reviewing the components of an off the shelf bike set up. If I want to know about brakes, I'll read a brake review. if the tires it came with are not to your liking, mention you changed them, but I realy don't care about the tires them selfs. every one dials a bike in to fit them personal. when I read a bike review I'm more conserned with the handling and workings of the the frame and suspention set up. brakes, tires saddles, bars, drive train will be all in the indivduals personal set up. or maybe thats just my prespective being that I like to dial my bikes in and seldom does it remain bone stock for long.

just my 2c.

[corey@nsmb.com]

Originally posted by Joe Dick

some thing I don't get about bike reviews, why spend so much time reviewing the components of an off the shelf bike set up. If I want to know about brakes, I'll read a brake review. if the tires it came with are not to your liking, mention you changed them, but I realy don't care about the tires them selfs. every one dials a bike in to fit them personal. when I read a bike review I'm more conserned with the handling and workings of the the frame and suspention set up. brakes, tires saddles, bars, drive train will be all in the indivduals personal set up. or maybe thats just my prespective being that I like to dial my bikes in and seldom does it remain bone stock for long.

just my 2c.


We cover these items because when you are going to spend big bucks on a new steed, you should know what's stock equipment. When you are looking at a couple different bikes, the stock spec may be a deciding factor in which bike you get, and which bike is a better value for the dollar.

It makes sense to discuss how these components work, because the components are what complete the package and make the frame "work".

There are bikes out there that shine because of the stock build. Until recently, the tires that came as oem on bikes, for the most part, were not the greatest. Things are changing, and when a bike comes with stellar rubber, it deserves to be noted.

All that being said, we don't focus on those parts anyway. We mention that which is worth mentioning (as they somehow played a special role in the test), but we focus on the bike itself.

You are unique in a way that you switch things around when you buy new. Not everyone does, and the manu's are making the packages better and better so there is less need to do so. It also costs more to do so. I'm a gear nut, if I reviewed bikes based on how I build them, I'd be reviewing a different product than you'd be able to purchase, which doesn't make a lot of sense when you are writing for the masses.

:)

[Joe Dick]

I got ya, and I understand. I don't think I'm that unique in the fact that I change things around on a new bike. not average or more educated would be better wording maybe. I think these days a lot of buyers are more educated about the bikes they are buying, especialy if they are buying a bike in the price range I've seen reviewed on this web site. if a bike comes with (enter crap part here) it's not going to change my opinion of the over all bike, I'll just expect to have to change it as part of buying the bike. If I want a 5-6 inch travel closed front triange frame to use in freeride/ large milage back country appications with a $3000 price tag celling then I've got a small hand full of bikes to chose from. the biggest desiding factors will come down to function at the of the heart of the beast, avallibility and a shop willing to deal with me on my terms. things I know are going to be crap and will be lucky to find on the bike I'm going to buy are wheels, tires, bearrings, often brakes will be well below par and drive train (8 speed on an off the floor bike in the $3000 price range, yeah right.)

maybe it's just that I know what I want, most likely I'm just rambling.

:cool:

[axisofevelknievel]

I for one want to hear about the quality of the stock components on a bike. Any given person will make some changes when they buy a new bike, but I want to know whether I'm changing a single $20 item or if I would need to replace several higher cost items. Basically, the more the reviewer changes the bike, the less useful the review is.

[peachy-B]

totally!

Originally posted by axisofevelknievel
I for one want to hear about the quality of the stock components on a bike. Any given person will make some changes when they buy a new bike, but I want to know whether I'm changing a single $20 item or if I would need to replace several higher cost items. Basically, the more the reviewer changes the bike, the less useful the review is.