Xplane 10 Blog - No More Reviews?

[Simmo W]

A very open statement from Chip and I. And you wondered why there aren't many reviews in the Xplane community..

http://xplane10.wordpress.com/2010/08/16/to-review-or-not-review/

[nrhh]

Very, very sad to see this happen. Well, I can understand that developers who put in much time and love into their work, dont want to see critical comments or that they think their work is not done justice in comparission to other developers work. But if you release products you have to face those circumstances - ever more so if you release payware.

That said I own a lot of free and payware aircraft. Bought either from x-aviation or the .org or from the developer direct. And I have always found that your reviews were fair and objective and where in line to my peronal experience as CUSTOMER and USER with those aircraft you reviewed and which I own. I think the work you did with your reviews is/was a great contribution to the x-plane community, even though some developers may disagree for personal or business reasons. Remember that no developers offer trial versions so your reviews are a opportunity to get a informed and IMHO objective oppinion on the various payware offerings.

So please, continue with your payware reviews.

Regards,

[tkyler]

Simon,

I for one definitely would like you to continue as I said before.  I've re-read my communications to you and still stand by what I said to you.  For the record to others though, I did not describe his work as destructive though I apologize if this is the way it was interpreted.  I think after dealing with criticisms, I can understand your frustrations Simon.  Your presentations are professional and promote x-plane is a very positive light.  Regarding my thoughts, you have them and I hope in the calm of the aftermath, you can reread it and perhaps see it for what it is.

[Oliver]

Please continue your great review guys! I loved reading everyone of them and I'd be sad to never read another one  :'(

[iamfallen]

My budget for payware is lacking, so I have greatly enjoyed your well written and thought out reviews. I think it would be a disservice to discontinue your reviews. If you are concerned about the backlash, perhaps you could offer a developer rebuttal section in your review to allow the developer a chance to share why they did things the way they did.

[Baber20]

Loved your reviews. Please keep them coming.

[Kesomir]

You need a tough skin for reviewing as no-one will write anything nice about you for doing so, everyone will chip in with their criticisms and disagreements about your review, but won't write their own reviews.

That said; reviews shouldn't be pre-approved - it's their independence which gives them credence. Some companies / individuals may supply aircraft free for reviewing purposes, but they can't dictate what you write about them.

You should keep reviewing because:

1. It helps people determine whether they would enjoy a product and reduces disappointing buys + excites others to make a good purchase.

2. It raises the profile of products and x-plane as a whole.

Sod the developers, here and there  - keep your chin up, you're doing a good job - would be better if it was on the web rather than pdf imho, but whatever.

People do find your reviews useful, they are just the silent majority (yup I pulled that stat out of my ...)

If others want to review products then they can get off their asses and do it themselves to offer their view on it.

[woweezowee]

Simon, the decision comes down to:

option a: doing it for the fun of it, in favour of the hobby. Then you avoid what spoils the fun. Concentrate on what really is fun.

option b: go a more professional route. define your standards and stand up to them. do what YOU want, even so if the fun gets spoiled now and then by people that complain.

Myself, I'm on option b, earning my money as editor-in-chief of an online magazine (not flight sim related). I certainly understand your decision, although I have to say that there could have been the potential to make the whole thing something bigger, more professional. It was a good start, but there are decisions to be made. All I would ask for is: Don't let those decisions be dicated by others.

Example: Not accepting "free review copies" like you made clear in your mission statement. There's no reason to discuss that at length. You either deal with professionals "on the other end" also, and they have to deal with the outcomings of your reviews .... or they try to belittle and influence you. At least, that way you'll know who to avoid. Yes, YOU avoid them completely instead of trying to find a compromise, wich in the end would be a dictated desicion. Or you bash them in another article. But as we all want to see xplane flourish, that's not the smartest of ideas

[Goran_M]

Don't let those decisions be dicated by others.

Wise words.

[XPAfrica]

Keep on reviewing - before xplane10's blog, selling a payware item for lpx-ane would rely on a couple of screenshots, a description in a forum and maybe a couple of enthusiastic beta-tester comments. In the past, i've bought payware content that i should never have bought, just because of that.

I understand the sadness, i've read most of the comments at the .org about the reviews and frankly it's so low that it doesn't even deserve paying attention to.

As for telling someone not to talk about his products, that is censorship in my opinion. Plus, i've never had the feeling that any product you have reviewed made the oher one look bad, be it free or pay, quite the contrary actually.

As i said to chip in private earlier, you guys are just the beginning: in the future and with the expansion of the community, the x-plane developpers will have to face reviews and critics even by paper magazines, like it is today in the FS world, and the impact of those, as well as the reviews that will be posted on big FS sites likely to turn to X-Plane, will be far superior to the influence xplane10 may have on sales or image. Then they'll be facing the "usual" practices from any other such business (in my experience, the music press): "You buy a quarter page publicity in my magazine, i write a review, you buy half a page, i write a better review, you buy the centerfold, you'll get the best review and a fat title on the frontpage." ...

For the rest i'm completely in line with Kesomir's recommandations, especially the pdf thingie - it would be great to have the review on the site and be able to dowload it in pdf.

keep 'em coming!

[Lukasz]

Guys, don't give up because of some haters! You're in the spearhead of X-Plane publicity and it's widely known, that's the position with the highest chances to get a medal or a bullet Or both. Try paying attention to people who like your work - I'm positive there are many of them - and don't bother with the rest. The life is too short for doing unnecessary stuff. Be like you were before - enthusiastic, polite, honest and unbiased, with authentic praise for high quality hard work and a dose of healthy constructive criticism (even if not said straight, the smart folks will read between the lines) for anything less than that.

It's my general policy not to speak publicly direct criticism against any developer, as I'm aware that making even simple and medium to low quality aircraft is a lot of work and time involved, but I'll make an exception this time, for a higher good. I'm familiar with Mr Chandler's website and his planes and helicopters, I've been there even before the time I've discovered X-Aviation. Also I'm quite interested in some of his works - but judging by specific aircrafts types only. Every time I've come to the website and wanted to make a purchase, I've been asking myself a question, while looking at screenshots, "Will I have a good time sitting for an hour or two behind this generic looking panel, in this dark and lifeless cockpit?". The answer was always "No, I won't". Not because of lack of eyecandy, but mostly because lack of immersion and believability, so inherent to for example Sundowner and Falco or freeware (!) MD500 and An-2. I believe the key to raising Mr Chandler's sales, is to update his aircraft to be on par with "others", instead of lowering prices and crying over "unfair reviewers". Actually, you were quite "patient" and polite while reviewing the Bonanza.

I really prefer to praise other people's work and to keep any ctiticism to private conversations or even to myself only, but this time I had enough. If we can't have truly free media even in a world of enthusiasts of a "niche computer game", or other hobby for that matter, then this world is really going to an end.

Don't give up guys, I've complimented you once and I'll stand by it! You're doing good job and a favor to all the community.

We have wasted enough time, what aircraft the next review will be about? MD 500? Corvalis? Or maybe the Duchess?

[Testpilot]

Please keep it up! Your articles are seem to some of us as an oasis of unbiased, peacemaking, honest and fun loving information that is actually relevant and useful. May I suggest that the only reason they have generated some hate is because others are recognizing their power to shape users and reach beyond "domains".

[Cameron]

Alright, so I've been called out in this little letter here. Guess it's my turn, huh?

In your letter you state that Tom and I "have told us our reviews are destructive because they are not critical enough, and that we allow mediocre developers to pawn-off their wares on a gullible public, and that has to be accounted for."

After reading that, I think it takes a little while to interpret what's being said here. I believe what you mean is that is your general consensus of what was said, and surely NOT that we told you that your reviews were destructive.

What may surprise people here is what WAS really said by me, mainly because people are used to the censorship mentality from developers trying to protect their product. So, I'm going to use the good ole' number system and just state my points in a simple, yet (hopefully) effective manner and get out of here. So, here goes:

1. I believe you should be able to say whatever you want to say about a product, especially if you buy it. Regardless of such, I believe you should be able to say whatever you want to say even if you DON'T buy it! This is where I feel you guys fall short, simply because you are too worried about burning a bridge. We (at least at X-Aviation), have consistently told you to be as critical as you feel. X-Aviation takes this risk day in and day out with magazine reviews, online reviews (like Avsim.com), and even our own store review system. We have a live online store review system for people to share their thoughts and help other people make purchasing decisions.

2. I feel that the MSFS crowd needs to see what X-Plane can do from an honest point of view. You will earn respect and credibility from the masses when you're honest in your positive and negative views of a product. Remember...you're 'customers' are the users of X-Plane, NOT developers, so stop worrying about what we think when it comes to saying something good or bad about a product! It's like someone else said here: free media.

3. If your review is negative, chances are you aren't the only one that thinks so. You'll have a backing. If a developer truly cares about their customers and the product they are selling they will try and fix the product to meet the masses wants. This is similar to how we get bug fixes done. They are like mini-reviews from people that inform us what they feel is wrong with the product and where we need to improve. How do you think people would feel if we left everything unattended to?

These three simple points are all I really have to say right now. You guys can do what you want with it. In my eyes, at this time, the only people being destructive are you guys to your own blog by not pushing forward amidst the raucous. Have faith...it works...in some form or another.

[AADX]

.

[Simmo W]

Thanks everyone for your messages, I'll make sure Chip sees them all since he's the one most personally affected by this, me a close second. rest assured I plan to keep it going, at least until xplane11!!

[Nova]

I always found your reviews a bit too good. But maybe my expectations towards a product are too high.

Never the less, it was a good way to get some sort of "big picture" what was going on/market situation.

I for myself have been, and still am sometimes, a reviewer of Mac games for a german gaming community.

Never had to deal with such a harsh criticism, although some people didn't always agree with my opinion.

So hopefully you continue.

I think, some are too scared to be compared to each other, since they know, that their product is not good enough.

Never the less, they want to make money out of it, and bad reviews don't help.

We never had to show the game publishers our reviews, although it's a common thing on the PC market,

where reviews are held back, until the publisher likes the outcome. I really hope, your not going this way!

[ksgy]

...and who cares about the developers? If they do poor quality, they have to face it... And by the way, what about the "freedom  of  speech "? Xp10 is a BLOG, isn't it?

[Cameron]

...and who cares about the developers? If they do poor quality, they have to face it... And by the way, what about the "freedom  of  speech "? Xp10 is a BLOG, isn't it?

+1

[Beej]

Simon,

Keep the reviews coming along.  Reviews are by definition opinion pieces and the most useful ones are those that actually do take a position on a product/service then justify that position.  I'm definitely not Org bashing here but I find the completely non-critical approach to reviewing there a bit of a nonsense.  People really need to get independent information about what's good, and what's bad (and why...). I think the most important thing in any review process is consistency in application of standards and making certain that the "ratings" match the commentary.

I certainly wouldn't worry about getting critical comment - it goes with the territory.  Do you think that restaurant critics (think John Lethlean in The Australian - an example you may know) don't receive vigorous "feedback" from the people that they eviscerate? I think it's important to listen to the feedback (if it appears), evaluate it, decide whether you have been true to your own values then move on to the next review.

As far as certain developers insisting you don't review their product.  Well, if you've purchased it and want to put an opinion on a blog, that's free speech!  More power to you, and keep reviewing that same developer as long as you feel you can maintain "objectivity".  A final word on objectivity to bring us back full circle - reviewing is subjective - we all have pet hates (ie some won't rate an aircraft at all if it doesnt have a 3D cockpit..) just as long as "pet hates" are out there for all to see - it's all fair, especially if people have the opportunity to publicly respond - a possibility, I might mention, denied to most movie , restaurant etc reviews.

Keep at it and have at it, full steam ahead!  Be fearless, fair, and fun!

Beej

[YYZatcboy]

Please keep the reviews coming. Smart developers will look at it and say, "well Gee, I never thought of that" and fix it, or explain why they chose to do what they did and defend it. . Stupid developers will get angry and insult the review and the authors with ad hominem attacks.

Ignore those people. We need you to keep everyone honest.

You have my full support.

Best

Joe

[steven winslow]

You have my vote for complete, fair, blunt, honest, possibly brutal reviews.  Just remember, live by the sword....die by the sword.  Or keyboard in this case.

[Paraffin]

My $.02... I've enjoyed reading the reviews and think you should continue. I'm especially interested in the new stuff on scenery creation because I might take a small stab at that soon, but the payware plane reviews are also very helpful. That said, I would suggest a few things about your review process:

First, I don't think it's a good idea to get pre-approval from the developer. It's a trade-off. On the positive side, there is always a risk that you'll misunderstand something about the model, or miss an important feature that the developer could help correct for the review. On the downside, it inevitably puts you in the position of thinking about what the developer will think of your review, before you even make it public to anyone else. That puts the developer too much "in bed" with the reviewer, in my opinion. The best way to handle any errors or omissions in the review is with an addendum, where you can tag on any updates.

Another thing... I feel the most trust in reviews by people who buy things anonymously with their own money, and then write about them. It's the "Consumer Reports" model, basically. You have maximum distance from the source, and you're on the same level I am, when I buy a payware plane.

You may get some heat from being honest and critical, but that goes with the territory. Just don't say anything negative about the aliasing on Austin's latest work on v10, or you'll REALLY be in hot water. (cough, cough)

As a last comment... one of the first complaints in the .org thread was about linking directly to your blog's home page instead of to the review. That wasn't framed well and it was unnecessarily nasty, but he had a point. It takes time to hunt through a blog if something isn't prominent on the front page. If I can't find something quickly that's been linked to, I'll bail out. Also some people may find a message like this months or even years later, so that's another reason for a direct link. A link to your blog's home page is difficult to distinguish from blog spam, and some people are very sensitive about that.

Anyway, keep up the good work.

[XPAfrica]

I'm very happy to see that there seems to be a general consensus coming up on this subject, also very happy to see how everybody here is keeping things cool and not getting into useless passing of some kind of personal frustration towards the one or the otherperson/developper/forum/whatever. Very happy also, to see Cameron and Tom issue clear,  positive and open minded points of view about the subject (not that i have doubted it but it's always nice to see that happen.).

In short, i'm very happy things turn out for the best and to read that Simon and Chip will keep on going  - your work is useful and those who don't see it today will definitely end up seeing it sometime soon.

cheers

[MdMax]

Thanks everyone for your messages, I'll make sure Chip sees them all since he's the one most personally affected by this, me a close second. rest assured I plan to keep it going, at least until xplane11!!

Great news !  ;D Thank you Simon & Chip for your reviews, and I really hope add-on developers will see your blog, and every X-Plane site and review as a chance for their work to come into the light. They all work hard to improve our X-Plane experience. You work hard to show this. So it's great if you keep it going !

8)

[Pete_SMS]

...and who cares about the developers? If they do poor quality, they have to face it... And by the way, what about the "freedom  of  speech "? Xp10 is a BLOG, isn't it?

Couldn't agree more! If developers are producing poor products, the customers should know about it. Why should poor products or products with significant flaws get good reviews? Reviews are all over the internet and I use them often, before I buy something. Having reviews is just raising quality in the end and that is what we all want I guess. A review has to be fair, but also be honest.

So please keep on making these reviews, and its great to read that you do, at least to XP11.