Mel Choyce

Ratings

I was thinking about WordPress themes this morning, and how hard it is to find a good theme. There are tons of themes that look great, but once I install them, I either have no idea how to set up the theme to look like the demo, or I’m presented with so many customization options that I say “whelp, nevermind!” and go try to find another one.

In theory, ratings are supposed to help you find better themes, but they’re so open to interpretation that you really only end up getting unfocused opinions. I like this, therefore, five stars. I hate Gutenberg, therefore, zero stars. It’s not really helpful.

What if theme ratings were more granular? For example:

Aesthetics
☆☆☆☆☆

Setup
☆☆☆☆☆
◻ Someone set it up for me

Works as intended
☆☆☆☆☆
Did you encounter any bugs?:

Has the features I need
☆☆☆☆☆
Elaborate:

Helpful support
☆☆☆☆☆
◻ I haven’t contact support

Optional Comments:

Every person using a theme is, of course, biased — so in some ways the ratings would still be arbitrary — but by providing some semblance of categorization, we might at least help people think about the theme experience.

You might be saying (because I am also thinking) — “oh, but Mel, it’s already hard to get people to fill out ratings. By adding more questions, wouldn’t it make it even harder?”

Partially, yes. Someone’s gonna look at that form and be like “nah, pass.” But for some, the additional structure might make them more likely to review. I hate being presented with a single star rating field and a comment field because it feels so unstructured, I never know what to say. Since my name is always included, I feel like I need to have a smart response or else someone’s going to come along and be like, “wow, Mel’s an idiot.” Having a guided form like the above helps me at least rationalize how I feel about something, and by breaking it down into specifics, I feel like I can provide a more accurate rating.

Anyway, just random a Sunday morning afternoon thought.

11 responses to “Ratings”

  1. Torres 126 Avatar
    Torres 126

    I think that the main issue is that it’s really tricky to get themes to look like the demo, and users just get frustrated and assume it’s not possible so give it a low rating, yet I feel like only those who really love the theme will take the time to leave a higher review. It leaves a tricky balance, but honestly, I don’t like the idea of complicating leaving a review too much. Perhaps checkboxes would be nice: “Did you get the theme to look as it was presented?”, “Have you contacted support?” etc.

    Honestly, the more granular experience you mentioned probably would strike a good balance between not being overcomplicated but being more fair to theme developers.

    Reply
    1. mel Avatar
      mel

      Yeah, absolutely agree that it’s hard to get any theme to look like its demo. It’s a challenge we’ve been tackling for years on WordPress.com, and I feel like we’re only now just starting to break ground thanks to Gutenberg.

      Checkboxes are also a good idea!

      Reply
      1. Rich Tabor Avatar
        Rich Tabor

        By themselves, themes are inherently difficult to setup (as is WordPress in general, realistically). But with something like Merlin WP – it’s a cinch. I’ve had so many folks complimenting the wizard. Gutenberg will help to a degree, but without some sort of helper to guide folk the issue will continue to persist.

        Reply
        1. mel Avatar
          mel

          Totally — and I think there’s also a lot of potential to improve theme setup once Gutenberg templates are more standardized and fleshed out.

          Reply
  2. Birgit Pauli-Haack Avatar
    Birgit Pauli-Haack

    The more nuances review style resembles the restaurant rating where people are asked to judge the friendliness of the service, the quality of the foot and the order and payment experience. It will have various degrees of helpfulness, especially when something is really not working right. Then the Themes author knows where to invest more time, and the other theme searchers know to stay away from it. Similar to your experience, I walk away if a theme has too many options and allows me to give color to every single piece of content. I am definitely a fan of Color palettes, and if the home page doesn’t look like the demo, I am ok with it at first, but after 15 minutes,I should have a working homepage. I think the world of black and white, right or wrong, i would like to get back to a more nuanced discourse. Let’s start with WordPress themes:-)

    Reply
  3. Will Ellington Avatar
    Will Ellington

    I’ve been toying with this issue of review criteria myself for quite some time. I review WordPress themes and plugins on my blog. You’re right that any review is ultimately going to be subjective, as is the reason for choosing a theme, and then deciding whether it fits your project or not.

    I think your idea of a “granular” or more personable approach to theme reviews is on the right track. As others have mentioned, the problem is finding a balance between the specific and the general.

    All that being said, I still think that some review criteria are universal in appeal, including but not limited to the following:

    Security = Is the theme well coded? Does it have regular updates? Does it require many 3rd party plugins thus opening the way for security vulnerability?
    Installation & Setup = Does the theme work right out of the box? Or does it have a learning curve? This is tricky, because if you want specific features then you’re going to have to be willing to go the extra mile in terms of installation. But I still think that a one-click install is the way to go.
    Usability = Can you actually do what you need to do without applying for a Udemy course?
    Speed = Does the theme take a decade to load? Or is it lightweight and snappy?
    Functionality = Does the theme work across all major platforms from desktops to smartphones? So many themes I download look great on my 27″ iMac but when I turn to my iPhone they go into Frankenstein mode!

    Perhaps it would be helpful to draft a “white paper” for review criteria. Circulate it and get feedback.

    Reply
    1. mel Avatar
      mel

      All good ideas. If you write it, I’d love to read it!

      Reply
      1. Will Ellington Avatar
        Will Ellington

        Hi Mel, this is just to follow up on my previous comment. After giving the whole theme review things some more thought, I came up with a tentative list of criteria for what a useful WordPress theme review might consist of: http://wpliving.net/what-should-a-wordpress-theme-review-consist-of

        I know this doesn’t strictly apply to the WordPress.org theme repository review system, although I do mention it in the post, but hopefully some of the thinking might help towards improving that system too. I’d appreciate any feedback you might have. Thanks very much.

        Reply
        1. mel Avatar
          mel

          Thanks for sharing! I’ll give it a read ✹

          Reply
  4. Mark Root-Wiley Avatar
    Mark Root-Wiley

    Friction definitely has its place in UX, and I see longer forms as a great example. I suspect that the least useful reviews would be weeded out, and negative reviewers would be forced to break down their complaints. A 1 star review in one section of a review like you propose is wildly more useful than a 1 star oveall review.

    Reply
  5. What should a WordPress theme review consist of? – Wpliving

    I want to make more WordPress theme reviews on this blog, but I’m still not sold on what a WordPress theme review should consist of. In this post, I set out my list of review criteria, but I know it’s far from perfect. I would appreciate any feedback you might have as a fellow reviewer or as someone who might appreciate reading a review before buying/installing a WordPress theme.

    Post contents

    A. What makes a useful review?

    The Amazon model
    The WordPress.org model
    The “expert” review model
    What I learnt from these models

    B. My WordPress theme review criteria

    Review introduction
    Rubric 1: Installation
    Rubric 2: Usability
    Rubric 3: Speed
    Rubric 4: Support
    Review conclusion

    C. Final Word

    A. What makes a useful review?

    When it comes to the overall purpose of a review, I think there are two general questions that need addressing: (1) which criteria are most useful to prospective users of the theme? (2) Which criteria will allow the reviewer to be as transparent and impartial as possible? To answer the first question, let’s take a step back and look at how product reviews work in other contexts. We’ll return to the question of transparency at the end.

    The Amazon model

    The most widely read user reviews of any website I know are customer product reviews on Amazon. Take, for example, the reviews of Lisa Sabin-Wilson’s popular book, WordPress for Dummies. Each one consists of a star rating, a title and an open comment box. Readers can see whether the reviewer purchased the item and vote on how helpful the review content was in helping them with their purchase.

    An example of a user review on an Amazon product.The authority of this type of review is based on two factors: the first is whether the user has actually purchased and therefore used the product. This falls under the “usability” heading in my review criteria below.

    The second is whether prospective buyers found the reviewer’s opinions useful. In the case of a bias or contentious review, readers have the ability to leave comments and argue against the reviewer. This is a very nice “check and balance” system that keeps reviews relevant and (relatively) trustworthy.

    In the case of WordPress theme reviews on this blog, I am also going to introduce a “was this review helpful?” thumbs up/down system, and of course I will leave comments open.

    The WordPress.org model

    Let’s jump back to the WordPress context. In the WordPress.org theme repository, which is by far the largest collection of free WordPress currently available for download, there’s a similar type of rating system in use.

    However, unlike Amazon, the WordPress review system doesn’t have the ability to verify whether the reviewer has actually used the theme under review, or whether he/she is posting an arbitrary rating. This weakens the authority and credibility of the reviews.

    Current user oriented theme review system on WordPress.org/themes.As designer Mel Choyce recently pointed out, this lack of specificity in the WordPress theme review system makes the ratings arbitrary and is of little use to newcomers. Choyce suggests a more “granular” approach, with a wider range of review criteria that correspond to real world usage issues. I think that’s right and that’s the general direction I’m heading in with review on this blog too.

    The “expert” review model

    The last type of review I want to mention, are reviews written by “experts,” often for online newspapers and magazines. One of my favourites in this field is Digital Photography Review, a long-running site that reviews digital cameras.

    A typical review on DPreview.com.As you can see in the image above, expert reviews tend to be split into multiple sections and span multiple pages. In the DPReview model, they start with an overview of the camera, looking at its design and feature set, they move on to benchmark testing of image and video quality, and they end with samples.

    They give each camera a bronze, silver, gold or platinum rating based on a long list of criteria. They also include a list of “pros” and “cons” of the camera and a list of its potential uses.

    A typical summary section on DPReview.In this case, what gives the review its authority is on the one hand the reviewer’s expert knowledge, and on the other hand the level of detail from technical information to benchmark testing and usability.

    What I learnt from these models

    What works well with the Amazon model is the ability of the reviwer to give personal “real world” insight into using the product and the ability of the community to hold the reviews to account in terms of accuracy and usefulness.

    What works well with the expert review model is the level of technical detail, the benchmark testing and the ability to compare the product with similar items in the same field. However, I think the level of detail that we see in the DPR reviews would be overkill for a WordPress theme review. Digital cameras and WordPress themes have completely different degrees of technicality.

    So, I think we need a combination of both models. WordPress theme reviews should be written from a “hands on” perspective with as much practical information about how the theme actually functions (vs. the developer’s ideal view of how it functions) as possible.

    The review should cover the positive and negative aspects of the theme. The negative aspects should be backed up with concrete examples and there should be scope for readers to challenge the validity of the review through a rating and comment system. The review should also give the reader a clear idea of the sorts of applications the theme could be used for and it should reference similar reviews in the field.

    B. My WordPress theme review criteria

    So here are my suggestions of criteria for WordPress theme reviews. I have narrowed it down to 6 steps, including an introduction and conclusion, and 4 main rubrics: installation, usability, speed, and support. Each main rubric is given a rating from 1 (lowest) to 5 (highest). Let’s take a quick look at each part.

    Review introduction

    The opening section of each review will give the reader a quick impression of what the theme consists of and what it can do. It will describe the theme’s distinguishing features, its looks (I was debating on whether to include this as one of the main rubrics, but since aesthetics is such a subjective affair, I have left it in a secondary position) and the developer’s expectations for the theme.

    Rubric 1: Installation

    This rubric has two goals: first, to inform the reader on how easy or difficult the theme is to install. Is this a one-click job or does it require multiple steps? The second goal is to give the reader an idea of how easy or difficult it is to get the theme looking and working like the developer’s demo site. So many themes fail on this front. The theme looks brilliant when you try the demo, but looks terrible when you install it.

    1
    2
    3
    4
    5

    Difficult to install and/or setup without help from the developer.
    Installation and setup require extensive consultation of documentation.
    Installation and setup require a bit of tweaking, but nothing unusual.
    Installation and setup were easy, but still require some basic
    Installation and setup were a breeze.

    Rubric 2: Usability

    This rubric looks at the day-to-day operation of the theme. The main concern here is the efficiency of the admin side of the theme after installation and setup are complete. So for example, does the theme have too many configuration options that makes its use overwhelming? Or does its UX merge seamlessly with the WordPress architecture (post/page composition including Gutenberg, widgets, theme options, plugin operation).

    1
    2
    3
    4
    5

    The theme has a steep learning curve.
    The theme requires some consultation of the documentation and some trial and error to operate smoothly.
    The theme functions as expected. Yes, there is a bit of tweaking, but nothing unusual.
    The theme is easy to operate and does what it says on the tin.
    The theme is so easy to use, even my grandmother can use it.

    Rubric 3: Speed

    In this rubric, I will do some benchmark testing with two aims in mind. First to assess the speed of the theme in both desktop and mobile environments. I will use Pingdom and GTMetrix for the former and Google Mobile Speed Test for the latter. The second is to assess the overall code quality of the theme with its SEO potential in mind. I will use the W3C Markup Validation check and the theme check plugin to do this.

    1
    2
    3
    4
    5

    Very slow loading time and very poor coding quality.
    Slow loading time and poor coding quality.
    Average loading time and standard coding quality.
    Fast load times and no code issues.
    Super fast load times and no coding issues.

    Rubric 4: Support

    The final rubric in the review is theme support. This is one of the most important elements in choosing a WordPress theme, particularly a paid theme. While I will not be able to assess the long-term credibility of developer support, I will use available information based on reviews and comments from current theme users to assess the support situation. I will look at how quickly the developer responds to support issues, whether there is an adequate support system in place, and whether there is adequate theme documentation and resources.

    1
    2
    3
    4
    5

    No support or completely dysfunctional support system.
    Limited support leading to an unsatisfactory end user experience.
    Adequate support.
    Good support, fast and efficient.
    Excellent support and resources.

    Review conclusion

    The conclusion will do four things. It will give a list of the theme’s pros and cons. It will assign a rating for each of the 5 main rubrics in the review resulting in a minimum score of 1 and a maximum of 10. It will list the potential uses of the theme. Finally, it will provide links to similar themes in the field.

    C. Final Word

    So that’s where I’m at so far in establishing WordPress theme review criteria. As I said at the beginning of the post, this is still a work-in-progress and I am sure there are many things I have overlooked or not fully thought through yet.

    In that sense, I would be hugely grateful for any feedback you might have about any of this. So please feel free to leave critical comments and I’ll be happy to revise my criteria accordingly.

    The next step will be to apply these criteria to an actual review and see whether it works. Just as the development of a WordPress theme comes with updates and bug fixes, so too will these review criteria change as I streamline the review experience. Thanks so much for reading!

    Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)