User:Phyzome/Proposals/Integrated rating system

Please confine any content edits to the comments section below, though typo-fixing and wikification are welcome anywhere. Discussions about this page itself should go on the talk page. Thanks!

Moderation in MediaWiki

Overview

A system of moderation to flag bad edits for review and bad users for warning or banning. This system does not apply to anons. Ideally, this would not be used as a criterion for adminship or de-adminship, but simply as a method for sifting through good and bad edits more rapidly.

Concept

Each user has several scores: accuracy, honesty, calmness, impulsiveness, neutrality, humor. These comprise the 6 dimensions of editing.

Each edit is scored on the same dimensions.

A user may review an edit and rate it along each dimension. The rating affects the edit's scores, as well as the scores of the user responsible for the edit.

Each user has an overall weight score that determines how heavily they may affect other users' scores. This is determined by their own scores and history.

Users with high neutrality, calmness, honesty, and/or accuracy and/or low impulsiveness have more overall weight to their ratings of edits. Additionally, users with high neutrality have more weight in the neutrality dimension of their rating of another, etc.

The humor dimension's uses are left unspecified.

Traits of the dimensions:

• Neutrality: Ability to see multiple perspectives, ability to mediate.
• Accuracy: Knowledge in chosen field(s) of editing, resourcefulness in research, ability to find and distinguish reputable sources.
• Calmness: Ability to remain unruffled even when personally attacked or not getting one's way.
• Humor: 'nuff said.
• Honesty: Truthfulness in both general and serious metters. Sticks to promises.
• Impulsiveness: Low threshold for action. Not necessarily a bad thing. Too little can lead to ignoring problems, too much can create new problems.

Implementation

Events

When a user reviews any edit, they can choose to assign a rating in each dimension (or abstain from any or all). This will affect the scores of the editor to some degree.

New users start off with each score at zero.

When an edit is made, its score is initialized with the editor's score. The firmness, however, is set low, but proportional to the editor's firmness.

Anon users' edits have a lower accuracy to begin with, to help flag them.

Serious impersonation of another user, specific vandalism of comments of others, or other injuries to the basic mechanism of MediaWiki causes an admin-drop in honesty and neutrality, as well as in weight.

Mechanics

Each dimension is represented by a level (the actual index) and a firmness (the weight-adjusted number of ratings). When a new rating comes in, each dimension is adjusted by an amount proportional to the rater's weight and inversely proportional to the firmness. This means that untrusted or new users have less effect on ratings, and thoroughly-rated users are less affected by individual ratings.

Variables: current rating is R, current firmness is S, new rating is A, weight of rater is W (0 to 1).

An unweighted average would be computed like this:

(R*N+A)/(N+1) -> T (new R)
(N+1) -> new N

To weight it, apply W. T-R gives us the change, but we only want part of it. W(T-R) gives us the weighted change. R+W(T-R) is our weighted new R. N+W is our weighted new N.

This formula may be simplified:

(R*N+W*A)/(N+W) -> R
N+W -> N

The rating system applies to both users and their edits.

Abuse prevention and moderation: Negative feedback loop

Admins retain their admin flag and special abilities, as do sysops and all other stations and ranks. Admins naturally achieve good scores very quickly, as they have already shown their good qualities (a social prerequisite for adminhood). Admins who are rated poorly by others of good standing may have their admin flag called into question.

Users who repeatedly give ratings contradicting a vast majority may have their ratings flagged for admin-review. Admins may examine the record and make a decision to decrease the weight of that user.

Interface/Visualisation

The RecentChangesNeedingAttention list presents changes in a combination of reverse chronological order (oldest at the top) and how-often reviewed order (least reviewed at the top).

Considerations

Vandalbots should be caught quickly by racking up spectacularly poor scores in a short period of time. Perhaps an auto-banning system could be set up to temporarily ban users exhibiting this sort of behavior?

Stealth-trolls (e.g. GNAA members) who make good edits interspersed with vandalism should be caught by this system. Specifically, their vandalisms will attract ratings-attention and give them low weight and general untrustedness.

Both wikignomes and wikifaeries whould be promoted by this system by acquiring firm scores.

Comments

yak, yak, yak...