Instructions for document image quality evaluation

Introduction

Your task is to evaluate the subjective visual quality of Arabic text in binary document image. Each HIT can be completed in around 60 seconds.

Payment

We have methods that analyze the consistency of your answers with respect to themselves, to those of your fellow workers and to references we know to be accurate. We will use these methods to rank the submitted assignments according to quality.

For this experiment we will pay a base reward of $0.02/HIT for every accepted HIT. We have made available a set of 15 different HITs. You will receive a bonus of:

· $0.03/HIT (for a total of $0.05/HIT) if you submit all 10 HITs or

· $0.08/HIT (for a total of $0.10/HIT) if you submit all 10 HITs and your results are among the top 50% or

· $0.18/HIT (for a total of $0.20/HIT) if you submit all 10 HITs and your results are among the top 10%.

Bonuses will be paid up to 7 days after submission, because we can only rank the submissions once we have a statistically significant number of answers. The base reward will always be paid within 24 hours of submission.

Instructions

Each file should be given a score according to the following scale, known as the MOS (mean opinion score) scale:

Score	Quality	Impairment
5	Excellent	The text is clear and easily readable. The strokes are sharp. White connected components are not filled. No touching strokes.
4	Good	Just perceptible smudged stroke, but not annoying.
3	Fair	Perceptible smudged stroke and slightly annoying.
2	Poor	Perceptible smudged stroke and very annoying.
1	Bad	The smudged stroke is impossible to be recognized.

The following references illustrate the meaning of each score. Please note that you may encounter other types of distortions. Therefore, these examples do not exhaust the range of conditions you can expect to see.

Examples

Reference score: 4.0

Reference score: 3.5

Reference score: 3.0

Reference score: 2.0