Instructions for document image quality assessment

Introduction

Your task is to evaluate the quality of Arabic text images. Text images can be degraded in different ways. Factors affect the quality of text images include background noise, fatten stroke, broken stroke, bad writing style and so on. In this task, you should solely focus on evaluating stroke quality of text image. Each HIT can be completed in around 60 seconds.

Payment

We have methods that analyze the consistency of your answers with respect to themselves, to those of your fellow workers and to references we know to be accurate. We will use these methods to rank the submitted assignments according to quality.

For this experiment we will pay a base reward of $0.01/HIT for every accepted HIT. We have made available a set of 30 different HITs. You will receive a bonus of:

Bonuses will be paid up to 7 days after submission, because we can only rank the submissions once we have a statistically significant number of answers. The base reward will always be paid within 24 hours of submission.

Instructions

Each file should be given a score according to the following scale, known as the MOS (mean opinion score) scale:

Score Quality
5 Excellent
4 Good
3 Fair
2 Poor
1 Bad

The following references illustrate the meaning of each score. Please note that you may encounter other types of distortions. Therefore, these examples do not exhaust the range of conditions you can expect to see.

Examples


Reference score: 5.0
The strokes are sharp. The text is easily readable.



Reference score: 4.0



Reference score: 3.0



Reference score: 2.0



Reference score: 1.0

Approval/Rejection Policy

Your results will be collected and evaluated for consistency. We (the requesters) have an estimate of each file's subjective quality that conforms with the references above. Thus, we can detect if someone submits random scores or does not rate according to these instructions, which can lead to work being rejected. You can rest assured that your work will be approved if you rate according to the table and examples above.

Answers will be either reviewed or automatically approved within 24 hours.