This is how you help us:
There are two ways in which we ask for your opinion on the relevance of documents for particular searches:
Task 1: Document Judgment
A Document Judgment page shows a query and several documents. The order of the documents is random and should not influence your judgment. For each of the documents, please determine how relevant it is to the query on a 3-level scale:
Green highlighting means that the meaning of this word corresponds to the meaning of a query term.
and highlighting means that the word‘s meaning is related to the query terms‘ meaning. You can get an explanation of how they are related by hovering over the highlighted word.
Black highlighting means that this word appears in the query (but possibly with a different meaning).
Task 2: Ranking Judgment
In a Ranking Judgment, we want you to judge the result lists as a whole. You will see three columns with document excerpts, each representing the order of documents as returned by one search algorithm. Please think about which ranking is best (and which is second best) overall, i.e. you would prefer to see after entering the query in a search engine, click the appropriate button.
When you have completed both task, you will get the same tasks again for a different query.
Then let's go! You will be asked to choose a user name, so you can stop the evaluation at any time and continue it later,
as well as to provide some basic statistical data.