Quick Instructions:

One step: run 'ant'
 This should download collections and place them in the dist/collections directory

Under a given collection 'example' there would be the following structure:
 queries.txt: Trec topics file
 judgements.txt: Relevance judgements file
 corpus.gz: Gzipped corpus. 

See FILEFORMATS.txt for more information on the structure of these files.


How to use with Lucene-java benchmark package?

Step 1: create a contrib/benchmark/conf/openrelevance.alg

### START OF FILE: just an example
content.source=org.apache.lucene.benchmark.byTask.feeds.TrecContentSource
content.source.log.step=2500
doc.term.vector=false
content.source.forever=false
content.source.encoding=UTF-8
directory=FSDirectory
doc.stored=true
doc.tokenized=true
content.source.excludeIteration=true
ResetSystemErase
CreateIndex
{ AddDoc } : *
CloseIndex
### END OF FILE

Step 2: place the corpus.gz into the contrib/benchmark/work/trec folder.
 Or alternatively configure this to a different location in the .alg file

Step 3: from contrib/benchmark, run ant run-task -Dtask.alg=conf/openrelevance.alg
 This will create an index in contrib/benchmark/work/index

Step 4:  
 java -Dfile.encoding=UTF-8 -cp lucene-core-3.0-dev.jar:lucene-benchmark-3.0-dev.jar org.apache.lucene.benchmark.quality.trec.QueryDriver queries.txt judgements.txt submission.txt contrib/benchmark/work/index

 This will print a bunch of information, finally a summary output.

 You can also take the resulting submission.txt, along with judgements.txt, 
 and run trec_eval to get "official" calculations.