Towards a fair benchmark for image browsers

Bibtex entry :

@inproceedings { VG:MMM2000c,
    author = { Wolfgang M{\"u}ller and St{\'e}phane Marchand-Maillet and Henning M{\"u}ller and Thierry Pun },
    title = { Towards a fair benchmark for image browsers },
    booktitle = { SPIE Photonics East, Voice, Video, and Data Communications },
    year = { 2000 },
    address = { Boston, MA, USA },
    month = { nov 5--8 },
    url = { http://vision.unige.ch/publications/postscript/2000/MuellerWMarchandMuellerHPun_spie2000.ps.gz },
    abstract = { The recent literature has shown that the principal difficulty in multimediaretrieval is the bridging of the "semantic gap" between the user'swishes and his ability to fomulate queries. This insight has spawnedtwo main directions of research: Query By Example (QBE) with relevancefeedback (i.e. learning to improve the result of a previsously formulatedquery) and the research in query formulation techniques, like browsingor query by sketch. Browsing techniques try to help the user in findinghis target image, or an image which is sufficiently close to thedesired result that it can be used in a subsequent QBE query. Fromthe feature space viewpoint, each browsing system tries to permitthe user to move consciously in feature space and eventually reachthe target image. How to provide this functionality to the user ispresently an open question. In fact even obtaining objective performanceevaluation and comparison of these browsing paradigms is difficult.We distinguish here between deterministic browsers, which try tooptimise the possibility for the user to learn how the system behaves,and stochastic browsers based on more sophisticated Monte-Carlo algorithmsthus sacrificing reproducibility to a better performance. Presently,these two browsing paradigms are practically incomparable, exceptby large scale user studies. This makes it infeasible for researchgroups to evaluate incremental improvement of browsing schemes. Moreover,automated benchmarks in the current literature simulate a user bya model derived directly from the distance measures used within thetested systems. Such a circular reference cannot provide a seriousalternative to real user tests. In this paper, we present an automaticbenchmark which uses user-annotated collections for simulating thesemantic gap, thus providing a means for automatic evaluation andcomparison of the different browsing paradigms. We use a very preciseannotation of few words together with a thesaurus to provide sufficientlysmooth behaviour of the annotation-based user model. We discuss thedesign and evaluation of this annotation as well as the implementationof the benchmark in an MRML-compliant script with pluggable moduleswhich allow testing of new interaction schemes (see http://www.mrml.net). },
    owner = { steph },
    timestamp = { 2008.05.04 },
    url1 = { http://vision.unige.ch/publications/postscript/2000/MuellerWMarchandMuellerHPun_spie2000.pdf },
    vgclass = { refpap },
    vgproject = { viper },
}