ECS Research
Publications
Intranet Tools

nb. next round of REF2013 will NOT be using data from eprints.ecs, but the central university REF interface.

RSS 1.0 Feed
RSS 2.0 Feed
Atom Feed
 

A Transfer Report on the Development of a Framework to Evaluate Search Interfaces for their Support of Different User Types and Search Tactics

Wilson, M. L. (2008) A Transfer Report on the Development of a Framework to Evaluate Search Interfaces for their Support of Different User Types and Search Tactics. Technical Report 2008032515642, School of Electronics and Computer Science, University of Southampton.

Download

[img]
Preview
 Accepted Version
 PDF (PhD Upgrade Report)
2662Kb

Abstract

As the understanding of search systems, user needs and seeking strategies is developing, the design of search user interfaces is evolving to support more complicated and exploratory forms of search. With the design of new search features that enable these richer modes of exploration, comes the need to better understand the support they provide. In this report a new evaluation framework is presented that analyses search features for how they a) contribute to an overall interface, b) allow users to carry out different search tactics, and c) support different types of users and their needs. The novel contributions of the framework improve on some of the limitations of typical user studies, and allow search systems to be systematically analysed in much more detail and in much less time. The presented evaluation framework is then validated in three ways. First the validity of the models used as the building blocks of the framework are investigated through related work. Second the method of integrating these building-block models is validated and strengthened by consensus of expert opinion. Third, the overall approach is validated by comparing its analyses to the results of previously carried out user studies. The validation process has shown both the value of the framework and identified areas of future work that should be addressed for the framework to be completed. This report concludes with the set of contributions that the framework makes, and why the remaining work will be challenging, but critical to the final design.

Item Type:Technical Report
Creator/Authors:
Max L. Wilson
Research Group:Old ECS Groups > Intelligence, Agents, Multimedia
Date:2008
Information about this record:
Performance Indicator:EZ~01~01~06
Downloads (2010):109
ID Code:15642
Last Modified:23 Sep 2011 10:36
Deposited On:29 Apr 2008 11:29 by Wilson, Max

Tools & Metadata

Download Statistics

Last month

Last year

Members of ECS may view the download statistics dashboard for this record.

References in Article

Select the SEEK icon to attempt to find the referenced article. If it does not appear to be in this archive you will be forwarded to the paracite service. Poorly formated references will probably not work.

M. J. Bates. Idea tactics. Journal of the American Society for Information Science, 30 (5):280–289, 1979a.

Marcia J. Bates. Information search tactics. JASIS, 30:205–4214, 1979b.

Marcia J. Bates. Where should the person stop and the information search interface start? Inf. Process. Manage., 26(5):575–591, 1990. ISSN 0306-4573.

Marcia J. Bates. The cascade of interactions in the digital library interface. Inf. Process. Manage., 38(3):381–400, 2002. ISSN 0306-4573.

Steven M. Beitzel, Eric C. Jensen, Abdur Chowdhury, David Grossman, and Ophir Frieder. Hourly analysis of a very large topically categorized web query log. In SIGIR ’04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 321–328, New York, NY, USA, 2004. ACM Press. ISBN 1-58113-881-4.

N. J. Belkin, P. G. Marchetti, and C. Cool. Braque: design of an interface to support user interaction in information retrieval. Inf. Process. Manage., 29(3):325–344, 1993. ISSN 0306-4573.

N.J. Belkin, C. Cool, A. Stein, and U. Thiel. Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems. Expert Systems with Applications, 9(3):379–395, 1995.

Spring Hull Bonnie Lida and Katie Pilcher. Breadcrumb navigation: An exploratory study of usage. Usability News, 5(1), 2003.

P. Borlund. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3):8–3, 2003.

S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30(1-7):107–117, 1998.

Bertram C. Brookes. The foundations of information science: Part I. Philosophical aspects. Journal of Information Science, 2(3-4):125–133, 1980.

M. K. Buckland. Design for an adaptive library catalog. Proceedings of the ASIS MidYear Meeting, 165:71, 1992.

R. Capra, G. Marchionini, J.S. Oh, F. Stutzman, and Y. Zhang. Effects of structure and interaction style on distinct search tasks. In Proceedings of the 2007 conference on Digital libraries, pages 442–451. ACM Press New York, NY, USA, 2007.

Ed H. Chi, Peter Pirolli, Kim Chen, and James Pitkow. Using information scent to model user information needs and actions and the web. In CHI ’01: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 490–497, New York, NY, USA, 2001. ACM Press. ISBN 1-58113-327-8.

Colleen Cool and Nicholas J. Belkin. A classification of interactions with information. In CoLIS 4: Proceedings of the 4th international conference on Conceptions of Library and Information Science, pages 1–15, 2002. ISBN 1-59158-016-1.

B. Dervin. From the mind’s eye of the user: The sense-making qualitative-quantitative methodology. Qualitative research in information management, pages 61–84, 1992.

Natalya Godbold. Beyond information seeking: towards a general model of information behaviour. Information Research, 11(4):paper 126, 2006.

Richard Gong and David Kieras. A validation of the goms model methodology in the development of a specialized, commercial software application. In CHI ’94: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 351–357, New York, NY, USA, 1994. ACM Press. ISBN 0-89791-650-6.

Wayne D. Gray, Bonnie E. John, and Michael E. Atwood. The precis of project ernestine or an overview of a validation of goms. In CHI ’92: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 307–312, New York, NY, USA, 1992. ACM Press. ISBN 0-89791-513-5.

Jonathan Grudin. The case against user interface consistency. Communications of the ACM, 32(10):1164–1173, 1989. ISSN 0001-0782.

M. Hammersley and P. Atkinson. Ethnography: Principles in Practice. Routledge, 1995.

D. K. Harman. The trec conferences. Morgan Kaufmann Multimedia Information And Systems Series, pages 247–256, 1997.

Marti A. Hearst. Next generation web search: Setting our sites. IEEE Data Engineering Bulletin: Special Issue on Next Generation Web Search, 2000.

Marti A. Hearst. Clustering versus faceted categories for information exploration. Commun. ACM, 49(4):59–61, 2006. ISSN 0001-0782.

K. Holtzblatt and H.R. Beyer. Requirements gathering: the human factor. Communications of the ACM, 38(5):31–32, 1995.

I. Hsieh-Yee. Search tactics of web users in searching for texts, graphics, known items and subjects: Asearch simulation study. Electronic Resources: Use and User Behavior, 1998.

Ingrid Hsieh-Yee. Effects of search experience and sub ject knowledge on online search behavior: Measuring the search tactics of novice and experienced searchers. Journal of the American Society for Information Science, 44(3):161–174, 1993.

I. Huvila and G. Widen-Wulff. Perspectives to the classification of information interactions: the cool and belkin faceted classification scheme under scrutiny. Proceedings of the 1st international conference on Interaction in context, pages 144–152, 2006.

Kalervo J ̈arvelin and Peter Ingwersen. Information seeking research needs extension towards tasks and technology. Information Research, 10(1), 2004.

B.E. John. Extensions of GOMS analyses to expert performance requiring perception of dynamic visual and auditory information. Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, pages 107–116, 1990.

Bonnie E. John and Allen Newell. Predicting the time to recall computer command abbreviations. SIGCHI Bul l., 17(SI):33–40, 1987. ISSN 0736-6906.

Bonnie E. John, Paul S. Rosenbloom, and Allen Newell. A theory of stimulus-response compatibility applied to human-computer interaction. SIGCHI Bul l., 16(4):213–219, 1985. ISSN 0736-6906.

K.S. Jones et al. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21, 1972.

J. Koenemann and Nicholas J. Belkin. A case for interaction: a study of interactive information retrieval behavior and effectiveness. In CHI ’96: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 205–212, New York, NY, USA, 1996. ACM Press. ISBN 0-89791-777-4.

Sascha Kriewel. Finding and using strategies for search situations in digital libraries. Bulletin of the IEEE Technical Committee on Digital Libraries (TCDL), 2(2), 2006.

Carol C. Kuhlthau. Inside the search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science, 42(5):361–371, 1991.

Gary Marchionini. Exploratory search: from finding to understanding. Commun. ACM, 49(4):41–46, 2006. ISSN 0001-0782.

Catherine C. Marshall and III Frank M. Shipman. Spatial hypertext and the practice of information triage. In HYPERTEXT ’97: Proceedings of the eighth ACM conference on Hypertext, pages 124–133, New York, NY, USA, 1997. ACM Press. ISBN 0-89791-866-5.

C. R. McClure and P. Hernon. Improving the quality of reference service for government publications. American Library Association Chicago, 1983.

M. K. Moody. Documents search strategies and general reference search strategies: an analysis and comparison. The Reference librarian, (32):57–69, 1991.

Jakob Nielsen and Rolf Molich. Heuristic evaluation of user interfaces. In CHI ’90: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 249–256, New York, NY, USA, 1990. ACM. ISBN 0-201-50932-6.

Heather L. O’Brien and Elaine G. Toms. What is user engagement? a conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6):1–18, April 2008.

H.L. O’Brien and E.G. Toms. Engagement as Process in Human-Computer Interactions. In Annual Conference of of American Society for Information Science and Technology, 2005.

H.L. O’Brien and E.G. Toms. Evaluating engagement in interactive search. In SIGIR2007 Workshop on Web Information Seeking and Interaction, 2007.

R.A. Peterson. Constructing Effective Questionnaires. Sage Publications, 2000.

N. Pharo. Web information search strategies: A model for classifying web interaction. In Proceedings of the 3rd International Conference on the Conceptions of the Library and Information Science, pages 207–218, 1999.

J. Piaget. The stages of the intellectual development of the child. Bul l Menninger Clin, 26:120–8, 1962.

Peter Pirolli and Stuart Card. Information foraging in information access environments. In CHI ’95: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 51–58, New York, NY, USA, 1995. ACM Press/Addison-Wesley Publishing Co. ISBN 0-201-84705-1.

V. Raghavan, P. Bollmann, and G.S. Jung. A critical investigation of recall and precision as measures of retrieval system performance. ACM Transactions on Information Systems (TOIS), 7(3):205–229, 1989.

M. Rettig. Prototyping for tiny fingers. Communications of the ACM, 37(4):21–27, 1994.

SE Robertson. On term selection for query expansion. Journal of Documentation, 46 (4):359–364, 1991.

SE Robertson, S. Jones, et al. Relevance Weighting of Search Terms. Journal of the American Society for Information Science, 27(3):129–46, 1976.

SE Robertson, S. Walker, MM Beaulieu, N. Square, E.C.V. London, and M. Gatford. Okapi at TREC-4. DIANE Publishing, 1998.

Peter Ro jas. Entelligence: Understanding the success of the ipod and ipod mini. Online, 2004.

R. Rubinstein and H. Hersh. The Human Factor: Designing Computer System for People. Digital Press, Burlington, Mass, 1986.

G. Salton and C. Buckley. Improving Retrieval Performance by Relevance Feedback. Journal of the American Society for Information Science, 41(4):288–97, 1990.

Tefko Saracevic. The stratified mode of information retrieval interactino: Extension and applications. In Proceedings of the American Society for Information Science, volume 34, pages 313–327, 1997.

m.c. schraefel, Max L. Wilson, and Maria Karam. Preview cues: Enhancing access to multimedia content. Technical report, School of Electronics and Computer Science, University of Southampton, August 2004. http://eprints.ecs.soton.ac.uk/9253/.

m.c. schraefel, Max L. Wilson, Alistair Russell, and Daniel A. Smith. mspace: improving information access to multimedia domains with multimodal exploratory search. Communications of the ACM, 49(4):47–49, 2006. ISSN 0001-0782.

m.c. schraefel, Yuxiang Zhu, David Modjeska, Daniel Wigdor, and Shengdong Zhao. Hunter gatherer: interaction support for the creation and management of within-webpage collections. In WWW ’02: Proceedings of the 11th international conference on World Wide Web, pages 172–181, New York, NY, USA, 2002. ACM Press. ISBN 1-58113-449-5.

Ben Shneiderman, Benjamin B. Bederson, and Steven M. Drucker. Find that photo!: interface strategies to annotate, browse, and share. Commun. ACM, 49(4):69–71, 2006. ISSN 0001-0782.

Ben Shneiderman and Catherine Plaisant. Designing the user interrface: Strategies for effective human-computer interaction (4th Ed.). Addison-Wesley, 2005.

Ben Shneiderman and Catherine Plaisant. Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In BELIV ’06: Proceedings of the 2006 AVI workshop on BEyond time and errors, pages 1–7, New York, NY, USA, 2006. ACM. ISBN 1-59593-562-2.

S. J. Shute and P. J. Smith. Knowledge-based search tactics. Information Processing and Management, 29(1):29–45, 1993.

P. J. Smith, S. J. Shute, B. Galdes, and M. H. Chignell. Knowledge-based search tactics for an intelligent intermediary system. ACM Transactions on Information Systems (TOIS), 7(3):246–270, 1989.

L.T. Su. Evaluation Measures for Interactive Information Retrieval. Information Processing and Management, 28(4):503–16, 1992.

Linda Tischler. The beauty of simplicity. Fast Company, (100):Online, November 2005.

C. Wharton, J. Rieman, C. Lewis, and P. Polson. The cognitive walkthrough method: a practitioner’s guide. Usability inspection methods table of contents, pages 105–140, 1994.

H. D. White and K. W. McCain. Visualizing a discipline: An author co-citation analysis of information science, 1972-1995. Journal of the American Society for Information Science, 49(4):327–355, 1998.

Ryen W. White, Bill Kules, Steven M. Drucker, and m.c. schraefel. Introduction. Commun. ACM, 49(4):36–39, 2006. ISSN 0001-0782.

B. M. Wildemuth, R. de Bliek, S. He, and C. P. Friedman. Search moves made by novice end users. Proceedings of the 55th annual meeting on Celebrating change: information management on the move: information management on the move table of contents, pages 154–161, 1992.

B. M. Wildemuth, Jacob, Fullington, R. de Bliek, and C. P. Friedman. A detailed analysis of end-user search behaviors. Proceedings of the ASIS Annual Meeting, 28: 302–12, 1991.

Max L. Wilson. A nine month report on progress towards a framework for evaluating advanced search interfaces considering information retrieval and human computer interaction. Technical report, School of Electronics and Computer Science, University of Southampton, 2007.

Max L. Wilson, Paul Andre, Daniel A. Smith, and m.c. schraefel. Spatial consistency and contextual cues for incidental learning in browser design. Technical report, School of Electronics and Computer Science, University of Southampton, September 2007.

Max L. Wilson and m.c. schraefel. A longitudinal study of exploratory and keyword search. In ACM/IEEE-CS Joint Conference on Digital Libraries. IEEE Computer Society, June 2008.

Max L Wilson, m.c. schraefel, and Ryen W. White. Evaluating advanced search interfaces using established information-seeking models. In submission to the Journal of the American Society for Information Science and Technology, 2008.

Max L. Wilson, Alistair Russell, Daniel Smith, and m.c. schraefel. mspace mobile: Exploring support for mobile tasks. In British Conference on Human-Computer Interaction 2006, 2006.

Max L. Wilson and m.c. schraefel. mspace: What do numbers and totals mean in a flexible semantic browser. In The 3rd International Semantic Web User Interaction Workshop, 2006.

Max L. Wilson and m.c. schraefel. Bridging the gap: Using ir models for evaluating exploratory search interfaces. In First Workshop on Exploratory Search and HCI at SIGCHI07, 2007.

T.D. Wilson. Models of information behaviour research. Journal of Documentation, 55 (3):249–270, 1999.

Ka-Ping Yee, Kirsten Swearingen, Kevin Li, and Marti Hearst. Faceted metadata for image search and browsing. In CHI ’03: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 401–408, New York, NY, USA, 2003. ACM Press. ISBN 1-58113-630-7.

Junliang Zhang and Gary Marchionini. Evaluation and evolution of a browse and search interface: relation browser. In dg.o2005: Proceedings of the 2005 national conference on Digital government research, pages 179–188. Digital Government Research Center, 2005.

Corrections

ECS staff and postgraduates may modify this record

   Welcome from Deputy Head of School (Research) Research Prospectus Industrial Partnerships New Research Students Notes for Guidance New Research Students Notes for Guidance
The ECS EPrints Repository supports OAI 2.0 with a base URL of http://eprints.ecs.soton.ac.uk/cgi/oai2

EPrints is free software developed by the University of Southampton to facilitate Open Access to research.		 EPrints