Assessment of E-Negotiation Users and Processes
Project 2 began in January 2002. At this stage the project is mainly concerned with empirical analysis of available negotiation data sets as well as the design and preparation of new experiments. The main data source is the database created using the Inspire ENS since 1996. In addition, experiments were performed with two student groups in Austria and Taiwan in March 2003 using two different ENSs, Inspire and SimpleNS. Furthermore, in collaboration with project 5, an e-negotiation tournament using six different ENSs was performed in November/December 2003, and data from this tournament will also be available for analysis shortly.
Impact of negotiators' characteristics on process, outcomes and ENS adoption
The Inspire database contains records of all negotiations carried out using the Inspire ENS since the system was made available on the Web in 1996. It currently contains records of over 2500 negotiations with participants from over 50 countries. Thus, it provides a unique opportunity to study negotiator behaviour in a multicultural context. However, in order to provide a consistent data base for analysis, the original questionnaires, which have been developed at early stages of the Inspire project to obtain some feedback from users, had to be retained. Furthermore, the user population represented in the Inspire database is rather heterogeneous, since utilization of Inspire has not been not a control variable and access to Inspire has been open to anyone. Hence, empirical analysis of the Inspire data is comparable to an explorative field study rather than the analysis of a carefully controlled experiment.
Since the environment represented by the Inspire negotiation records is rather different from the controlled and homogeneous setting in which user assessment studies are usually carried out, it is not possible to use established models from the area of information systems evaluation like the Technology Adoption Model (TAM). We therefore ad to develop our own Assessment Model for Internet-based Systems (AMIS). AMIS builds on key concepts from the TAM model, like usefulness, ease of use and intentions to use an information system, and provides the extensions and modifications necessary for the environment in which Inspire negotiations are carried out. Specifically, the following points are taken into account:
The development and initial empirical studies of AMIS have been documented in several research reports and journal publications. Early empirical tests have led to two important results:
These two empirical phenomena have led to the formulation of Culture-Technology Link Model about the different impacts that various features of an ENS might have on negotiators from different cultures. The model contains four hypotheses:
These four hypotheses will form the basis of our future empirical studies, which will be carried out using Inspire as well as other ENSs.
Computational Linguistic Perspective
The following summarizes the work performed till the end of 2003.
Assessment of Negoisst negotiators and negotiations
So far, Negoisst has mainly been evaluated in real-life negotiations between members of cooperative construction teams. The negotiation tournament offered a different possibility of assessing the system's capabilities and shortcomings. Based on the pre-negotiation and post-negotiation questionnaire, we obtained information about the system's usefulness regarding the support of efficient negotiations.
Furthermore, we are also interested in the quality aspect of negotiation support since Negoisst is strongly based on communication management and dynamic support of user flexibility. To this end, the negotiation logs were examined and compared to the previous negotiations in the construction sector.
The main goal of the assessment and examination of negotiation steps is to improve and validate our approach to communicative support and enrichment of complex (written) negotiation processes conducted with Negoisst. The assessment is based on a framework of speech act elements that are taken, revised, and combined from Habermas' Theory of Communicative Action and Searle's Theory of Speech Acts. The framework has been successfully used in cooperative document management in healthcare and as a theoretical basis for the development of Negoisst. Each utterance is classified into one class of speech acts representing the sender's intention, e.g. assertive, commissives, directive. The particular illocutionary force (i.e. offer, request, information, question etc.) needs to be adapted to the field of negotiations. One of our aims is to refine the existing classification by evaluating whether the forces offered as message types so far are sufficient. Each class of acts then carries certain obligations. For example, a commissive represents a speaker's/sender's intention to carry out a particular action. Such obligations are important for the contract that is automatically derived from the messages that are exchanged.
Further research includes an assessment of potential application branches of Negoisst and continuous evaluations both within and beside the negotiation tournament.
Comparative study of two ENSs use
While e-negotiations gain in popularity there have been few studies comparing the efficacy of different ENSs. We have initiated such a study with the comparison of negotiations performed with Inspire and the Simple NS systems. The novelty of this study requires both theoretical and empirical research. Both issues are addressed in ongoing work within this project.
The main theoretical issue arising out of this project is the ex post evaluation of negotiator performance. In NSS without analytical support, like SimpleNS, no utility functions are elicited from negotiators. Therefore it is not possible to compare negotiation results using utility values. To overcome this problem, the Extended Dominance Relation (EDR) method for the evaluation of negotiation outcomes has been developed. This approach starts from the dominance relation of multiattribute decision analysis, which provides an incomplete ranking of alternatives without requiring information about preferences of decision makers (negotiators).
In the EDR approach, the dominance relation is augmented by observed preferences between alternatives discussed during the negotiation. The method generates a scalar measure of negotiator performance, which does not require experimental subjects to estimate a utility function and thus can be used to evaluate outcomes achieved with any ENS. In a first empirical analysis using this approach, different variants of EDR performance measures have been successfully tested against utility ratings available from Inspire. Results of this study, as well as the theoretical development of the EDR approach, is documented in a working paper, and submission to a peer-reviewed journal is being prepared.
While the EDR approach can be used to measure the performance of individual negotiators, additional methods are required to determine the efficiency (Pareto-optimality) of negotiation results. For this problem, an extension of Data Envelopment Analysis as well as a probability-based approach were developed. Theoretical development and implementation of the software necessary to test these approaches have been completed and the initial results were presented at the project workshop in Prague. It is expected that this stream of research will lead to a journal submission in 2004.
For the empirical comparison of Inspire and SimpleNS, a set of controlled experiments has been conducted in March 2003 with 80 Austrian and Taiwanese participants using two ENS: SimpleNS and Inspire. Results of these experiments are being analyzed using statistical methods for quantitative characteristics of the users, negotiation processes and outcomes. In addition, content analysis methods are employed to obtain a deeper understanding of communication patterns and the negotiation processes.
Preliminary results of the quantitative analysis indicate that while there is little difference in the users' assessment of the SimpleNS and Inspire negotiations, a significantly greater number of Inspire users reached an agreement than the SimpleNS users. This would indicate that support based on analytical models does not have negative influences on the negotiation process but significantly impacts the negotiation effectiveness.
The content analysis performed on the negotiation logs of these experiments aims to describe and analyze web-based negotiation processes in more detail. In particular we want to:
We use qualitative research methods, i.e. content, process and interaction analysis to analyze the data. Currently, the negotiation process transcripts of 40 negotiations are separated in communication propositions, i.e. the unit of analysis, and coded by two independent coders. For this purpose, we have adapted the Bargaining Process Analysis (BPA) model for electronic negotiation processes. Additionally, we apply personal pronoun analysis. Preliminary results are expected in May 2004, to be followed by a research report and a conference presentation in July 2004.
Ph.D. and Master's theses:
Research will continue along the lines described above. For the empirical analysis of Inspire and further development of the AMIS model, the questionnaires currently in use in Inspire will be redesigned to more closely reflect constructs of the AMIS model. This change should lead to results which are more comparable with the existing literature on IS evaluation. The revised AMIS model will be tested using data obtained from the e-negotiation tournament performed in December 2003 as well as a new tournament, which will take place in January 2004.
Development of performance measures will also continue as outlined above. We expect a journal submission of the EDR method paper to be ready in the first months of 2004 and a working paper on the DEA and probability-based approach to be completed by mid 2004. Initial tests of the EDR method have identified a surprisingly high level of inconsistent behaviour by subjects: in more than a third of all negotiations analyzed, negotiators finally accepted a compromise, which, according to the utility values calculated by Inspire, was worse for them than a previous offer from their opponents. An empirical study of possible explanations for thus behaviour is planned for late 2004 and will continue throughout 2005. The performance measurement approaches developed will be employed to analyze empirical results obtained with different NSS throughout the project.
Work on text analysis will continue throughout 2004 and 2005. In the next several months, we will:
The longer-term plans include the following:
Project members at the University of Vienna (R. Vetschera, S. Koeszegi) have started a cooperation with the departments of Software engineering at the Technical University of Vienna, Austria and the University of Linz, Austria to launch a joint research program on the use of decision and negotiation analysis in software engineering projects. A project proposal for funding of such a program has been submitted. Initial research in the preparation phase of this project has focused on identifying opportunities to use negotiation support in software requirements analysis (Grünbacher et al., 2003).
S. Koeszegi and R. Vetschera will organize the meeting of the INFORMS Section on Group Decision and Negotiation in Vienna 2005.