AFRL/IISI Workshop on Mixed Initiative Decision Making

October 20-21, 2003

 

Final Report

Henry Kautz

University of Washington

 

The AFRL/IISI Workshop on Mixed Initiative Decision Making brought 35 computer scientists, psychologists, decision making analysts, and military personnel to the Intelligent Information Systems Institute at Cornell University to discuss needs opportunities for research on critical decision making.  The group developed a Gap Analysis for each of four areas:

 

A short summary of the outcomes of the discusses and analyses is:

·        Traditional decision-support tools are of little use for the kinds of real-world critical decision making tasks that DoD faces, because such tools assume that the decision problem is precisely formulated in advance.

·        Teams of experts from the field of naturalistic decision-making can create effective domain-specific decision-making tools, but this process is often slow and costly.

·        Recent advances in algorithms for representing and reasoning about preferences and uncertainty provide a core technology for creating effective real-world decision making tools much more rapidly and with less manual effort.

 

In short: the workshop identified key research opportunities to develop technology that will be broadly useful for DoD decision support applications.

                       

Purpose of the Workshop

 

Classical decision theory assumes that:

 

 

Real-world strategic decision making situations rarely meet all these requirements.  The initial decision problem is typically ill defined, and the model of the decision problem grows through many iterations as flaws and inconsistencies are revealed.  The user's utilities are not always known in advance, but may only be determined incrementally as he accepts or rejects candidate solutions.  Furthermore, a useful solution to a problem is not just a decision, but rather a defensible reason for making that decision, in terms of the facts and assumptions built into the model.  Finally, both the human and machine effort needed to make the decision must be taken into account according to the broader decision context, which can range from long-range planning to immediate action under fire.

 

Research in Artificial Intelligence deals with many of these issues through work on interactive planning, preference elicitation, resource-bounded reasoning, and algorithms for single and multi-agent decision problems.

 

Some researchers in psychology and decision science (such as work in judgment under uncertainty and naturalistic decision making) also address situations with poorly defined goals, missing data, stress, high stakes, time pressure, and uncertainty.  Studies in fields such as military command and control show that the strategies human experts employ are quite distinct from the simple classical model.  For example, classifying a situation as an instance of a previously solved problem is a more prevalent strategy than systematic weighing of alternatives. 

 

The purpose of the workshop is to bring together researchers in AI and decision science to discuss how real-world decision-making could be improved though the creation of effective, interactive human-machine decision-making systems

 

Outcomes

 

The workshop began with a series of talks on current research on decision-making.  We then broke into four working groups to develop gap analyses of major sub-areas of the field.

 

The Human Decision Making group identified the following user needs and associated gaps:

  1. Information searching: user interfaces often overload user with information.  Increasing the amount of information presented can decrease the usefulness of the information.  Research is needed on new ways to present rich information to humans in a way that reduces demands on short-term memory, such as graphical multi-resolution data visualization and automated summarization.
  2. Building an effective decision support system requires creating a cognitive task analysis of the decision problem, a lengthy and costly process.  Research is needed on tools that help interactively perform such an analysis.
  3. Current decision support systems require the human to create the set of possible hypothesis.  This can be problematic, because people often fail to see the inadequacies of their first hypothesis.  Research on systems that can critique hypothesis and suggest when all known hypotheses could help address this.
  4. Training high-risk decision making experts is a lengthy, domain-specific process.  More psychologically based research on transfer of expertise between domains could help speed up training.
  5. Human decision makers often get conflicting recommendations from different decision support tools.  Research on how humans perceive the trustworthiness of a machine’s recommendations will provide a basis for developing better strategies for decision makers to use when faced with such situations.

           

The Preferences group identified the following user needs and associated gaps:

  1. Any automated decision support system needs a formal model of user preferences, but current systems use impoverished representations.  Extensive basic research on practical and expressive preference representations and algorithms is needed.  This includes the problem of validating the consistency of a user’s preferences, and tracking a user’s preferences as they may change over time.
  2. Research on automatically generating explanations for a system’s recommendations is important for making such systems trustworthy.
  3. Decision support systems for organizations like DoD must model organizational objectives and experience, not just the preferences of a single agent.  Research on dealing with preferences within a structured group, while supporting both vertical and horizontal inter-organization relationships, is needed.
  4. Research on algorithms and user interfaces for interactive preference elicitation is needed to support all of needs.

 

The group concentrating on Time Critical Reasoning under Uncertainty identified the following user needs and associated gaps:

  1. Time-critical decision support systems need to function in an “anytime” fashion, because an answer may be needed before there is time to perform a complete detailed analysis.  Basic research on design principles for interactive anytime systems is needed. 
  2. Decision making situations are inherently uncertain, and therefore any particular model inherently has some inaccuracies.   Research is needed on model evaluation, including the problem of evaluating the usability tradeoffs between simple models and more accurate but more complex.

 

Finally, the Multi-Agent Systems group identified the following user needs and associated gaps:

  1. Communication within a team of human and software-agent decision makers requires a common vocabulary and ontology.   Research on systematic ways to build and extend ontologies is needed.
  2. Current decision-support systems have inflexible control of initiative.  This can be problematic – for example, the system may have control of initiative and be demanding that the user fill out an online form, but in order to get the information to do so the user needs to take back initiative and query the system, but cannot.  Research is needed on variable initiative systems, including strategies of how control be indicated, passed, given up, and taken away.

 

Summary of Research Needs

 

  1. Psychologically-grounded techniques for presenting and visualizing information
  2. Automated explanation and human/machine trustworthiness
  3. Eliciting, reasoning about, and aggregating preferences across organizations
  4. Anytime reasoning systems with flexible control of initiative

 

Participants

 

David Artman (Applied Research Assoc.) <DArtman@ara.com>

Ronen Brafman (Ben-Gurion)   <brafman@cs.bgu.ac.il>

Tim Busch (AFRL/IFS)    <Timothy.Busch@rl.af.mil>

Joe Carozzoni (AFRL/IFS)     <Joe.Carozzoni@rl.af.mil>

Donald Cox (Klein Associates)       <donald@decisionmaking.com>

David Diller (BBN)              <ddiller@bbn.com>

Carmel Domshlak (Cornell/Technion)                   <dcarmel@cs.cornell.edu>

Jon Doyle (NCSU)           <jon_doyle@ncsu.edu>

Jerry Dussault (AFRL/IFS)    <Jerry.Dussault@rl.af.mil>

George Ferguson (Rochester)     <ferguson@cs.rochester.edu>

Nort Fowler (AFRL/IF)      <Northrup.Fowler@rl.af.mil>

Judy Goldsmith (U Kentucky)  <goldsmith<at>cs.uky.edu>

Carla Gomes (Cornell)           <gomes@cs.cornell.edu>

Don Gossink (AFRL/Australian Def. Sci. Tech Org.)      <Don.Gossink@data-net.org>

John Graniero (AFRL/IFB)     <granieroj@rl.af.mil>

Joe Halpern (Cornell)           <halpern@cs.cornell.edu>

Robert Hillman (AFRL/IFT)    <Robert.Hillman@rl.af.mil>

Mike Hinman (AFRL/IFE)    <Michael.Hinman@rl.af.mil>

Earl Hunt (U Washington)     <ehunt@u.washington.edu>

Henry Kautz (U Washington)                        <kautz@cs.washington.edu>

Kevin Kwiat (AFRL/IFG)    <Kevin.Kwiat@rl.af.mil>

Jamie Lawton (AFRL/IFT)    <James.Lawton@rl.af.mil>

Chet Maciag (AFRL/IFG)    <Chester.Maciag@rl.af.mil>

Barry McKinney (AFRL/IFB)    <Barry.McKinney@rl.af.mil>

John McNamara (AFRL/IFS)    <John.McNamara@rl.af.mil>

Janet Miller (AFRL/HECA)                       <Janet.Miller3@wpafb.af.mil>

Sibabrata Ray (U Alabama)   <sibu@cs.ua.edu>

Dale Richards (AFRL/IFT)    <Dale.Richards@rl.af.mil>

John Salerno (AFRL/IFE)    <John.Salerno@rl.af.mil>

Eugene Santos, Jr. (U Conn)    <eugene@eng2.uconn.edu>

Meinolf Sellmann (Cornell)          <sello@cs.cornell.edu>

Bart Selman (Cornell)           <selman@cs.cornell.edu>

Walt Tirenin (AFRL/IFG)    <Wladimir.Tirenin@rl.af.mil>

Mike Wessing (AFRL/IFE)    <Mike.Wessing@rl.af.mil>

Shlomo Zilberstein (U Mass)               <shlomo@cs.umass.edu>

                       

Attachments

 

  1. Gap Analysis: Human Decision Making
  2. Gap Analysis: Preferences
  3. Gap Analysis: Time Criticality and Uncertainty
  4. Gap Analysis: Multi-Agent Systems
  5. Gap Analysis: Overall Summary