UNCLASSIFIED Reynard Proposers Day IARPA-BAA-09-05 Overview Dr Rita Bush Intelligence Advanced Research Projects Activity IARPA Office of the Director of National Intelligence ODNI 4 19 2009 UNCLASSIFIED 1 UNCLASSIFIED Disclaimer This presentation is provided solely for information and planning purposes The Proposers' Day Conference does not constitute a formal solicitation for proposals or proposal abstracts 4 19 2009 UNCLASSIFIED 2 UNCLASSIFIED Today's Topics Program Overview Program Metrics and Milestones Award Information Eligibility Information Application Review Information 4 19 2009 UNCLASSIFIED 3 UNCLASSIFIED Reynard Program Proposers Day PROGRAM OVER VIEW 4 19 2009 UNCLASSIFIED UNCLASSIFIED Introduction Growth in Virtual Worlds by Year 350 300 Active VWs 250 200 150 100 50 0 New VWs Cumulative VWs UNCLASSIFIED UNCLASSIFIED Reynard Hypothesis Starting from the premise that Real World RW characteristics are reflected in Virtual World VW behavior the Reynard program seeks to determine behavioral indicators in the realm of Massive Multiplayer Online Games MMOGs and VWs that are predictive of RW characteristics of the users UNCLASSIFIED 6 UNCLASSIFIED Behavioral Indicators Quantitative statements regarding RW characteristics of the user based on VW cues in a given context Context Quantitative Assessment In EVE R Online across all servers 90% of the users who display a Russian flag in their user profile are RW Russian VW Cue RW Characteristic Example In Second Life TM Just Leap In TM World of Warcraft TM and Warhammer TM across all servers 80% of users who chat in punctuated English sentences are 35 years or older and have at least a high-school education 4 19 2009 UNCLASSIFIED 7 UNCLASSIFIED Reynard Virtual World Definition Virtual Worlds or VW will be used to refer to both MMOGs and VWs MMOGs typically quest-oriented VWs primarily social UNCLASSIFIED UNCLASSIFIED Reynard Virtual World Definition Has the following characteristics graphical landscape avatars persistent massive goals Not constrained by client hardware PCs consoles mobile phones Enables or inhibits expression of user behaviors UNCLASSIFIED UNCLASSIFIED Related Research Recent research indicates that RW behaviors do emerge in VWs e g the norms for politeness Much published research is qualitative Peer-reviewed quantitative research just emerging UNCLASSIFIED UNCLASSIFIED Heuristics Example heuristics Person is a RW adult or a RW child based on complexity of their vocabulary Person is a technologist or scientist in the RW based on names of groups or guilds Person is Russian based on the national flag shown in their user profile www Eve-Online com UNCLASSIFIED UNCLASSIFIED Example Research Three example quantitative studies Personal space use in Second Life TM World of Warcraft TM guild size Economic decision making in Second Life TM Customer Spending In-World 2 09 500 001 - 1 000 000 L$ 50 001 - 100 000 L$ 5 001 - 10 000 L$ 501 - 2 000 L$ Total Customer Spending 0 100000 200000 300000 400000 500000 http secondlife com statistics economy-data php 4 19 2009 UNCLASSIFIED 12 UNCLASSIFIED Program Research Areas RW Attributes of interest Gender Approximate age child young adult older Economic status low middle upper Educational level primary secondary college Occupation management technical service labor educator military student Ideology or world view e g pacifist militancy multilateralism unilateralism etc Degree of influence Digital native vs digital immigrant Approximate physical geographic location e g North America South America Europe Asia Australia Native language e g English German French Chinese Japanese Korean Arabic Russian other Culture e g Western Asian Middle Eastern Russian Researchers are encouraged to propose additional attributes 4 19 2009 UNCLASSIFIED 13 UNCLASSIFIED Program Research Focus Literal mapping between RW and VW not always expected Example Furries in VW may never dress as animals in the RW But furries in VWs may have occupations in the creative arts http francisanderson wordpress com 2008 01 28 links-for-2008-01-29 4 19 2009 UNCLASSIFIED 14 UNCLASSIFIED Framework for Reynard Research Questions Groups Money and Economy Group Formation and Dynamics Individuals Things Avatars Do Communication Avatars and Representation Culture 4 19 2009 UNCLASSIFIED 15 UNCLASSIFIED Understanding Cultural Differences Looking for study of non-U S based players Compare similarities differences in motivation usage patterns and behaviors across multiple cultures www time com time photogallery 0 29307 1817271_1728550 00 html Sydney Morning Herald www smh com au news breaking gamers-should-get-an-earful-of-thisgadget 2006 05 18 1147545447078 html 4 19 2009 http news gotgame com tag girl-gamers UNCLASSIFIED 16 UNCLASSIFIED Specific Research Questions--Individuals Avatars and Representation What characteristics of an avatar may reveal something about the real-life person behind it What motivates avatar choice Can we predict when an individual might select certain characteristics over others Does avatar selection reveal something about an individual's real-life personal characteristics attitudes customs groups or culture Verbal Communication Can we determine whether the person is a native-language speaker of the language e g English being used Is the use of language in text chat characterized by a signature similar to that found in other studies of authorship Can the path of viral ideas be traced through the chat history of the residents as it apparently can in exclusively text-based media which may then allow inferences about the degree of influence of the propagators of the ideas 4 19 2009 UNCLASSIFIED 17 UNCLASSIFIED Specific Research Questions--Individuals Nonverbal Communication When and why is nonverbal communication used Does the use of nonverbal communication cues in VWs vary from RW culture to RW culture What if anything might the use of VW nonverbal communication reveal about the RW person Things Avatars Do 4 19 2009 To what extent might cues in activities provide indicators about the RW person Do individuals from different RW cultures choose to engage in different sorts of VW activities UNCLASSIFIED 18 UNCLASSIFIED Specific Research Questions--Groups Group Formation What types of groups form what are their characteristics and what supports their continuance Does group membership reveal anything about the RW individuals who belong to them 4 19 2009 UNCLASSIFIED 19 UNCLASSIFIED Specific Research Questions--Groups Group Dynamics Given the multi-national nature of some VWs are individuals from different countries more likely to form relationships than in the RW Do individuals recognize others as from the same culture or different cultures Do VW residents establish more or less numerous relationships than in their RW relationships To what extent are VW relationships driven by RW relationships 4 19 2009 UNCLASSIFIED 20 UNCLASSIFIED Specific Research Questions--Economics Economics Do certain economic decisions or activities vary depending on the culture or nationality of the user Money Does handling of VW currency vary depending upon the RW culture of the user What inferences can be drawn about the RW individual based on their treatment of VW goods and currencies 4 19 2009 UNCLASSIFIED 21 UNCLASSIFIED Other Research Proposal Considerations Demonstrate knowledge of VW and its suitability for the research Non-U S -based players - data sources Overall demographics of the VW Demographically heterogeneous set of VWs vs single demographically homogeneous VW 4 19 2009 UNCLASSIFIED 22 UNCLASSIFIED Other Research Proposal Considerations Minimum of 4 RW attributes Multiple hypotheses or approaches for deriving behavioral indicators 4 19 2009 UNCLASSIFIED 23 UNCLASSIFIED Out Of Scope Large investment in construction of new VWs New VW software applications VW software architectures Development of new interface mechanisms such as hardware peripherals that are alternatives to the keyboard and mouse Research on PC console or mobile platform improvements for VWs Enhancing the game or VW experience through use of artificial intelligence or improved graphics Study of the impact of the VW environment on RW attitudes beliefs and behaviors 4 19 2009 UNCLASSIFIED 24 UNCLASSIFIED Acceptable Research Methodologies Some example research methods include Ethnographic methods such as participant observation or interviews in VWs Experimental studies in VWs Experimental studies in a Real-World laboratory environment with a stand-alone VW s Analysis of log data from VWs Large-scale survey research Case Studies Specify mitigation strategy for the weaknesses of chosen method Validate the RW characteristics of subjects 4 19 2009 UNCLASSIFIED 25 UNCLASSIFIED Other Research Method Considerations - 1 Must provide own virtual and or real laboratory facilities Specify Number of human subjects Number of unique characters and or end-users Planned length of the data gathering period Quantitative research methods Qualitative methods may be used as an adjunct to inform the direction of the quantitative research or to help interpret the results of the quantitative research Sample size Control for experiment farming 4 19 2009 UNCLASSIFIED 26 UNCLASSIFIED Other Research Method Considerations - 2 Technique s for safeguarding privacy An important evaluation criteria in proposal review Safeguard data sets from accidental release or malicious intrusions Must comply with all U S laws End User License Agreements EULA Terms of Service ToS and Privacy Policies of the VW s Must include appropriate techniques for safeguarding Personally Identifiable Information PII including avatar names See the section on Civil Liberties and Privacy Protection Measures for more details 4 19 2009 UNCLASSIFIED 27 UNCLASSIFIED Other Research Method Considerations - 3 IARPA will not receive any raw data National Institute of Standards and Technology NIST is Reynard's Validation Team Data set access may be provided to NIST as needed NIST will retain data access no longer than 12 months after the conclusion of Period of Performance 4 19 2009 UNCLASSIFIED 28 UNCLASSIFIED Civil Liberties and Privacy Protection Measures Must address the following as applicable Comply with federal policy for protection of human subjects in research Receive approval from their Institutional Review Board Obtain informed consent from subjects of active experimentation Employ techniques to protect privacy and confidentiality Compliance with the End User License Agreement EULA Terms of Service ToS and Code of Conduct CoC Annual review by Reynard PM and ODNI Civil Liberties and Privacy Office 4 19 2009 UNCLASSIFIED 29 4 19 2009 UNCLASSIFIED Questions UNCLASSIFIED 30 UNCLASSIFIED Reynard Program Proposers Day Program Metrics Milestones 4 19 2009 UNCLASSIFIED 31 UNCLASSIFIED Behavioral Indicators again Quantitative statements regarding RW characteristics of the user based on VW cues in a given context Context Quantitative Assessment In EVE R Online across all servers 90% of the users who display a Russian flag in their user profile are RW Russian VW Cue RW Characteristic Example In Second Life TM Just Leap In TM World of Warcraft TM and Warhammer TM across all servers 80% of users who chat in punctuated English sentences are 35 years or older and have at least a high-school education 4 19 2009 UNCLASSIFIED 32 UNCLASSIFIED Research Program Metrics o 3 years 2 Phases of 18 months each o Phase 1 and Phase 2 Metrics RW Characteristic Target Minimum Probability Value Phase 1 Target Minimum Probabillity Value Phase 2 Gender 0 75 0 90 Approximate Age minor young adult older 0 70 0 85 All other RW Characteristics 0 60 0 75 Performers should also report on the confidence limits associated with the estimated probabilities Proposers should suggest additional specific metrics that will enable the program manager to assess their progress 4 19 2009 UNCLASSIFIED 33 UNCLASSIFIED Reynard Research Milestones Milestones 1 2 3 4 5 6 7 8 Months After Kickoff 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Phase 1 Phase 2 Phase 1 Kickoff Site Visit #1 Pilot Test by Performer Site Visit #2 Assess Progress PI Meeting w Final BIs Reported Phase 1 Final Report Phase 2 Kickoff Site Visit #3 Pilot Test by Performer Site Visit #4 w Prelim BIs Reported PI Meeting w Final BIs Reported Phase 2 Final Reports The Reynard Program is envisioned to begin in the fourth quarter of Fiscal Year 2009 and end by 2012 4 19 2009 UNCLASSIFIED 34 4 19 2009 UNCLASSIFIED Questions UNCLASSIFIED 35 4 19 2009 UNCLASSIFIED Reynard Program Proposers Day A WA RD INF ORMA TION UNCLASSIFIED 36 UNCLASSIFIED Award Plan 3-year Program starting FY4Q2009 Phase 1 - Base Period - 18 months Phase 2 - Option Period - 18 months Criteria for moving to Phase 2 sufficient progress in achieving Phase 1 metrics Multiple awards anticipated depending upon quality of the proposals received availability of funds 4 19 2009 UNCLASSIFIED 37 UNCLASSIFIED Reynard Program Proposers Day ELIGIBILITY INFORMATION 4 19 2009 UNCLASSIFIED 38 UNCLASSIFIED Eligible Applicants Collaborative efforts teaming strongly encouraged Content communications networking and team formation responsibility of proposers Foreign organizations and or individuals may participate Must comply with Non-Disclosure Agreements Security Regulations Export Control Laws etc as appropriate 4 19 2009 UNCLASSIFIED 39 UNCLASSIFIED Ineligible Organizations Other Government Agencies Federally Funded Research and Development Centers FFRDCs University Affiliated Research Centers UARCs and any organizations that have a special relationship with the Government including access to privileged and or proprietary information or access to Government equipment or real property are not eligible to submit proposals under this BAA or participate as team members under proposals submitted by eligible entities 4 19 2009 UNCLASSIFIED 40 UNCLASSIFIED Reynard Program Proposers Day APPLICATION REVIEW INFORMATION 4 19 2009 UNCLASSIFIED 41 UNCLASSIFIED Evaluation Criteria Evaluation criteria in descending order of importance are Overall Scientific and Technical Merit Effectiveness of Proposed Work Plan Relevance to IARPA Mission and Reynard Program Goals Relevant Experience and Expertise Cost Realism 4 19 2009 UNCLASSIFIED 42 UNCLASSIFIED Meeting and Travel Requirements Plan for 1 Program Kick-off meeting and 2 Program Review in each phase in Metropolitan DC area to facilitate an open exchange among all Program participants Each meeting 2-3 days Periodic site visits 4 19 2009 UNCLASSIFIED 43 UNCLASSIFIED Human Subjects Must provide evidence of or a plan for review by an Institutional Review Board IRB upon final proposal submission to IARPA Allot ample time to complete the approval process No IARPA funding can be used towards human subjects research until ALL approvals are granted 4 19 2009 UNCLASSIFIED 44 UNCLASSIFIED Publication Approval Pre-publication approval may be required for sensitive information Any award may include a requirement for pre-pub review Soft copy to the IARPA Reynard Program Manager and the Contracting Officer Representative COR 4 19 2009 UNCLASSIFIED 45 UNCLASSIFIED Reporting Minimum requirement monthly technical and financial status reports Final report 4 19 2009 UNCLASSIFIED 46 UNCLASSIFIED Point of Contact Dr Rita M Bush Program Manager IARPA Incisive Analysis Office Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington DC 20511 Phone 301-226-9100 Fax 301-226-9137 Electronic mail dni-iarpa-baa-09-05@ugov gov include IARPA-BAA-09-05 in the Subject Line Website www iarpa gov UNCLASSIFIED Thank You Any Final Questions UNCLASSIFIED Second Life TM SL TM and Lindex TM and are trademarks of Linden Labs World of Warcraft TM is a trademark of Blizzard Entertainment Just Leap in TM is a trademark of Leap In Entertainment WarHammer TM is a trademark of EA Mythic Entertainment EVE R Online is a registered trademark of CCP hf HIPIHI TM is a trademark of HIPIHI Co Ltd Kiwi Heroes TM is a trademark of Massively Me Inc Lineage II R and The Chaotic Throne TM are trademarks of NCSoft Corp 4 19 2009 UNCLASSIFIED 49 4 19 2009 UNCLASSIFIED                     National Security Archive    Suite 701  Gelman Library  The George Washington University    2130 H Street  NW  Washington  D C  20037    Phone  202 994‐7000  Fax  202 994‐7005  nsarchiv@gwu edu