Adaptive Intelligent agent in real-time strategy games

ADAPTIVE INTELLIGENT AGENT IN REAL-TIME STRATEGY GAMES

A Hybrid Online Case-Based Planning &Reinforcement Learning Approach

Omar Enayet

Amr Saqr

Ahmed Atta

Abdelrahman Al-Ogail

Dr. Mostafa

Aref

Dr. Ibrahim Fathy

PROJECT MEMBERS

Project Research Area & Domain Motivations. Problem Definition Objectives Related Work Our Methodology.

Offline Stage. Online Stage.

Testing and Results. Conclusion and Future Work. Demo. References.

AGENDA

AI Learning

Make the machine learn.

AI Planning

Plan then re-plan according to new

givens.

Knowledge

SharingLet everyone know instantly what you

knew through experience.

PROJECT RESEARCH AREA

RTS GamesReal-Time Strategy Games.

Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions

PROJECT DOMAIN

RoboticsFor interest for

military which uses battle simulations in training programs.

War Simulation

For the corporation of robots.

Experimental Relevance

They constitute well-defined environments

to conduct experiments.

MOTIVATIONS

Experience Loss Static ScriptsComputer AI relies on

static scripting techniques.

The Absence of sharing experience costs a lot

PROBLEM DEFINITION

Predictability Non-Adaptability

Computer Opponent doesn’t adapt to

changes in human actions.

Computer Opponent actions easily

predicted.

PROBLEM CHALLENGES

Adaptive A.I.Making the Computer

Opponent adapt to changes like human do.

Mobile Experience

Import/Export your experience !

OBJECTIVES

•Eric Kok introduced : Adaptive Reinforcement Learning Agents in RTS Games, which merged BDI Agents technology with Reinforcement Learning, 2009

•Santi Ontanon introduced Darmok 2 which is an Online Case-Based Planning system designed to play Wargus in 2010.

•M.Johansen devised a CBR/RL system for learning micromanagement in real-time strategy games, 2009

RELATED WORK

ARCHITECTURE OVERVIEW

CASE REPRESENTATIONCase

3

0.6Performance

Behavior

BuildBase1TrainForce(TinyLandForce)TrainForce(TinyAirForce)Attack(NearWoodPeon)

Snippet

UnitExist(Peon)

Preconditions

PlayerBuildings > 0

Alive conditions

Rush-AttackStrategy

WinWargusGoal

NumberOfPlayerPeons = 10NumberOfPlayerCanonTower = 3

PlayerHasFortress = 1..

Shallow Features

PathExists = 1DistanceToEmeny = 40

.

.

Deep Features

0.8Prior Confidence

BeginningSituation

Eligibility

PERCEPTION

GAME STATE ANALYZER

Offline Stage:Learning from

human demonstration before shipping

game

CASE ABSTRACTION

CASE ABSTRACTION (CONT’D) Simplify case complexity. Increase the flexibility.

Abstractor

Point(10, 137) Unguarded Region

CASE ACQUISITION

Generates Cases from Human’s game play.

AcquisitionAbstract Trace Casebase

CASE ACQUISITION (CONT’D)

Online Stage:Learning what’s the

best to do while playing

CASE RETRIEVAL

CASE RETRIEVAL – CONT’D

RetrieverCase Base

Goal

Case

CASE ADAPTATION

Adapts Behaviors according to current game state. Removal of unnecessary actions. Adaptation for unsatisfied preconditions.

AdaptationBehavior Adapted Behavior

CASE ADAPTATION(CONT’D)

PlanBuild(Barracks)

PlanBuild(Great-Hall)

Train(Peon)Build(Barracks)

ONLINE PLAN EXPANSION & EXECUTION

ONLINE PLAN EXPANSION & EXECUTION (CONT’D)

ACTION CONTROLLER

CASE CONCRETIZATION

CASE CONCRETIZATION (CONT’D) Adapt the abstract actions to suit current

situation.

Concreter Point(10, 137)Unguarded Region

CASE REVISION

CASE REVISION (CONT’D)

Uses reinforcement learning, TD-learning SARSA( )λ

ReviserCase

Used Case Evaluation

TESTING AND RESULTS

Offline Le

arning

Adap

tation

Retriev

al

Abstr

action

Concre

tizati

on

Build

ing Pl

acemen

t

Respon

ds to

attack

s

Attack

s

Resourc

e Gath

ering

Traini

ng Fo

rces

0102030405060708090

100

Success Percentage

CONCLUSION A Hybrid Architecture of case based reasoning and

reinforcement learning was introduced to play strategy games.

The architecture merged online case based planning with Sarsa(λ) with eligibility traces. The system showed promising simulation of human behavior; however it still needs a lot extra effort and testing to become industrially capable.

Also, the concept of an abstract case base was introduced which opens the door for generic AI engines for games which is never implemented till the date of writing of this document.

Demo!

FUTURE WORK1) Cooperative AI Agents.

2) Opponent Modelling.

3) Strategy visualization tool.

4) Generic situation assessment.

5) Learn weights of Game State through neural network.

6) Online I-Strategizers.

7) Generic Abstraction/ Concretization.

WEB RESOURCES To get introduced for the whole project journey, evolution,

technical summaries, presentations, discussions, meeting minutes and others visit project blog:

http://rtsairesearch.wordpress.com/

For full materials of papers, technical summaries, documentations, articles, external links and running version of WARGUS (our test best) use the repository link:

svn checkout http://rtsairesearch.googlecode.com/svn/trunk/ rtsairesearch-read-only

For downloading the latest source code for the I-Strategizer Project, please use the following:

svn checkout http://istrategizer.googlecode.com/svn/trunk/ istrategizer-read-only

REFERENCES [1] Martin Johansen Gunnerud. A CBR/RL system for learning

micromanagement in real-time strategy games. In Norwegian University of Science and Technology, 2009

[2] Santi Ontañón, Neha Sugandh, Kinshuk Mishra, Ashwin Ram. On-Line Case-Based Planning. In Computational Intelligence, 26(1):84-119, 2010.

[3] Brain Schwab. AI Game Engine Programming. Charles River Media, 2009.

[4] Santi Ontañón and Kane Bonnette and Prafulla Mahindrakar and Marco A. G´omez-Mart´ın and Katie Long and Jainarayan Radhakrishnan and Rushabh Shah and Ashwin Ram. Learning from Human Demonstrations for Real-Time Case-Based Planning. In AAAI 2008

[5] Ralph Bergmann and Wolfgang Wilke, On the role of abstraction in case-based reasoning

REFERENCES – CONT. [6] Kinshuk Mishra, Santiago Santi Ontañón, and Ashwin Ram.

Situation Assessment for Plan Retrieval in Real-Time Strategy Games. In 9th European Conference on Case-Based Reasoning (ECCBR 2009), Trier, Germany.

[7] Neha Sugandh and Santiago Santi Ontañón and Ashwin Ram . On-Line Case-Based Plan Adaptation for Real-Time Strategy Games. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008)

[8] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, An Introduction. MIT press, 2005.

[9] Wikipedia, the free encyclopedia. http://www.wikipedia.com

[10] Michael Buro, Call for Research in RTS AI, AAAI 2004