Jan 20, 2021
Optimisation Problems: Definition
Find values of a given set of decision variables:X=(x1, x2, …., xn) which maximises (or minimises) thevalue of an objective function: x0 = f(x1, x2, …., xn),subject to a set of constraints
Any vector X, which satisfies the constraints is calleda feasible solution and among them, the one whichmaximise (or minimise) the objective function iscalled the optimal solution
Maximise: x sin(x)subject to 0 ≤ x ≤ 30
Optimisation Problems: terminology
f(X)
X
local maximumsolution
global maximumsolution
( )
Neighbourhood ofsolution
globalmaximum value
Y
Optimisation Problems: Difficulties
For most of real world problems
An exact model cannot be built easily;
Number of feasible solutions grow exponentiallywith growth in the size of the problem.
A
C
D
B
E
F
Methods of optimisation
Mathematical optimisation
Based on Mathematical techniques to solve theoptimisation problem exactly or approximatelywith guarantee for quality of the solution;
Examples:
Simplex method: by far the worlds most widelyused optimization algorithm!
Lagrange multipliers, branch and bound,cutting planes, interior point methods, etc;
Methods of optimisation
Mathematical optimisation
+: Guarantee of optimality
- : Unable to solve larger instances of difficultproblems due to large amount of computationaltime and memory needed;
A
C
D
B
E
F
Constructive Heuristics
Using simple minded greedy functions to evaluatedifferent options (choices) to build a reasonablesolution iteratively (one element at a time);
Examples: Dijkastra method, Big M, Two phasemethod, Density constructive methods forclustering problems, etc;
Methods of optimisation
A
C
D
B
E
F
Constructive Heuristics
+: Ease of implementation;
-: Poor quality of solution;
-: Problem specific.
Methods of optimisation
Methods of optimisation (cont.)
Local Search algorithms
A neighbourhood search or so called local searchmethod starts from some initial (complete)solution and moves to a better neighbouringsolution until it arrives at a local optimum, onethat does not have a better neighbour.
Examples: k-opt for TSP, etc;A
C
D
B
E
F
Methods of optimisation (cont.)
Local Search algorithms
+: Ease of implementation;
+: Guarantee of local optimality usually in smallcomputational time;
+: No need for exact model of the problem;
-: Poor quality of solution due to getting stuck inpoor local optima;
Methods in G52AIM
Meta-heuristics
These algorithms guide an underlying heuristic /local search to escape from being trapped in alocal optima and to explore better areas of thesolution space;
Methods in G52AIM
Meta-heuristics
Local search based approaches
Hill climbing “Run uphill/downhill and hope you find the
top/bottom”
Simulated annealing “Shake it up a lot and then slowly let it settle”
Tabu search “Don’t look under the same lamp-post twice”
GRASP, VNS, etc
Methods in G52AIM
Meta-heuristics
Population based approach
Genetic algorithms “survival of the fittest”
Ant algorithms “wander around a lot and leave a trail”
Genetic programming Learn to program
Evolutionary algorithms, etc
Methods in G52AIM
Meta-heuristics
+: Able to cope with inaccuracies of data andmodel, large sizes of the problem and real-timeproblem solving;
+: Including mechanisms to escape from localoptima of their embedded local search algorithms,
+: Ease of implementation;
+: No need for exact model of the problem;
-: Usually no guarantee of optimality.
Problems in G52AIM
Problem domain: optimiation
Quality of solutions: given by evaluation function(objective function, fitness, etc)
Aim: minimize or maximize this objective
Initially focus on algorithms that might beclassed as “iterative improvement”
Take a candidate complete ‘solution’ and thentry to fix or repair it
Simplest version of this is “local search”
Local Search Methods
A neighbourhood function is usuallydefined by using the concept of a move,which changes one or more attributes of agiven solution to generate another solution.
Definition
A solution x is called a local optimum with respectto the neighbourhood function N, if f(x) < f(y) forevery y in N(x).
Local Search Methods
Local Search Methods
Why local search?
Exponential growth of the solution space for mostof the practical problems;
Ambiguity of the model of the problem for beingsolved with exact algorithms;
Ease of use of problem specific knowledge indesign of algorithm than in design of classicaloptimisation methods for a specific problem.
Local Search Methods
Elements
Representation of the solution;
Evaluation function;
Neighbourhood function
To define solutions which can be consideredclose to a given solution.
For example: For optimisation of real-valuedfunctions in elementary calculus, for a currentsolution x0, neighbourhood is defined as aninterval (x0–r, x0+r).
Local Search Methods
Elements
Neighbourhood search strategy
Random and systematic search;
Acceptance criterion
first improvement; best improvement; best ofnon-improving solutions; random criteria
Hill Climbing - Algorithm
1. Pick a random point in the search space
2. Consider all the neighbours of the current state
3. Choose the neighbour with the best quality andmove to that state
4. Repeat 2 thru 4 until all the neighbouring states areof lower quality
5. Return the current state as the solution state
Hill Climbing - Examples
Problem
TSP: A travelling salesman is visiting n cities
Constraint: He can visit each city only once, andback to the starting city
Objective: Find the shortest tour
Task
Apply the hill climbing method to solve TSP
Hill Climbing – Exercise
Hill Climbing – Exercise
How do we define the problem as a searchproblem?
What is a “point” in the search space?
How to represent a solution?
What is the search space?
Hill Climbing – Exercise
How do we design the HC for TSP?
What is your evaluation function?
What is your neighbour?
What is your search strategy?
What is your acceptance criteria?
Hill Climbing – Exercise
What’s the problem in HC?
First improvement
Within the neighbourhood, if a point x’ is animprovement then immediately terminate andjump straight to it
Best Improvement
Search the entire neighbourhood, and find thex’ that gives the largest improvement
What to do in tie-breaks?
Hill Climbing – local optima
How can bad local optima be avoided?
Summary
Optimisation problems
Definition
Methods
Local search algorithms
Elements
Hill climbing
Learning objectives
Terminology
Local vs. global optima
Neighborhood
Feasibility
Local search algorithms
Concepts
Elements
Basic hill climbing