11. Optimum Design Concepts - CAUisdl.cau.ac.kr/education.data/DOEOPT/11.opt.design.concepts.pdf · 11. Optimum Design Concepts. SCHOOL OF MECHANICAL ENG.-1-CHUNG-ANG UNIVERSITY Introduction

Hae-Jin ChoiSchool of Mechanical Engineering,

Chung-Ang University

11. Optimum Design Concepts

SCHOOL OF MECHANICAL ENG.

CHUNG-ANG UNIVERSITY-1-

Introduction to basic ideas, concepts, and

theories used for optimization

Once the problems from different fields

have been transcribed into mathematical

statements, they become the same

mathematical problem to solve using

optimization theories.

Optimization methods are categorized into

indirect or direct (search) methods

Indirect methods: seeking solution from

optimality criteria

Direct methods: seeking solution from

initial estimation and iterative process

DOE and Optimization

Introduction

Optimization

methods

Indirect

methods

Direct

methods

Constrained

problem

Unconstrained

problem



Global (absolute) Minimum

A function f(x) of n variables has global

(absolute) minimum at x* if f(x*)≤ f(x) for all

x in the feasible region.

Local (relative) Minimum

A function f(x) of n variables has a local

(relative) minimum at x* if f(x*)≤ f(x) for all

x in a small neighborhood N of x* in the

feasible region, where


Global and Local Minimum

{ | with }N S *x x x x

x

f(x)

x

f(x)

x=a x=b



Geometrically, the gradient vector is normal to the tangent plane at

the point x* for a function. -> the direction of maximum increase in

the function

At a given point (x*), the gradient vector is


Gradient Vector

1

2

1 2

( )

( )( ) ( ) ( )

( ) ....

....

( )

T

n

n

f

x

ff f f

f xx x x

f

x

*

*

* * **

*

x

xx x x

x

x



Hessian matrix is obtained

differentiating the gradient vector

once again where all derivatives are

calculated at a given point x*

Hessian is an n x n matrix usually

denoted as H or .

Hessian is always an symmetric

matrix since

Hessian plays an important role in the

sufficiency condition of optimality


Hessian Matrix

2

2 2 2

2

1 1 2 1

2 2 2

2

2 1 2 2

2 2 2

2

1 2

where i=1 to n and j=1 to n

or

...

...

... ... ... ...

...

i j

n

n

n n n

f

x x

f f f

x x x x x

f f f

x x x x x

f f f

x x x x x

H

2 f

2 2

i j j i

f f

x x x x



Approximated function by polynomial in a neighborhood of any

point in terms of its value and derivative.

Taylor’s series expansion about the point x* is

where R is remainder (higher order terms)

Let x-x*=d


Taylor Series Expansion

* 2 ** * * 2

2

( ) 1 ( )( ) ( ) ( ) ( )

2

df x d f xf x f x x x x x R

dx dx

* 2 ** * 2

2

( ) 1 ( )( ) ( )

2

df x d f xf x d f x d d R

dx dx



For two variables

In general


Taylor Series Expansion

* * * ** * * *1 2 1 2

1 2 1 2 1 1 2 2

1 2

2 * * 2 * * 2 * ** 2 * * * 21 2 1 2 1 2

1 1 1 1 2 2 2 22 2

1 1 2 2

( , )) ( , ))( , ) ( , ) ( ) ( )

( , ) ( , ) ( , )1( ) 2 ( )( ) ( )

2

f x x f x xf x x f x x x x x x

x x

f x x f x x f x xx x x x x x x x R

x x x x

*

1( ) ( ) ( ) ( ) ( )

2

1( ) ( )

2

T T

T T

f f f R

or

f f f R

* * * *

*

x x x - x x - x H x - x

x + d x d d Hd



Quadratic form is a special nonlinear function having only second-order

terms. e.g.,

In general

or

Replacing P with a symmetric matrix A

H in the Taylor Series Expansion is associated with the quadratic form


Matrix of Quadratic Form

2 2 2

1 2 3 1 2 2 3 3 1( ) 2 3 2 2 2f x x x x x x x x xx

1 1

1( ) are known constants

2

n n

ij i j ij

i j

f p x x where px

1( )

2

is called matix of quadatic form

Tf

where

x x Px

P

1 1( )

2 2

T Tf x x Px x Ax


CHUNG-ANG UNIVERSITY-8-DOE and Optimization

Example: Matrix of Quadratic Form

Identify a matrix associated with the quadratic form

Dividing the coefficients equally between pij and pji,

2 2 2

1 2 3 1 1 2 1 3 2 2 3 3

1( , , ) (2 2 4 6 4 5 )

2F x x x x x x x x x x x x

1 1

1 2 3 2 1 2 3 2

3 3

2 2 4 2 0.5 11 1 1

( ) [ ] 0 6 4 [ ] 1.5 6 62 2 2

0 0 5 3 2 5

T

x x

F x x x x x x x x

x x

x x Px

1

1 2 3 2

3

2 1 21 1

( ) [ ] 1 6 22 2

2 2 5

T

x

F x x x x

x

x x Ax



Matrix of Quadratic Form

If for all x except x=0, then the

quadratic form is positive definite

If for all x except x=0, then the

quadratic form is negative definite

If for all x (at lease one that

makes ) then the quadratic form

is positive semidefinite

If for all x (at lease one that

makes ) then the quadratic form

is negative semidefinite

0Tx Ax

0Tx Ax

0Tx Ax

0Tx Ax

x 0

0Tx Ax

x 0

0Tx Ax

2 2 2

1 2 3

1 2 3

2 0 0

0 4 0

0 0 3

2 4 3 0

for all unless 0

Therefore is positive definite

T x x x

x x x

A

x Ax

x

A

2 2 2

1 2 1 2 3

2 2

3 1 2

1 2 3

1 1 0

1 1 0

0 0 1

( 2 )

( ) 0

for all and

0 when and 0

Therefore is negative semidefinite

T

T

x x x x x

x x x

x x x

A

x Ax

x

x Ax

A



F(x)=xTAx is a positive definite if and only if all eigenvalues of A are

strictly positive, i.e., λi > 0, i=1 to n

F(x) =xTAx is a positive semidefinite if and only if all eigenvalues of A

are non-negative, i.e., λi ≥ 0, i=1 to n (at least one eigenvalue is

zero)

F(x) =xTAx is a negative definite if and only if all eigenvalues of A are

strictly negative, i.e., λi < 0, i=1 to n

F(x) =xTAx is a negative semidefinite if and only if all eigenvalues of A

are non-positive, i.e., λi ≤ 0, i=1 to n (at least one eigenvalue is

zero)

F(x) =xTAx is indefinite if some λi < 0 and some other λi > 0


Method of Checking - Eigenvalues



To differentiate the Quadratic Form with respective to xi

Therefore, the Gradient of the quadratic form is

Differentiating once again with respect to xj, we get

The Hessian matrix is


Differentiation of a Quadratic Form

1 1

1( )

2

n n

ij i j

i j

F a x xx

1

n

ij j

ji

Fa x

x

( )F x Ax2

ij

i j

Fa

x x

H A



Calculate gradient and Hessian of following quadratic form


Example

2 2 2

1 1 2 1 3 2 2 3 3

1( ) (2 2 4 6 4 5 )

2F x x x x x x x x xx

1

1 2 3 2

3

1

2

3

2 1 21 1

( ) [ ] 1 6 22 2

2 2 5

2 1 2

( ) 1 6 2

2 2 5

2 1 2

1 6 2

2 2 5

T

x

F x x x x

x

x

F x

x

x x Ax

x Ax

H = A



Minimization of f(x) without any constraint on x.

Not a practical case, but good for understanding constrained

optimization concept.

Optimality Condition for Function of Single Variable

First-order necessary condition

Sufficient condition

Optimality Condition for Functions of Multiple Variables


Unconstrained Optimum Design Problem



Necessary condition for optimality: this

condition must be satisfied for a point to be

optimum

If any point does not satisfy the necessary condition, it

cannot be optimum.

Satisfaction of the necessary condition does not guarantee an

optimum point

Sufficient condition for optimality: this condition

provides tests to distinguish between optimum and

non-optimum points

If a candidate optimum point satisfies the sufficient

conditions, then it is indeed optimum

Even if no point that satisfies the sufficient condition, we

still have a chance to have an optimum.


Optimality Conditions

Optimum condition

Sufficient

condition

Necessary condition



* 2 ** 2

2

( ) 1 ( )( ) ( )

2

df x d f xf x f x d d R

dx dx


Necessary Condition for Function of Single Variable

Taylor series expansion near x*

Change in the function near x*

* 2 ** 2

2

( ) 1 ( )( ) ( )

2

df x d f xf f x f x d d R

dx dx

The change must be non-negative value (≥0) when x* is local minimum

Therefore, ignoring the remainder R

Since d could be positive or negative value,

f

*( )0

df xd

dx

*( )0

df x

dx

x* is stationary

point

that could be

maximum,

minimum, or

saddle point

The first term is the dominant term



Sufficient Condition for Function of Single Variable

2 *2

2

1 ( )

2

d f xf d R

dx

Since d2 is always positive, for x* to be minimum, ignoring

the remainder, R.

2 *

2

( )0

d f x

dx

Since f’(x)=0

Now, this term is dominant term



For the general case of a function of multiple variables, f(x) where x is an

n-vector,

The necessary condition using the multidimensional form of Taylor series

expansion is

Since the first term is dominant, the necessary condition is

With the second term the sufficient condition is


Optimality Condition for Functions of Multiple

Variables

* 1( ) ( ) ( *) ( *)

2

T Tf f f f R*x + d x x d d H x d

( *)f x 0

( *) is poisitive definiteH x

0Td H(x*)d



Find a local minimum point for the function

With necessary condition


Example: Local minima for a function

1 2

1 2

(4.0 06)( ) 250

Ef x x

x xx

2

1 2

2

1 2

2 2

1 2 1 2

2 2

1 2 1 2 1 2 1 2

1 2

4.0 061

0( )

4.0 06 0250

Solving the following two equations

4.0 06 0 250( ) 4.0 06 0

250( ) ( 250 ) 0

250

Substituting this t

E

x xf

E

x x

x x E and x x E

x x x x x x x x

x x

x

1 2

o the original equation,

1000 and 4

The stationary point is (1000, 4)

x x



Find a local minimum point for the function

(solution)

With necessary condition



1 2

1 2

(4.0 06)( ) 250

Ef x x

x xx

2

1 2

2

1 2

2 2

1 2 1 2

2 2

1 2 1 2 1 2 1 2

1 2

4.0 061

0( )

4.0 06 0250

Solving the following two equations

4.0 06 0 250( ) 4.0 06 0

250( ) ( 250 ) 0

250

Substituting this t

E

x xf

E

x x

x x E and x x E

x x x x x x x x

x x

x

1 2

o the original equation,

1000 and 4

The stationary point is (1000, 4)

x x

General approach for

solving simultaneous

non-linear equations

Newton-Raphson method

(See Appendix)



The Hessian Matrix

Checking sufficient condition at the stationary point (1000, 4)

H is positive definite since the two eigenvalues of the Hessian matrix are

0.006 and 500.002, which are all positive values.

Therefore, (1000, 4) is indeed local minimum, and the local minimum is

f(1000, 4) = 3000



2

1

2 3

11 2

2

21

4.0 06

21

x

xE

xx x

x

H

2

0.008 1(4.0 06)(1000,4)

1 5004000

EH






Appendix: Newton-Raphson

Numerical method for simultaneous non-linear equations.

For a one-variable system, the Taylor series approximation and resulting Newton-Raphson equations are:

For a two-variable system,

)()()()( 11 iiiii xfxxxfxf 1

( )

( )

ii i

i

f xx x

f x

f1,i1 f1,i x1,i1 x1,i f1,i

x1 x2,i1 x2,i

f1,i

x2x1,i1 x1,i

f1,if2,ix2

f2,if1,ix2

f1,ix1

f2,ix2

f1,ix2

f2,ix1

f2,i1 f2,i x1,i1 x1,i f2,i

x1 x2,i1 x2,i

f2,i

x2x2,i1 x2,i

f2,if1,ix1

f1,if2,ix1

f1,ix1

f2,ix2

f1,ix2

f2,ix1




1[ ]{ } { } i iZ x x f

n

ininin

n

iii

n

iii

x

f

x

f

x

f

x

f

x

f

x

f

x

f

x

f

x

f

Z

,

2

,

1

,

,2

2

,2

1

,2

,1

2

,1

1

,1

][

1, 2, ,{ }T

i i i n ix x x x

1 1, 1 2, 1 , 1{ }T

i i i n ix x x x

1, 2, ,{ }T

i i n if f f f

1, 1,

1, 1 1, 1, 1 1, 2, 1 2,

1 2

2, 2,

2, 1 2, 1, 1 1, 2, 1 2,

1 2

i i

i i i i i i

i i

i i i i i i

f ff f x x x x

x x

f ff f x x x x

x x

1[ ]{ } { } [ ]{ }, where [ ] i iZ x f Z x Z Jacobian matrix






Ex.Use the Newton-Raphson method to determine the roots of the equations. Use the initial guesses of x1 =1.5 and x2 = 3.5.

573

10

2

212

21

2

1

xxx

xxx

5.32)5.3)(5.1(6161 75.36)5.3(33

5.1 5.65.3)5.1(22

21

2

0,222

2

1

0,2

1

2

0,1

21

1

0,1

xxx

fx

x

f

xx

fxx

x

f

6.5(32.5) 1.5(36.75) 156.125Jacobian




5.210)5.3(5.1)5.1( 2

0,1 f

625.157)5.3)(5.1(35.3 2

0,2 f

The values of the functions can be evaluated at the initial guesses as

These values can be substituted to give

03603.2125.156

)5.1(625.1)5.32(5.25.11

x

84388.2125.156

)75.36)(5.2()5.6(625.15.32

x

The computation can be repeated until an acceptable accuracyis obtained.


11. Optimum Design Concepts - CAUisdl.cau.ac.kr/education.data/DOEOPT/11.opt.design.concepts.pdf · 11. Optimum Design Concepts. SCHOOL OF MECHANICAL ENG.-1-CHUNG-ANG UNIVERSITY Introduction

Documents