A SENSITIVITY ANALYSIS OF EVOLUTIONARY ALGORITHMS …

A SENSITIVITY ANALYSIS OF EVOLUTIONARY ALGORITHMS IN GENERATING SECURE CONFIGURATIONS (VULNERABILITIES)

Shuvalaxmi Dass

Akbar Siami Namin

CYBER-PHYSICAL SYSTEMS

• Cyber-Physical Systems are emerging technology and platforms in modern ubiquitous computing.

• It is an integration of computation, networking, and physical devices in a closed form.

• It is composed of mixture of different hardware and software components.

• CPS = Hardware components + software components

• HW: sensors, actuators, and embedded systems

• SW: varied collections of proprietary and commercial third-party software products which are in charge of control and monitoring.

INTRODUCTION

• Integration of highly complex cyber-physical components .

• Interaction between complex characteristics often put the privacy and security of the system at risk.

• We focus on the security risks caused by the software component of CPS.

• Particularly, vulnerabilities caused due to improper parameter settings in the software product.

MOTIVATION

• Problem: From software testing perspective, enumerating all parameter settings and then verifying whether the given software is vulnerable to certain attacks is infeasible.

• Solution: A systematic configuration testing framework and mechanism needed in identifying a reduced set of configuration test inputs to inspect the given software.

• How: Develop a specific adequacy criterion for covering vulnerabilities and then examine the software under test for those selected vulnerabilities.

• We extend the initial idea of our paper1, in adapting evolutionary algorithms on vulnerability coverage, to explore it further through Sensitivity Analysis (SA).

• 1. Shuvalaxmi Dass and Akbar Siami Namin. 2020. Vulnerability coverage for adequacy security testing. In Proceedings of the 35th Annual ACM Symposium on Applied Computing (SAC ’20).

PREVIOUS WORK

• We proposed the concept of vulnerability coverageas an adequacy criterion for testing the wide-rangingvulnerabilities present in the underlying softwareapplications.

• Main goals:

• Identify a set of vulnerability patterns/configurations that belong to a certain level of CVSS score.

• Key idea: utilize CVSS as a fitness metric.

• Implemented GA and PSO, to generate a pool of vulnerability patterns for adequacy testing of underlying system.

• then choose the representative sets of vulnerabilities with similar vulnerability vector patterns for further inspection of the SUT.

Figure shows a snapshot of vulnerability patterns generated by

GA along with their CVSS scores.

CONTRIBUTIONS

• Conduct a sensitivity analysis on parameters involved in evolutionary algorithms.

• We observed that the parameters involved in GA such as mutation rate and population size have little to no impact on the performance of genetic algorithms.

• Unlike GAs, we observed that the parameters involved in Particle Swarm Optimization (PSO) play an important role in the generation of number of instances of secure configurations.

BACKGROUND

Genetic Algorithm(GA)

• A search-based optimization technique that mimics biological process of evolution: reproduction and natural selection, to solve for the ‘fittest’ solutions.

• 3 main operations in breeding process : selection, crossover, and mutation.

• In our domain:

config/chrom: [AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H]

fitness score: 9.8 (cvss score)

Elements of GA

Process

Particle Swarm Optimization(PSO)

• Based on the social behavior of bee swarms and bird flocking.

• Each particle in the swarm has:

1) data: possible solution,

2) velocity: how much the data can be modified, and

3) fitness value: evaluated by the fitness function.

• These parameters guide the particles (i.e., birds) towards the global optimum by updating swarms.

• In every iteration, each particle is updated based on two best scores:

1) personal best fitness score it has achieved so far, and

2) the global best fitness score achieved by any particle in the swarm so far.

Common Vulnerability Scoring System (CVSS)

• captures the principal characteristics of a vulnerability and produce a numerical score reflecting its severity.

• composed of 3 main metric groups:

• (1) Base,

• (2) Temporal, and

• (3) Environmental,

each consisting of a set of sub-metrics.

• Without loss of generality and to demonstrate the feasibility of the proposed approach, the GA and PSO algorithms are only applied to the Base metric group.

SENSITIVITY ANALYSIS (SA)

PSO

• Parameters that cooperate in order to locate the best possible configurations:

• Velocity

• Fitness

• In order to understand whether any of these parameters contribute more to the final best solution or if there is any interaction between them, we performed a sensitivity analysis on three scenarios, as follows:

1. Keeping particle fitness value range constant between[2,3)

2. Keeping particle velocity value range constant as [0,1].

3. Changing both velocity and particle fitness value range simultaneously.

GAWe carried out SA on GA to see the effect on the number of secure (score = 2.0) patterns generated by:

1. varying values of mutation rate from 0.2 to 1.0

2. varying values of population size from 100 to 500 with an interval of 100.

RESULTS

PSO

Max # best particles in [0; 1] = 11

Min # best particles in [5; 6] = 1

Max # best particles in [2; 3) = 14

Min # best particles in [8; 9) = 3

Max # best particles in [6; 7) : [4; 5]

= 9

GA

No clear pattern observed

CONCLUSION

• The results of SA indicate that the GA’s parameters (mutation rate and population size) are less influential on the performance of the algorithm.

• On the other hand, the parameters involved in PSO algorithms are more influential in generating more secure configurations.

• In other words, changing the values of velocity and fitness values can affect the number of instances generated by the algorithm and thus should be taken into account when designing such evolutionary-based MTD platforms.

• Furthermore, our results show that employing PSO might be a better choice compared to GA because of the influence of changing its parameters on improving the performance/

RELATED WORK

• Evolutionary algorithms have been widely used in the field of security. Crouse and Fulp developed Moving Target Defense (MTD) platforms using Genetic Algorithm to ensure security in computer systems. [A moving target environment for computer configurations using genetic algorithms. In 2011 4th Symposium on Configuration Analytics and Automation (SAFECONFIG), pages 1–7, Oct 2011.]

• Crouse and Fulp improved their MTD approach further by introducing a modified variation of GA-based MTD technique and proposed a new approach called chromosome pool management. This technique was introduced to address the issue of stagnancy in the pool of configurations. [Improving the diversity defense of genetic algorithm-based moving target approaches. 2012.]

• John and Furp compared and contrasted two different genetic algorithms to conduct Moving Target Defense, namely GA+PVM and GA+PDM. In GA+PVM, mutation operator mutates the parameter values based on its type (integer, option, and bit); Whereas, in GA+PDM, mutation operator changes the domain of the parameter using machine learning algorithms by removing the insecure setting from the parameter’s domain.[Evolutionary based moving target cyber defense. In Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, GECCO Comp ’14, pages 1261–1268, New York, NY, USA, 2014. ACM.]

• Do et al. conducted sensitivity analysis of the finite moving averages test, which was used for detecting cyber physical attacks on Supervisory Control And Data Acquisition (SCADA) systems. This analysis calculates the likelihood of wrong decisions with respect to variation of operational parameter. These results are then utilized to detect an attack scenario on a SCADA water network. [V. Long Do, L. Fillatre, and I. Nikiforov. Sensitivity analysis of the sequential test for detecting cyber-physical attacks. In 2015 23rd European Signal Processing Conference (EUSIPCO), pages 2261–2265, 2015.]

THANK YOU!