An Alternative Approach for Formula Modelling in Security Metrics

AN ALTERNATIVE APPROACH FOR FORMULA MODELLING INSECURITY METRICS

Rodrigo Sanches Miani, Felipe Marques Pires and Leonardo deSouza MendesDepartment of Communication, School of Electrical and Computer Engineering, State University of Campinas

Av. Albert Einstein, 400, Cidade Universitria ”Zeferino Vaz”, Distrito Baro Geraldo, Campinas, SP, [email protected], [email protected], [email protected]

Keywords: Security metrics, Network security, Security analysis.

Abstract: This paper proposes an alternative approach to modelling the formula attribute within the context of securitymetrics. This approach seeks to correct past errors by treating a security metric like a set, and inserting acomponent that addresses the set intersection between the security elements. The work consists in to definethe model, explain the differences to the previous model andvalidate it, with examples from the metrics foundin literature and also with the results of a case study applied in Metropolitan Broadband Access Network inPedreira, a city located in the state of So Paulo, Brazil.

1 INTRODUCTION

A widely used concept in the information securityscope, is the security metric. Metrics can be definedas a set of measures that can generate a quantitativeapproach about a problem. (Lowans, 2002).

The primary goal of a metric is to convert rawdata into information capable of analysis. In theinformation security world, large organizations asCERT (Computer Emergency Response Team), SANS(SysAdmin, Audit, Network, Security) and NIST (Na-tional Institute of Standards and Technology) developand recommend the implementation of security met-rics.

The metrics are usually defined from a series ofattributes. Among them we highlight: purpose, fre-quency, data source, measures and formula. The for-mula attribute, in particular, is important to describethe calculations that will be performed to quantify themetrics in a numerical expression (Swanson et al.,2003). From the result of the formula, the metricsvalue or indicator is obtained, and it is usually ex-pressed in percentage terms.

This paper will present an alternative approachthat was proposed in (Miani et al., 2008) for themodelling of security metrics formulas, aiming to in-crease the reliability degree from the results obtainedby the formulas. We will show the differences be-tween the two approaches and a comparative study ofthe research results of a case study performed in the

Metropolitan Broadband Access Network of Pedreira.This paper is organized as follows. Section 2

brings some related works of security metrics. Thesection 3 presents the mainly motivations and definesthe concepts that will be studied in this work. In sec-tion 4 we will introduce the basis of the proposedmodel and the differences between the model that willbe compared. In section 5 we present one model ap-plication example and the results of a metrics imple-mentation case study developed in the MetropolitanBroadband Access Network of Pedreira. The section6 brings the conclusion and some future works.

2 RELATED WORKS

The study of security metrics and their applicationsin the IT scenarios are targets of several discussions(Rosenblatt, 2008). An increase in failure rate ofcomponents, discovered vulnerability in software andcommunication network attacks may cause a big con-cern on questions related to information security. Todeal with these problems it is necessary to invest in se-curity controls implementation and security policies.The characteristics of these investments must be care-fully accounted for and can be defined from measuresand analysis of information security structure. Thisprocess is formalized using a security metrics appli-cation (Weiss et al., 2005). Through a combination

381

of predefined objectives, collection and data analysis,metrics can indicate the actual level of security wemust aim at, directing the actions network adminis-trators must take to secure the network (Payne, 2006).

Jaquith (Jaquith, 2007), Swanson et al. (Swansonet al., 2003), Payne (Payne, 2006) and the ISO/IEC27002 (ISO, 2005) standard, contributed to the devel-opment and formalization of attributes that constitutea security metric.

Herrera (Herrera, 2005) examined the indicatorsdevelopment from the security metrics, affirming thatthere is no magical formula to establish the perfect in-dicator, each organization shall determine which indi-cators are useful according to their business and howto get to the results. However, the growth and the needfor security metrics has created a gap in the conceptsstandardization. Every security framework providesits own security indicator method and this fact mayaffect the development of security indicators.

The CVSS (Common Vulnerability Scoring Sys-tem) (Mell et al., 2007) is an initiative in this direc-tion. Its goal is the creation of standard indicators forsecurity vulnerabilities from the equations that are di-vided in three groups of measures: base, temporal andenvironmental. Popular vulnerability scanners suchas Nessus, already use the CVSS in their database, aswell as the NVD - National Vulnerability Database ofthe U.S. government, maintained by NIST.

Other efforts in the standardization of measures inthe information security area can be found in (Jelenand Williams, 1998). In this work, Jelen and Wil-ians argue that assurance is an integral part of the riskand security management process. They proposes aformulation intended to be universal in the measuringassurance area. Weiss et al. (Weiss et al., 2005) pro-poses a model for the security level calculation of anorganization through the percentage of lost assets.

Miani et al. (Miani et al., 2008) defines a modelfor formula calculation of security metrics. The at-tributes that constitute the model are: objective, met-ric, measure, data source, frequency, metrics clas-sification and formula. The model standardizes thenomenclature of terms relative to security metrics,and proposes the formula calculation in a generic waycontributing to the decreasing of subjective criteria inthe metrics formulation. The model define the cal-culation only using the arithmetic mean. This workproposes a model that using set theory and the inclu-sion of a new component, the intersection component,that seeks to correct possible flaws in the model inter-pretation proposed by Miani et al.

3 METRICS FORMULATION

The security metrics model, proposed in (Miani et al.,2008) has a different characteristic from the tradi-tional approaches, that is from the grouping of multi-ple metrics in a common group, calculate the securityindicator of this group. For example, consider the fol-lowing security metrics proposed by ISO/IEC 27002:

• P1 = Percentage of communication channels con-trolled by the organization that have been securedin accordance with policy.

• P2 = Percentage of mobile users who accessenterprise facilities using secure communicationmethods.

• P3 = Percentage of workstation firewalls, hostfirewalls, sub-network firewalls, and perimeterfirewalls configured in accordance with policy.

Although the metrics are organized into a com-mon group called “ Network access control”, they areindividually treated. Each metric has its own formulaand there are no recommendations on how to analyzethe whole group. The model proposed by Miani etal. consists in, to combine the three metrics in onlyone, aiming the overall group analysis by calculatinga single formula representing the “ Network accesscontrol” level.

The grouping is important because it unifies sev-eral results in only one number, easing the resultsinterpretation of the non-technical organization staff.When necessary the calculation of individual metricscan also be performed.

In this case, even according to the proposal, theformula of the group “Network access control” itwould be calculated as follows:

Consider the components,P1, P2 andP3. The nextstep is to examine the component security.P1 is se-cure, because when the number of secure communica-tion channels increase, the risks of security problemsdecrease. Analogously,P2 andP3 are secure compo-nents too. Thus, the formula is given by the arithmeticmean betweenP1, P2 eP3.

The simplicity and possible flaws in the interpre-tation of results, are the main motivating factors forthe development of a new reliable model. Take thefollowing example:

Example 1. Consider a metricM which aims tomeasure the security between the connections of theMBAN buildings. LetAt be the set of buildings thatconstitute the network. Consider two secure compo-nentsA1 andA2 of the metric such that:A1 is a subsetof At , whereA1 represents the number of buildingsthat have firewall resources or another logical accesscontrol andA2 is a subset ofAt , whereA2 represents

SECRYPT 2009 - International Conference on Security and Cryptography

382

the number of buildings that have ciphered connec-tions.

Miani et al. states that the metrics formula, in thiscase would be the arithmetic mean between the com-ponents. However, we can consider the existence ofanother subsetA3 with A3 = A1∩A2, in other wordsA3 is the set of buildings that have firewall resourcesand encryption between connections. This new pa-rameter must be part of the formula. For this considerthe number of security resources in a system, in gen-eral, more quantity of security resources implies inmore security. In our case, the setA3 has more secu-rity resources then the setsA1 andA2 and hence, thesetA3 will have a greater weight in relation to othersets. Therefore, we can conclude that the intersectionquantity of a component must affect its weight.

This is the major motivation of this work, to de-velop a standardized model capable of correcting theinaccuracies of the model proposed by Miani et al.and encourage its use in any kind of security metric.

4 MODEL DESCRIPTION

Take the set of the componentsa1,a2,a3, ...,an. Foreachai , let at be the maximum value that this mea-sure assumes. Rewriting this sentence using the settheory notation, we have one set for each compo-nent a1,a2,a3, ...,an. The correspondentat will bethe set that contains the respectiveai . Then,a1 ⊂ at1,a2 ⊂ at2 and so on.

The model objective is to increase the reliabilityof the security index calculation. For this, the compo-nents in the formula calculation will be balanced byusing different weights and another factor will be pre-sented: the intersection component between the sets.

The first step to update the calculation of the for-mula is to verify if the metric has a maximum compo-nent or maximum value set, with two or more subsets.Then, you should classify every metrics component.

The model proposed by Miani et al. states that,given a metricM: i) M is composed only by securecomponents, ii)M is composed only by insecure com-ponents and iii)M is composed by insecure and se-cure components

Let us first consider the case where the metricis composed only by secure components. The othercases will be deducted from this.

In cases where maximum value sets with two ormore related subsets do not exists, the formula cal-culation is reduced to the mean between the compo-nents. Here we only consider the existence of maxi-mum value sets with at least two related subsets.

We begin the formula construction for the case

where the number of subsets of a maximum value setis equal to 2, then the case where such number is 3and at last the formula will be generalized. Considera metricM, consisting of a maximum value setT andtwo setsA1 andA2 such thatA1 ⊂ T andA2 ⊂ T. LetI1,2 be the set formed by the intersection betweenA1andA2. The cardinality, of these sets are:a1 = #A1,a2 = #A2, i1,2 = #I1,2 andt = #T.

The formula for the case where the number of sub-sets is 2, will be built using a weighted mean betweenthe subsets. The weights will be distributed as fol-lows: 2 for the intersection component and 1 for theother components. Note that the weight 2 representsthe number of subsets ofT. Then the formula will bewritten as:

F2 =(2)(

i1,2t )+(1)(

a1t )+(1)(

a2t )

(2+1+1)

It is important to make an analysis of the maxi-mum and minimum of the formula. A metric withformula equal to 0 represents that no security require-ments have been accomplished. Similarly, a metricwith formula equal to 1 represents that the securityrequirements have been accomplished. However, for-mula equal to 1, does not mean that security is fullyaccomplished.

The maximum security is achieved when the num-ber of elements ofI1,2 is equal to the number of ele-ments ofT, that is, i1,2 = t. On the other side thisis only possible whenA1 = A2. If A1 = A2 thenI1,2 = A1 = A2 and a1 = a2 = i1,2 = t. Calculatingthe formula, we obtain 1.

The minimum security is achieved if no securityrequirements were met, that is, ifA1 = A2 = I1,2 = ∅.Calculating the formula, we obtain 0.

In other words, the maximum and minimum anal-ysis shows that the developed formula is consistentwith the defined requirements.

Consider a metricM, composed by a maximumvalue setT and three setsA1, A2 eA3 such thatA1 ⊂ T, A2 ⊂ T andA3 ⊂ T and the intersection setsI1,2, I1,3,I2,3 andI1,2,3. The cardinality, of these sets are:a1, a2,a3, i1,2, i1,3, i2,3, i1,2,3 andt.

The weights will be distributed as follows: 3 forthe intersection setI1,2,3, 2 for the other intersectionsets,I1,2, I1,3 andI2,3 and at last 1 for the other sets.Thus,

F3 =(3)(

i1,2,3t )+(2)(

i1,2t +

i1,3t +

i2,3t )+(

a1t )+(

a2t )+(

a3t )

(3+2+2+2+1+1+1)

The same analysis of maximum and minimumshould be made here. Note that the results do notchange because the requirements for the maximumsecurity level is thatI1,2,3 = t that is,A1 = A2 = A3.

Now we can generalize the formula calculation forthe case where the number of subsets ofT is n.

AN ALTERNATIVE APPROACH FOR FORMULA MODELLING IN SECURITY METRICS

383

Consider a metricM which is composed by onemaximum value setT with A1,A2, ...,An subsets ofT. We denote the cardinality of this sets as follows:a j = #A j .

The formula will be built using a general rule forobtaining each of the terms. The term of weightn isobtained by the ratio between the cardinality of the in-tersection of then subgroups and the cardinality of thesetT. The term of weight(n−1) is obtained addingall the ratios between the cardinality of the intersec-tion of n−1 sets and the cardinality of the setT. Con-tinuing this process, all terms will be obtained. Thedenominator is formed by the sum of the weights ofeach of the terms. Each of the combinationsCn

k rep-resents the number of subsets in each of 1 ton terms.A generalized version of the formula is:

Fn =n

(

i1,...,nt

)

+(n−1)

(

i1,...,n−1t +...+

i2,...,nt

)

+...+

n(Cnn)+(n−1)(Cn

n−1)+...+(2)(Cn2)+(1)(Cn

1)+

++...+2

(

i1,2t +...+

in,n−1t

)

+( a1t +...+ an

t )

n(Cnn)+(n−1)(Cn

n−1)+...+(2)(Cn2)+(1)(Cn

1)

However, we need to recall one last detail. Theformula is valid only for one maximum value set. Form sets we should do the calculation for each one, andthen calculate the arithmetic mean between the re-sults.

4.1 Differences between the Models

This section aims to show that the inclusion ofweights makes the model presented here more pre-cise than the model proposed in (Miani et al., 2008).In other words, the formula results presented in thiswork are always smaller than the formula results pro-posed by Miani et al. Consider the case where thenumber of sets is equal to 2.

Let M be a metric composed by one maximumvalue setT and two setsA1 andA2 such thatA1 ⊂ TandA2 ⊂ T. Let I1,2 be the set composed by the inter-section between the setsA1 andA2.

Note that the following inequalities are valid:i1,2 ≤ a1 e i1,2 ≤ a2.

The formula presented by Miani et al. is given byF1 = a1+a2

2t , and the formula proposed in this work is

given byF2 =2(i1,2)+a1+a2

4t .Therefore, we would like to demonstrate that

a1+a22t ≥

2t+a1+a24t . Using the proof by contradiction,

we obtaina1 +a2 < 2i1,2

This contradicts our assumption because, if wesum the inequalitiesi1,2 ≤ a1 and i1,2 ≤ a2 we havea1 + a2 ≥ 2i1,2. In this case, the formula proposed

in this work is always less or equal than the formulaproposed by Miani et al.

For the other cases the demonstration is analogue,using the proof by contradiction in the obtained in-equalities. The case that the number of sets is equalto 3, for instance, the following inequality must beproved:

a1+a2+a33t ≥

3(i1,2,3)+2(i1,2)+2(i1,3)+2(i2,3)+a1+a2+a312t

regarding the validity of the following inequali-ties: i) i1,2,3 ≤ a1 , i1,2,3 ≤ a2 and i1,2,3 ≤ a3, ii)i1,2 ≤ a1 andi1,2 ≤ a2, iii) i1,3 ≤ a1 andi1,3 ≤ a3 andiv) i2,3 ≤ a2 andi2,3 ≤ a3.

5 RESULTS AND APPLICATIONEXAMPLE

In this section will be presented an application exam-ple of the proposed model in security metrics foundin (ISO, 2005) and the case study, using the proposedmodel in the Metropolitan Broadband Access Net-work (MBAN) of Pedreira.

5.1 Application Example

Consider the metrics of theCommunication andOperations Managementgroup, proposed in (ISO,2005). Within this group we can identify three metricsthat can address the same security control: backup.In our proposal, will be created a new group called “Backup Policy” containing such metrics. The metricsdefinition are,

1. Assets backed up: measures the percentage ofsystems with critical information assets that havebeen backed up in accordance with policy.

2. Assets backup validated: measures the percentageof systems with critical information assets whererestoration from a stored backup has been suc-cessfully demonstrated.

3. Assets backup offsite: measures the percentage ofbackup media stored offsite in secure storage.

The metrics, separately, would be calculated asfollows. Considerat = total number of assets,a1 =number of assets backed up,a2 = number of assetswith backup procedures and validateda3 = number ofassets whose backups are stored off-site. So, for thefirst metric, we havea1

at, for the second metrica2

atand

finally for the third metrica3at

. Note that the three com-ponents,a1, a2 anda3, have the same total value com-ponent, number of assets, allowing that the proposed

SECRYPT 2009 - International Conference on Security and Cryptography

384

model may be applied here. Applying the model pre-sented in this work, the new formula of the group “Backup Policy” will be calculated like this:

F =(3)(

i1,2,3t )+(2)(

i1,2t +

i1,3t +

i2,3t )+(

a1t )+(

a2t )+(

a3t )

(12)

Thus, we have a security indicator for the whole“Backup Policy” task. The model can be applied toany set of metrics within requirements, easing the elu-cidation of the results and producing an efficient andbalanced overview of a security question.

5.2 Metrics Application in the MBAN ofPedreira

Metropolitan broadband access networks (MBAN)can be defined as the convergence of services, appli-cations and infrastructure to create a community com-munications network of a city. This implements thepublic information highway, characterized for highbandwidth transmission capacity and data aggrega-tion of several types (Mendes, 2006). Further infor-mations about this kind of network can be found in(Alexiou et al., 2006).

The MBAN of Pedreira is a project that has beingdeveloped by the State University of Campinas (UNI-CAMP) and by the government of the city of Pedreira.The project started in 2005 and officially launchedin 2007. The detection of security vulnerabilities inPedreira’s network was the main motivation for thedevelopment of particular metrics that could quantifythe high amount of data generated by technical reportsand management software. In next will be presentedthe results of three security metrics applied in this pe-riod. Also will be showed a comparison between thetwo models discussed in this work. The metrics are:i) Security between the MBAN buildings, ii) Securityrequirements in the VoIP network and iii) Availabilityand reliability in the MBAN servers.

Security between the MBAN BuildingsThe aim here is to analyze and to increase securitylevel among the MBAN buildings. The formula com-ponents are the following:at1 = total number ofbuildings, a1 = number of buildings that use fire-wall resources or logical access control in their con-nections,a2 = number of buildings that use cipheredresources in their connections andi1,2 = number ofbuildings that have both firewall resources and en-crypted connections. The formula also have a factorp that varies accordingly the size of the cryptographicprotocol used, attached to thea2 component. The for-mula is given by:

F1 =2

i1,2t1

+a1t1

+pa2t1

4

Security Requirements in the VoIP NetworkThe objective here is to analyze the security require-ments of the VoIP network in a MBAN. The formulacomponents are the following:at1 = total number ofVoIP branches,at2 = total number of VoIP calls ina specific period,a1 = number of VoIP branches ci-phered,a2 = number of VoIP branches which are inseparated networks from the data network,a3 = num-ber failed calls andi1,2 = number of VoIP branchesboth ciphered and in separated networks. The formulais given by:

F2 =

2

(

i1,2at1

)

+( a1at1 )+( a2

at1 )(4)

+(

1−(

a3at2

))

2

Availability and Reliability in the MBAN ServersThe aim here is to evaluate the impact of the un-planned downtime in the services deployed by theMBAN servers. The formula components are the fol-lowing: at1 = total number of servers,at2 = totalnumber of hours,a1 = number of servers with re-dundancy resources,a2 = number of servers that arein the backup program,a3 = number of servers thatstores the backups in security offsite,a4 = uptimemean of servers andi1,2 = number of servers withboth redundancy and in the backup program. The for-mula is given by:

F3 =

2

(

i1,2at1

)

+( a1at1 )+( a2

at1 )(4)

+(

a3a2

)

+(

a4at2

)

3

Table 1 shows the result of each one of the metricsand also compares with the model proposed by Mianiet al. We denoteModel 1for the model proposed byMiani et al. andModel 2 for the model proposed inthis work.

Table 1: Metrics results and comparison.

MetricFormulaModel 1

FormulaModel 2

Decrease

Securitybetween the

MBANbuildings

0.5411 0.3117 42.39%

VoIP securityrequirements

0.7296 0.6046 17.13%

Server’savailability

andreliability

0.7496 0.7217 3.72%

According to what was showed in section 4.1, themodel proposed here achieved lower results. The col-umn “Decrease” illustrates the difference between theresults of the two models. This difference is obtained

AN ALTERNATIVE APPROACH FOR FORMULA MODELLING IN SECURITY METRICS

385

from the values of the component intersection. Highervalues imply in the decrease of distance between theformulas. Similarly, lower values imply in the in-crease of distance between the formulas.

Besides the intersection component, the way thatthe metrics formula is obtained can also influence onthe difference between the models. If the formula hascomponents that require in its composition the inser-tion of additional arithmetic mean, such as have si-multaneously secure and insecure components (met-ric 2) or have components outside the intersection(metric 3), these components will act as follows: val-ues near to 1 decrease the difference and values nearto 0 increase the difference.

6 CONCLUSIONS

Security metrics are modern tools and with high re-search potential. They are extremely important for thesecurity level understanding of the organization whenproperly developed and applied.

A classic security metric has several components,including: objective, data source, frequency, classifi-cation and formula. The purpose of the formula, inparticular, is to describe the calculations to be per-formed for quantify the metrics in a numerical expres-sion. That is, the metrics results are investigated fromthe formula. It is important that this task be accom-plished in a clear, robust and generic way.

The model proposed in this work sought to cor-rect the inaccuracies of the model proposed by Mianiet al. developing a new component, which deals withsets intersections of security measures. This compo-nent plays an important role in the model, distribut-ing the weights in the proposed formula. Besides theformula, the whole nomenclature and the logic con-struction developed in this work can be reused to buildother models in this area.

The model validation it was obtained in two ways:from the metrics application found in literature andwith a case study. Classic security metrics as foundin (Jaquith, 2007), (Swanson et al., 2003) and (ISO,2005) are easily migrated to our model. One of thebenefits is the aggregation of various measures in onlyone, easing the overview and the results interpretationof the non-technical organization staff. Besides that,the proposed model was used in three security metricsthat were implemented in the MBAN of Pedreira. Theresults showed that the model proposed here achievedlower results when compared to Miani et al. modeland could also explain how the numerical differencesbetween the models are established.

Future works includes the model utilization in

other security metrics, aiming to create its own cata-log, suchlike what is developed in the Metrics CatalogProject (MetricsCenter, 2008) and the application ofnew case studies to refine the proposed model in pri-vate institutions, government and other MBANs en-abling the development of a security metrics database.

REFERENCES

Alexiou, A., Bouras, C., and Primpas, D. (2006). Designaspects of open municipal broadband networks. InAcessNets ’06: Proceedings of the 1st internationalconference on Access networks, page 20, New York,NY, USA. ACM Press.

Herrera, S. (2005). Information security management met-rics development. InSecurity Technology, 2005.CCST ’05. 39th Annual 2005 International CarnahanConference on, pages 51–56.

ISO (2005). Code of practice for information security man-agement - iso/iec 27002.

Jaquith, A. (2007).Security Metrics - Replacing Fear, Un-certainty and Doubt. Addison-Wesley.

Jelen, G. and Williams, J. (1998). A practical approach tomeasuring assurance. InComputer Security Applica-tions Conference, 1998, Proceedings., 14th Annual,pages 333–343.

Lowans, P. W. (2002). Implementing a network securitymetrics program. Technical report, SANS.

Mell, P., Scarfone, K., and Romanosky, S. (2007). A com-plete guide to the common vulnerability scoring sys-tem version 2.0. http://www.first.org/cvss/.

Mendes, L. S. (2006). Infovia Municipal - Um novoParadigma em Comunicaes. Universidade Estadualde Campinas.

MetricsCenter (2008). http://www.metricscenter.org/index.php/plexlogicmetricviewer. Accessed in24/02/2009.

Miani, R. S., Zarpelo, B. B., de Souza Mendes, L., and Jr.,M. L. P. (2008). Metrics application in metropolitanbroadband access network security analysis. InSE-CRYPT 2008 - International Conference on Securityand Cryptography, pages 473–476.

Payne, S. C. (2006). A guide to security metrics. SANS Se-curity Essentials GSEC Practical Assignment Version1.2e.

Rosenblatt, J. (2008). Security metrics: A solution in searchof a problem.EDUCAUSE Quarterly, 3:8–11.

Swanson, M., Bartol, N., Sabato, J., Hash, J., and Graffo, L.(2003). Security metrics guide for information tech-nology systems. Technical report, NIST Special Pub-lication 800-55.

Weiss, S., Weissmann, O., and Dressler, F. (2005). A com-prehensive and comparative metric for information se-curity. In Proceedings of IFIP International Confer-ence on Telecommunication Systems, Modeling andAnalysis (ICTSM2005), pages 1–10.

SECRYPT 2009 - International Conference on Security and Cryptography

386