Interdependency of critical digital services can be modeled in the form of a graph with exactly known structure but with edge weights subject to estimation errors. We use standard and custom centrality indexes to measure each service vulnerability. Vulnerability of all nodes in the graph gets aggregated in a number of ways into a single network vulnerability index for services whose operation is critical for the state. This study compares sensitivity of various centralities combined with various aggregation methods to errors in edge weights reported by service operators. We find that many of those combinations are quite robust and can be used interchangeably to reflect various perceptions of network vulnerability. We use graphs of source files’ dependencies for a number of opensource projects, as a good analogy for real critical services graph, which will remain confidential.
Correct operation of digital services and infrastructures has since long become critical for societies, and therefore demands coordinated actions for maintenance and incident response. The Directive on Security of Network and Information Systems (NIS [
Complete, i.e., possessing information about all critical services in the country
Automated, i.e., minimizing human factor in daily operations as well as in network model construction
Coupled, i.e., exchanging information at international level
Researchers, industry, and regulators stay aware of the above challenges and come up accordingly with ideas of such systems (cf., e.g., [
Apart from privacy and organizational obstacles, filling a questionnaire can be a challenge of its own for an operator. For a given own service, an operator is asked to report services preconditioning its correct operation, and to provide estimates of their impact on own service in terms of availability, confidentiality, and integrity (CIA) [
Our goal is to examine how the above process is sensitive to incorrect information about mutual service impact as reported by operators, with the assumption that the structure of the network is known fully and correctly. Such information is crucial because that the scalar index value will be reported to SOCs and, consequently, will play the role of the main threat indicator.
We organized the paper as follows. A network model of services is presented in the remaining part of this section. A suite of methods for calculation of service vulnerability and for aggregation of vulnerabilities into a scalar vulnerability index are described in Section
The network of interdependent digital services is modeled as a directed graph:
Such graph model extension with edge weights represented actually by a matrix of up to nine aspects of impact demands developing new graph algorithms—or picking up one of the aspects, like it is done in this paper. It makes the model universal enough to accommodate both digital services and physical infrastructure elements. In the latter case, one refers to just the availability aspect. For example, availability of backup power supply may influence availability and integrity of the physical access control system; hence, an operator has to address the influence in two aspects:
Topology of a service graph represents existence of service interdependencies, while edge weights stand for intensity of those interdependencies. When combined, they make it possible to calculate the overall vulnerability of each service. There are many ways such vulnerability could be formulated; we express its definition as
While
The major practical problem concerns credibility of
Therefore, we must assume that, contrary to structure of service dependencies that is known and correct, the reported impact values
Star ratings have been commonplace practice in many fields where user feedback is required. While facilitating the questioning process from a psychological perspective, it complicates analysis of statistical properties of responses, as it has been reported in [
In our case, we kept the original 10star scale as proposed by the NPC riskanalysis team. Such scale leaves operator no “middle” option, unlike grade “3” on 5star scale. Indeed, we do not want operators to answer neutrally because, opposite to, e.g., hotel ranking, there is no “neutral” answer other than absence of the edge connecting the two services. Moreover, finer scale makes room for elaborating more precise instructions on selfassessment and answering in the future. As regards the choice of distribution for
The main aim of this paper is to evaluate sensitivity of various definitions of service vulnerability,
There exist a number of recognized and widely known definitions of vertex structural importance that can be used as candidates for
Service vulnerabilities calculated above are based on incoming edges and in fact have the meaning of service susceptibility to failure.
Vulnerabilities can be aggregated by equation (
For any instance of reported impact matrix,
In the context of difference between two sets of services, we may introduce yet another measure based on difference in ordering of the most important services there:
In practice, the service graph
Graph of real dependencies between 33 services run by 17 operators in 3 branches of national economy.
We found that networks of source code dependencies are a close analogy. First, they represent software components, on a much smaller scale though. Second, the dependency between modules can be relatively easily tracked by static code analysis. Third, failure or malfunction of one software module influences the operation of all modules that depend on it, although differently. Fourth, module dependencies in opensource projects appear not in predefined way but represent current needs of programmers, as already reported in [
All networks analyzed in this study describe software module dependencies in Javascript (JS) projects available from hosting platform
Properties of projects used for analysis.
Project name  Modules  Number of circular dependencies  Project url 

Airbnb  22  0 

Fcc  426  18 

Nodejs  9507  27 

Omi  475  0 

React  507  0 

Vue  419  8 

Screenshot of a sample exemplary graph of module dependencies in a part of Airbnb project, displayed by Madge.
Formula (
Mean average absolute error,
Mean average relative error,
Standard deviation of error,
Standard deviation of error, relative to true value,
They all are comprehensive measures of how errors of operators impact estimation affects errors of network vulnerability, given any of the proposed formulas of
All the reasoning provided above concerns a single instance of
Scatter plots of vulnerability estimation error (left), and standard deviation (middle) vs. true vulnerability. Also (right), standard deviation histogram—for experiments carried out for Airbnb network, with
Figures
Results in Figure
Sensitivity of Airbnb graph,




 

(a) 


































(b) 































Sensitivity of Fcc graph,




 

(a) 


































(b) 































Sensitivity of Omi graph,




 

(a) 


































(b) 































Sensitivity of React graph,




 

(a) 


































(b) 































Sensitivity of Vue graph,




 

(a) 


































(b) 































The figures given in Tables
Small total error
Big sensitivity
Small standard deviation
Candidate combinations of
Values of
As analyzed combinations form a cloud in 3D space, we may find a Pareto front, i.e., a set of nondominated combinations. They are
It should be reminded that research reported here is done in context of a large project aiming to build a nationwide model of critical services network. While integrity of the resulting graph can be obtained by careful automated inspection of questionnaires filed by service operators, the estimated reported impact between services will be biased and inherently erroneous. Therefore, it was worth to study sensitivity of some candidate synthetic metrics of overall network vulnerability with respect to incorrect input. We felt it correct to use networks of software module dependencies because of their functional and structural similarity to network of critical services, let alone that such real networks will probably remain confidential.
The study shows that all three proposed formulas for individual service vulnerability calculation are valuable. This is rather a positive observation, as each of them has its own specifics and can be used under various circumstances. Also, almost all proposed ways of vulnerability aggregation into a single vulnerability index are useful (except the Levenshtein distance, which shows much variation and has turned out to be useless). Naturally, combinations of formulas appropriate for capturing “extreme” phenomena, as
The main takeaway is that it is safe to apply mean or median aggregation of individual service vulnerability, whatever is the formula for importance calculation. Such aggregated value may serve as a single, comprehensive vulnerability index. Note that being robust to errors in graph edge weights, it will be affected by major structural graph changes—e.g., edge removal as result of realtime detected failure. Our previous work has shown that networks of autonomous systems (AS) can be really badly affected by just one link failure, contrary to widespread belief in Internet robustness [
One should remember that results reported here were based on the sound assumption of analogy between critical services and software modules. This assumption will eventually get verified in practice, once the national cybersecurity platform is operational and filled with data. We look forward to compare properties of vulnerability calculation formulas calculated here by random sampling with careful expert judgment and postmortem analyses for real services graph.
The open source code used to support the findings of this study is publicly available on
The author declares that there are no conflicts of interest regarding the publication of this paper.
The work presented in this paper has been supported by the Polish National Centre for Research and Development grant (CYBERSECIDENT/369195/I/NCBR/2017).