Differentially Private Attributed Network Releasing Based on Early Fusion

Vertex attributes exert huge impacts on the analysis of social networks. Since the attributes are often sensitive, it is necessary to seek effective ways to protect the privacy of graphs with correlated attributes. Prior work has focused mainly on the graph topological structure and the attributes, respectively, and combining them together by defining the relevancy between them. However, these methods need to add noise to them, respectively, and they produce a large number of required noise and reduce the data utility. In this paper, we introduce an approach to release graphs with correlated attributes under differential privacy based on early fusion.We combine the graph topological structure and the attributes together with a private probability model and generate a synthetic network satisfying differential privacy. We conduct extensive experiments to demonstrate that our approach could meet the request of attributed networks and achieve high data utility.


Introduction
Social network is an important topic in many subjects, such as sociology, economics, and informatics. Many applications need to achieve useful information through analyzing social networks. Social networks contain sensitive personal information, including the individual information and the relationships among them. However, vertices usually contain sensitive attribute information and we need to protect the privacy of not only the graph topological structure but also the vertex attributes. We call the network with vertex attributes as attributed network. In this paper, we study the privacy protection method of releasing the attributed networks. Network data can be released for public with sanitization by a substituted synthetic network generated from a private network generation model.
As a common privacy protection method, anonymization technique (e.g., k-anonymity [1] and l-diversity [2]) could easily be susceptible to new privacy attacks. Recently, differential privacy has been proposed to address such privacy problem. Unlike the anonymization technique, differential privacy provides strong theoretical guarantees by adding calibrated noise against adversaries with prior knowledge. e standard technique to ensure differential privacy of social network is to "sanitize" the topological structure and neglect the vertex attributes. ese vertex attributes affect not only the vertex information but also the topological structure. For example, vertex attributes are closely related to the connection between vertices. When the topological graph has high assortativity coefficient, vertices with similar attributes connect with each other with a higher probability. Adversaries with vertex attributes could infer the topological structure effectively.
A straightforward approach to release the attributed network under differential privacy is to sanitize the vertex attributes and topological structure, respectively, and then fuse them together to get the differentially private version of the network. We call this kind of method as late fusion. On the contrary, when we deal with the vertex attributes and topological structure simultaneously, we call it as early fusion. In this paper, we study the attributed network releasing method under differential privacy through early fusion.
We propose a probability model of attributed network under differential privacy, called DP-ANP (i.e., differentially private attributed network probability model). DP-ANP contains the vertex attribute information and the topological structure.
e DP-ANP records connection probabilities between any pair of vertices and attribute value probabilities of each vertex. It allows to draw a sample model from the model's space through joint probability distribution, and it is equivalent to maximizing the probability of the model. In particular, to satisfy the differential privacy, we add noise to the parameters of the probability model and generate the private synthetic network using DP-ANP. e core of DP-ANP is to add noise to the model parameters of each vertex. However, due to the large scale of vertices, DP-ANP may lead to a large amount of noise. erefore, we propose an improved DP-ANP, called DP-ANPHP (i.e., differentially private attributed network probability model based on hyperparameter). In DP-ANPHP, we construct a model of network based on hyperparameters and add noise to them instead of the model parameters in DP-ANP. e scale of hyperparameters in DP-ANPHP is much less than the parameters in DP-ANP. It makes DP-ANPHP produce less noise and achieve high data utility. In summary, we present several contributions: (1) We introduce the probability model of attributed network under differential privacy based on early fusion, called DP-ANP. is model sanitizes the model parameters of the overall information of the network instead of vertex attributes and topological structure separately. Compared with the late fusion method, DP-ANP reduces the complexity of computing.
(2) We develop an improved DP-ANP model based on hyperparameters, called DP-ANPHP. DP-ANPHP reduces the scale of noise. (3) rough privacy analysis, we prove that DP-ANP and DP-ANPHP satisfy differential privacy. We conduct extensive experimental study over synthetic datasets and real datasets. e results demonstrate that DP-ANP outperforms the late fusion method and DP-ANPHP outperforms DP-ANP. e rest of the paper is organized as follows. Section 2 provides a literature review of differential privacy of social network. Section 3 presents necessary concept on differential privacy and attributed network. Section 4 describes early fusion and late fusion. Section 5 describes the DP-ANP model. Section 6 describes the DP-ANPHP model. Section 7 reports the experimental results. Section 8 concludes the paper.

Related Work
2.1. Differentially Private Network Analysis. Dwork et al. [3] firstly proposed a differentially private method to answer the query functions by disturbing the outcome directly. Hay et al. [4] proposed a differential privacy in a post-processing phase to compute the output which retained the accuracy and privacy at the same time. ey used this method to estimate the degree distribution. Nissim et al. [5] introduced the concept of smooth sensitivity and protected individuals' privacy by adding a small amount of noise to the released statistics, such as triangle count. Karwa et al. [6] expanded this concept and calculate k-star count of a network. Zhang et al. [7] analyzed the released statistics through a ladder function and reduced the sensitivity effectively. Cheng et al. [8] presented a two-phase differentially private frequent subgraph mining algorithm, named DFG. In DFG, frequent subgraphs are privately identified in the first phase, and the noisy support of each identified frequent subgraph is calculated in the second phase. Ding et al. [9] published the private triangle counts with the triangle count distribution and the cumulative distribution. Sun et al. [10] studied fundamental problems related to extended local view. ey formulated a decentralized private scheme named DDP, which requires that each participant consider both her own privacy and her neighbors' privacy involved in her ELV.

Differentially Private Network
Publishing. Sala et al. [11] proposed a private graph model which extracts a graph's structure into dK-graph and generated a synthetic graph. Mir and Wright [12] used maximum likelihood estimation to privately estimate the parameters based on the stochastic Kronecker graph model. Xiao et al. [13] proposed a private network publishing method which computes an estimator of the graph in the hierarchical random graph (HRG) model. ey sampled possible HRG structures via Markov chain Monte Carlo (MCMC) through exponential mechanism. Qin et al. [14] investigated techniques to ensure local differential privacy of individuals while collecting structural information and generating representative synthetic social graphs. ey proposed LDPGen which incrementally clusters users based on their connections to different partitions of the whole population and adapted existing social graph generation models to construct a synthetic social graph. Chen et al. [15] presented a method for publishing private synthetic graphs, which preserves the community structure of the original graph without sacrificing the ability to capture global structural properties. Wang et al. [16] presented a differential privacy method for weighted network through structuring a private probability model.

Differential Privacy for Attributed Network.
Qian et al. [17] proposed to deanonymize social graphs and infer private attributes leveraging knowledge graphs, which carry both graph structural information and semantic information. Ji et al. [18] conducted the first attribute-based anonymity analysis for attributed network under both preliminary and general models and proposed a new deanonymization framework, which takes into account both the graph structure and the attribute information to the best of our knowledge. Jorgensen et al. [19] proposed a method for publishing private attributed network which adapted existing graph models and introduced a new one and then showed how to augment them with differential privacy. Yin et al. [20] defined a new type of attack called attribute couplet attack and proposed a new anonymity concept called kcouplet anonymity to achieve privacy preservation under attribute couplet attacks. Kiranmayi et al. [21] designed an algorithm named couplet anonymization by using the node addition approach to reduce the misleading fake relations.
Meanwhile, some research studies focus on the privacy of high-dimensional data. ese methods are also closely related to the differential privacy for attributed network. Chen et al. [22] proposed a novel solution to preserve the joint distribution of a high-dimensional dataset. ey developed a robust sampling-based framework to systematically explore the dependencies among all attributes and identified a set of marginal tables from the dependency graph to approximate the joint distribution based on the solid inference foundation of the junction tree algorithm. Zhang et al. [23] presented a differentially private method for releasing highdimensional data named PrivBayes. PrivBayes injected noise into the low-dimensional marginals instead of the highdimensional dataset to circumvent the curse of dimensionality. ey also introduced a novel approach that uses a surrogate function for mutual information to build the model more accurately. Ren et al. [24] developed a local differentially private high-dimensional data publication algorithm, LoPub, by taking advantage of our distribution estimation techniques. Correlations among multiple attributes were identified to reduce the dimensionality of crowdsourced data.
In conclusion, only a few number of research studies focus on differential privacy for attributed network and are generally based on late fusion. In this work, we develop a differentially private attributed network publishing method based on early fusion.

Background
3.1. Differential Privacy. Given a graph G, differential privacy ensures the outputs to be approximately same even if any edge is arbitrarily added or deleted in the graph. us, the presence or absence of any edge has a negligible effect on the outputs. We define two graphs G 1 � (V 1 , E 1 ) and G 2 � (V 2 , E 2 ) to be neighbors if they satisfy V 1 � V 2 , E 1 ⊂ E 2 and |E 2 | � |E 1 | + 1. ε-Differential privacy is defined as follows.
Definition 1 (ε-differential privacy) (see [3]). A randomized algorithm A is ε-differential privacy if for any two neighboring graphs G 1 and G 2 , and for any output O ∈ Range(A), Differential privacy is based on the concept of global sensitivity of a function f. It is used to measure the maximum change in the outputs of f when any edge in the graph is changed.
e global sensitivity of f is defined as Differential privacy can be achieved by Laplace mechanism and exponential mechanism. e Laplace mechanism is mainly used for functions whose outputs are real values. Differential privacy can be achieved by adding properly noise drawn randomly from Laplace distribution to the true answer.
Theorem 1 (laplace mechanism) (see [3]). For any function satisfies ε-differential privacy, where Lap i (Δf/ε) are i.i.d Laplace variables with scale parameter Δf/ε. e exponential mechanism is mainly used for functions whose outputs are not real numbers. e main idea is to sample the output data O from the output space O according to the utility function u.
e global sensitivity of u is Theorem 2 (exponential mechanism) (see [25]). Given a graph G and a utility function u: (G × O) ⟶ R, the arithmetic A whose output is with probability proportional to exp(ε · u(G, O)/2Δu) satisfies ε-differential privacy.

Any subset D iter sampled from D satisfying each data point is included independently with probability p. If algorithm
is a set of edges, and Λ � a 1 , a 2 , . . . , a S is the set of attribute S. Given the integer K, N vertices are divided into K disjoint subsets. Let e ij be an undirected edge and G be an undirected attributed network.

Early Fusion and Late Fusion
We classify the methods of publishing differentially private attribute network into two categories: early fusion and late fusion.
e main idea of early fusion is constructing a private network model which combines the vertex attributes and topological structure into a whole structure and allows two parts to interact with each other. Early fusion makes the network model private and uses it to release the private vertex attributes and topological structure. Figure 1 shows the structure of early fusion.
Late fusion needs to consider that vertex attributes affect the topological structure. Adversaries might pose threats to the privacy through the correlation between them without considering that vertex attributes affect the topological structure. As a result, late fusion needs to design the parameters which represent the correlation between vertex attributes and topological structure and sanitize them. However, it results in additional privacy cost and increases the complexity of the algorithm. Figure 2 shows the structure of late fusion.
Compared with late fusion, early fusion constructs the network model and combines the vertex attributes and Security and Communication Networks topological structure into a whole structure which contains the correlation. ere is no need to design the correlation parameters. In this paper, we study the attributed network releasing method under differential privacy through early fusion.

Differentially Private Attributed Network Model
In this section, we introduce a kind of attributed network probability (ANP) model and enable generation of synthetic attributed graph with differential privacy.

Privacy Analysis of Straightforward Approach.
In practice, we know that vertices with similar attributes connect with each other with a higher probability. For example, in social network, people with similar interest in a certain field may follow a well-known person in that field in common. ey are all connected with this vertex in social network. To simplify the complexity, we divide these vertices into the same group. Vertices in the same group are stochastically equivalent. ey have similar attributes and play equivalent roles in generating the network's structure. In particular, they have the attributes which obey the same distribution and connect with other vertices with the same distribution. We assume that this kind of partition exists but is unknowable. us, it can be treated as hidden variable and cannot be observed directly.
ANP is denoted by a triple (X, Y, Z). Adjacency matrix X � [X ij ] is an N × N symmetric random matrix, where X ij is a binary random variable which takes value from 0, 1 { }. When there is an edge between vertices v i and v j , where Z i is a random variable denoting the label of the group which v i belongs to. ANP outputs a sample from all possible attributed graphs. ANP uses the parameters α, θ, and ϕ; a compact graphical representation is given in Figure 3. e label of the group Z i is sampled from a multinomial distribution with its parameter α representing the probability of vertex belonging to the group. It is defined as where α is a K-vector parameter and α � (α 1 , α 2 , . . . , α k ). α k denotes the proportion of vertex belonging to the group k and satisfies α k ∈ [0, 1] and K k�1 α k � 1. Suppose vertices in the same group have similar attributes and their attributes are samples from the same distribution. Given the label of the group Z i , the attribute Y t i is sampled from a multinomial distribution with its parameter θ t Z i representing the probability of attribute values. It is defined as where θ s are connected with each other from the same distribution. Given the label of the group Z i , X ij is sampled from a Bernoulli distribution with its parameter ϕ Z i Z j representing the probability of whether there is an edge between vertices Z i and Z j . It is defined as where We use maximum likelihood estimation (MLE) to specify the values of α, θ, and ϕ. e group label Z is treated as hidden variable and cannot be observed directly. Adjacency matrix X and attributes Y can be observed directly and could be used for the calculation. We take the expectation maximization (EM) algorithm to address this issue: E step: estimate the distribution of hidden variable Pr(Z|X, Y, α, θ, ϕ t ).
M step: get the parameters through finding the maximize expectations, and the parameters are computed as follows:

Differentially Private ANP.
Our goal is to publish a differentially private attributed network G which matches the topological structure and attribute distribution of the original network as much as possible. We identify the private ANP model and generate G from it.
To identify the private ANP model, we need to add noise to the three kinds of parameters and get the noisy parameters.
en, we use the noisy parameters to generate the sanitized network G(X, Y, Z). Our differentially private attributed network publishing algorithm is shown in Algorithm 1.
To satisfy differential privacy, we need to calculate the noisy parameters during each iteration process of parameter estimation. We add noise to all the parameters and the scale of noise depends on the sensitivity of the parameters. In each iteration process, we run the EM algorithm and obtain the parameters α, θ, ϕ (lines 4-5). We firstly add noise to the group parameter α (line 6). α k denotes the proportion of the vertices belonging to group k, and the sensitivity of α is calculated as the maximum change in α k if any one vertex changes its group label. Suppose the number of vertices in group k is l k , we have α k � l k /N. If one vertex from group k changes its group label to other groups or one vertex from other group changes its group label to k, α k changes to (l k − 1)/N or (l k − 1)/N. e global sensitivity of α is 1/N. en, we sample Z through the multinomial distribution with the parameter α (line 6) and add noise to the attribute parameter θ (line 8). e parameter θ of each group is different to each other. θ Z i m denotes the proportion of vertices in group Z i that take the attribute value as m, and the sensitivity of θ is calculated as the maximum change in θ Z i m if any one vertex in group Z i changes its attribute. Suppose the number of vertices with the attribute equal to m in group Z i is q and the number of vertices in group Z i is l Z i (line 7), we have θ Z i m � q/l Z i . If one vertex from group Z i changes its attribute m to other value or one vertex changes its attribute value to m, θ Z i m changes to (q − 1)/l Z i or (q + 1)/l Z i . e global sensitivity of θ is 1/l Z i . At last, we add noise to the parameter ϕ (line 9). e sensitivity of ϕ is calculated as the maximum change in ϕ Z i Z j if any one vertex disappears or appears. Suppose the number of vertices in group Z i and Z j is l Z i and l Z j , respectively, and there are r edges between group Z i and Z j , we have ϕ Z i Z j � 2r/l Z i · l Z j . If one vertex from group Z i connects with s vertices in group Z j , after deleting this vertex, we have ϕ Z i Z j � 2(r − s)/l Z i · l Z j . e maximum change of ϕ Z i Z j is 2s/l Z i · l Z j . When this vertex connects all the vertices in group Z j , s takes the maximum value, and s � l Z i . us, the global sensitivity of ϕ is 2/l Z i .

Security and Communication Networks
After getting the noisy parameters α, θ, ϕ , we sample vertex attribute from the multinomial distribution with the parameter θ(line 11) and sample edges from the Bernoulli distribution with the parameter ϕ (lines 12-14).

Privacy Analysis of DP-ANP.
We prove that DP-ANP satisfies ε-differential privacy based on the sequential composition property.

Differentially Private Attributed Network
Model Based on Hyperparameter

Differentially Private Attributed Network Model Based on
Hyperparameter (DP-ANPHP). As discussed in the previous section, we need to add all the parameters α, θ, ϕ to satisfy differential privacy in DP-ANP. To reduce the amount of noise and relieve the effect of the noise, we propose a differentially private attributed network publishing method based on hyperparameter, called DP-ANPHP. In DP-ANPHP, we learn the parameters by Bayesian theory and propose an efficient differential privacy method by sanitizing a part of hyperparameters. us, the scale of parameters we need to sanitize decreases, and we could generate synthetic attributed network effectively. At last, we prove that DP-ANPHP satisfies differential privacy.
However, some difficulties lie in the calculation of the posterior distribution of hidden variable when we use probabilistic inference model in Bayesian theory. erefore, we use variational Bayesian algorithm to address this problem. We treat parameters α, θ, and ϕ as random variables, and let them have prior distribution. We place a Dirichlet distribution over α as the prior distribution with hyper-parameter ξ, and we also place a Dirichlet distribution θ as the prior distribution with hyper-parameter c. We place Beta distribution over ϕ as the prior distribution with hyperparameter (μ, ]). As Dirichlet distribution and Beta distribution are the conjugate priors for multinomial distribution and Bernoulli distribution, respectively, the posterior distribution of α and θ is still Dirichlet distribution, and the posterior distribution of θ is still Beta distribution. is will result in mathematical convenience.
Furthermore, the variational distribution q(z, α, θ, ϕ) can be factorized as where i � 1, 2, . . . , N. We further convert the variational distribution to the hyperparameter form. Based on the conjugate distribution, q(α|ξ) and q(θ|c) are defined as Dirichlet distribution with hyperparameters ξ and c, respectively, and q(ϕ|μ, ]) is defined as beta distribution with hyperparameter (μ, ]). And we also introduce a new variational parameter β i to replace α as the parameter of group label Z i . e reason is that according to equation (10), we require Z i to no longer depend on α. So, we treat β i as the hyperparameter of Z i . q has the following form: To maximize G(q), we take the derivatives of G(q) with respect to ξ, c, μ, ], and β and set them to zeros: ] t+1 where i � 1, 2, . . . , N; k � 1, 2, . . . , K; l � k + 1, k + 2, . . . , K; m � 1, 2, . . . , M. δ(Y i , a m ) is Kronecker delta function. f(·), g(·), and h(·) are the linear functions with parameters ξ t+1 k , c t+1 km , and (μ t+1 kl , ] t+1 kl ), respectively. To satisfy differential privacy, we need to add noise to hyperparameters during each iteration process. In practice, we found that the functions of ξ, c, μ, and ] are with respect to the parameter β. e coefficients of β are δ(Y i , a m ), X ij , and (1 − X ij ), respectively. ey are set to 1 or 0, and we could treat them as the range of the sum of β. Hence, we only need to add noise to β during the iteration process to satisfy differential privacy. In this way, we could reduce the scale of parameters which we need to add noise to and reduce the amount of noise required by differential privacy. Similar to the parameter α in DP-ANP, the global sensitivity of β is 1/N. e differentially private attributed network publishing algorithm based on hyperparameter is shown in Algorithm 2.

Privacy Analysis of DP-ANPHP.
Based on the sequential composition property, we prove that DP-ANPHP satisfies ε-differential privacy. Theorem 6. DP-ANPHP satisfies ε-differential privacy.
Proof. Suppose that DP-ANPHP makes the parameters converge after iterating J times. From Algorithm 2, we know that during each iteration process, β satisfies ε 1 -differential privacy. Based on the sequential composition property, J times iteration process satisfies J · ε 1 -differential privacy. Furthermore, generating synthetic network does not consume any privacy budget. In conclusion, DP-ANPHP satisfies ε-differential privacy and ε � J · ε 1 .

Experimental Results
Existing work on differentially private attributed network publishing algorithm has mainly focused on the late fusion method. For comparing the results effectively, Algorithm 3 shows a kind of late fusion method to compare DP-ANP and DP-ANPHP. Firstly, we need to add noise to the connection probability directly (line 9) and perturb the attributes using exponential mechanism (line 12). We emphasize that these two steps are independent.
We evaluate the performance of DP-ANP and DP-ANPHP with LN. As the Laplace mechanism differential privacy produces random noise, we measure the accuracy of the result using the median relative error where we run the Laplace mechanism for 10 times. 7.1. Datasets. We evaluate the utility over one synthetic dataset and two real-life datasets. e synthetic data consist of 100 vertices, and there are a number of edges randomly between them. It have two attributes. One includes two attribute values, and the other one includes three attribute values. Real-life datasets are Cora and Flickr. As these two datasets contain a large number of attributes, we sample 5 discrete attributes as our experimental data. e statistics of datasets are given in Table 1.
e fill of a network is the proportion of edges to the number of all possible edges, and it denotes the probability that an edge is present between two randomly vertices.

Evaluation under Different ε.
We compare the utility of LN, DP-ANP, and DP-ANPHP through the measure average clustering coefficient, number of triangles, normalized degree distribution, and modularity. We fix K at 4 and allocate the privacy budget ε as follows: 0.25, 0.5, 0.75, and 1. We represent the relative error of these measurements as where A is the differentially private output. In Figures 4 and 5, we compare the correlated error of average clustering coefficient and number of triangles. We can see that DP-ANP and DP-ANPHP outperform LN, and DP-ANPHP outperforms DP-ANP. is means that DP-ANPHP performs better in terms of representing the density level. Furthermore, with the increase of ε, the correlated error of DP-ANPHP had greater decreases, and the decrease of LN and DP-ANP is smaller. e reason is that in DP-ANPHP, we only need to add noise to the hyperparameter β. When the private strength is small, the noise affects the output more slightly. We need to add noise to all the parameters of the model in LN and to add noise to a part of the parameters in DP-ANP. e noise affects the output more greatly. In Figure 6, we compare the correlated error of normalized degree distribution. We can see that DP-ANPHP outperforms DP-ANP and DP-ANP outperforms LN.
We use the modularity as the measure of cluster analysis to compare DP-ANPHP with DP-ANP. Modularity is a measurement for evaluating the result of community division. It reflects the concentration of nodes in the same group. e higher the modularity, the tighter the nodes in the same group and the looser the nodes in different groups. Modularity has a range of [−1, 1]. In Figure 7, we can see that the modularity of DP-ANPHP is higher than that of DP-ANP. is is mainly Input: node dataset V, attribute dataset Λ, group number K, privacy parameter ε 1 , maximum iterations J. Output: sanitized attribute network (X, Y, Z).
(1) Initialize the attribute network (X, Y, Z) from V, Λ and K.
(2) Initialize the parameters α, ϕ 0 (3) for t � 1,. . .,J do (4) E step: given α, ϕ t− 1 , compute log Pr(X, Z| α,   because DP-ANPHP updates the hyperparameters during the iteration process and has a better result on clustering. Moreover, with the increase of ε, DP-ANP and DP-ANPHP have higher value of modularity. is means when the strength is small, community division has more accurate result. Figures 8-10, we show how the correlated error of average clustering coefficient, number of triangles, and normalized degree distribution vary under different K. We also compare the modularity in Figure 11. We allocate K as follows: 2, 4, 6, and 8. We can see that DP-ANP and DP-ANPHP have better performances      when K is larger. e reason is that the network is divided into more groups when K is larger, and the model has more exact parameters.

Conclusions
In this paper, we investigate the problem of attribute network differential privacy. We use the idea of early fusion. In early fusion, it constructs a probability model combining the topological structure and the attributes together and uses it to generate a private synthetic network. We propose two attribute network differential privacy methods based on early fusion, named DP-ANP and DP-ANPHP. DP-ANP sanitizes the model parameters during the iteration process to satisfy the differential privacy. As an improvement of DP-ANP, DP-ANPHP sanitizes the model hyperparameters during the iteration process and has a better performance.
e results of extensive experiments show that both DP-ANP and DP-ANPHP generate private synthetic attribute network with a high accuracy. In the future, we focus on the research of attributed network differential privacy with higher data utility.

Data Availability
e data used to support the findings of this study can be accessed from http://konect.cc/networks/.

Conflicts of Interest
e authors declare that they have no conflicts of interest.