Blockchain-Assisted Distributed Fog Computing Control Flow Attestation

,


Introduction
With the rapid development of IoT technology, a large number of embedded devices appear in our lives. A large part of it is deployed in critical information infrastructure and plays an important role. Once these embedded devices are maliciously attacked, it will pose a great threat to our lives. In recent years, control flow hijacking attacks have caused great security threats to embedded devices by tampering with the runtime behavior of programs. Control flow integrity (CFI) [1] was proposed to defend against this threat. e CFI obtains the control flow graph (CFG) of the program by analyzing the normal control flow of the program so that the control flow is transferred within the range limited by the control flow graph, and the execution process of the program is guaranteed to be safe and credible.
Remote attestation is a method of verifying the integrity of software on a remote device. It usually consists of two entities, a verifier who wants to know the state and one or more provers who provide reports of their state. Typically, there is agreement between a verifier and a prover. e verifier accepts a report of the hash value of the running state of the software to be executed signed by the security chip inside the device (such as the Trusted Platform Module, TPM) sent by the prover to verify whether the software state meets expectations. Remote attestation transfers the most expensive part of the entire attestation process to the verifier, thereby reducing the performance overhead of the prover.
In the early years, people used static measurements to verify the state of programs. In this way, the prover obtains a static measurement that is usually a signature or MAC calculated from the hash value of the program code and sends them to the verifier. However, static attestation cannot detect return-oriented programming (ROP) [2] and jumporiented programming (JOP) [3] in control flow hijacking attacks. e ROP attack is the most common code reuse attack today. It does not need to call the complete function block but calls the program code segment and the gadget that ends with the ret instruction in the dynamic link library code. By operating the program running stack, the program control flow is controlled so that the program jumps to the corresponding instruction segment when the execution function returns to achieve the purpose of attack. e JOP attack is achieved by using a chain of gadgets that ends with a Jmp instruction. In recent years, it has been proposed to measure and attest the integrity of the program runtime control flow through the prover [4][5][6] in order to more accurately verify the runtime program control flow. is adds overhead as it gets more contextual information at the basic block level. To balance runtime overhead and control flow security, some people have proposed a mutable control flow attestation scheme based on probability prediction [7] and a granularity adaptive control flow attestation scheme based on Non-dominated Sorting Genetic Algorithm-II (NSGA-II) [8]. A log-based control flow attestation scheme [9] has also been proposed to deal with ROP [2] attacks and uses Physical Unclonable Functions (PUF) as a lightweight root of trust for the prover.
None of the above control flow attestation schemes can adapt to today's explosive growth of embedded devices. ey are all single-server and single-database architectures that cannot handle real-time attestation requests from massive embedded devices, and because they are stand-alone databases, they cannot resist a centrally managed database being tampered with. Once the database is tampered with, it will affect the attestation results of the entire system. Moreover, the resources of the embedded devices are limited. Although MGC-FA [7] and GACFA [8] reduce runtime overhead to a certain extent, it comes at the cost of reducing control flow security. We use a simplified control flow representation model to effectively reduce the runtime overhead without reducing the security of the control flow, making the whole scheme more suitable for resource-constrained embedded devices. Moreover, the control flow attestation schemes mentioned above all use the challenge-response method for control flow attestation, and only perform one-way identity authentication, that is, the verifier authenticates the prover, which will cause great security risks.
In recent years, blockchain technology has been widely used in the Internet of ings [10,11]. A blockchain is a distributed ledger distributed throughout a distributed system where multiple nodes maintain the same information without requiring a central authority. erefore, this technology can not only mitigate the tampering attack of the centrally managed database but also reduce the communication overhead between the data center and the regional manager.
In this paper, we propose a blockchain-assisted distributed fog computing control flow attestation scheme to alleviate the above problems. e main contributions of this paper are as follows: (1) We propose a blockchain-assisted distributed fog computing control flow attestation scheme, which can adapt to today's explosively growing embedded devices and mitigate centralized database tampering attacks. (2) We use fog computing to deploy verifiers and blockchain nodes to the edge of the network, thereby reducing the communication overhead between verifiers and provers and improving the real-time nature of control flow attestation. (3) We propose a simplified control flow representation model by simply using the <source address, destination address, number of jumps> of the jump instruction to represent the program control flow, thereby effectively representing the control flow of the program and reducing the runtime overhead of the prover during the attestation process. (4) We propose a lightweight bidirectional control flow attestation protocol based on elliptic curves, which can greatly ensure the communication security between the verifier and the prover and does not generate excessive performance overhead and communication cost. (5) We use SGX technology to protect the control flow remote attestation, and protect the integrity and confidentiality of the data of the verifier and the prover during the attestation process.

Remote Attestation.
Remote attestation obtains the running status of software on a resource-constrained prover through a resource-rich verifier. is reduces the attestation overhead for resource-constrained devices. e early remote attestation is mostly based on static, which can only protect the binary code when the program is started, but cannot prevent the control flow hijacking attack during the program execution. Later, the proposed C-FLAT [4] realized a more comprehensive runtime remote attestation of the program, and completed the work that could not be done by static remote attestation. e program control flow is protected from alteration by computing the cumulative hash of the basic blocks of program code. During the running of the program, all control flow instructions are intercepted by the runtime tracker trampoline and transmitted to the safe area for hash operation, and finally a hash value representing the execution state of the current program control flow is obtained. e resulting LO-FAT [5] uses a microcontroller to intercept instructions, instead of using a runtime tracker tool for software to intercept instructions in C-FLAT, allowing C-FLAT to be implemented with a lower performance overhead. ATRIUM [6] is a hardware-based runtime attestation protocol that not only checks the control flow of the program but also checks the specific instructions. It provides resilience against software-and hardware-based Time of Check Time of Use (TOCTOU) attacks while incurring minimal area and performance overhead. Although the hardware-based control flow attestation scheme reduces the attestation overhead of the prover, the hardware implementation increases the cost and reduces the scalability of the scheme. MGC-FA [7] uses the probability model in machine learning to predict the fragile probability of each function in the program, thus distinguishing the normal part from the fragile part of the program. It is then subjected to lightweight coarse-grained checks and expensive finegrained checks, respectively, balancing runtime overhead and control flow security. GACFA [8] optimizes the security and performance overhead of the attestation of the control flow of the program using the NSGA-II algorithm and then adaptively performs coarse-and fine-grained checks on the functions in the program according to the optimization results, thus reducing the performance overhead. Liu et al. [9] proposed to record the control flow through a program status log, and the verifier can effectively verify whether the target program has been damaged using the information in the log. At the same time, for more secure storage, they used a lightweight root of trust based on on-chip SRAM Physical Unclonable Functions in the attestation. However, this scheme log records function pointers and function return addresses, so it may be powerless against JOP attacks and attacks against decision data (branch variables or loop variables) in the program. However, in today's environment of explosive growth of embedded devices, none of these solutions can adapt to new challenges-real-time attestation requests of massive embedded devices and attacks on centralized databases are tampered with. Hence, we use blockchain technology and fog computing to deal with these problems. Moreover, there are certain problems in their attestation protocols, so we propose a lightweight bidirectional control flow attestation protocol based on elliptic curve encryption.

Blockchain Technology.
e blockchain is a chain data structure in which blocks are linked in chronological order and are protected by cryptographic algorithms, and the security of the ledger in the entire blockchain network is jointly maintained by means of distributed accounting. At present, according to the open authority of the network [12], the blockchain system can be divided into the public blockchain, the alliance blockchain, and the private blockchain. In this paper, we mainly study the consortium blockchain that is jointly managed by several different organizations or institutions. Unlike the public blockchain, the entities of the consortium blockchain are no longer a single individual but multiple organizations, so as long as most of the organizations in the consortium obtain a consensus, the data can be operated and managed. In a consortium blockchain, it is not the nodes participating in the chain that have permission to access data, but need to be preapproved by the institution to gain access, and because the consortium blockchain is semi-centralized, it is more efficient than the public blockchain.
In the past few years, due to the rapid development of blockchain technology, blockchain technology has been widely used in a large number of fields, such as finance, medical care, Internet of ings, edge computing, etc., for example, the application of blockchain technology in the field of cross-data center authentication of vehicular fog services (VFSs) [13]. By effectively combining modern cryptography and blockchain technology, the communication between service managers during user authentication is eliminated because the records of all service managers are updated synchronously and can effectively resist the attack of tampered database managed by a central because the public ledger is maintained by all service managers. ere are also applications of blockchain technology to distributed data systems IoT [14]. To solve the conflict between the operation performance and security of the blockchain system, the conflict between transparency and privacy, and the compatibility problem of a large number of IoT devices that run together, a distributed data system for IoT based on blockchain technology is proposed. It provides a new system architecture for different industrial IoT devices to deploy high-performance blockchain systems in many scenarios.
ere is also the application of blockchain technology to the radio frequency identification (RFID) supply chain authentication protocol in the 5G mobile edge computing environment [15]. By applying 5G and blockchain technology to the supply chain, the supply chain process can be simplified and allow automatic payment upon receipt of goods. It will help save the company millions of dollars in operating costs by eliminating the need for distributors who have to handle accounts receivable and pay bill with department personnel to track unpaid invoices. Additionally, this will also help avoid the legal fees that it may incur in disputes. However, our extensive literature survey shows that no one has applied blockchain technology to the realm of program control flow attestation.
is article adopts blockchain as the database to improve the security and performance of BDFCFA.
Consensus algorithms are used mainly in distributed systems to ensure data consistency. e consensus algorithm used in blockchain is to solve the "block conflict" problem that may arise when a new transaction block is added to the blockchain. Nowadays, common consensus algorithms in blockchain include proof of work (PoW) [16], proof of stake (PoS) [17], DPoS (delegated proof of stake), practical byzantine fault tolerance algorithm (PBFT) [18] and RAFT algorithm [19], and so on. ese five consensus algorithms have their own advantages, and their performance comparisons are shown in Table 1. First of all, since the management departments in BDFCFA are all credible, the consensus algorithm we choose does not need to consider Byzantine fault tolerance, only crash fault tolerance. Second, since the blockchain in BDFCFA is mainly managed by multiple management departments, the selected consensus algorithm does not need to have a high degree of decentralization. Finally, due to the high real-time requirement of control flow attestation, the selected consensus algorithm should have a lower communication complexity, faster verification speed, and higher throughput. After considering the Byzantine fault tolerance, crash fault tolerance, degree of decentralization, communication complexity, verification speed, and throughput of each algorithm, we chose the RAFT algorithm. Although the scalability of the RAFT algorithm is not strong, its resource consumption is low, and its other properties are also very suitable for the high realtime and high security scenarios of control flow attestation.

Fog Computing.
e concept of fog computing was first proposed by Cisco. By allowing devices deployed at the edge of the network to provide computing, storage, and network transmission services for terminal devices in a small area, the efficiency of data analysis and processing can be improved, and latency and network transmission pressure can be reduced. Today, the rapidly developing IoT faces many new challenges, such as strict latency requirements, network bandwidth constraints, resourceconstrained devices, etc. [20], which cannot be adequately addressed by today's cloud computing and mainframe computing models alone. erefore, to adapt to these new challenges, we adopt fog computing in BDFCFA to deploy verifier and blockchain nodes to the edge of the network, thereby improving the real-time performance of control flow attestation and alleviating the performance bottleneck of the central authority.

2.4.
Intel SGX Technology. SGX (Software Guard Extensions), the Intel Software Guard Extensions, is a set of instruction sets supported by Intel since 2013 [21]. SGX provides an isolated Trusted Execution Environment TEE (Trusted Execution Environment) called enclave, which protects the safe operation and data of legitimate software from malicious attacks. No one, but the CPU, can access the code and data in it. SGX can now provide a trusted execution environment for many application scenarios, such as by introducing a trusted execution environment in edge computing, that is, software protection extension technology, to ensure the confidentiality of the medical IoT data analysis process [22]. ere are also applications of SGX to verifiable confidential cloud computing, which guarantees code and data confidentiality, as well as the correctness and integrity of its results [23]. Confidentiality and integrity are preserved even when large components such as Hadoop, the operating system, and the hypervisor are compromised, and it outperforms Hadoop without SGX protection. ere is also the use of SGX technology in the authentication of the IoT end device, enabling shielded execution of measurements and attestation procedures. e sensitive data in the authentication process are hidden through the specific key of the SGX enclave, which ensures the security of the sensitive data [24]. We enable SGX on the verifier and the prover, and the verifier and the prover process the remote control flow attestation related data in the enclave area, such as attestation request, program control flow data, and attestation report. Because the data processing process is carried out in the enclave area provided by SGX, malicious programs outside the enclave area cannot view the data, let alone tamper with them.

Elliptic Curve.
e application of elliptic curves in cryptography first appeared in 1986, proposed by Miller [25] and Koblitz [26] based on the elliptic curve logarithm problem. After that, elliptic curve cryptography (ECC) became a popular research direction in cryptography and gradually became the mainstream of public-key cryptography.
e security of ECC relies on the discrete logarithm problem of the elliptic curve (ECDLP), that is, for two points P, Q on an elliptic curve over a finite field, solve k so that k·P � Q holds. On a classical computer, the time complexity of solving the discrete logarithm problem of the elliptical curve is exponential [27], which guarantees the security of the encryption of the elliptical curve. Compared to encryption techniques such as DSA, RSA, DH, etc., ECC is a more efficient security encryption technique that uses a shorter key to provide the same level of security. erefore, ECC is often used in the authentication of the identity of devices constrained by resources, such as the Internet of ings. For example, Mahmood et al. [28] applied elliptic curve encryption to the communication authentication of smart grid. eir scheme not only provides mutual authentication with low computational and communication costs but is also resistant to most attacks, while also providing anonymity and privacy. However, Sadhukhan et al. [29] proposed an ECC-based lightweight remote user authentication scheme for IoT devices and remote user authentication, which uses three-factor authentication to protect user privacy and data confidentiality from multiple attacks. Rostampour et al. [30] proposed an efficient scheme to secure communication between IoT edge devices and cloud servers, where the authors used an ECC-based authentication protocol. We design a lightweight two-way control flow attestation protocol based on an elliptic curve. By using elliptic curve encryption, we can effectively ensure the security of communication between the verifier and the prover and at the same time reduce the performance overhead and communication cost as much as possible.

Problem Statement and Assumptions
At present, the implementation of control flow hijacking attacks mainly includes controlled data attacks and noncontrolled data attacks. Among them, control data attacks can be divided into code injection attacks and code reuse attacks according to the source of the attack code. In code injection attacks, the attacker uses the input operation of the software to inject malicious code into the memory space of the target software and then tampers with the control flow data to make the program jump to the malicious code. In code reuse attacks, attackers use existing instructions in programs and shared libraries to achieve attack goals, such as ROP [2] and JOP [3]. Non-controlled data attack is to modify the user identity data, configuration data, user input data, decision data, and other data in the program and then hijack the program control flow to complete the attack, such as (Data-oriented Programming, DOP) [31]. ere are two assumptions about our scheme. First, we assume that the embedded device deploys the DEP (Data Execution Prevention) scheme that makes the executable area read-only, which is a built-in protection scheme commonly used on embedded platforms. erefore, we do not consider code injection attacks. Second, we assume that attackers can carry out code reuse attacks and attacks on decision data (branch variables or loop variables) in the program as well as replay attacks, man-in-the-middle attacks, and impersonation attacks in network attacks. However, the attacker cannot perform any other attack such as pure data attack and physical attack. Figure 1 shows the system framework designed in this paper, which consists of multiple areas. Each zone includes a peer node, multiple fog verifiers, and multiple embedded devices.

BDFCFA System Model
ere are five types of entities in the system.

Management Department.
e management department is a completely credible institution and is a member of the blockchain network.
ere are multiple management departments in the entire system. e management sector includes software service providers and control flow attestation service providers. e software service provider is responsible for registering programs that require attestation of the control flow. When registering a program, first instrument the target program, obtain the control flow data of each possible control flow path, and measure the control flow data of each possible control flow path, respectively, to e measurement calculation formula is shown in equation (1), where H is the final measurement value, and H(0) is the initial hash value, which is set by the management department, and is generally 0. E represents the jump edge in the control flow graph, the data in the edge includes the source address of the jump instruction and the destination address of the jump instruction, and F � f i |i ∈ n is the set of execution times of all the jump instructions in the current path (see Section 5.1 for details).

Key Generation Center (KGC).
e key generation center is another trusted entity in the system, responsible for generating a public-private key pair for the fog verifier and prover in BDFCFA, which is used for identity authentication between a fog verifier and prover in control flow attestation. For the specific key generation process, see Section 5.3.

Peer Node.
A peer node is a member of the blockchain network. In BDFCFA, not every fog verifier needs to maintain the blockchain ledger but establishes a peer node in an area to join the blockchain network, reducing the number of nodes in the blockchain network, thereby reducing the cost of deploying the infrastructure. rough smart contract settings, peer nodes cannot push data to the blockchain in the blockchain network but can only query the data on the blockchain, thereby reducing the possibility of peer nodes becoming malicious nodes. Peer nodes provide query services to fog verifiers by providing services.

Fog Verifier.
e fog verifier provides program control flow attestation services for embedded devices within a certain physical range. Before performing control flow remote attestation, fog verifiers need to register with the key generation center. During attestation, the fog verifier needs to query the blockchain for all expected measurement values of the program to be verified and the program input range corresponding to the expected measurement values. e query process is shown in Figure 2.
First, the fog verifier sends a query request to the peer nodes in the area to which it belongs, and then randomly selects to send a query request to 2n nodes in the blockchain network. Finally, after receiving all the results of the request, the final result is determined according to most principles, and according to this result, we attest to the target program.
ere is a special case, where the randomly selected nodes include the nodes of the management department, and the final result is the result of the node of the management department because the management department is a completely trusted authority. rough such a query mechanism, the harm of attackers tampering with a single database can be greatly alleviated.

Prover.
e prover is an embedded device that needs to attest the control flow of the program running on it. It can be industrial control equipment and other important embedded devices deployed in critical information infrastructure. e program on the prover is provided by the software service provider. Before remote attestation of the control flow, it needs to be registered with the key generation center.

Simplified Control Flow Representation Model.
To authenticate the program control flow, the verifier needs to measure the program running path on the prover. However, it is obviously not feasible for the prover to directly transmit every executed instruction to the verifier. is requires the prover to store lengthy control flow data, resulting in a large performance overhead and attestation delay. To reduce the complexity of program control flow representation, thereby reducing the performance overhead and attestation delay of the prover, this paper proposes a simplified control flow representation model. e model is specifically defined as follows: e simplified control flow representation model is a directed graph G that represents the control flow of a program, represented by a quadruple 〈V, S, D, F〉.  Definition 3. V � v i |i ∈ N , V is the set of all vertices in the directed graph G, and a node represents a basic block. e basic block is a sequence of code instructions with only one entry and one exit without branching, so the basic block ends with a jump instruction. Hence, s i is the exit of v i and d i is the entrance of v i+1 .
is the set of all edges in the directed graph G; there are two types of edges, jump edges and sequential edges. e jump edge indicates that the jump instruction at the exit of the basic block executes the jump, while the sequence edge indicates that the jump instruction at the exit of the basic block does not jump but executes the next instruction. A jump edge is represented by a two-tuple 〈S, D〉.
eoretically, the two-tuple 〈S, D〉 can fully represent the control flow of a program. However, a large number of basic block jump edges will be generated by loops and recursive calls during program operation, resulting in a large number of repetitions of control flow data information and increasing performance overhead. erefore, when we represent the control flow, we choose to use the triple 〈S, D, F〉 to represent it.
Taking Figure 3 as an example, we explain the control flow model and its measurement results.
As shown in Figure 3, the control flow graph has two paths, and their expected measurement results are as follows: H H(H(H(H(H(E1, H(0)), E2), E3), E4), E5), F 1 , H(H(E1, H(0) where F 1 and F 2 are the set of execution times of all jump instructions in the two paths, respectively. Since in the second path, E6 and E7 are sequential edges and no jumps are performed, they are not added to the measurement calculation. e specific measurement calculation formula is mentioned in Section 4. en, we use the simple program shown in Figure 3 as an example to demonstrate the detection of five types of attacks, including the four types of attacks mentioned in Section 3: (1) ROP attack: the attacker tampered with the return address of the function func1 so that the jump edge E5, which should have returned B7 from the exit of the node to the entry of the B6 node, pointed to an illegal malicious code address. erefore, the destination address of the edge E5 will be different from expected, thus detecting that the program has been attacked by control flow hijacking. (2) JOP attack: the attacker tampered with the jump address of the Jmp class instruction so that the jump edge E3, which should have jumped from the exit of node B2 to the entrance of node B5, pointed to an illegal malicious code address. erefore, the destination address of the edge E3 will be different from expected, thus detecting that the program has been attacked by control flow hijacking. (3) Branch variable attack: the attacker tampers with the prover's input so that the sequential edge that should be executed sequentially from node B2 to node B3 becomes a jump edge to jump to the entry of node B5, causing the control flow to enter an unexpected but legal path. However, the input of the prover does not conform to the program input range corresponding to this legal path, so it can also be detected that the program has been attacked by control flow hijacking.  address. e destination address of the edge E1 will be different than expected, thus detecting that the program has been attacked by control flow hijacking.
In summary, our method can detect these five attacks. Figure 4, SGX-based control flow attestation includes two roles of the fog verifier and prover. Each role is divided into a secure area and an insecure area. e SGX enclave is the security area inside the role. We deploy the request generation part and the report verification part of the fog verifier in the SGX enclave. Guarantee the security of the fog verifier request and report the verification process and key information during the remote attestation process of the control flow. At the same time, we deploy the runtime tracking part of the program in the unsafe area in the prover, and deploy the authentication, counting, measurement, and report generation part in the SGX enclave to protect the security of the attestation report, key information, and control flow measurement process. First, the fog verifier generates the attestation request and session key in the request generator and sends the session key to the report verifier and then the attestation request to the prover through the network communicator. e request contains the timestamp, program id, program input in, authentication information, and auxiliary information of encrypted session key. After the prover's network communicator receives the attestation request, it forwards the request to the authenticator. e authenticator first verifies whether the timestamp meets a certain threshold, otherwise the session is terminated. en verify the identity information of the fog verifier and ensure that the verification does not pass the session termination. Finally, the auxiliary information of the session key will be decrypted through the attestation request to calculate the session key, which will be forwarded to the report generator, and the runtime tracker will execute the corresponding program and instrument program according to the program id and program input in the attestation request. When the application runs to the jump instruction, the interceptor in the runtime tracker will intercept it, and then judge whether the instruction is executed for the first time; if so, send the instruction address and destination address to the measurer. e measurer performs hash operation according to formula (1); if not, it sends a signal that the instruction count is incremented by one to the counter, and the counter increments the execution times of the target instruction by one according to the signal. When the program ends, the counter sends the set of execution times of the jump instruction to the measurer for the final hash operation to obtain the final measure, which is then sent to the report generator. e report generator generates an attestation report and sends it to the fog verifier through the network communicator. e fog verifier's network communicator receives the attestation report and sends it to the report verification. e report verification first verifies whether the timestamp meets a certain threshold, otherwise the session is terminated. en verify the identity information of the prover, and the verification does not pass the session termination. en use the session key calculated by the request generator to verify the information in the attestation report, and if the verification fails, the session is terminated. Finally, query the blockchain for the expected measurement value corresponding to the input of the program to be verified and check whether the measurement value calculated according to the attestation report is the same as the expected measurement value. e specific calculation process in the above process is shown in Section 5.3.

SGX-Based Control Flow Attestation. As shown in
e key modules are described in the following: (1) Request generator: this part uses the fog verifier's private key, the prover's public key, timestamp, program id, program input, and a random number to generate the attestation request and session key and send to the network communicator and report verification device, respectively. (2) Report verification: authenticates the attestation report received by the network communicator from the prover and verifies whether the control flow measurement value meets the expectations, thereby determining whether the program on the prover is subject to a control flow hijacking attack. (3) Network communicator: the part of the fog verifier and prover that performs network communication, providing network communication and data forwarding functions for modules in the SGX enclave. (4) Authenticator: the part of the prover that verifies the identity of the fog verifier. e identity of the fog verifier is verified through the attestation request, and the session key is further calculated to generate the attestation report.
(5) Runtime tracker: this part is used to track the target application. We rewrite the Pin tool in Intel Pin-3.15 as a tool for intercepting jump instructions when the program is running.
(6) Interceptor: intercepts the jump instruction when the target application is running, and judges whether it is a jump instruction that has been intercepted. If it is, it will send a signal to the counter that the number of executions of the instruction is incremented by one. If not, it will intercept the received address information of the instruction being sent to the measurer for measurement.
(7) Counter: the part that obtains the execution times of the jump instruction and puts it in the enclave container to protect the execution times of the jump instruction from being tampered with. Because the address information will be hashed after interception and the set of execution times needs to be completed after the program runs, the counter is deployed in the safe area. (8) Measurer: when a hash operation is performed in the enclave container, the security of the measurement process is guaranteed. After the final measurement h is calculated, it is sent to the report generator.
8 Security and Communication Networks (9) Report generator: using the prover's private key, the fog verifier's public key, and the session key calculated by the authenticator, an attestation report is generated and sent to the verifier through the network communicator.

Protocol Scheme Design.
is section proposes a lightweight bidirectional control flow attestation protocol scheme based on elliptic curves. e scheme is divided into three stages: initialization stage, registration stage, and attestation stage. e initialization stage is executed only once by the KGC during the system establishment process. e symbols used in this paper are shown in Table 2.
(1) Initialization stage: in this stage, the KGC performs the following operations to initialize a set of system parameters.
(1) Set a system security parameter k and generate two large security prime numbers p and q. (2) Choose an elliptic curve y 2 ≡ x 3 + ax + b(mod p), denoted by E P (a, b), where a, b ∈ Z p and 4a 3 + 27b 2 mod p ≠ 0.
(3) Choose an additive cyclic group G of order q, and choose a generator P for G, which contains the elliptic curve points defined by E P (a, b).

Security and Communication Networks
(2) Registration stage: this stage completes the registration of fog verifiers and provers, as shown in Figure 5. It is executed only once in the lifetime of each fog verifier and prover. e fog verifier registration process will do the following. e prover registration process is similar to that of the fog verifier, which will not be described here.
(1) e fog verifier uses the hardware information on the device (such as CPU serial number, motherboard serial number, hard disk serial number, physical MAC, etc.) to calculate its unique ID V , and transmits it to the key generation center through a secure and trusted channel. (2) After receiving the unique identity ID V of the fog verifier, the key generation center randomly selects an integer R, connects ID V and R, and uses the formula SK V � H1(ID V � � � �R) to calculate the private key of the fog verifier SK V , SK V ∈ Z * q . en, the public key PK V of the fog verifier is calculated by PK V � SK V •P. Finally, the key generation center transmits PK V and SK V to the fog verifier using a secure and trusted channel.
(3) After the fog verifier receives the keys PK V and SK V generated by the key generation center, the registration process ends.
(3) Attestation stage: this stage completes the attestation of the program control flow on the prover by the fog verifier, as shown in Figure 6. Among them, the fog verifier attests the operation of the program on the prover through a series of operations and judgments, that is, judges whether the running path of the program on the prover is safe and credible. e attestation protocol stipulates that both the fog verifier and the prover can access the binary of the program, and through the traditional static attestation protocol, the program being executed on the prover is guaranteed to be unmodified and complete. e specific process of the attestation stage is as follows: (1) e fog verifier first calculates the shared key DHK used for identity authentication according to DHK � PK p •SK v , and then randomly generates a number n, n ∈ Z * q . N is calculated by elliptic curve multiplication N � n•P, and session key S is calculated using S � N•SK v . Use h1 � H3(in‖T‖id) to hash the program id, program input in, and current timestamp T1 to get h1.
en calculate the XOR of h1, n and DHK, C1 � h1⊕n⊕DHK, and then perform hash operation C2 � H2(C1, S) on C1 to get C2. Finally, the input in of the program, timestamp T1, program id, and identity of the authentication information C1 and C2 are spliced together to form an attestation request Req, which is sent to the prover.
(2) After the prover receives the attestation request Req from the fog verifier, it first checks the freshness of the timestamp. If the difference between timestamps exceeds a certain threshold, the session abruptly ends. en use the secure hash function H3 to calculate h1 � H3(in‖T‖id), and then calculate the shared key DHK′ used for identity authentication through DHK′ � PK v •SK p . rough n ′ � C1 ⊕ h1 ′ ⊕DHK ′ , n' is calculated, which is used to calculate the session key. Use S ′ � n•PK v to calculate the session key S′, and then perform a hash operation C2 ′ � H2(C1, S ′ ) on C1 to obtain C2'.
en Comp(C2, C2 ′ ) compares whether C2 and C2′ are the same; if not, the session is terminated. So far, the prover has completed the identity authentication of the fog verifier.
en the prover executes the target program aid according to the program id and the program input in and obtains the address information [Src 0 , Dest 0 ], ···, [Src n , Dest n ] and the set F of the execution times of the jump instruction.

Protocol Security Analysis.
e lightweight bidirectional control flow attestation protocol based on the elliptic curve in this paper has the following security features: (1) Two-way authentication: the protocol proposed in this paper realizes two-way authentication between the fog verifier and the prover, and the identities of both parties are authenticated before the control flow attestation. Both the fog verifier and the prover use their own private key and the other party's public key when calculating the shared key DHK. It is almost impossible for an attacker to calculate the shared key DHK only knowing their public key. e prover verifies the fog verifier by verifying whether C2 and C2' are the same. e fog verifier verifies the prover by verifying whether Cr and Cr' are the same. e calculation of C2 and Cr will use DHK. erefore, as long as the public key infrastructure is secure, only legitimate fog verifiers and provers can perform control flow attestation.
(2) Anti-impersonation attack: the private key SK V of the fog verifier is strictly kept secret. Although the public key PK V and the generator P are open to the public, it is extremely difficult to calculate the private key SK V from the public key PK V and the generator P, which belongs to solving the elliptic-curve discrete logarithm problem. It is also extremely difficult for an attacker to directly calculate the private key SK V through the ID v of the fog verifier. Because a random number R is added when calculating the private key SK V , it is almost impossible to calculate the private key SK V without knowing the random number R. e prover is similar to the fog verifier, which will not be described here. erefore, in the protocol proposed in this paper, attackers cannot pretend to be fog verifiers and provers and can prevent impersonation attacks. (3) Anti-man-in-the-middle attack: when A and B communicate, the attacking host C becomes a forwarder in the middle, and the information between them is forwarded by C. C can not only eavesdrop on the communication of A and B but also tamper with the information. en pass it on to the other party. We assume that the middleman C intercepts the attestation request Req = in ||T1 || id ||C1 ||C2 and forwards it after maliciously tampering with the attestation request. If the middleman tampers with the data in in, T1 or id, then the prover will not be able to get the expected value by calculating h1' � H3(in‖T‖ id), resulting in failure to compare C2 and C2'. If the middleman tampered with the data of C1, the prover will not be able to calculate the correct n' through n' � C1⊕h1'⊕DHK', resulting in the termination of the session. If the middleman tampered with the data of C2, it will directly lead to the failure of comparing C2 and C2'. Similarly, if the middleman C intercepts the attestation report Rep = T2 || hs || Cr, and forwards it to the fog verifier after maliciously tampering with the attestation report. Assuming that the middleman has tampered with the data of T2, the fog verifier will make an error in calculating Cr' � H2(T2‖h', DHK), resulting in a failure to compare Cr and Cr'. If the middleman tampered with the data of hs, the fog verifier will not be able to decrypt the final measurement value h' with h' � hs⊕S, resulting in the termination of the session. If the middleman tampered with the Cr data, it will directly lead to the failure of comparing Cr and   erefore, for the protocol proposed in this paper, the attacker cannot achieve the purpose of the attack through the man-in-the-middle attack. (4) Anti-replay attack: the replay attack is that the attacker sends an authentication information that the destination host has received to deceive the other party. In the protocol proposed in this paper, both the attestation request and the attestation report contain the timestamp T, which is not only sent in clear text but is also hidden in C1 and C2 in the attestation request and Cr in the attestation report. erefore, if an attacker replays the attestation request from the fog verifier or the prover's attestation report, the fog verifier and the prover can identify it by checking the freshness of T. If the attacker replaces a new timestamp T′ in the attestation request or attestation report, the identity authentication of the prover or fog verifier will also fail. Because the prover makes an error when calculating C2, which causes the authentication to fail. e same applies to fog verifiers. e protocol proposed in this paper ensures that the attestation request or report of each transmission is different and there is no leakage of any valuable information, so the attacker cannot deceive the other party by replaying the intercepted message. (5) Known session key security: the fog verifier uses the random number n, n ∈ Z * q to calculate N, and then uses the private key SK v and N to calculate the session key S. Since n for each calculation of the session key is randomly selected, the attacker cannot obtain other session keys through the known session key. It is not even possible to obtain any valid information from the known session key because the session key needs to use the private key SK v of the fog verifier in addition to N, and it is almost impossible to obtain any valid information only knowing S.

Query Mechanism Performance Evaluation.
We use Hyperledger Fabric version 1.4.12 to build a blockchain network for query mechanism performance evaluation. e configuration of the blockchain network built by Hyperledger Fabric is shown in Table 3.
In this network, we use fog verifiers to randomly query 1, 3, 5, and 7 peer nodes 45 times, respectively, and get the average query time, as shown in Table 4. It can be seen that although the increase in the number of query peer nodes will increase the query time, in fact, compared with the entire attestation process, this overhead is not large. During the attestation process, we ask fog verifiers to randomly query 3 peer nodes in the blockchain network. e average attestation overhead of obtaining fog verifiers after 45 remote attestations is 29.50 ms. e overhead of attestation includes the cost of the fog verifier to generate the attestation request and the cost of the fog verifier to verify the attestation report, which includes the query time. is overhead is acceptable for fog verifiers. In summary, our query mechanism does not incur too much overhead, while mitigating the harm of database tampering by attackers.

Runtime Overhead Evaluation.
Since our BDFCFA scheme is aimed at embedded devices, to evaluate the performance of BDFCFA, we use the SNU real-time benchmark [32] to test BDFCFA. ere are many C files in this benchmark, such as adpcm-test.c and fft1.c for embedded platforms. In addition, since the runtime overhead is mainly determined by the number of control flow events, we mainly compare the number of control flow events for each scheme when evaluating the runtime overhead.
e experimental results of the runtime overhead are shown in Table 5. We compare the MGC-FA [7] and GACFA [8] with our BDFCFA scheme. e experimental data on the runtime overhead of MGC-FA come from the literature [7]. e scheme with p = 0.3 in MGC-FA is the scheme proposed in the literature [7]. e scheme when p = 0 is the same as the control flow checking scheme in the literature [33], and the scheme when p = 1 is the same as the control flow checking scheme in the literature [8]. Runtime overhead is the average value obtained after the experiment is repeated many times. As can be seen from the table, the number of BDFCFA control flow events is less than or equal to the number of control flow events when the probability threshold p = 0 of the MGC-FA scheme. Among them, the number of control flow events of the three programs adpcmtest.c, fft1k.c, and lms.c is very different and is not of the same order of magnitude. Although the probability threshold for the MGC-FA scheme is p = 0.3 or 1, there are some programs with fewer control flow events than our BDFCFA. However, when the probability threshold is p = 0.3 or 1 of the MGC-FA schemes, they reduce the number of control flow events by sacrificing the control flow security, thereby reducing the runtime overhead. Because the number of control flow events directly affects the timeto check and measure control flow events, and the time overhead of checking and measuring control flow events accounts for the vast majority of the runtime overhead. Similarly, GACFA makes its control flow events smaller than BDFCFA on the premise of sacrificing control flow security. In summary, our BDFCFA scheme effectively reduces runtime overhead without sacrificing control flow security, making the whole scheme more suitable for resource-constrained embedded devices.

Attestation Protocol Performance Evaluation.
Since our attestation protocol is used for control flow attestation, we compare the performance of the challenge-response based unidirectional control flow attestation protocol used in [4][5][6][7][8] with our elliptic curve-based bidirectional control flow attestation protocol. We use two attestation protocols separately in the BDFCFA scheme for comparison. Among them, the elliptic curve used by the bidirectional control flow attestation protocol based on the elliptic curve is 256 bits, and the secure hash function is also 256 bits. e challengeresponse based unidirectional control flow attestation protocol uses 256-bit elliptic curve based ECSDA and 256bit hash function.

Communication Cost Performance
Analysis. First, we theoretically analyze the communication cost of the two protocols. In the challenge-response-based unidirectional control flow attestation protocol, the verifier first needs to send a challenge C containing the program id, the program input in, and the random number N to the prover. e prover needs to return an attestation report R containing the expected measurement value h, the challenge C, and the digital signature S obtained by signing the expected measurement value and the challenge. In the bidirectional control flow attestation protocol based on the elliptic curve proposed in this paper, the fog verifier first needs to send an attestation request Req containing the input in of the program, the timestamp T1, the id of the program, and the identity authentication information C1 and C2. e prover needs to return an attestation report containing the timestamp T2, the encryption measurement value hs, and the identity authentication information Cr. e theoretical communication costs of the two protocols are shown in Table 6.
We assume that the size of program id, program input in, timestamp, and random number N are all 4 bytes, so the size of challenge C is 12 bytes. Since the bidirectional control flow attestation protocol based on an elliptic curve uses a 256-bit elliptic curve, the sizes of C1, C2, and Cr are all 32 bytes, so the size of the attestation request Req is 76 bytes and the size of the attestation report Rep is 68 bytes. erefore, the communication bandwidth consumption of the bidirectional control flow attestation protocol based on the elliptic curve is 144 bytes. e challenge-response-based unidirectional control flow attestation protocol uses 256-bit elliptic curve-based ECSDA and a 256 bit hash function, so the size of the attestation report R is 108 bytes. e communication bandwidth consumption of the challenge-response-based unidirectional control flow attestation protocol is 120 bytes. It can be seen from the data that the communication bandwidth consumption of the two protocols is almost the same.

Protocol Operational Efficiency Analysis.
We use the SNU real-time benchmark [32] to compare the runtime overhead of the two protocols on the prover. By performing multiple experiments on nine programs selected in the SNU real-time benchmark and averaging them, we obtain a comparison chart of the runtime overhead of the two protocol provers, as shown in Figure 7. It is obvious from the figure that the prover's runtime overhead difference between the bidirectional control flow attestation protocol based on the elliptic curve and the unidirectional control flow attestation protocol based on the challenge response is very small, only one to two milliseconds. erefore, the runtime overhead of the two protocols for the prover is almost the same.
We also compared the attestation time of the verifiers of the two protocols. e attestation time here refers to the time when the verifier generates the attestation request plus the time when the verifier verifies the attestation report, and does not include the time caused by the communication between the verifier and the prover. Any program in the SNU real-time benchmark will only generate an attestation request of the same size for the verifier and will only receive an attestation report of the same size. erefore, we used the two protocols to perform 45 repeated experiments on the adpcm-test.c program in the SNU real-time benchmark test   and obtained the comparison chart of the attestation time of the verifiers of the two protocols, as shown in Figure 8. It can be seen from the figure that the verifier attestation time of the unidirectional control flow attestation protocol based on the challenge response is shorter than that of the bidirectional control flow attestation protocol based on the elliptic curve. Among them, the average attestation time of the verifier of the bidirectional control flow attestation protocol based on the elliptic curve is 29.50 ms, and the average attestation time of the verifier of the unidirectional control flow attestation protocol based on challenge response is 25.53 ms. e difference between the two is close to 4 ms, and the difference is not big.
By analyzing the communication cost and operation efficiency of the two protocols, the communication cost and operation efficiency of our proposed bidirectional control flow attestation protocol based on the elliptic curve are very small compared to the unidirectional control flow attestation protocol based on the challenge response. However, our proposed protocol is more secure and can greatly ensure the communication security between the verifier and the prover. erefore, our protocol is more suitable for program remote control flow attestation in the field of software security.

Security Performance Evaluation.
We compare the security of our scheme BDFCFA with some control flow remote attestation schemes in recent years, as shown in Table 7. Since the security of the GACFA [8] scheme is related to specific procedures, there is no security comparison here. Both C-FLAT [4] and MGC-FA [7] can detect all control flow hijacking attacks proposed in the table when the probability threshold p � 0. However, it is not resistant to impersonation attacks, there is no two-way authentication, and there is no way to maintain the security of the known session key. However, LO-FAT [5] and ATRIUM [6] cannot resist man-in-the-middle attacks except for the security features that C-FLAT does not have. Because the attestation protocols of these two schemes only digitally sign the random number in the attestation request when generating the attestation report, the attacker can completely intercept the attestation request and tamper with the id or input of the program to achieve the purpose of the attack. When MGC-FA has probability threshold p � 1, in addition to not having the security characteristics of probability threshold p � 0, it cannot detect the JOP attack, branch variable attack, and loop variable attack in control flow hijacking attacks. e reason for this is because MGC-FA only measures function pointers and function return addresses in the program when The elliptic curve based bidirectional control flow attestation protocol The challenge-response based unidirectional control flow attestation protocol Average attestation time for elliptic curve-based attestation protocol Average attestation time for challenge-response-based attestation protocol the probability threshold p � 1. Liu et al. [9] used PUF as a lightweight root of trust for the prover, which can resist impersonation attacks and achieve known session key security. However, since it only digitally signs the log, it is also not resistant to man-in-the-middle attacks. Moreover, it does not perform two-way authentication and only records the function pointer and function return address in the program in the log, so only the ROP attack and function pointer attack in the control flow hijacking attack can be detected. Our scheme BDFCFA not only can detect all control flow hijacking attacks proposed in the table but also resist network attacks such as impersonation attacks, replay attacks, and man-in-the-middle attacks, and at the same time realizes two-way authentication, ensuring the security of known session keys, and each session key is different.

Conclusion
is paper proposes a blockchain-assisted distributed fog computing control flow attestation (BDFCFA), which can not only adapt to today's explosive growth of embedded devices, reduce the communication overhead between the verifier and the prover, and improve the real-time performance of control flow authentication but also mitigate the attack of the centralized database being tampered with. At the same time, we use SGX to protect the integrity and confidentiality of the verifier and prover data during the certification process. In addition, the query mechanism adopted by the fog verifier when querying the measurement data of the program from the blockchain network can spend less time overhead, thereby mitigating the harm of the attacker tampering with the database. Compared to MGC-FA [7], the simplified control flow representation model used in our scheme can effectively represent the control flow of the program under the premise of ensuring the security of the control flow, thus reducing the runtime overhead of the prover in the attestation process.
rough comparative experiments with the challenge-response-based unidirectional control flow attestation protocol, it can be inferred that our proposed bidirectional control flow attestation protocol based on the elliptic curve can greatly protect the communication security between the verifier and the prover and does not generate excessive performance overhead and communication costs. is protocol is more suitable for program remote control flow attestation than the challenge-response-based unidirectional control flow attestation protocol used in the program control flow scheme by previous researchers. Finally, by comparing the security of BDFCFA with some remote control flow attestation schemes in recent years, it can be seen that the BDFCFA scheme has the highest security and can better protect the security of program control flow attestation. In summary, BDFCFA can adapt to today's explosive growth of embedded devices, improve the real-time performance of control flow attestation, alleviate the harm of attackers tampering with the database, reduce the runtime overhead of the prover during the attestation process, and greatly protect the security of the communication between the verifier and prover, and does not produce excessive performance overhead and communication cost.
In the future, we will study remote control flow attestation based on the combination of dynamic and static measurements. Because of the current runtime control flow measurement, only the control flow data when the program is dynamically running are measured. However, if the attacker tampers with the binary code of the program on the premise of ensuring the original control flow, the existing control flow remote attestation will not be able to detect this behavior.

Data Availability
No data were used to support this study.

Conflicts of Interest
e authors declare no conflicts of interest in this work.