Computer systems and applications on the internet provide services to outsiders and, at the same time, the vulnerabilities may be exploited by attackers and leak some sensitive private information. To collect and monitor the service information provided by the network environment such as IoT (Internet of Things), vehicular networks, cloud computing, and cloud storage, it is particularly important that a system can provide faster service discovery for discovering and identifying specific network services. The current service discovery systems mainly use port scanning technology, including Nmap, Zmap, and Masscan. However, these technologies hard code the service features and only support common services so that cannot cope with real-time updates and changing network services. To solve the above problems, this paper proposed a customizable distributed network service discovery system based on stateless scanning technology of Masscan and proposed a customizable interactive pattern set syntax. The system used random destination address technologies to scan for Ipv4 address allocation and used a distributed deployment scheme. Experimental results show that the system has high scanning speed and has high adaptability to new services and special services.
With the growth of internet devices and applications, various large scale cyberattacks continue to emerge, and internet vulnerabilities also show a surging trend [
The IoT, vehicular networks, cloud computing, cloud storage, and other environments can provide users with flexible and convenient service access [
In order to solve the above problems, we designed a customizable distributed network service discovery system (CDNSDS) in this paper. The main contributions of this work are as follows:
We designed a system architecture, which includes three subsystems: central control subsystem (CCS), schedule control subsystem (SCS), and scanning proxy subsystem (SPS). The CCS is the brain; it receives the user’s instructions and manages and assigns tasks to the SCS. The SCS is the bridge connecting CCS and multiple SPS. The last subsystem is the SPS, the key factor for performance. We optimized Masscan, the most efficient scanning tool currently, and used a distributed program to improve concurrency To be customizable, we had compiled a pattern set of syntax conventions. The syntax conventions can convert the user’s customized services description, including interactive service, to standard syntax which is accepted by a scanning tool in the SPS
The rest of the paper is organized as follows: a related work is described in Sections
In the study of empirical security, fast Internet-scale network service discovery has opened a new avenue, while scanning technology plays an important role. One of the earlier scanning tools is Nmap [
Although Zmap has greatly improved in performance, scanning technology is still in progress. The fastest internet port scanner Masscan [
Except for the above famous scanners, a number of research efforts focus on empirical security. In order to scan anonymously, Rodney et al. [
The traditional service discovery systems may cause issues such as triggered IDS alarm and single-node detection poor performance. In this paper, we design and implement the CDNSDS and the architecture is shown in Figure
System architecture.
CDNSDS includes three subsystems.
Central control subsystem (CCS): it is the brain, which receives a user’s instructions and manages and assigns tasks to the SCS. Users can get the task process and results and manage the attribute and state of the scanning node. In this subsystem, we design a pattern set of syntax conventions to support customizability. Schedule control subsystem (SCS): it is a bridge connecting CCS and multiple SPS. It provides task division, scheduling management, and results of the temporary service. Scanning proxy subsystem (SPS): it is the key factor for performance. The SPS consists of several distributed agent modules. Each agent is a scan node with optimized Masscan that performs real-time scan task from the SCS.
The CCS provides users with customizable service probe interfaces. A pattern set of syntax conventions is defined for customization as follows.
We use
Table
The property of pattern set.
Name | State | Description | Format | Essential |
---|---|---|---|---|
msg | After the state is transferred to this node, the text in the msg attribute needs to be filled into the application layer load; the data packet is constructed and probed | Hex/string | Y | |
waiting | After sending the probe packet in this state, state will be transferred to receive state when receives a response packet | Y | ||
isbanner | Output application layer load to output file | True/false | N | |
patterns | Probe pattern set, which is used to guide state transitions | Hex/string | N | |
len | Limit of payload length | Range | N | |
goto | The state is after filtering of patterns and len | {index, state_with_id} | Y |
State transition diagram.
In this example, there are three kinds of state node: (1) the green solid nodes
The SCS is aimed building an efficient and reliable communication environment between the CCS and SPS, while providing intermediate data storage and high-speed read service.
The SCS contains three modules: state management module, message queue module, and cache module.
Status management module: to better understand the survival status and scanning progress of each scanning agent node, the SCS is logically responsible for building the communication environment between the central control system and each scanning node. At the same time, to deal with the problems such as downtime of the CCS and change of server address, the SCS also provides an interface to dynamically manage the connection configuration of the scanning nodes to ensure the normal delivery of heartbeat packets and scanning progress packets. Message queue module: it contains task queue management and result queue management. The detection tasks issued by the CCS will split into specified slices for smaller granularity and detection. Each scanning agent node consumes only one slice at a time, and these task slices will be handed over to the task queue management. From the scanning agent’s point of view, each slice represents a scanning task, and the result of the task may be success, fail, or timeout. The task queue manages the various results that may exist after each task slice is received. The slice that fails to scan is reenlisted for other scanning nodes to probe again. When the task is successfully scanned, the result data will be passed back from the scanning node to the result queue, which will store that result slice temporarily for the CCS. Similarly, the result queue will also manage the status of the results processed by the central control system as described above. Cache module: after the scanning node successfully detects the target address set, it will pass the result slice back to the result queue and then open the next detection task. Due to the large number of potential detection nodes, if each slice’s results are stacked in the CCS, it will increase the pressure on its storage and processing. Therefore, the module provides a dumping service to the cache and notifies the CCS to consolidate, deduplicate, and persist the results after receiving. Thus, the cache module provides memory-level high-speed data processing functions.
For each packet received, the SCS will determine whether it is a task slice, heartbeat data, or task result.
Task slice packet: it is handed over to the “task queue” to manage and monitor the execution (dispatch) result of this slice. Heartbeat packet: it is forwarded to the central control system to update the survival and progress status. Task slice result data: it is temporarily stored in the cache module. The task slice result data is temporarily stored in the cache module to remind the central control system for integration.
When the SPS performs a scan task, there are four situations after sending the first TCP handshake request.
SYN-ACK: the host port of corresponding target is open, and we can continue to probe service. RST: the target is open, but the destination port is close. ICMP unreachable: the target is close. No response: connection timed out.
It is obviously that in the first case we can keep detecting while other cases can be directly abandoned.
Normally, scanners based on semiconnected state will send RST to close connection after receiving SYN-ACK. Such a scheme is not suitable for interactive detection. This issue can be resolved by two possible solutions.
Using the operating system protocol stack, reestablish the connection to the open target port for deep probing Send ACK to finish three-way handshake instead of RST
The first solution theoretically provides reliable connection, but the number of connections is limited. The second solution requires a user-mode protocol stack and is more efficient than the former. Fortunately, Masscan already provides this functionality. Therefore, the second solution is adopted in this paper.
In order to record all active connections, a TCP connection table is needed to maintain the management of Transmission Control Block (TCB) which contains all the important information about the connection, as shown Figure
TCP connection table in user-mode protocol stack.
The interactive service detection hierarchy is displayed in Figure
Interactive service detection hierarchy.
During service discovery, packets need to bypass the original system stack; otherwise, the original system stack will send RST packet because of the absence of connections. This paper proposed two solutions.
ARP cheat: send an ARP packet with an unreal IP in same subnet to router. Modify Linux iptables: drop traffic with a specified port.
The CCS delivered tasks in the form of fragments. Under the premise of address randomization, in order to avoid duplication of detection intervals of all nodes, the system sets the range as
Denote IP segment
Scan range mapping set is follows:
Using the above mapping set, the conversion from the index of the range to host addresses and ports can be achieved according to equations (
After serialization, let us suppose fragment range set is
Due to the condition
1. 2. If 3. Return 4. Else 5. Return 6. EndIf
1. 2. For 3. If 4. 5. Else 6. 7. EndIf 8. 9. EndFor 10. If 11. Return 12. Else 13. Return b 14. EndIf
This method is a modification of Feistel encryption, function is
The process of randomize address recovery is as follows.
1. 2. If 3. return 4. Else 5. return 6. EndIf
1. If (r is odd) Then 2. 3. Else 4. 5. End 6. For 7. If 8. 9. Else 10. 11. EndIf 12. 13. EndFor 14. Return
In order to satisfy cluster operations and distributed schedule, the system adopted Docker Swarmkit and deployed in 5 nodes. Among them, in Docker Swarm Mode, the CCS and SCS are deployed in the admin node, and another five SPS are deployed in other nodes. The system deployment diagram is shown as Figure
System deployment diagram.
In this paper, we choose a single node to probe HTTP service, the target host segment is 169.54.23.0/16, and the probe port is 80. Each set of experiments is the average of three testing. The testing results are shown as Table
The results of interactive service detection.
Speed (pkt/s, k: 103, w: 104) | Service | Port | Service time (s) | Port time (s) |
---|---|---|---|---|
0.1k | 3078 | 3167 | 852 | 713 |
0.5k | 3190 | 3154 | 178 | 150 |
1k | 3239 | 3240 | 95 | 79 |
2k | 3151 | 3157 | 52 | 46 |
3k | 3033 | 3080 | 38 | 34 |
5k | 2189 | 2251 | 26 | 25 |
1w | 1218 | 1054 | 17 | 17 |
According to the data, we can get the following charts. As Figures
Interactive service detection results chart (k: 103, w: 104).
Interactive service detection time chart (k: 103, w: 104).
There are a total of five nodes for testing; the pattern set comes from the analysis of the protocol icoco with port 80. In order to compare the distribute platform and single node, we test in the following two aspects.
Scanning range is fixed, and sending rate changes Sending rate is fixed, and scanning range changes
For the first aspect, the target host segment is 169.54.23.0/10 and port is 80. The results are shown as Table
The results of testing in multimode.
Speed (k: 103, w: 104) | Multinode: results/time (s)/target | Single-node: results/time (s)/target |
---|---|---|
0.5k | 25641/1743/3 | 25691/8630/3 |
0.7k | 25764/1248/3 | 25669/6174/3 |
1k | 25666/877/3 | 25576/4334/3 |
2k | 25709/448/3 | 25496/2174/3 |
4k | 25510/230/3 | 25389/1094/3 |
6k | 25246/157/3 | 25373/734/3 |
8k | 25377/122/3 | 25520/557/3 |
1w | 25459/101/3 | 25527/447/2 |
2w | 25046/57/3 | 25026/230/2 |
3w | 24968/43/3 | 25212/160/2 |
4w | 24844/36/3 | 25137/124/3 |
5w | 23063/31/3 | 25118/103/2 |
6w | 21221/28/3 | 25132/88/3 |
7w | 19018/26/2 | 24998/74/2 |
8w | 16642/25/3 | 25277/69/3 |
9w | 15190/24/2 | 25169/64/3 |
10w | 13740/25/3 | 25203/59/3 |
20w | 10590/25/1 | 25317/47/2 |
Fixed scanning range (k: 103, w: 104).
For the second aspect, the fixed sending rate is set 1000 pkt/s, the results for different scope are shown in Table
The impact of scope on the result.
Scope | Multinode: results/time (s) | Single-node: results/time (s) |
---|---|---|
/24 | 18/13 | 15/13 |
/22 | 89/14 | 89/14 |
/20 | 199/14 | 200/17 |
/18 | 842/16 | 854/34 |
/16 | 2862/29 | 2854/96 |
/14 | 8271/74 | 8247/323 |
/12 | 13226/237 | 13405/1129 |
/10 | 25961/878 | 26281/4341 |
/8 | 40920/3427 | 40632/17102 |
/6 | 343482/13881 | 338102/69277 |
Fixed sending rate.
As can be seen from Figure
Consider ratio changes at the same sending rate of the single-node and multinode, as shown in Figure
Single-node and multinode ratio diagram.
As can be seen from Figure If the detection range is fixed, the lower the sending rate, the easier it is to approach the ideal ratio If the sending rate is fixed, the larger the detection range, the easier it is to approach the ideal ratio
Based on this conclusion, for better detection results, the system parameters can be adjusted by three factors: the number of nodes, the range of detected host, and network bandwidth.
The current service discovery system cannot deal with real-time updates and changing network services. Existing scanners only support the probing of common public protocols. This paper designed a Customizable Distributed Network Service Discovery System (CDNSDS) to solve the issue. CDNSDS consists of three subsystems: CCS, SCS, and SPS. In the CCS, a pattern set of syntax conventions is defined to assist users in customizing scan features. At the same time, the SPS provides an efficient Masscan-based scanning module. In the SPS, we describe interactive detection technology, including TCP connection management, and randomize target address in detail. Finally, the Docker Swarm Mode is used to distribute container choreography, and the experiment shows that the CDNSDS has high efficiency and accuracy, especially in industrial control protocols.
As a future work, the system should extend the syntax of the pattern set to make it better adapted to the changing protocol, such as dynamically constructing the sending packet for the reply packet. At the same time, it is necessary to calculate the relationship expressions of the transmission rate, transmission range, and the optimal ratio mathematically to arrange the distributed nodes to conduct detection with higher timeliness.
The data used to support the findings of this study are available from the corresponding author upon request.
The authors declare that there is no conflict of interest regarding the publication of this paper.
This work is supported by the National Key R&D Program of China (2016QY05X1000) and the National Natural Science Foundation of China (201561402137).