Real-Time Testing of Synchrophasor-Based Wide-Area Monitoring System Applications Acknowledging the Potential Use of a Prototyping Software Toolchain

This article presents a study on real-time testing of synchrophasor-based “wide-area monitoring system’s applications (WAMS application).” Considering the growing demand of real-time testing of “wide-area monitoring, protection, and control (WAMPAC)” applications, a systematic real-time testing methodology is formulated and delineated in diagrams. The diagrams propose several stages through which an application needs to be assessed (sequentially) for its acceptance prior to implementation into a production system. However, only one stage is demonstrated in this article which comprises the use of a prototyping software toolchain and whose potential is assessed as suﬃcient for preliminary real-time testing (PRTT) of WAMS applications. The software toolchain is composed of two components: the MATLAB software for application prototyping and other open-source software that allows ingesting prerecorded phasor measurement unit (PMU) signals. With this software toolchain, a PRTT study is presented for two WAMS applications: “testing of the PMU/phasor data concentrator (PDC)” and “testing of wide-area forced oscillation (FO) monitoring application.”


Introduction
e main use of synchrophasor technology has been in the field of "wide-area monitoring, protection and control" (WAMPAC) [1][2][3][4]. However, the full potential of "phasor measurement units (PMUs)-enabled wide-area network (WAN)" has not been fully realized in the wide-area monitoring and control (WAMC) center, while many novel and advanced WAMPAC applications are proposed yearly, and the rate of adoption of new applications in WAMC centers is much lower. In other words, the adoption of new applications in WAMC centers is slow despite their availability in the literature.
is is because the transmission system operator (TSO) in WAMC demands the application to be tested and approved based on a certain eligibility criterion before its real-time implementation [5,6]. is eligibility criterion differs from application to application based on how each specific application operates. However, there is one of the foremost eligibility criteria that must be met by all WAMPAC applications, that is, "real-time testing." e demand for real-time testing continues to grow, and this trend is unlikely to change with technological advancement and the deployment of low-cost PMUs [7,8] at a large scale. e other reason behind the slow adoption of WAMPAC applications is the lack of a clear generalized and systematic real-time testing methodology in the literature.
Real-time testing poses several challenges, and it is not only needed for application testing but also for power system component testing as well [9]. Encoding an algorithm into a software application in a real-time environment requires additional expertise than what is required for offline software applications. Figure 1 depicts the gap between the TSO's demand and researchers' limitations which is the major roadblock in real-time testing of WAMPAC applications and thus is a roadblock in its frequent upgradation in WAMC. Indeed, the WAMPAC applications should meet real-time performance requirements, but more importantly, the approach to test applications should also be systematic while at the same time considering the researchers' limitations as depicted in Figure 1. However, staff from the National Institute of Standards and Technology (NIST) of the United States have proposed a requirement test framework for applications that has not yet gained adoption, and it does not formalize a testing methodology [10]. To the best knowledge of the authors, there is no clear systematic testing methodology reported yet that could be used for real-time testing of WAMPAC applications in a time-efficient manner so that the gap depicted in Figure 1 can be reduced.
One successful method for thorough testing requires a laboratory testbed consisting communication-enabled "realtime-simulator" which may be coupled with hardware-inloop (HIL), software-in-loop (SIL), power HIL (PHIL), and control HIL (CHIL) setup [11][12][13][14][15]. In [15], a synchrophasorbased voltage stability monitoring application was tested with (1) positive-sequence based (PSB) simulations, (2) PMU data from real-time-HIL, and (3) PMU data from a real Nordic grid transmission system, sequentially. e authors shared their learned lessons that any synchrophasorbased application must be tested thoroughly with actual PMU measurements so that the developed application with PSB simulations can be modified to become robust for more realistic features such as noise and outliers. However, a more common approach is to ignore all aspects of data communications, i.e., a communication-less approach and perform an application's testing only by playing back simulation data or measurement records [16]. e major issue to incorporate communication-based real-time testing in 2 nd and 3 rd stages lies in the fact that such testing requires all-time access to signal steam/broadcast until the application is tested satisfactorily. Unlike prerecorded data, such all-time access to signal steam/broadcast is difficult to attain in 2 nd and 3 rd stages due to limited laboratory access hours and security-protocol issues, etc.
With the hype of advanced laboratories, the potential of exploiting a software toolchain remained unacknowledged in the literature which can be very helpful in communication-based preliminary real-time testing (PRTT) before attempting testing in an advanced laboratory. It allows to prototype "wide-area monitoring system's applications (WAMS applications)," enhance, and test them (preliminarily) without the need of a laboratory. e PRTT stage is proposed herein as a part of a real-time testing methodology as will be discussed in detail later. e main contributions of this article are as follows: (i) Formulation of a real-time testing methodology for synchrophasor-based WAMPAC applications (ii) Establishing an approach to exploit a software-only toolchain for the PRTT stage of the proposed realtime testing methodology (iii) Demonstration of the software toolchain for PRTT of WAMS/WAMS applications e remainder of the paper is organized as follows: in Section 2, with brief discussion on the WAMPAC system, a methodology is formulated for real-time testing of WAM-PAC applications. In Section 3, open-source software for real-time synchrophasor communication is discussed. Section 4 presents the software toolchain. Section 5 demonstrates the software toolchain for PRTT of WAMS/WAMS application, and lastly, the conclusions are drawn in Section 6.

WAMPAC System and Methodology
Formulation for Its Real-Time Testing 2.1. WAMPAC System. Typical deployment of the WANenabled WAMPAC system is shown in Figure 2. Generally, the WAMPAC system and involved applications are divided into three categories: (1) wide-area monitoring system (WAMS), (2) wide-area protection system (WAPS), and (3) wide-area control system (WACS). Classifying how synchrophasor-based applications are tested results in two categories: (1) WAMS and (2) WAPS/WACS as depicted in Figure 2 in the red oblong label as "WAMPAC." e differences between these categories will be discussed in the sequel. In many cases, system operators may apply protection and control actions after assessing the WAMS application's results. Such an assessment based on the operator's own experience is generally referred 'situational awareness' [17], as depicted in Figure 2. Among the two categorizations previously mentioned, the latter one, i.e., the synchrophasor-based WAPS/WACS is still in its infancy as far as real-time grid operation is concerned, and there are relatively few applications deployed in wide-area systems [18]. On the other hand, the first categorization, i.e., the synchrophasor-based real-time WAMS is a widely studied part of the WAMPAC system and is also deployed in many power systems, majorly for situational awareness purposes. e developed application under both categorizations should be tested systematically, and thus, a real-time testing methodology is formulated next.

Formulation of the Real-Time Testing Methodology.
is section provides a testing methodology to reduce the gap depicted in Figure 1 by formulating a systematic real-  time testing methodology. If T is the specified domain of a performance metric that must be achieved by a WAMPAC application A in its testing, then it can be deemed successfully tested if the following condition holds true: where T is the operator that is operated on A to achieve its performance metrics T and T is the actually achieved performance. In practice, T can be referred as a testing methodology and T could be the set of performance criteria, e.g., execution time < specified time, application's output index < pre-defined threshold, etc. Achieving equation (1) can be regarded equivalent to the stamp of 'tested and approved' on A.
For generalized testing of any A, the existing literature does not report any method to formulate T. Conventionally, this testing methodology T requires a costly laboratory testbed. Indeed, such a testbed is needed to test whether meets the performance criteria or not, under multiple realistic power grid conditions, before it could be deemed successfully tested. However, performing laboratory tests with naively implemented WAMPAC applications might be difficult to justify especially at initial development stages. However, a communication-enabled software toolchain can greatly help in PRTT and debugging of WAMS/WAMS applications, and thus, a significant amount of time and effort can be reduced prior to laboratory tests. Figure 3 shows the formulated methodology for real-time testing of WAMPAC applications involved in WAMPAC systems, which will be described next in detail.

Proposed Testing Methodology.
e legend shown at the top left corner of Figure 3 should be perused first before following the remainder of Figure 3. is legend specifies a "green box," a red asterisk, and three different arrows, boldred, dotted-red, and bold-blue. ese arrows describe the paths on which methodology is applied from the one testing stage to another after passing the preceding stage. As mentioned earlier, WAMPAC applications A are categorized into two categories based on the involved testing methodology, i.e., A 1 : WAMS applications and A 2 : WAPS/WACS applications and A � [A 1 , A 2 ]. Similarly, the testing methodology T can also be segregated into two parts, i.e., T 1 : testing methodology for A 1 and T 2 : testing methodology for A 2 and T � [T 1 , T 2 ]. By virtue of this categorization, Figure 3 is segregated into two parts where the light-orange colored part indicates T 1 and the light-blue colored part indicates T 2 . Both parts comprise multiple testing stages grouped in to "offline," "pseudo-online," and "online" stages and these groups are represented by gray colored boxes. Now, A 1 , A 2 needs to be passed/moved along the shown path for their approval by their respective parts T 1 , T 2 comprised their respective grouped-stages. Formulated methodology can be understood by thorough observation shown in Figure 3 in conjunction with the following noteworthy points: (i) e "testing" and "enhancement" of A are interlinked to each other. Testing results may call for enhancements to the application and enhancement would call for testing. erefore "testing" and "enhancement" are a recursive process that ends when testing results yield specified performance metrics or until equation. (1) is achieved.
(ii) As depicted in the column's head in Figure 3, the application not only is being tested in the offline/ online stages but will also go through debugging (enhancements). How fast an application performs to meet real-time requirements, depends on how A is implemented in the software prototype. (iii) Before moving to laboratory testing (red asterisk boxes in Figure 3) with A, it is desirable to apply the testing stages (unasterisk boxes in Figure 3) to debug A, which will minimize the cost and effort of performing laboratory tests (iv) Differences in T 1 and T 2 are as follows: (1) as apparent from Figures 2 and 3, the online stages of T 1 require PMU signals to be broadcasted as input, whereas the stages of T 2 require the ascertained monitoring results by T 1 as input.
(2) T 2 essentially requires the power system modelling data in all its stages, whereas it is not required in all stages of T 1 , as apparent from Figure 3 because synchrophasorbased monitoring not necessarily needs the power system modelling data. (v) e findings of WAPS/WACS depend on the monitoring results that are the output of the WAMS applications. ese results are important as the performance metrics of A 1 aid in bounding those of A 2 , as can be observed in Figures 2 and 3. (vi) For A 1 to be tested in T 1 , the attributes to be monitored in the PMU signals must be known a priori by ancillary studies, e.g., eigenvalue analysis, online test case libraries [19], or self-created attributes via simulation. (vii) Condition in equation (1) can be divided into T 1 IT 1 (corresponding to A 1 ) and T 2 IT 2 (corresponding to A 2 ). Meeting such conditions implies that requirements of A 1 and A 2 have been fulfilled in their PRTT, as depicted in the upper right corner of Figure 3. e stage highlighted by the box with a pink outline in T 1 (see Figure 3) is identified as the most important stage (PRTT stage) in the formulated testing methodology which avails the facility of the pseudo online environment. is stage utilizes a software toolchain to test A 1 that can broadcast the prerecorded PMU data similar to how commercial PMUs stream data and where the data can be retrieved by an application A in real time. Even though the PRTT stage is not sufficient for commercial real-time testing because it uses the prerecorded data, still, it helps in preliminary testing and enhancement of A 1 before entering the final testing stage (red asterisk box in T 1 ). e remainder of this article presents a study that is focused only on the PRTT stage. (1) signal broadcaster (publisher and server) and (2) signal retriever (subscriber and client). e former type serves to broadcast prerecorded/stored PMU signals as similarly performed by commercial PMUs/PDCs, and the latter type helps retrieve the broadcasted signal. Table 1 lists such pieces of open-source software, some of which were briefly reviewed by the authors of [12], which could be followed there if required. e detailed comparative study between these listed pieces of software is out of the scope of this article. A software toolchain can be formed by combining one signal broadcaster and one signal retriever, and one of each type is discussed next.

PUPO: Signal Broadcaster.
Among the signal broadcasters listed in Table 1, 'PMU-PDC stream simulator Or' (PUPO) [24] is found to be a suitable signal broadcaster to form the software toolchain, which is developed in C++. To briefly describe PUPO, the main GUI is shown in Figure 4. e merits of PUPO over others are summarized as follows: (i) Easy distribution and deployment (by an '.exe' file or small size). (ii) A simple and intuitive user-friendly graphical user interface (GUI).

International Transactions on Electrical Energy Systems
Being a PMU/PDC emulator, the inner workings of PUPO should not be mistaken with a conventional software PDC. PUPO was designed to emulate the PMU/PDC data streaming functionalities by broadcasting the prerecorded signals by following the same protocols used by commercial PMUs/PDCs. e current version of PUPO consumes the prerecorded data in a.csv file (editable in any tool, e.g., Notepad/Notepad++) and uses an extension name, ".phcsv". e prerecorded data file can be imported by choosing the option 'file-based input' given under each phasor tab. e polar form of the PMU data given in the "csv" file has to be first converted in a rectangular form before entering it into the input file. e method of preparing the input file and other necessary steps are described in [24]. By default, the PUPO has one PMU station in which three phasor signals can be imported. However, both of these numbers can be increased/decreased by clicking on the 'folder/cross' icon as depicted in Figure 4.

SADF: Signal Retriever.
e recently developed "synchro-measurement application development framework (SADF)" [12] is the MATLAB-based software that enables the retrieval of TCP, UDP, or mixed TCP/UDP synchromeasurement data. To the best of the authors' knowledge, SADF is the only MATLAB-based open-source toolbox available for such a purpose to date. It enables MATLAB to retrieve the PMU data in real time with additional facility to perform parallel computations leveraging the parallelization tools within MATLAB. SADF can be used for real-time prototyping of WAMPAC applications [12]. Due to the promising potential of SADF and vast functionalities available with the widely used MATLAB, SADF was selected from the signal retrievers given in Table 1 to form the software toolchain. e script, "SADF_setting" allows us to configure the PMU/PDC connection settings such as the TCP/IP, port, device ID, and maximum time of retrieval. e main script, Table 1: Open-source software for real-time communication.

Signal broadcaster
Signal retriever (i) PMU-PDC stream simulator (C++) [24] (i) SADF (MATLAB) [12] (ii) PyPMU (Python) [20] (ii) S3DK (Labview) [25] (iii) iPDC (Linux) [22] (iii) Babelfish (Labview) [23] (iv) OpenPDC (Java) [26] (iv) Phasor toolbox (Python) [21] (v) OpenPDC (Java) [26]  International Transactions on Electrical Energy Systems 'SADF_run' allows us to embed a custom code for a WAMPAC application [12]. e user is required to develop an understanding of the default script, 'demo_WAMS' to restructure and encode their WAMPAC application as per the compatibility with SADF. is default script is given to plot the retrieved signal in real time with its specifications mentioned on the plot. e retrieved signal is being stored in the MATLAB workspace which can also be utilized later for the offline analysis.

PUPO-SADF Software Toolchain
As indicated before, PUPO and SADF are the simplest software in their respective categories ( Table 1) that can communicate with each other through the TCP/IP protocol (Internet) on a single computer. By the combination of these, a software toolchain is formed named, 'PUPO-SADF toolchain,' which is found suitable for the PRTT stage (pinkoutlined box in Figure 3). is software toolchain provides a real-time environment for testing and enhancement of any A 1 at the PRTT stage. e flowchart for the complete PUPO-SADF toolchain is shown in Figure 5. Firstly, the prerecorded PMU data given in the standard "csv" file need to be edited in any text editor (e.g., Notepad/Notepad++). is file is then imported into the PUPO, and then, the broadcast can be started as per IEEE C.37.118.2 compliance by pressing the 'start simulator' button as depicted in Figure 5. en, SADF can retrieve the data in real time and can execute A 1 (with parallelization, if needed) consuming the retrieved data segment/window w(n). e time length of this window T w can be defined by the user as per their choice at the beginning of the encoded application A 1 . e computed results can be monitored in real time through the MATLAB plot of an individual's choice as written in the encoded application A 1 .

Monitoring Resolution: Speed in Real-Time Monitoring.
For any application, A 1 to be executed in real-time, the computational time, T A 1 c must be lower than T w i.e. T A 1 c < T w [27]. e authors [28] have used the term 'near real-time' instead of 'real-time.' Indeed, one application can achieve real-time requirements better than another based on T A 1 c . erefore, a new index, i.e., "monitoring resolution" Ψ is defined herein as follows: Ψ relates to the speed of the A 1 in real time and Ψ � 100 % is practically impossible because no application can produce the results in zero seconds. erefore, the value of Ψ lies in the range of 0 < Ψ < 100, and a high value of Ψ signifies that real-time performance is being met for A 1 . However, the index Ψ should not be mistaken for the overall performance of any application. Ψ is just one index among several indices required to govern overall performance. For example, the application developed in [29] for oscillation monitoring utilized a mode decomposition technique, and thus, it will give low Ψ due to the heavy computation burden involved in comparison to a simple power spectral density (PSD) operation. On the other hand, the application developed by authors of [29] would give better oscillation detection results subjected to mode mixing problems [29]. In other words, if there exists any detection accuracy index D, then the application described in [29] would have a better score than the PSD operation, but Ψ would be compromised. It is obvious that the application's overall performance would be a function of Ψ and D, as well as other indices. ere are no sets of indices defined in the literature to govern the global performance of a particular synchrophasor application yet, and further discussion on this aspect is also omitted here. As mentioned before, the proposed software toolchain provides a real-time environment for testing and enhancement of any A 1 at the PRTT stage shown in Figure 3. After qualifying this stage, the application moves to the next and final testing stage (red asterisk box) where a real-time-HIL laboratory testbed is required. If any A 1 is qualified/ approved in the PRTT stage, then it is expected to meet the performance requirements of the final stage, i.e., the red asterisk box/stage of T 1 in Figure 3 or at least it will assist in the final testing stage. Next, the PUPO-SADF toolchain for the PRTT is demonstrated.

Real-Time Testing of Synchrophasor-Based WAMS Using the PUPO-SADF Toolchain
Before continuing, it is important to mention here that the PUPO-SADF toolchain is exclusively utilized in the PRTT stage; however, the SADF can be utilized in another online testing also where the signal stream is accessible in real time.
A demonstration of the PUPO-SADF toolchain is presented in two WAMS applications (1) testing/validation of PDC emulation and (2) testing of wide-area forced oscillation (FO) monitoring application. Even though the first one is not related to application testing, it helps to verify that PUPO can be used to develop monitoring applications with it. e prerecorded PMU data from North American synchrophasor initiative (NASPI) [30] are considered for the analysis. e data file "NASPI-2014-Workshop-Oscillation-Case2.csv," which belongs to "oscillation detection: test Case 2", consists of 30 signals of 10 min sampled at 60 Hz. e prerecorded voltage phasor signals from Bus-06 and Bus-01 are broadcasted using PUPO which can now be treated as if they were wide-area PMU signals streaming in real time from a commercial PDC.

Testing of PUPO for Authentic WAMS Results.
Testing of PMU/PDC compliance is one of the important parts of WAMPAC application's testing [31]. e testing of commercial PDC requires a rigorous approach considering several compliance aspects; however, in the case of PUPO's testing in this article, the objective is to ensure validity of results for real-time WAMS applications. e broadcasted signals are retrieved by the SADF by inputting TCP/IP and port settings of PUPO. Testing of PUPO is a postretrieval process. e retrieved 10 min voltage magnitude signals along with time stamps are stored in the MATLAB workspace in the variable, "DATA.Magnitude' and 'DATA.-Timestamp." Now, the testing of PUPO includes two aspects to validate real-time monitoring results as follows.

Check for Synchronized
Overlapping. e recorded NAPSI signals are already synchronized in its data file (.csv) [30], and it is necessary to ensure that the signals while being broadcasted by PUPO should also remain synchronized.
is check can be performed by plotting retrieved and broadcasted signals in one figure. If the retrieved signals perfectly overlap the respective broadcasted signals, then it can be accepted that the PUPO had broadcasted the signals correctly. To this end, 'array alignment' (time-frame shifting) of the retrieved signals with prerecorded (PUPObroadcasted) signals needs to be assessed. is is because the PUPO broadcasts the imported array/signals (".phcsv") in a loop without encountering any time delay in streaming from the end sample to the first one, and thus, it is unlikely that the first sample of the retrieved array/signal in the MATLAB workspace is exactly the same as that of the inputted signal in PUPO. Whatever array-alignment rule is followed for one array (signal) should also be followed by all other arrays. If the same array-alignment rule can overlap all retrieved signals with respective broadcasted signals, then the 'synchronized overlapping check' can be passed. Figure 6(a) shows the plot of both prerecorded ('.csv') and both retrieved signals by following the one single 'array-alignment' rule. It can be seen that both the retrieved signals overlap with the respective broadcasted signals; therefore, this check can be passed for PUPO.

Check for the Sampling Rate.
e purpose of this check is to verify that the PUPO has broadcasted the prerecorded signals at its original sampling rate, F s , and if it is uniform throughout. is is an important test as A 1 may require F s to compute the monitoring results, e.g., A 1 comprising the PSD operation will require F s . is check can be performed by plotting the retrieved timestamps stored in the workspace variable, "DATA.TimeStamp." Figure 6(b) shows the plot of retrieved timestamps versus the sample number in which it is apparent that the PUPO had indeed broadcasted at the original sampling/data rate F s � 60 Hz as 60 samples (xaxis) have been retrieved every one second (y-axis), and also, this F s remained uniform. erefore, this check is also passed for PUPO. us, PUPO has been tested "OK" for the testing objective defined earlier. However, Figure 6(b) suggests one bug in the current version of PUPO that does not affect the defined testing objective, but it will create a problem in plotting the retrieving signal. e identified bug is further discussed and rectified next.

Discussion on an Identified Bug and Its Rectification.
Looking at the x-axis and y-axis in Figure 6(b), it can be ascertained that the retrieved timestamps are following the staircase pattern instead of a linear pattern because the PUPO is streaming the same timestamp 60 times in one second. is is because the GPS-clock quality of PUPO is limited to seconds (HH:mm:ss) and not up to milliseconds (HH:mm:ss:sss).
is corresponds with PUPO's own repository [24] which states that the "commit button does not gray out for setting PMU station settings." is is also shown here with the help of Figure 7 where it can be seen that due to nonfunctioning of the highlighted commit button, the clock quality of PUPO cannot be more granular. To further understand the problem with this bug, notice that the bug has caused the repeated entries at the y-axis/array in Figure 6(b) which correspond to the timestamps at the x-axis in the realtime signal. erefore, the real-time plot would not be possible unless the repeated timestamps, i.e., staircase patterns in Figure 6(b) are corrected with a linear pattern.
Notice that this rectification is equivalent to improving the clock quality of PUPO. is is achieved by using the 'linspace' command on retrieving the data window w(n) in the embedding A 1 , which is as follows: w timestamps (n) � linspace w timestamps (1), w timestamps (end), T w × F s .

(3)
Readers may follow next to next figure before reading further to observe how the real-time signal retrieving plot is made possible by rectifying the bug as described above. Notice that the PUPO's bug is not rectified in PUPO but is rectified in the SADF. Next, the PUPO-SADF toolchain as a whole provides a suitable mean for PRTT of any A 1 .

FO Monitoring Application.
In this section, the FO monitoring application utilizes the magnitude-squared coherence (MSC) tool [16] for spectral analysis. e MSC estimate is a powerful spectral tool for oscillation monitoring that has been shown to yield better results as compared to the PSD [16]. e authors in [27] indicated that without having any prior information it is very challenging to distinguish oscillations from ambient responses by only International Transactions on Electrical Energy Systems using PSDs and spectrums.
e MSC estimate C xy , is a function of frequency f which reflects how well the sequence x(n) correlates to another sequence y(n) at any frequency f where f ∈ R. e region of space R maps the space [0(F s /2)] where F s is the common sampling frequency of both the sequences x(n)andy(n). e mathematical expression of MSC is written in the following equations [27,32]:  where P xx (f) and P yy (f) are the PSDs of x(n)an dy(n), respectively, and P xy (f), is the cross PSD (CPSD) between x(n)an dy(n). e values of the MSC estimate are real and lie between 0 and 1 as indicated in equation. (5). If x(n)an dy(n) are two sinusoidal signals of frequency f x and f y , respectively, then the following equation holds true: e application utilizing the MSC estimate is encoded in the script, "smart_WAMS" (A 1 ) using the 'mscohere' function of the MATLAB's 'signal processing toolbox' that computes the MSC estimate between two signals. Before MSC, the application first preprocesses the signals by using the 1 st order high-pass Butterworth filter with a cut-off frequency of 0.01 Hz. Increasing the window length T w increases the frequency resolution but reduces the time resolution [27]. e time length of the data window T w is considered 30 sec herein. A Hamming window is chosen to minimize leakage noise [27]. e application also incorporates an automatic peak detector with the help of the function "findpeaks". is function detects the peaks in the MSC estimate in real time and shows it on the MSC plot to provide real-time visualization of the FO frequency. e threshold C thres xy (f) for peak detection is set to 0.1; i.e., if C xy (f) ≥ C thres xy (f),then the peak will be detected in MSC indicating the FO detection at its corresponding frequency. e major steps in the encoded 'smart_WAMS' application are delineated in the flowchart shown in Figure 9.

Real-Time FO Monitoring Using the PUPO-SADF
Toolchain.
e script "smart_WAMS" is embedded in the main script of SADF, i.e., "SADF_run". e script "SAD-F_run" is edited to compute the MSC and plot it which is being updated in real time. e script 'smart_WAMS' also plots the segmented MSC in real time. e method to plot the segmented MSC is the same as for the segmented PSD and segmented self-coherence described in [32]. Here, the 30 sec window w(n) is slid 30 times for the next retrieving 30 sec. As a result, 30 MSC estimates are obtained and plotted as a segmented MSC. e recorded screen capture for this real-time retrieval and monitoring can be viewed in [33]. As the SADF begins retrieving and plotting the signals, the MSC plot (colormap) begins showing the results only after the retrieval of the predefined length of the window T w . e retrieving signals and their MSC estimate are also shown in Figures 8(a) and 8(b), respectively, for one instant in the real-time monitoring.
As the embedded script "smart_WAMS" runs and updates the figures in real-time [33], it is also made to save the colormap (segmented MSC) every 30 secs by using the "saveas" command placed in the script. e purpose of performing this is to show the real-time monitoring [33] by a set of relevant figures (video frames), as shown in Figure 10. Figure 10 Results. From Figures 6(a) and 10 and the recorded screen [33], it can be noticed that the first frame ( Figure 10) is associated with the early sec of the signals (Figure 6(a)) wherein the 13.3 Hz FO is observable. e second frame (Figure 10) shows the monitoring of pre-event seconds (Figure 6(a)) wherein it can be noticed that the spectrum band of the 13.3 Hz FO has widened. e occurrence of an event at 272 seconds ( Figure 6(a)) triggered another 1.25 Hz FO which was sustained over the event's duration. e third frame ( Figure 10) shows the monitoring of the signals during the period when the first outliers appear in the data (see Figure 6(a)). e presence of outliers can be International Transactions on Electrical Energy Systems noticed along with the newly triggered 1.25 Hz FO. When outliers are present, the MSC estimate does not provide useful information. It can be noticed in the fourth frame ( Figure 10) that after the first outlier is passed, both FOs sustained, and the widened spectrum band of 13.3 Hz FO has reverted to narrow. e last frame is the monitoring results during the period when additional/second outliers (Figure 6(a)) finished passing through the window w(n). It can be noticed that after the outliers are passed and after the event stopped at around 587 seconds (Figure 6(a)), the 1.25 Hz FO disappears and 13.3 Hz FO still sustains. e discussed analysis can be verified from the video described in [33]. e presence of these FOs is also confirmed by the authors of [34] in an offline study.  Figure 3).

5.2.5.
Discussion. Next, the additional potential of the software toolchain to be used in the final testing stage is discussed. As mentioned before, the PRTT stage is the only stage that is demonstrated in this article; however, there are potential uses in the final testing stage, i.e., the red asterisk   box/stage of T 1 in Figure 3. After being approved in the PRTT stage, the designed 'smart_WAMS' application is ready to move to its final testing stage, where the only modification required is to change the connection IDs in the SADF's script from PUPO's IDs to laboratory PMU/PDC's IDs. No further efforts have to be made in a laboratory in rewriting the application unless it fails tests when subjected to multiple realistic power grid conditions. e software toolchain can help here too by storing the PMU/PDC signals recorded from the failed tests and replaying them using the toolchain to debug the application while avoiding the cumbersomeness and limited access time at the laboratory. In this way, the PRTT stage incorporating the PUPO-SADF toolchain is highly beneficial in professional testing of WAMPAC applications.

Conclusions
is article presents a study on real-time testing of synchrophasor-based WAMSPAC applications. A real-time testing methodology is formulated and specified in multiple stages. A few offline testing stages are incorporated along with online testing stages, and it is expected that the application passes (for approval) all the stages sequentially, making the testing process more efficient and less cumbersome. e potential uses of a software toolchain to assist in the testing process were acknowledged and found sufficient for PRTT of WAMPAC applications. e software toolchain is formed using MATLAB and the PUPO opensource software which is referred as the PUPO-SADF toolchain. With this software toolchain, the PRTT is demonstrated in two WAMS applications, i.e., 'testing of PMU/ PDC' and 'testing of wide-area FO monitoring application'. In the first one, a minor bug was found and rectified in custom encoding within the SADF, while in the latter one, FOs at frequencies 13.33 Hz and 1.25 Hz are detected and monitored in real time using the recorded PMU data provided by NASPI.

Abbreviations:
WAMPAC: Wide-area monitoring, protection, and control PMU: Phasor measurement unit PDC: Phasor data concentrator FO: Forced oscillation PRTT: Preliminary real-time testing WAN: Wide-area network WAMC: Wide-area monitoring and control TSO: Transmission system operator HIL: Hardware-in-loop SIL: Software-in-loop PHIL: Power HIL CHIL: Control HIL WAPS: Wide-area protection system WACS: Wide-area control system PUPO: PMU-PDC-stream simulator SADF: Synchro-measurement application development framework NASPI: North American synchrophasor initiative MSC: Magnitude-squared coherence PSD: Power spectral density CPSD: Cross PSD T, achieved: Performance A: WAMPAC application lim x⟶∞ T: Testing methodology T: Specified domain of performance-eligibilities A 1 : WAMS-applications A 2 : WAPS/WACS applications T 1 : Testing methodology for A 1 T 2 : Testing methodology for A 2 C xy : MSC estimate P xx (f): PSD T w : Window length T c : Computation time Ψ: Monitoring resolution.

Data Availability
e utilized data can be sought by approaching NASPI in the link https://www.naspi.org/.