Random Error Reduction Scheme for Combinational Stochastic Circuit

In conventional stochastic computation, all the input streams are Bernoulli sequences (BSs), which may result in large random error. To reduce random error and improve computational accuracy, some other sequences have been reported as alternatives to BSs. However, these sequences only apply to the specific stochastic circuits, have difficulties in hardware generation, or have length constraints. To this end, new sequences without these disadvantages should be considered. This paper proposes the random error analysis method for stochastic computation based on autocorrelation sequence (AS), which is more general than the conventional one based on BS. The analysis results show that we can use the proper ASs as input streams of stochastic circuits to reduce random error. On the basis of that conclusion, we propose the random error reduction scheme based on maximal concentrated autocorrelation sequence (MCAS) and BS, both of which are ASs. MCAS and BS are applicable to any combinational stochastic circuit, are easily generated by hardware, and have no length constraints, which avoid the disadvantages of sequences in the previous work. Moreover, we apply the proposed random error reduction scheme into several typical stochastic circuits as case studies. The simulation results confirm the effectiveness of the proposed scheme.


Introduction
Stochastic computation, first introduced in 1960s [1], has been widely used in many fields: neural network [2,3], digital image processing [4][5][6][7], channel decoding [8][9][10], MIMO detection [11,12], reliability evaluation [13], and so on.The main idea of stochastic computation is to represent continuous values with stochastic bit streams, which makes it possible to perform complex arithmetic computations with simple bitwise operations.One obvious advantage of stochastic computation is that the hardware implementation of the stochastic arithmetic is much simpler than the traditional arithmetic.For example, as shown in Figure 1, multiplication and scaling addition can be performed with an AND gate and a multiplexer, respectively.The simplicity of hardware also brings short critical path and high clock rate.Moreover, all the bits in a stochastic bit stream have the same significance.That means the stochastic arithmetic is unary coding arithmetic which has better fault tolerance than the traditional binary arithmetic.Recently, skew tolerance has also been introduced as another inherent advantage of stochastic computation.
Despite the above advantages, a disadvantage of stochastic computation is that the random fluctuations of stochastic bit streams introduce the random error, which causes the computation result to be a random variable.In [4], it is suggested that, in most cases, the random error will dominate the other errors such as quantization error and approximate error.Thus, the random error should be carefully investigated and addressed in stochastic computation area.
Thus far, many statistical models for random error analysis have been proposed.They are all based on variance evaluations.In the early work, it is assumed that all the input streams of stochastic circuits are BSs.Qian et al. [4] propose the variance evaluation model base on BSs.However, in real application, the random error for stochastic arithmetic based on BSs is often too large.Therefore, people have made effects to reduce random error by using other sequences as input streams of stochastic circuits.
Braendler et al. [3] employ two deterministic bit streams (DBSs) as input streams for the stochastic circuit AND gate.The output stream is also deterministic, which means there are no random fluctuations.Although it eliminates random fluctuations and improves computational accuracy, the use of these DBSs is narrowly restricted to the single AND gate.They can not apply to more complex stochastic circuits.
Alaghi and Hayes [14] propose a general framework for analyzing stochastic circuits with correlated inputs.They demonstrate that using correlated inputs can significantly reduce the random error for some stochastic circuits.However, the use of correlated inputs has its limitation.For example, as illustrated in [15], if the two input streams applied to an AND gate are correlated, say, each has the same bitpattern of value , then the output will also have value  instead of  2 .Hence, the AND gate will not perform multiplication if its inputs are correlated.Similarly, many other commonly used stochastic computational elements can not perform the expected arithmetic with correlated inputs, which largely restricts their applications in stochastic computation systems.
Han et al. [13] use fixed-ones random-permutation sequences (FRSs) as input streams for stochastic circuits.They present the variance evaluation model for the basic stochastic computational elements, including AND gate and invert.Furthermore, they prove that the use of FRSs as input streams reduces random error for stochastic computation compared to the conventional use of BSs.FRSs can be wildly used in stochastic arithmetic.They apply to any combinational stochastic circuit.In [13], FRSs are generated by software and used as input streams of the stochastic circuit for reliability evaluation.However, the low cost hardware generation of FRS is difficult to implement.Thus, FRSs are not suitable for stochastic circuits with input streams generated by hardware.
From above, we can see that the main issues of stochastic arithmetic based on different input streams are as follows: (i) BSs: large random error; (ii) DBSs or correlated inputs: only applicable for the specific stochastic circuits; (iii) FRSs: difficulties in hardware generation; (iv) PWM signals: length constraints.
To avoid these disadvantages, we need to use other proper sequences as input streams.Considering the autocorrelation between different bits of a sequence, we propose the notion of AS.And then we analyze the random error of stochastic arithmetic on the assumption that input streams are ASs.By definition, BS can be seen as a special AS with equivalent autocorrelation coefficients.Thus, the proposed random error analysis method based on AS is more general than the conventional one based on BS.
According to our analysis results, if proper ASs are used as input streams, the random error can be reduced.Based on that conclusion, we investigate how to reduce random error for any combinational stochastic circuit by employing MCASs and BSs, both of which are ASs, as input streams.Then, We adopt several typical stochastic circuits as case studies and find that the use of MCASs and BSs can significantly improve the calculation performance.MCAS has no difficulties in hardware generation and the hardware cost of the MCAS generator is merely the same as that of the conventional BS generator.Moreover, from the simulation results, the random error reduction scheme based on MCAS and BS is valid for any input stream length and has no length constraints.Hence, by using MCASs and BSs as input streams, we avoid the disadvantages in the previous work and achieve the following goals: (i) Random error reduction: the random error is reduced compared to using only BSs as input streams; (ii) Wide applications: the random error reduction scheme based on MCAS and BS is applicable to any combinational stochastic circuit; (iii) Easy hardware generation: the MCAS and BS generator are easily implemented by hardware; (iv) Flexible length: the MCAS and BS have no length constraints.
Parts of this paper are based on our prior work entitled "Random Error Analysis and Reduction for Stochastic Computation Based on Autocorrelation Sequence," which has been presented at the IEEE International Symposium on Circuits and Systems (ISCAS), 2014.In the conference paper, we briefly present a general random error analysis method for stochastic computation and illustrate how to reduce random error for a typical stochastic circuit.Compared with the conference version, this journal manuscript has more new contents updated and more insightful discussion, which are stated as follows: (i) Detailed illustration of the general random error analysis method is provided.
(ii) The random error reduction scheme for any combination stochastic circuit is proposed.
(iii) The complete mathematical proof for the effectiveness of the proposed random error reduction scheme is given.
(iv) More typical stochastic circuits are presented as case studies.
The rest of the paper is structured as follows: Section 2 analyzes the random error based on AS.Section 3 illustrates the random error reduction scheme based on MCAS and BS for any combinational stochastic circuit.Section 4 presents several typical case studies.Section 5 gives simulation results.Finally, we give conclusions and future directions of research.(

Random Error Analysis Method
The expectation of  is defined by and the variance of  is defined by Consider an arithmetic operation  with  normalized input numbers  1 ,  2 , . . .,   and a result , which can be expressed by  = ( 1 ,  2 , . . .,   ).In the stochastic implementing of , each input number   , 1 ⩽  ⩽ , is represented by the corresponding stochastic bit stream, denoted by   , where    =   .We employ these stochastic bit streams as input streams of stochastic circuit.Then, the stochastic computational result, denoted by   , can be obtained from the output stream  as follows: There are two important statistics for stochastic arithmetic.The first one is the expectation of output stream.Since   should be an unbiased estimator of the actual result , we have From ( 2) and ( 4), ( 5) can be rewritten as   =  = ( 1 ,  2 , . . .,   ).Thus, the expectation of output stream indicates the arithmetic operation performed by stochastic circuit.
Another important statistic is the variance of output stream.Considering that   is an unbiased estimator of the actual result , random error can be calculated in the form of mean square error (MSE) as MSE =  ((  − ) 2 ) =  ((  −  (  )) 2 ) = Var (  ) .
By ( 3) and ( 6), MSE = Var  .Therefore, the random error for stochastic computation can be measured by the variance of output stream.

Autocorrelation Sequence.
For a bit stream  with length of , the correlation between two different bits can be measured by autocorrelation coefficient, which is defined by    = (()()), 1 ⩽  <  ⩽ .For example, if  is a BS, its autocorrelation coefficients are    = (  ) 2 , 1 ⩽  <  ⩽ ; that is, all the autocorrelation coefficients of BS are equivalent to the square of its expectation.
In the practical applications, input streams may have a variety of autocorrelation coefficients.Considering that, we propose AS, whose autocorrelation coefficients are not necessarily equivalent to each other.Definition 1.A bit stream , which represents the probability , is called an AS on the following condition:   = .
From Definition 1, AS is more general compared to BS and BS can be seen as special case of AS whose autocorrelation coefficients are equivalent to the square of its expectation.

Random Error Analysis Based on AS.
For clarity, we define the following notations: let BI denote the case that all the input streams are BSs; let AI denote the case that all the input streams are ASs; let (BI) denote the expectation of output stream  in the case of BI; let (AI) denote the expectation of output stream  in the case of AI; let Var(BI) denote the variance of output stream  in the case of BI; let Var(AI) denote the variance of output stream  in the case of AI.
For any stochastic circuit, we expect that the arithmetic computed in the case of AI is the same as that in the case of BI.As illustrated in Section 2.1, the expectation of output stream indicates the arithmetic operation performed by stochastic circuit.Hence, the expectations of output stream should be equivalent to each other in the two cases and we have Equation ( 7) is the essential condition for the random error analysis in the case of AI.On the premise of ( 7), the random error base on AS can be measured by the variance of the output stream  as follows: Example 2. As shown in Figure 1, two stochastic bit streams  1 and  2 are employed as the input streams for an AND gate.When  1 and  2 are BSs, ( The expectation of output streams can be calculated as Now, consider that the input streams  1 and  2 are ASs with the following properties: Without losing generality, we suppose that ( 1 ()) =   1 .The expectation of output stream can be calculated as By ( 9) and ( 10), ( 7) is satisfied.Thus, the random error can be calculated by (8) as follows: Hence, the variance of output stream in the case of AI is determined by the expectations and autocorrelation coefficients of input streams.

AI Affection Evaluation.
In order to evaluate how much AI affects the random error, we propose differential variance, denoted by DV, where DV = Var(AI) − Var(BI).As suggested in [4], the variance of output stream in the case of BI can be calculated by Then we have the following theorem.
Theorem 3. The range of DV is as follows: Proof.See our prior work [17].
According to Theorem 3, the upper bound of DV is nonnegative and the lower bound of DV is nonpositive.Thus, DV can be positive, which indicates that the improper use of AS will result in the increase of random error.On the other hand, DV can be negative, which indicates that the proper use of AS will reduce random error.Therefore, we get the following conclusions.
(i) Since the random error in the case of AI can be less than that in the case of BI, the random error reduction scheme based on AS is feasible.
(ii) Since the random error in the case of AI can be greater than that in the case of BI, it is important to select proper AS to avoid this situation.

Random Error Reduction Scheme
In the last section, we find that the proper use of AS is able to reduce the random error for stochastic computation.On the basis of that conclusion, we will discuss how to employ ASs as input streams for the combinational stochastic circuit to reduce the random error.

Combinational Stochastic Circuit.
For any combinational stochastic circuit, the relationship between input streams and output stream can be expressed by where   1, is a constant integer and   ∈ {0, 1}, 1 ⩽  ⩽ .
For convenience of illustration, we denote  1, by the subscript The set of subscripts of PISs in stochastic circuit is denoted by  pis .

Mathematical Model Based on MCAS and BS Definition 6.
If a bit stream  has the following properties: where  is the length of , it is called an MCAS.
Consider the combinational stochastic circuit with the  input streams  1 ,  2 , . . .,   and an output stream .In order to indicate the sequence type for each input stream, we introduce the state vector V = ( 1 ,  2 , . . .,   ), where There are totally 2  possible states for V.For instance, if  = 2, the four possible states are follows: (MCAS, MCAS), (MCAS, BS), (BS, BS), and (BS, MCAS).The expectation and variance of output stream , when state vector is V, are denoted by (V) and Var(V), respectively.As discussed in Section 2.3, in order to guarantee that the arithmetic computed by the combinational stochastic circuit, when state vector is V, is the same as that in the case of BI, V should satisfy On the premise of (21), if the state vector V satisfies then the random error is not greater than that in the case of BI.Thus, the state vector V satisfying (21) and ( 22) represents a random error reduction scheme for the combinational stochastic circuit.

Lemma 7. For any combinational stochastic circuit, if the state vector V satisfies
then one has that (V) = ().
Proof.See Appendix A. Lemma 8. Two input streams  1 and  2 , which are both MCASs, have the following properties: Proof.See Appendix B.
Theorem 9.For any combinational stochastic circuit, if the state vector V satisfies then one has (V) = (), and Var(V) ⩽ Var().
Proof.See Appendix C. Lemmas 7 and 8 will be used in the proof.
Theorem 10.For any combinational stochastic circuit, if the state vector V satisfies then one has (V) = (BI), and Var(V) ⩽ Var(BI).
Proof.See Appendix D. Lemmas 7 and 8 will be used in the proof.
Theorems 9 and 10 give two different sufficient conditions for the state vector satisfying (21) and (22), respectively.On the basis of the two theorems, we will propose the random error reduction scheme for any combinational stochastic circuit in the next subsection.

Random Error Reduction Scheme Based on MCAS and BS.
In this subsection, we illustrate how to get the state vector V which represents the random error reduction scheme based on MCAS and BS for any combinational stochastic circuit.Specific steps are stated in Algorithm 1. Now, we prove that the state vector V obtained from Algorithm 1 satisfies (21) and ( 22).Consider the following two cases: (I)  pis is a null set; (II) otherwise.In case (I), the valid steps are (1), ( 2), (3), and (5).From criterion (a) in step (3), ( 27) is satisfied.According to Theorem 9, the state vector V obtained from these steps satisfies (21) and ( 22).In case (II), the valid steps are (1), ( 2), (4), and (5).From criterion (a) and (b) in step ( 4), ( 28) and (29) are satisfied.According to Theorem 10, the state vector V obtained from these steps also satisfies (21) and ( 22).Therefore, in both cases, the state vector V obtained from the corresponding steps satisfies (21) and (22).

Hardware Cost and Power Consumption.
In conventional stochastic computation, the input streams for stochastic circuits are all BSs.However, in the proposed random error reduction scheme, some input streams are specified as MCASs.Different input streams are generated by different generators.The hardware structures of the MCAS generator and the BS generator are shown in Figure 2. The MCAS generator consists of an -bit comparator and an -bit upcounter, while the BS generator consists of an -bit comparator and an -bit Liner Feedback Shift Register (LFSR), where 2  is the length of the input stream to be generated.In stochastic computation system, all the MCAS generators can share the same -bit upcounter.By contrast, each BS generator requires a different -bit LFSR.Thus, the total hardware cost of MCAS generators is usually lower than that of BS generators.That is to say, the proposed random error reduction scheme does not require any additional hardware cost.
Moreover, due to a lower switching activity, MCAS generator will have less dynamic power consumption than BS generator, which is another inherent advantage of our random error reduction scheme.

Case Study
For better understanding of the random error reduction scheme based on MCAS and BS, we take several typical stochastic circuits as case studies.We assume that all the input streams are uncorrelated.

Case Study 1: AND Gate.
In this case study, we illustrate how to get the random error reduction scheme when  pis is not a null set.An AND gate in stochastic logic implements a binary operator multiplication as  =  1  2 .We obtain the state vector V satisfying (21) and ( 22) from the steps given in Algorithm 1.

Requrie:
The combinational stochastic circuit implementing arithmetic; Ensure: The state vector V representing the random error reduction scheme; (1) Express the relationship between input streams and output stream in the form of ( 14).
(3) If  pis is a null set, we determine which input streams are specified as MCASs and obtain   by the following criteria: (a) Eq. ( 27) should be satisfied, i.e., only one input stream is specified as MCAS; (b) on the premise of (a), let |  | be maximal.Then, we skip step (4) and go to step (5).( 4) If  pis is not a null set, we determine which input streams are specified as MCASs and obtain   by the following criteria: (a) Eq. (28) should be satisfied, i.e., only PISs are likely to be specified as MCASs; (b) Eq. (29) should be satisfied, i.e., for each product term, at most one factor is specified as MCAS; (c) on the premise of (a) and (b), let |  | be maximal.Then, we go to step ( 5). ( 5) The input streams not belonging to   are specified as BSs.From above, we determine each input stream should be specified as MCAS or BS and obtain the state vector V.
Algorithm 1: Random error reduction scheme based on MCAS and BS.From above, the set   satisfying all the criteria in step (4) can be written as   = {1} or {2}.Without loss of generality, we set   = {1}, and  1 is specified as an MCAS.Then, go to step (5).( 5)  2 is specified as a BS.Thus, the state vector V = (MCAS, BS).

Case Study 2: XOR Gate.
In this case study, we illustrate how to get random error reduction scheme when  pis is a null set.An XOR gate in stochastic logic implements the arithmetic as follows:  =  1 +  2 − 2 1  2 .We obtain the state vector V satisfying (21) and ( 22) from the steps given in Algorithm 1.
(3) By criterion (a), only one of  1 and  2 can be specified as an MCAS.Hence, as shown in the first column of  4) and go to step (5).
(4) Since  pis is a null set, this step is skipped.
Similarly, we can get the state vector V satisfying (21) and ( 22) for the stochastic circuit implementing Bernstein polynomial of degree , which can be written as V = (MCAS, MCAS ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

Simulation Results and Comparison
The simulation platform for evaluating the random error in stochastic circuits is shown in Figure 4. Assume that the stochastic circuit is with  input streams  1 ,  2 , . . .,   , which represent  variables for the corresponding arithmetic.We randomly select 5000 sample points of   1 ,   2 , . . .,    from [0, 1].These sample points are used in both stochastic computation and float-point computation.The random error can be measured by the average of the absolute difference between the stochastic computation results (SCR) and the floating-point computation results (FCR) on these sample points.In order to simulate the generic and common circumstance, we used the same LFSR for all the input stream generators and just change the seed value.The size of LFSR equals the square root of length of input streams.For the sake of illustrating that the random error reduction scheme is valid for any input stream length, the length of input streams is chosen to be 2  , where  = 5, 6, . . ., 12.We compare the random error based on MCAS and BS with that based on other sequences on the following benchmarks.
(1) AND gate: the random error reduction scheme based on MCAS and BS for the stochastic logic AND gate is as follows:  1 is specified as an MCAS, and  2 is specified as a BS.For comparison, as shown in Figure 5, we give the simulation results in the following cases: (a)  1 is an MCAS, and  2 is a BS; (b)  1 and  2 are both BSs [4]; (c)  1 and  2 are both FRSs [14]; (d)  1 and  2 are DBSs as illustrated in [3].
(2) XOR gate: the random error reduction scheme based on MCAS and BS for the stochastic logic XOR gate is as follows:  1 is specified as an MCAS, and  2 is specified as a BS.For comparison, as shown in Figure 6, we give the simulation results in the following cases: (a)  1 is an MCAS, and  2 is a BS; (b)  1 and  2 are both BSs [4]; (c)  1 and  2 are both FRSs [14].
From the simulation results for the above benchmarks, we have the following conclusions.
(i) For all the benchmarks, the random error based on MCAS and BS is smaller than that based on BS with any sequence length.Hence, the proposed random error reduction scheme is effective for those stochastic circuits.(ii) In benchmark 1, DBSs have the best performance.
However, they are only suitable for this specific stochastic circuit.By comparison, the proposed MCASs and BSs can be used in any combination stochastic circuit.(iii) FRSs have better performance than MCASs and BSs in benchmarks 1 and 2, and vice versa in benchmark 3. Considering the difficulties in hardware generation for FRSs, the proposed MCASs and BSs, easily generated by hardware, have much wider applications.(iv) For all the benchmarks, the random error reduction scheme based on MCAS and BS is valid for any input stream length.Hence, the proposed random error reduction scheme has no length constraints.

Conclusion
In this paper, we propose a general random error analysis method based on AS.According to the analysis results, we find it feasible to reduce random error for stochastic computation by using proper ASs as input streams.Then, we present the random error reduction scheme based on MCAS and BS, which has the advantages of wide applications, easy hardware implementation, and flexible length.Both the theoretical analysis and simulation results confirm the effectiveness of the proposed scheme.In the future work, we will further study the features of ASs and apply them into more complex stochastic circuits.

C. Proof of Theorem 9
Proof.From (27), only one input stream is specified as an MCAS when the state vector is V. Thus, (23) is obviously satisfied.According to Lemma 7, we have that (V) = (BI).
Then we will prove that V also satisfies (22).

𝐸 (𝑃
Let   denote the input streams that are an MCAS.Then, (C.1) can be rewritten as ) . (C.3) For illustration purposes, we make the following notations: From above, Theorem 9 is proved.

D. Proof of Theorem 10
Proof.We can see that (29) is the same as (23).Thus, by Lemma 7, we have (V) = (BI).Then we will prove that V also satisfies (22).Let   1, denote the input stream that is an MCAS and a factor of    1, .Let  ℎ 1, denote the input stream that is an MCAS and a factor of   (D.1) For illustration purposes, we make the following notations:   From above, Theorem 10 is proved.

Figure 2 :
Figure 2: Hardware structure of the input stream generators: (a) MCAS generator; (b) BS generator.

Figure 4 :
Figure 4: Simulation platform for evaluating random error.