^{1}

^{2, 3}

^{2}

^{2}

^{1}

^{2}

^{3}

The present investigation deals with global instability of a general

Recently, there has been a rapidly growing interest in investigating the instability conditions of differential systems. The number of papers dealing with instability problems is rather low compared with the huge quantity of papers in which the stability of the motion of differential systems is investigated. The first results on the instability of zero solution of differential systems were obtained in a general form by Lyapunov [

Further investigation on the instability of solutions of systems was carried out to weaken the conditions of the Lyapunov and Chetaev theorems for special-form systems. Some results are presented, for example, in [

In the present paper, instability solutions of systems with quadratic right-hand sides is investigated in a cone dealing with a general

Unlike the previous investigations, we prove the global instability of the zero solution in a given cone and the conditions for global instability are formulated by inequalities involving norms and eigenvalues of auxiliary matrices. The main tool is the method of Chetaev and application of a suitable Chetaev-type function. A novelty in the proof of the main result (Theorem

In the sequel, the norms used for vectors and matrices are defined as

In this paper, we consider the instability of the trivial solution of a nonlinear autonomous differential system with quadratic right-hand sides

If matrix

We will give criteria of the instability of a trivial solution of the system (

In this part we collect the necessary material-the definition of a cone, auxiliary Chetaev-type results on instability in a cone and, finally, a third degree polynomial inequality, which will be used to estimate the sign of the full derivative of a Chetaev-type function along the trajectories of system (

We consider an autonomous system of differential equations

The zero solution

A set

A cone

The zero solution

Now, we prove results analogous to the classical Chetaev theorem (see, e.g., [

Let

Let

Let

Suppose to the contrary that this is not true and

Let

The proof is a modification of the proof of Theorem

Let

Suppose to the contrary that this is not true and

A function

Our results will be formulated in terms of global cones of instability. These will be derived using an auxiliary inequality valid in a given cone. Let

Let

We partition

If inequalities (

If inequalities (

In this part we derive a result on the instability of system (

Assume that the matrix

either

First we make auxiliary computations. For the reader's convenience, we recall that, for two

We will rewrite system (

It is easy to see that matrices

The full derivative of

Obviously

All the assumptions of Theorem

We will focus our attention to Lemma

Now we consider a particular case of the system (

Assume that

either

It is easy to see that the choice

Assume that

either

The set

The set

This research was supported by Grants nos. P201/11/0768 and P201/10/1032 of Czech Grant Agency, and by the Council of Czech Government nos. MSM 0021630503, MSM 0021630519, and MSM 0021630529, and by Grant FEKT-S-11-2-921 of Faculty of Electrical Engineering and Communication.