Invariant triple products

It is shown that the space of invariant trilinear forms on smooth representations of a semisimple Lie group is finite dimensional if the group is a product of Lorentz groups.


Introduction
Let G = PGL 2 (R) and let π 1 , π 2 , π 3 be irreducible admissible smooth representations of G. Then the space of G-invariant trilinear forms on π 1 × π 2 × π 3 is at most one dimensional. This has, in different contexts, been proved by Loke [10], Molchanov [11] and Oksak [12]. In this paper we ask for such a uniqueness result in the context of arbitrary semisimple groups. We give evidence that for a given group G uniqueness can only hold if G is locally a product of hyperbolic groups. For such groups we show uniqueness and for spherical vectors we compute the invariant triple products explicitly.
By a conjecture of Jacquet's, which has been proved in [3], triple products on GL 2 are related to special values of automorphic L-functions, see also [2,4,6,7]. The conjecture/theorem says that the existence of non-zero triple products is equivalent to the non-vanishing of the corresponding triple L-function at the centre of its functional equation.
The uniqueness of triple products in the PGL 2 -case mentioned above has been used in [1] to derive new bounds for automorphic L 2 -coefficients. This can also be done for higher dimensional hyperbolic groups, but, with the exception of the case treated in [1], the results do not exceed those in [9]. For completeness we include these computations in an appendix. I thank J. Hilgert and A. Reznikov for helpful comments on the subject of this paper.

Representations and Integral formulae
Let G be a connected semisimple Lie group with finite center. Fix a maximal compact subgroup K. LetĜ andK denote their unitary duals, i.e., the sets of isomorphism classes of irreducible unitary representations of G resp. K. Let π be a continuous representation of G on a locally convex topological vector space V π . Let V ′ π be the space of all continuous linear forms on V π and let V ∞ π be the space of smooth vectors, i.e., The representation π is called smooth if V π = V ∞ π . A representation (π, V π ) is called admissible if for each τ ∈K the space Hom K (V τ , V π ) is finite dimensional. LetĜ adm be the admissible dual, i.e., the set of infinitesimal isomorphism classes of irreducible admissible representations. A representation π inĜ adm is called a class one or spherical representation if it contains K-invariant vectors. In that case the space V K π of K-invariant vectors is one-dimensional. This is trivial for principal series representations (see below) and follows generally from Casselman's subrepresentation theorem which says that every π ∈Ĝ adm is equivalent to a subrepresentation of a principal series representation.
The Iwasawa decomposition G = ANK gives smooth maps a : G → A n : G → N k : G → K such that for every x ∈ G one has x = a(x)n(x)k(x). As an abbreviation we also define an(x) = a(x)n(x). Let g R , a R , n R , k R denote the Lie algebras of G, A, N, K and let g, a, n, k be their complexifications.
For x ∈ G and k ∈ K we define On the other hand, since G is unimodular, this also equals G g(xy) dx = G η(an(xy))f (k(xy)) dx = AN K η(an(anky))f (k(ky)) dank = K AN η(an(anky)) dan f (k y ) dk The second assertion follows from the first by replacing f withf (k) = f (k y ) and then y with y −1 .
Let M be the centraliser of A intersected with K, then P = MAN is a minimal parabolic subgroup of G. The inclusion map K ֒→ G induces a diffeomorphism M\K → P \G and in this way we get a smooth G-action on M\K. An inspection shows that this action is given by Mk → Mk x for k ∈ K, x ∈ G.

Trilinear products
Let π 1 , π 2 , π 3 be three admissible smooth representations of the group G and let T : for all v j ∈ V π j and every x ∈ G.
We want to understand the space of all trilinear forms T as above. In this paper we will only consider principal series representations, the general case will be considered later. So we assume that π 1 , π 2 , π 3 are principal series representations. This means that there are given a minimal parabolic P = MAN, irredicible representations σ j ∈M , and λ j ∈ a * for j = 1, 2, 3. Each pair (σ j , λ j ) induces a continuous group homomorphism P → GL(V σ j ) by man → a λ j +ρ σ(m), which in turn defines a G-homogeneous vector bundle E σ j ,λ j over P \G. The representation π j is the G-representation on the space of smooth sections Γ ∞ (E σ j ,λ j ) of that bundle. In other words, π j lives on the space of all C ∞ functions f : for all m ∈ M, a ∈ A, n ∈ N, x ∈ G. The representation π j is defined by π j (y)f (x) = f (xy). Every such f is uniquely determined by its restriction to K which satisfies f (mk) = σ j (m)f (k), i.e., f is a section of the Khomogeneous bundle E σ j on M\K induced by σ j . So the representation space Here ⊠ denotes the outer tensor product.
where φ is the kernel of T .
The group G is called a real hyperbolic group if it is locally isomorphic to SO(d, 1) for some d ≥ 2.
On Y = (P \G) 3 we consider the G 3 -homogeneous vector bundle E σ,λ given by E σ,λ = E σ 1 ,λ 1 ⊠ E σ 2 ,λ 2 ⊠ E σ 3 ,λ 3 . Next Y can be viewed as a G-space via the diagonal action and so E σ,λ becomes a G-homogeneous line bundle on Y .
We are going to impose the following condition on the induction parameters λ 1 , λ 2 , λ 3 . We assume that 3 j=1 ε j (λ j + ρ) = 0 for any choice of ε j ∈ {±1}. In other words this means • λ 1 − λ 2 + λ 3 + ρ = 0, and Theorem 2.1 Assume the parameters λ 1 , λ 2 , λ 3 satisfy the above condition. Let Y be the G-space (P \G) 3 . If there is an open G-orbit in Y , then the dimension of the space of invariant trilinear forms on smooth principal series representations is less than or equal to In particular, if π 1 , π 2 , π 3 are class one representations, then the dimension is less than or equal to the number of open orbits in Y .
There is an open orbit if and only if G is locally isomorphic to a product of hyperbolic groups. The Proof is based on the following lemma.
Lemma 2.2 Let G be a Lie group and H a closed subgroup. Let X = G/H and let E → X be a smooth G-homogeneous vector bundle. Let T be a distribution on E, i.e. a continuous linear form on Γ ∞ c (E). Suppose that T is G-invariant, i.e., T (g.s) = T (s) for every s ∈ Γ ∞ c (E). Then T is given by a smooth G-invariant section of the dual bundle E * .
Let (σ, V σ ) be the representation of H on the fibre E eH and let (σ * , V σ * ) be its dual. Then the space of all G-invariant distributions on E has dimension equal to the dimension dim V H σ * of H-invariants. So in particular, if σ is irreducible, this dimension is zero unless σ is trivial, in which case the dimension is one.
Proof: The equation T (g.s) = T (s), i.e., g.T = T ∀g ∈ G implies X.T = 0 for every X ∈ g R , the real Lie algebra of G. Let h R be the Lie algebra of H and choose a complementary space p R for h R such that g R = h R ⊕ p R . Let X 1 , . . . , X n be a basis of V and let We show that D induces an elliptic differential operator on E. By Ghomogeneity, it suffices to show this at a single point. So let P = exp(p R ).
In a neighbourhood U of the unit in G there are smooth maps h : U → H and p : The sections of E can be identified with the smooth maps s : We can attach to each section s a map f s on p with values in V σ by f s (Y ) = s(exp(Y )). The action of X ∈ p R on the section s is described by The first summand is of order zero and the second of order one. Moreover, the second summand at Y = 0 coincides with the coordinate-derivative in the direction of X. This implies that the leading symbol of D at eH is ξ 2 1 + · · · + ξ 2 n and so D is elliptic. The distributional equation DT = 0 then implies that T is given by a smooth section.
For the second assertion of the lemma recall that a G-invariant section is uniquely determined by its restiction to the point eH which must be invariant under H.
For the proof of the theorem we will need to investigate the G-orbit structure of Y = (P \G) 3 . First note that since the map Mk → P k is a K-isomorphism from M\K to P \G, the K-orbit of every y ∈ Y contains an element of the form (y 1 , y 2 , 1). Hence the P = MAN-orbit structure of (P \G) 2 is the same as the G-orbit structure of Y . By the Bruhat decomposition, the P -orbits in P \G are parametrized by the Weyl group W = W (G, A), where the unique open orbit is given by P w 0 P , here w 0 is the long element of the Weyl group. Note that the P -stabilizer of P w 0 ∈ P \G equals AM. This implies that the G-orbits in Y of maximal dimension are in bijection to the AM-orbits in P \G of maximal dimension via the map P xAM → (x, w 0 , 1).G. Again by Bruhat decomposition it follows that the latter are contained in the open cell P w 0 P = P w 0 N. So the G-orbits of maximal dimension in Y are in bijection to the AM-orbits in N of maximal dimension, where AM acts via the adjoint action. The exponential map exp : n R → N is an AM-equivariant bijection, so we are finally looking for the AM-orbit structure of the linear adjoint action on n R .
We will now prove that there is an open orbit if and only if G is locally a product of real hyperbolic groups. So suppose that Y contains an open orbit. Then n R contains an open AM-orbit, say AM.X 0 . Let φ + be the set of all positive restricted roots on a = Lie(A). Decompose n R into the root spaces On each n R,α install an M-invariant norm ||.|| α . This is possible since M is compact. Consider the map Since the orbit AM.X 0 is open, the image ψ(AM.X 0 ) of the orbit must contain a nonempty open set. Away from the set {X ∈ n R : ∃α : ||x|| α = 0} the map ψ can be chosen differentiable. Since the norms are invariant under M, one gets a smooth map

whose image contains an open set. This can only happen if the dimension of
A is at least as big as |φ + | and the latter implies that G is locally a product of real rank one groups. Now by Araki's table ( [5], pp. 532-534) one knows that these real rank one groups must all be hyperbolic, because otherwise there would be two different root lengths.
For the converse direction let G be locally isomorphic to SO(d, 1). We have to show that there is an open AM-orbit in n R . This, however, is clear as the action of AM on n R is the natural action of R × Let T be a G-invariant distribution supported on the closure of the orbit of This implies that T is of order zero along the manifold x 1 .G. By the G-invariance it follows that T is of the form where R is supported in [1,1,1]. Further, D is a differential which we can assume to be G-equivariant. Then where D 1 (m) is a differential operator in the variable a. Since D is Gequivariant, we may replace f with a 0 f for some a 0 ∈ A. Since x 1 a 0 = x 1 we get that the above is the same as This implies that D 1 must be of order zero and so T is of order zero. Restricted to the orbit x 1 .G ∼ = AM\G the distribution T is given by an integral of the form AM \G φ(y)f (x 1 y) dy. (Note that we use the notation without the dot again.) Invariance implies that φ is constant. If T is nonzero, then y → f (x 1 y) must be left invariant under AM, which implies We induce this in the notation by writing f (x 0 y) instead of f (x 0 .y). On a given orbit of maximal dimension, there is a standard invariant distribution which, by the lemma, is unique up to scalars and given by where α is a linear functional on the space of M o -invariants. In order to show that this extends to a distribution on Y , we need to show that the defining integral converges for all f ∈ Γ ∞ (E σ,λ ). This integral equals Since f is bounded on K 3 , it suffices to show the following lemma.

Conjecture 2.4
The assertion of the lemma should hold for any semisimple group G with finite center and n 0 ∈ N generic.
The conjecture would imply that if G is not locally a product of hyperbolic groups, then the space of invariant trilinear forms on principal series representations is infinite dimensional.
Proof of Lemma 2.3: Replace the integral over G by an integral over ANK using the Iwasawa decomposition. Since a(xk) = a(x) for x ∈ G and k ∈ K, the K-factor is irrelevant and we have to show that AN a(k 0 an) ρ a(w 0 an) ρ a ρ da dn < ∞.
Next note that w 0 a = a −1 w 0 and so we have a(w 0 an) ρ = a −ρ a(w 0 n) ρ as well as a(w 0 n 0 an) ρ = a −ρ a(w 0 n a 0 n) ρ , where n a 0 = a −1 n 0 a. We need to show AN a(w 0 n a 0 n) ρ a(w 0 n) ρ a −ρ da dn < ∞.
This is the point where we have to make things more concrete. Let J be the diagonal (d + 1) × (d + 1)-matrix with diagonal entries (1, . . . , 1, −1). Then SO(d, 1) is the group of real matrices g with g t Jg = J.
Note that the Lie algebra The Weyl element representative can be chosen to be So that with n = n(x) for x ∈ R d−1 we have We choose n 0 = n(1, 0, . . . , 0) and get with a = exp(tH), With n = d − 1 and a −ρ = e − n 2 t our assertion boils down to Consider first the case n = 1 and the integral over x < 0: Thus it suffices to show the convergence of Setting y = e −t we see that this integral equals is continuous, the integral over 0 < x < 1 converges. It remains to show the convergence of here we have used the substitution y = vx and the fact that for a, b ≥ 1 one has (1 + a)(1 + b) ≤ 3(1 + ab).
As above we can restrict to the case x > 1. We get As above, it suffices to restrict the integration to the domain x > 1. So one considers Choose 0 < ε < 1/2 and write For each λ ∈ a * let e λ be the class one vector in the associated principal series representation π λ given by e λ (ank) = a λ+ρ . Let λ, µ, ν ∈ a * be imaginary. Let T st be the invariant distribution on L (λ,µ,ν) considered in the last section. We are interested in the growth of T st (e λ , e µ , e ν ) as a function in λ. First note that the Killing form induces a norm |.| on a * .

An explicit formula
We write T st (λ, µ, ν) for T st (e λ , e µ , e ν ) and identifying a R to R via λ → λ(H 0 ) we consider T st as a function on (iR) 3 .
In this section we will prove the following theorem. where n = d − 1. So in particular, for fixed imaginary µ and ν. Then, as |λ| tends to infinity, while λ is imaginary, we have the asymptotic for some constant c > 0.
Proof: The asymptotic formula follows from the explicit expression and the well known asymptotical formula, as |t| → ∞, where the real part σ is fixed.
Now for the proof of the first assertion. The permutation group S 3 in three letters acts on (iR) 3 by permuting the co-ordinates. We claim that T st is invariant under that action. So let σ ∈ S 3 and let f = f 1 ⊗ f 2 ⊗ f 3 , then Since the open orbit is unique, there is y σ ∈ G with σ −1 (x 0 ) = x 0 .y σ , and so So in particular T st (λ, µ, ν) is invariant under permutations of (λ, µ, ν).
Replace the integral over G by an integral over ANK using the Iwasawa decomposition. Since a(xk) = a(x) for x ∈ G and k ∈ K, the K-factor is irrelevant and we have to compute AN a(w 0 n 0 an) λ+ρ a(w 0 an) µ+ρ a ν+ρ dadn.
Note that w 0 a = a −1 w 0 and so we have a(w 0 an) ρ = a −ρ a(w 0 n) as well as a(w 0 n 0 an) = a −ρ a(w 0 n a 0 n), where n a 0 = a −1 n 0 a. So the integral equals AN a ν−λ−µ−ρ a(w 0 n a 0 n) λ+ρ a(w 0 n) µ+ρ da dn.
Set n = d − 1 = 1, 2, 3, . . . and use polar co-ordinates to compute for n > 1, that and I 1 (a, b, c) to be equal to Then, for all n ∈ N, where c 1 = 1 and c n = (n − 1)vol(B n−1 ) for n > 1. So in particular, I n is invariant under permutations of the arguments.

Note that
and r n−2 = ∂ ∂r r n−1 n−1 . So, integrating by parts, for n > 1 we compute To get a similar result for I 1 note that

This implies
Let I ′ n (a, b, c) be the same as I n (a, b, c) except that there is a factor x in the integrand, i.e., Replacing x with −x first and then with x + e −t yields The last equation also holds for n = 1. Integration by parts gives So, + 1, b, c)).
Using (1) and (2) one gets for n > 1, + 1, b + 1, c), Finally we arrive at for n > 1 and These formulae amount to where d n = (n − 1)c n+2 /c n if n > 1 and d 1 = c 3 . A calculation using the functional equation of the Gamma-function, Γ(z + 1) = zΓ(z), shows that the right hand side of the claim in the theorem satisfies the same equation as n is replaced by n + 2. So the claim of the theorem for n implies the same claim for n + 2. Note that the formula A.5 in [1] implies the theorem for n = 1, where one has to take into account that in [1] a different normalisation is used. To get our formula from theirs, one has to replace λ j by −2λ j in [1].
To finish the proof of the theorem it therefore remains to show the claim for n = 2.
To achieve this, we proceed in a fashion similar to [1]. First note that the group G = SL 2 (C) is a double cover of SO(3, 1) 0 , so we might as well use this group for the computation. For λ ∈ C let V λ denote the space of all smooth functions f on C 2 with f (az, aw) = |a| −2(λ+1) f (z, w) for every a ∈ C × . Then G = SL 2 (C) acts on the space V λ via π λ (g)f (z, w) = f ((z, w)g). This is the principal series representation with parameter λ. For λ = 1 there is a G-invariant continuous linear functional L : V 1 → C, which is unique up to scalars and is given by where the integral runs over the standard sphere S 3 ⊂ C 2 with the volume element induced by the standard metric on C 2 ∼ = R 4 . Since S 3 equals (0, 1)K, where K = SU(2) is the maximal compact subgroup of G, the theory of induced representations shows that this functional is indeed invariant under G and is unique with that property. If λ is imaginary, then the representation π λ is pre-unitary, the inner product being given by f, g = L(fḡ). To compute T 2 st (λ 1 , λ 2 , λ 3 ), we will describe this functional on the space This kernel is invariant under the diagonal action of G and is homogeneous of degree 2(λ j − 1) with respect to the variable v j . From this point the computations run like in [1] A.4 and A.5. The theorem follows.

A Automorphic coefficients
Let Γ ⊂ G be a uniform lattice and consider the right regular representation of G on L 2 (Γ\G), which decomposes as a direct sum, where the isotypic component L 2 (Γ\G)(π) is zero for π outside a countable set of π ∈Ĝ and is always isomorphic to a sum of finitely many copies of π. By the Sobolev lemma one has L 2 (Γ\G) ∞ ⊂ C ∞ (Γ\G). A function ϕ is called pure, if ϕ ∈ L 2 (Γ\G)(π) for some π ∈Ĝ. For three pure smooth ϕ 1 , ϕ 2 , ϕ 3 the integral Γ\G ϕ 1 (x)ϕ 2 (x)ϕ 3 (x) dx exists. Moreover, fix π 1 , π 2 , π 3 ∈Ĝ and fix G-equivariant embeddings σ j : V π j → L 2 (Γ\G), then Let L 2 (Γ\G) K be the space of all K-invariant vectors in L 2 (Γ\G). Then there is an orthonormal basis (ϕ i ) i∈N of pure vectors in L 2 (Γ\G) K . For each i let L i be the G-stable closed subspace of L 2 (Γ\G) generated by ϕ i . Then the spaces L i are mutually orthogonal.
For fixed pure normalized ϕ, ϕ ′ ∈ L 2 (Γ\G) K we are interested in the growth of the sequence Let (λ i , µ, ν) be the induction parameters of the representations given by (ϕ i , ϕ, ϕ ′ ). First note that the numbers c i decay exponentially in |λ i |. So we define the renormalized sequence In [1] it is hown that for d = 2 one has and in [9], Theorem 7.6, that for d ≥ 3, We are going to reprove this result for d ≥ 4.
Theorem A.1 Let d ≥ 4. There is C > 0 such that for every T > 0, Proof: By the uniqueness of trilinear forms there exists a constant a i such that This constant depends on the embeddings of π λ , π ν , π µ into L 2 (Γ\G), but we can normalize these embeddings by insisting that the standard class one vectors e λ , e µ , e ν are mapped to ϕ i , ϕ, ϕ ′ . Thus the number a i only depends on ϕ i , ϕ, ϕ ′ . We also assume that V λ , V, V ′ are cuspidal, i.e., that fixed embeddings into L 2,∞ are given. Then the cuspidal trilinear form T induces a Hermitian form H aut λ on E.
Next consider the space L 2,∞ (Γ\G × Γ\G). Let H ∆ denote the Hermitian form on L 2,∞ (Γ\G × Γ\G) given by restriction to the diagonal, i.e., Let U be a Hilbert space with Hermitian form H. For a closed subspace L of U let P r L denote the orthogonal projection from U to L and let H L be the Hermitian form on U given by H L (u, v) = H(P r L (u), P r L (v)).
The map L → H L is additive and monotonic, i.e., Proof: By the uniqueness it follows H aut λ i = |a i | 2 H st λ i and since all the spaces L i are orthogonal one also deduces i H aut The group G × G acts on L 2,∞ (Γ\G × Γ\G) and thus on the real vector space of Hermitian forms on L 2,∞ (Γ\G × Γ\G). By integration, this action extends to a representation of the convolution algebra C ∞ c (G×G). Note that a non-negative function h ∈ C ∞ c (G × G) will preserve to cone of non-negative Hermitian forms.
Proof: The form h.H ∆ is given by integration over a smooth measure on Γ\G × Γ\G. Being smooth, this measure is bounded by a multiple of the invariant measure. Let u T be a nonnegative smooth function on (M\K) 2 with support in S such that u dxdy = 1, It is possible to construct such a function since the codimension of F equals dim(M\K) + 1. With these data define a positive functional ρ T on H(E) by Then one has Recall the standard trilinear form T st on V λ ⊗ V ⊗ V ′ and the corresponding linear map l Tst : V ⊗ V ′ →V λ . We identify the spaceV λ with C ∞ (M\K) and let z denote the point M1 ∈ M\K. The Dirac measure δ z at z is a continuous linear functional on C ∞ (M\K). We get an induced Hermitian form H z on E defined by H z (u, v) = l Tst (u)(z)l Tst (v)(z). Proof: This follows from the fact that the invariant Hermitian form onV λ is equal to K π(k)H z dk, whereH z is the Hermitian form onV λ given bỹ H z (u, v) = u(z)v(z).
Since the test function h was assumed to be K × K-invariant it follows that So in order to establish a lower bound for ρ T (H st λ ) it suffices to give a lower bound for δ u (x.H z ) = | π(x)f z , u | 2 for x ∈ D, where f z is the linear functional on E given by f z (u) = δ z (l Tst (u)).
Lemma A.6 There is C > 0 such that for T 0 large enough there exists an open, non-empty subset D 0 ⊂ D such that for T ≥ T 0 , |λ| ≤ T , and x ∈ D 0 we have | π(x)f z , u T | ≥ 1 C .
We next derive an explicit formula for φ on the open orbit. Recall that T st (f ) = G f (y, w ) y, w 0 n 0 y) dy = AN K f (ank, w 0 ank, w 0 n 0 ank) da dn dk, and that the last integrand equals a λ+ρ a(w 0 an) µ+ρ a(w 0 n 0 an) ν+ρ f (k, k(w 0 an)k, k(w 0 n 0 an)k) da dn dk.
Note in particular, that the function π(x)f z is invariant under M.
The set S is a tubular neighbourhood of the compact set F . We have natural local coordinates with F -directions, which can be identified with open parts of M-orbits and F ⊥ -directions, which can be viewed as AN-orbits. The above formula shows that the gradient of π(x)f z is zero along the M-orbits.
Along the AN-orbits this gradient is bounded by a constant times T , if we leave µ and ν fixed and restrict to |λ| ≤ T . Let C 0 > 0 be large and let D 0 ⊂ D be the subset of elements x ∈ D such that |π(x)f z | < C 0 . This set is open and nonempty if C 0 is large enpough and it does not depend on λ.
Lemma A.6 implies Proposition A.4 and the latter implies Theorem A.1 as follows.
Consider the inequality i |a i | 2 ρ T (H st λ i ) ≤ ρ T (H ∆ ). By Proposition A.4 the right hand side is bounded by CT 2 dim(M \K) and since dim M\K = dim P \G = dim N = d − 1 we get Restricting the sum to those i with |λ i | ≤ 2T and using the second assertion of Proposition A.4 we get According to Theorem 3.1 we have b i |λ i | 4−d ∼ |a i | 2 and so This implies the theorem.