Plancherel formula for Berezin deformation of L^{2} on Riemannian symmetric space

Yurii A. Neretin^{1}

0.Introduction

0.1. Kernel representations. Let G be a classical real group and let K be its maximal compact subgroup. Consider the Riemannian noncompact symmetric space G/K. There exists a hermitian symmetric space

G̃/K̃ ⊃ G/K

such that

dim_{ℝ} G/K = \frac{1}{2} dim_{ℝ} G̃/K̃

and G/K is a totally real submanifold in G̃/K̃ (the list of embeddings G/K → G̃/K̃ see in Section 6). We say that the symmetric space G̃/K̃ ⊃ G/K is the hermitization of the symmetric space G/K.

We define a kernel representation ρ of the group G as a restriction of an unitary highest weight representation ρ̃ of G̃ to the subgroup G. By wellknown Harish-Chandra construction, highest weight representations of G̃ are natural representations in spaces of (scalar-valued or vector-valued) holomorphic functions on G̃/K̃. We say that ρ is a scalar valued kernel representation if ρ̃ is realized in scalar-valued holomorphic functions on G̃/K̃.

The kernel representations are deformations of L^{2}(G/K) in some precise sense explained in Subsection 1.13.

The purpose of this paper^{2} is to obtain the Plancherel formula for scalarvalued kernel representations (see formula (2.6)–(2.15)).

There were different reasons for interest, which was attracted by kernel representations in last 5 years (see [2], [7], [8], [33]–[36], [38], [39]–[40], [52], [56]), and we will formulate reasons which are the most closed to the author. In first place, there are many explicit analytical formulas related to kernel representations (I hope that this paper also confirms this statement). Secondly, spectra of kernel representations are very rich^{3}. Thirdly, the kernel representations have some interaction with function theory^{4}, see [38], [34], [36]. Forth, the kernel representations also are closely related to Olshanskii constructions of representations of infinite dimensional groups U(p, ∞), O(p, ∞), Sp(p, ∞) (see [38]).

1supported by grants RFBR-98-01-00303 and RFBR 96-01-96249

2This work is continuation of works [35], [36] but logically it is independent on these papers.

Our main result was announced in [36].

3The most interesting spectral problems of noncommutative harmonic analysis which were intensively investigated in last 20 years are

0.2. Bibliographical comments. Let G/K be itself an hermitian symmetric space (i.e. G = U(p, q), Sp(2n, ℝ), SO^{∗}(2n), SO(n, 2)). Then its hermitization G̃/K̃ is G/K × G/K. A kernel representation of G in this case is a tensor product of a highest weight representation ρ̃_{α} of G and a lowest weight representation ρ_{β}^{∗} of G. In short paper [4] published in 1978 Berezin announced nice Plancherel formula for the sufficiently large parameter α = β of highest weight (see below Subsections 1.10-1.11). In this case^{5} the kernel representation is equivalent to the representation of G in L^{2}(G/K). Berezin died soon after this and the proof never was published^{6}. Berezin work didn’t attract serious interest in this time (see only papers [15],[46] on related subjects).

Secondly, the kernel representations appeared in G.I.Olshanskii and my work which was partially announced in [30], [43] and partially published in [38]. This work cocerned in vector valued kernel representations of the groups G = O(p, q), U(p, q), Sp(p, q) for small values of the highest weight. The main topic of our work was investigation of discrete part of spectra of the kernel representations and construction of “exotic” unitary representations of G by simple functional theoretical tools.

In the middle of 90-s interest to kernel representations increases (we list some publications: [52], [2], [33], [39], [40], [7], [8],[56]). In 1994 Upmeier and Unterberger [52] published proof of Berezin formula (see also [2])^{7}. Van Dijk and Hille [7] obtained the complete Plancherel formula for rank 1 groups. Olaffson and Orsted [39] proved that for a large highest weight a scalar valued kernel representation of G is equivalent to the representation of G in L^{2}(G/K).

In paper [35] there was defined B-function for arbitrary classical noncompact Riemannian symmetric space. For the symmetric cones GL(n, ℝ)/O(n), GL(n, ℂ)/U(n), GL(n, ℍ)/Sp(n) these B-integrals coincides with Gindikin Bfunction constructed in [11](1964) (see also exposition in [9]). The construction of [35] for special cases of parameters gives some integrals of Siegel [48], Hua Loo Keng [21], Unterberger–Upmeier [52], Arazy–Zhang [2]. The Plancherel formula for scalar-valued kernel representations of all classical groups for large values of parameter is easily reduced to these B-integrals. In this case a kernel representation is equivalent to the representation of G in L^{2}(G/K) and the spectrum of the kernel representation is supported by the principal nondegenerate unitary series.

— L^{2} on pseudoriemannian symmetric spaces

— Howe dual pairs (and the problem of decomposition of L^{2} on Stiefel manifolds which are in some sense equivalent to Howe dual pairs)

Each representation, which occurs in spectra of Howe dual pairs, occurs in spectra of some kernel representation. Converse statement is false. The a priori explanation of this phenomenon is contained in [33].

I think that spectra of kernel representations and spectra of L^{2} on pseudoriemannian symmetric spaces essentially differs. A priori embedding of spectra of L^{2}(U(p, q, 𝕂)/U(r, 𝕂) × U(p − r, q, 𝕂)) for 𝕂 = ℝ, ℂ, ℍ to spectra of kernel representations is discussed in [38], [33].

4For instance, in the work [38] we use functional theoretical arguments for construction of singular unitary representations of groups U(p, q, 𝕂)

5Tensor products for SL_{2}(ℝ) = U(1, 1) were earlier investigated by Pukanszky [45] and by Vershik, Gelfand and Graev [53], see also [27]

620 years later I heard some reminiscences about this proof but I can not reconstruct proof itself. It essentially differs from Unterberger–Upmeier [52] proof and my proof [35].

7Their result also covers groups E_{6}, E_{7}.

The case of small values of highest weight was discussed in paper [36]. In this case the spectrum of kernel representation is quite intricate and work [36] contains natural decomposition of a kernel representation on subrepresentations having relatively simple spectra.

The purpose of the present parer is to obtain the complete Plancherel formula for the kernel representations in the scalar-valued case.

0.3. Contents. Main part (Sections 1–5) of the paper deals with the series G = O(p, q)^{8}.

Section 1 of the paper contains preliminaries. We discuss the definition of the kernel representations and simple a priori properties of the Plancherel formula. We also formulate some necessary properties of spherical functions.

Basic results are formulated in Section 2. For large values of α (where α is the parameter of a highest weight) the Plancherel measure ν_{α} has the form

E(α) ∏_{k=1}^{p} |Γ(\frac{1}{2}(α + (p + q)/2 + s_{k})|^{2}ℜ(s)ds

where ℜ(s) is the Gindikin-Karpelevich density (see(1.42)–(1.43)),

s_{1}, …, s_{p} ∈ iℝ  (0.1)

(this notation means that Re s_{j} = 0) and E(α) is a meromorphic factor.

Assume q −p be sufficiently large. Let us move the parameter α from +∞ to 0. After passing across the point α = \frac{1}{2}(p + q) − 1 there appears an additional piece of the support of the Plancherel measure. This piece is defined by the conditions

s_{1} = α − \frac{1}{2}(p + q) + 1; s_{2}, …, s_{p} ∈ iℝ  (0.2)

After passing across the point α = \frac{1}{2}(p + q) − 2 the third piece of the support of the Plancherel measure appears:

s_{1} = α − \frac{1}{2}(p + q) + 1, s_{2} = α − \frac{1}{2}(p + q) + 2; s_{3}, …, s_{p} ∈ iℝ  (0.3)

8This case is the most complicated and all difficulties existing for other series exist also for O(p, q). For all other series our proof is more simple.

After passing across the point α = \frac{1}{2}(p + q) − 3 we obtain two additional components of the support

s_{1} = α−\frac{1}{2}(p+q)+1, s_{2} = α−\frac{1}{2}(p+q)+2, s_{3} = α−\frac{1}{2}(p+q)+3; s_{4}, …, s_{p} ∈ iℝ

and

s_{1} = α − \frac{1}{2}(p + q) + 3; s_{2}, …, s_{p} ∈ iℝ  (0.4)

etc. After passing across the point α = (q − p)/2 we obtain the first one-point piece

s_{1} = α − \frac{1}{2}(p + q) + 1; s_{2} = α − \frac{1}{2}(p + q) + 2, …, s_{p} = α − \frac{1}{2}(p + q) + p

This means that our representation has a subrepresentation entering discretely.

At the point α = p − 1 the component (0.1) of the support disappears. At the point α = p − 2 components (0.2), (0.4) also disappear, etc.

Theorems 2.2–2.4 contain the complete description of this process and give the Plancherel density on each component of the support. Interpretation of these pieces is given in [36], in the present paper this is not discussed.

The nature of spectra of kernel representations is explained in [36]^{9}.

For an integer negative α our construction gives the Plancherel formula for some finite dimensional representation of O(p, q) (see Subsection 2.6 of the paper).

Section 3 is based on [35] and contains evaluation of the B-integral (see formula (3.2)-(3.4)). For instance, in the case p = q our B-integral is given by

∫_{R+R^{t}>0} ∏_{j=1}^{p} \frac{det[(R + R^{t})/2]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + R]_{j}^{σ_{j}−σ_{j+1}}} · det(R + R^{t})^{−p}dR =  (0.5)

= const · \frac{Γ(λ_{k} − (p + k)/2 + 1)Γ(σ_{k} − λ_{k} − (p − k)/2)}{Γ(σ_{k} − p + k)}

where the integration is given over the space of dissipative p × p real matrices R and the symbol [A]_{j} denotes the left upper j × j block of a matrix A.

B-Integral allows to obtain the Plancherel formula for α > \frac{1}{2}(p + q) − 1. In Section 4 we construct the analytic continuation of the Plancherel formula to arbitrary α.

In Section 5 we prove positive definiteness of spherical functions which appears in the right side of the Plancherel formula.

9The discrete part of spectra of kernel representations consists of singular unitary representations having quite interesting properties. For instance, these infinite dimensional (non highest weight) representations have Gelfand–Tsetlin bases, see [43], [28]; the problems of decomposition of restrictions and tensor products for these representations also seems rich, see [30], [38]). In [43], [38] it was shown, that these representations admit inductive limits as q → ∞. Certain representations of this type appear in spectra discussed in [50], [25]

Section 6 contains a discussion of other series of classical groups. The Bintegrals for other series or real classical groups are evaluated in [35], and a generalization of the consideration of Subsections 1, 4, 5 to other series is quite trivial. Hence, we give only short remarks and also give the Plancherel formula in the form which slightly differs from Theorem 2.2. Author intentionally considers the series O(p, q) (and not so-called ’general case’) to do the exposition more or less self-closed. I try to avoid formal logical dependence on recent papers and also minimize using machinery of representation theory of semisimple groups as far as it is possible^{10}; I also try to avoid notations demanding long explanations.

Acknowledgements. I am very grateful to G.I.Olshanskii, V.F.Molchanov, and B.Orsted for numerous discussions of the subject. I thanks H.Schlichtkrull, G. van Dijk, and A.Dvorsky for discussions, comments and references.

1. Preliminaries.

A. Positive definite kernels.

The subject of the paper is the analysis in a family of hilbert spaces defined by positive definite kernels. The notion of positive definite kernel and associated machinery are quite old (see [51], [6], [26]) but not widely known. In this Section we briefly discuss elementary properties of the positive definite kernels and associated hilbert spaces.

1.1. Positive definite kernels. Let H be a hilbert space with a scalar product < ·, · >, let X be a subset in H. Consider the function L(x, y) on X × X defined by

L(x, y) =< x, y >

Obviously for all x_{1}, …, x_{n} ∈ X we have

det (\begin{matrix}L(x_{1}, x_{1}) & ⋯ & L(x_{1}, x_{n}) \\ ⋯ & ⋯ & ⋯ \\ L(x_{n}, x_{1}) & ⋯ & L(x_{n}, x_{n}) \end{matrix}) ⩾ 0  (1.1)

Let X be an abstract set. A function L(x, y) on X × X is called a positive definite kernel if it satisfies the conditions

1. L(x, y) = \overline{L(y, x)}

2. For any x_{1}, x_{2}, …, x_{n} ∈ X inequality (1.1) holds

Let L(x, y) be a positive definite kernel on X. Then where exists a hilbert space H = H[K] and a system of vectors v_{x} ∈ H enumerated by points x ∈ X such that 1. < v_{x}, v_{y} >_{H}= L(x, y)

10we needs in some basic properties of spherical functions, all necessary information is contained in Helgason book [17], chapter 4

2. the linear span of the vectors v_{x} is dense in H.

The family v_{x} is called a supercomplete basis^{11}.

This construction is natural in the following sense. Let H′ be another hilbert space and let v′_{x} be another system of vectors satisfying the same conditions. Then there exists the unique unitary operator U : H → H′ such that Uv_{x} = v′_{x} for all x ∈ X.

If X is a separable metric space and the kernel L(x, y) is continuous, then the hilbert space H[L] is separable.

In Subsections 1.2-1.3 and 1.4 we discuss two ways of “materialization” of the space H[L].

1.2. Scalar product in the space of complex-valued measures. Assume X be a separable complete metric space. Let μ be a complex-valued measure (charge) on X with a compact support. Consider a vector

v(μ) = ∫_{X} v_{x}dμ(x) ∈ H[L]

Thus, we obtain a way to represent elements of H[L]. Obviously

< v(μ), v(ν) >_{H[L]}= ∫_{X×X} L(x, y) dμ(x) d\overline{ν(y)}

Let us say the same construction more formally. Consider the linear space ℳ(X) of all compactly supported complex-valued measures on X. Consider the scalar product in ℳ(X) defined by the formula

< μ, ν >= ∫_{X×X} L(x, y) dμ(x) \overline{dν(y)}

We obtain a structure of a prehilbert space in ℳ(X) and the space H[L] is the hilbert space associated with the prehilbert space ℳ(X) (elements of the supercomplete basis corresponds to measures supported at points).

1.3. Scalar products in spaces of distributions. Assume X be a smooth manifold and the kernel L(x, y) be smooth. Denote by 𝒟 the space of compactly supported distributions on X. Consider the scalar product in 𝒟 given by the formula

< χ, φ >= \{K(x, y), χ(x) ⊗ \overline{φ(y)}\}  (1.2)

where brackets {·, ·} denote the pairing of smooth functions and distributions. Consider the hilbert space H associated with the prehilbert space 𝒟. Denote δ-distribution supported at a point x by δ_{x}. Obviously

< δ_{x}, δ_{y} >= L(x, y)

11Other terms for v_{x} are overfilled basis or system of coherent states.

Hence, we can identify H with H[L] and the vectors δ_{x} with elements of the supercomplete basis v_{x}.

Remark 1. The space of distributions equipped with scalar product (1.2) is not complete. This means that some vectors of H[L] can not be represented by distributions.

Remark 2. Scalar product (1.2) in 𝒟 can be degenerated. This means that a vector h ∈ H can be represented by a distribution in various ways.

1.4. The embedding of H[L] to the space of functions on X. For arbitrary h ∈ H[L] we consider the function

f_{h}(x) :=< h, v_{x} >_{H[L]}

on the space X. Obviously, the map h ↦ f_{h} is an embedding of H[L] to the space of functions on X. We denote the image of the embedding by H^{∘}[L]. By construction, the space H^{∘}[L] has structure of a hilbert space.

Lemma 1.1. Assume X be a separable metric space and the kernel L(x, y) be continuous. Let h_{j} converges to h. Then f_{h_{j}} converges to f_{h} uniformly on compacts.

Proof. Let Y ⊂ X be a compact set. Let x ∈ Y. Then

|f_{h_{j}}(x) − f_{h}(x)| = | < (h_{j} − h, v_{x} >_{H[L]} | ⩽

⩽ ∥h_{j} − h∥ · ∥v_{x}∥ = ∥h_{j} − h∥ · \sqrt{L(x, x)} ⩽ ∥h_{j} − h∥ · \sqrt{max_{x∈Y} L(x, x)} ⊠

Obviously, the function φ_{a}(x) ∈ H^{∘}[L] associated with the vector v_{a} ∈ H[L] is given by the formula

φ_{a}(x) = L(x, a)  (1.3)

Lemma 1.2. Let X be a locally compact metric space and the kernel L(x, y) be continuous. Then functions f_{h} ∈ H^{∘}[L] are continuous.

Proof. The linear span of the functions φ_{a} is dense in H^{∘}[L]. Then we apply Lemma 1.1. ⊠

Lemma 1.3. (Reproducing property) For any f ∈ H^{∘}[L], x ∈ X the following identity holds

f(x) =< f, φ_{x} >_{H^{∘}[L]}  (1.4)

Proof. Let f = f_{h}. Then

< f_{h}, φ_{x} >_{H^{∘}[L]}=< h, v_{x} >_{H[L]}= f_{h}(x) ⊠

Remark. Equation (1.4) gives a nonexplicit description of the scalar product in H^{∘}[L] and this description is sufficient for many purposes. Another way of description is the following identity. Let e_{n}(x) ∈ H^{∘}[L] be an orthonormal basis. Then

L(x, y) = ∑ \overline{e_{n}(x)}e_{n}(y)

(proof: consider < φ_{x}, e_{n} >)

Lemma 1.4. Let Ω be an open domain in ℂ^{n} and let L(z, u) be a positive definite kernel on Ω. Let L(z, u) be holomorphic in the variable u and anti-holomorphic in the variable z. Then all elements of the space H^{∘}[L] are holomorphic functions on Ω.

Proof. It is a corollary of Lemma 1.1. ⊠ 1.5. Operations with positive definite kernels. Lemma 1.5. a) Let L_{1}(x, y), L_{2}(x, y) be positive definite kernels on X. Then

L_{1}(x, y) + L_{2}(x, y) is a positive definite kernel

b) Let L_{1}(x, y), L_{2}(x, y) be positive definite kernels on X. Then the kernel L_{1}(x, y)L_{2}(x, y) is positive definite

c) Let L_{j}(x, y) be positive definite kernels and L_{j}(x, y) converges to L(x, y) point-wise. Then L(x, y) is a positive definite kernel.

d) Let K_{m}(x, y) be a family of positive definite kernels enumerated by points of some measure space M with positive measure μ. Assume that the integral

K^{∗}(x, y) = ∫_{M} K_{m}(x, y)dμ(m)

converges for all x, y ∈ X. Then K^{∗}(x, y) is positive definite.

e) Let L(x, y) be a positive definite kernel and let λ(x) be a function on X. Then the kernel M(x, y) = λ(x)\overline{λ(y)}L(x, y) is positive definite.

Proof. a) Let v_{x} (resp. w_{x}) be the supercomplete basis in H[L_{1}] (respectively H[L_{2}]). We consider the system of vectors v_{x} ⊕ w_{x} ∈ H[L_{1}] ⊕ H[L_{2}]. Then

L_{1}(x, y) + L_{2}(x, y) =< v_{x} ⊕ w_{x}, v_{y} ⊕ w_{y} >

b) Proof is similar, H[L_{1}L_{2}] ⊂ H[L_{1}] ⊗ H[L_{2}]

d) This is consequence of a) and c).

e) Indeed, H[M] = H[L] and the supercomplete basis in H[M] consists of vectors γ(x)v_{x} where v_{x} is the supercomplete basis in H[L]. ⊠

1.6. Positive definite kernels on homogeneous spaces. Let Γ be a group acting on X and let a positive definite kernel L(x, y) be Γ-invariant

L(gx, gy) = L(x, y)  for all  g ∈ Γ, x, y ∈ X

Obviously, for each g ∈ Γ where exists the unique unitary operator U(g) : H[L] → H[L] such that

Uv_{x} = v_{gx}  for all  x ∈ X

Then

U(g_{1}g_{2}) = U(g_{1})U(g_{2})

Hence, U(g) is an unitary representation of Γ.

Let X be a Γ-homogeneous space, X = Γ/K, let x_{0} be a H-fixed point. Let L(x, y) be a Γ-invariant function. Then L is completely defined by the

function

l(y) := L(x_{0}, y)

Indeed, let u, z ∈ X. Then u = gx_{0} for some element g in Γ and

L(u, z) = L(gx_{0}, z) = L(x_{0}, g^{−1}z) = l(g^{−1}z)

Moreover, for any γ ∈ K we have

l(y) = L(x_{0}, y) = L(γx_{0}, γy) = L(x_{0}, γy) = l(γy)

We see that the function l_{0} is a K-invariant function on Γ/K.

We also can consider a K-invariant function l(y) as function on double cosets K \ Γ/K.

We see that there is the canonical correspondence between 3 following sets:

– Γ-invariant functions on Γ/K × Γ/K

– K-invariant functions on Γ/K

– functions on K \ Γ/K.

We say that a K-invariant function on Γ/K or a function on K \ Γ/K is positive definite if the associated kernel on Γ/K × Γ/K is positive definite.

1.7. On K-invariant vectors in representations of Γ. Let Γ, K, x_{0} be the same as above. Let ρ be an unitary representation of Γ in a hilbert space H. Assume that there exists a K-invariant vector v ∈ H and assume v be a cyclic vector^{12}.

Consider the map Γ/K → H given by the formula

gx_{0} ↦ ρ(g)v

(the image of the map is the Γ-orbit of the vector v). Then the function

L(g_{1}x_{0}, g_{2}x_{0}) :=< ρ(g_{1})v, ρ(g_{2})v >_{H}

is a Γ-invariant positive definite kernel on Γ/K.

Hence, we can identify the hilbert space H with the space H[L]; the supercomplete basis in H consists of vectors ρ(g)v.

The function l(y) in our case is the matrix element < ρ(g)v, v > and our construction (Segal–Gelfand–Naimark construction) reconstructs the representation ρ by its matrix element.

12This means that the linear span of vectors ρ(g)v is dense in H.

B. Kernel representations.

Assume p ⩽ q

1.8. Pseudoorthogonal group O(p, q). Consider the linear space ℂ^{p} ⊕ℂ^{q} equipped with the indefinite hermitian form

J((x, y), (u, v)) = ∑_{j=1}^{p} x_{j}\overline{u}_{j} − ∑_{j=1}^{q} y_{k}\overline{v}_{k};  (x, y), (u, v) ∈ ℂ^{p} ⊕ ℂ^{q}  (1.5)

The pseudounitary group U(p, q) is the group of all linear operators g = (\begin{matrix}α & β \\ γ & δ \end{matrix})

in ℂ^{p} ⊕ ℂ^{q} preserving the form J(·, ·). In other words, a matrix g ∈ U(p, q) satisfies the condition

(\begin{matrix}α & β \\ γ & δ \end{matrix}) (\begin{matrix}1 & 0 \\ 0 & −1 \end{matrix}) (\begin{matrix}α & β \\ γ & δ \end{matrix})^{∗} = (\begin{matrix}1 & 0 \\ 0 & −1 \end{matrix})  (1.6)

The pseudoorthogonal group O(p, q) is the subgroup of U(p, q) consisting of real matrices. Below in Sections 1-5 by the symbol G we denote the group

G = O(p, q)

By K we denote the subgroup O(p) × O(q) ⊂ G consisting of matrices having

the form (\begin{matrix}α & 0 \\ 0 & δ \end{matrix}). It is a maximal compact subgroup in G.

1.9. Matrix balls. By B_{p,q}(ℂ) we denote space of all complex p×q matrices z having norm < 1 (where a norm is the norm of the operator v ↦ vz from the euclidean space ℂ^{p} to the euclidean space ℂ^{q}; remind that ∥z∥^{2} is the maximal eigenvalue of z^{∗}z).

By B_{p,q}(ℝ) we denote the space of real p × q matrices with norm < 1.

The group U(p, q) acts on the matrix ball B_{p,q}(ℂ) by fractional linear transformations

z ↦ z^{[g]} := (α + zγ)^{−1}(β + zδ)  (1.7)

This action is transitive and the stabilizer of the point z = 0 is the subgroup U(p) × U(q). Hence, B_{p,q}(ℂ) is the symmetric space

B_{p,q}(ℂ) = U(p, q)/U(p) × U(q)

In the same way, B_{p,q}(ℝ) is the symmetric space

B_{p,q}(ℝ) = G/K = O(p, q)/O(p) × O(q)

Arbitrary symmetric space admits unique up to factor invariant measure. For the space B_{p,q}(ℝ) the O(p, q)-invariant measure is given by the formula

dλ(z) = det(1 − z^{∗}z)^{−(p+q)/2}dμ(z)  (1.8)

where dμ(z) is the Lebesgue measure on B_{p,q}(ℝ).

1.10. Berezin kernels. Theorem 1.6^{13} . The kernel

L_{α}(z, u) = det(1 − z^{∗}u)^{−α}

on the matrix ball B_{p,q}(ℂ) is positive definite if and only if

α = 0, 1, 2, …, p − 1 or α > p − 1  (1.9)

Thus, for α satisfying the Berezin condition (1.9), we obtain the hilbert spaces H_{α} := H[L_{α}] and H_{α}^{∘} := H^{∘}[L_{α}]. The function L_{α}(z, u) is anti-holomorphic in z and hence by Lemma 1.4 the space H_{α}^{∘} consists of holomorphic functions on the matrix ball B_{p,q}(ℂ).

Remark. For α > p + q − 1 the scalar product in H_{α}^{∘} can be represented in the form

< f, g >_{α}= C(α) ∫_{B_{p,q}} f(z)\overline{g(z)} det(1 − z^{∗}z)^{α−p−q}dμ(z)

where dμ(z) is the Lebesgue measure on B_{p,q}(ℂ) and C(α) is the meromorphic factor defined by the condition < 1, 1 >_{α}= 1. In particular H_{p+q}^{∘} is the Bergman space. For α = q we obtain the Hardy space H^{2}. The scalar product in this case is given by the formula

< f(z), g(z) >_{q}= ∫_{z^{∗}z=1} f(z)\overline{g(z)}dν(z)

where dν(z) is the unique U(p) × U(q)-invariant measure on the set^{14} zz^{∗} = 1. For other values of parameters there exist integral formulas including partial derivatives but they are not simple (see [1]).

Remark. For α > p − 1 the space H_{α} contains all polynomials on B_{p,q}. For α = 0, 1, …, p−1 all functions f ∈ H_{α} satisfy some system of partial differential equations. For α = 0 our space contains only constants.

Proposition 1.7. ([3], [47]) a)For any g = (\begin{matrix}a & b \\ c & d \end{matrix}) ∈ U(p, q) the operator

T̃_{α}(g)(z)f(z) = f((a + zc)^{−1}(b + zd)) det(a + zc)^{−α}  (1.10)

is unitary in H_{α}^{∘} Remark. If α is not integer, then

det(a + zc)^{−α} = det a^{−α} det(1 + zca^{−1})^{−α} =

= | det a|^{−α} · e^{−α(i arg det a+2πki)} det(1 + zca^{−1})^{−α} (1.11) is a multi-valued function. It is easy to show that ∥ca^{−1}∥ < 1. Hence, (1 + zca^{−1})^{−α} is a well-defined single-valued function on the matrix ball B_{p,q}(ℂ). Hence, expression (1.11) has countable family of holomorphic branches on B_{p,q}(ℂ) and formula (1.10) defines a countable family of well-defined operators which differs by constant factors e^{2πkαi}.

13see Berezin [3] (1975), see also Gindikin [11], Rossi, Vergne [47], Wallach [55], see also a recent exposition in [9].

14Schtiefel manifold

Proof. Consider the supercomplete basis φ_{x}(z) = det(1 − x^{∗}z)^{−α} in H_{α}^{∘}. A calculation show that

T̃_{α}(g^{−1})φ_{x}(z) = det(a + zc)^{α}φ_{x^{[g]}}(z)

The simple identity

det(1 − x^{[g]}(y^{[g]})^{∗}) = det(1 − xy^{∗}) det(a + xc)^{−1} det(a + yc)^{−1}

implies

< φ_{x}, φ_{y} >_{H_{α}^{∘}}=< T̃_{α}(g^{−1})φ_{x}, T̃_{α}(g^{−1})φ_{y} >_{H_{α}^{∘}} ⊠

Obviously

T̃_{α}(g_{1})T̃_{α}(g_{2}) = e^{2πmαi}T̃_{α}(g_{1}g_{2}), where m ∈ ℤ

If α is integer, then T̃_{α} is a linear representation of U(p, q). If α is not integer, then T̃_{α} is a projective representation of U(p, q) or a linear representation of the universal covering group U(p, q)^{∼} of the group U(p, q).

1.11. Kernel representations of O(p, q). Kernel representation T_{α} of the group G = O(p, q) is the restriction of the representation T̃_{α} to the subgroup O(p, q). We also say that the function f(z) = 1 is the marked vector in H_{α}^{∘}. We denote this vector by Ξ.

Remark. A kernel representation is a linear representation. Indeed, we can wright | det a|^{−α}(1 + zca^{−1})^{−α} instead of (1.11).

Lemma 1.8. The vector Ξ is O(p, q)-cyclic.

Proof. Let Q ⊂ H_{α} be a subspace containing the G-orbit of Ξ. This orbit consists of functions (1.11) and hence the functions det(1 + zca^{−1})^{−α} are contained in Q. But the point ca^{−1} is the image of 0 under the fractional linear transformation (1.7). Since the action of O(p, q) is transitive on B_{p,q}(ℂ), the subspace Q contains all functions v_{u} = det(1 + zu^{∗})^{−α} where u ∈ B_{p,q}(ℝ). Furthermore, since the family v_{u} depends on u holomorphically, v_{u} ∈ Q for all u ∈ B_{p,q}(ℂ). But v_{u} is the supercomplete basis in H_{α}^{∘}. Hence, Q = H_{α}^{∘}. ⊠

Lemma 1.9. Any O(p, q)-invariant subspace in H_{α}^{∘} contains an O(p, q)- invariant vector.

Proof. Assume H_{α}^{∘} = R ⊕ Q where R, Q are invariant subspaces. Assume that R hasn’t an O(p) × O(q)-invariant vector. Then the projection of Ξ to R is zero, and hence Ξ ∈ Q. But Ξ is cyclic. Thus, Q = H_{α}^{∘}. ⊠

1.12. Another description of the kernel-representations. Let α satisfies Berezin conditions (1.9). By Lemma 1.5.e) the kernel

M_{α}(z, u) = \frac{det(1 − zz^{∗})^{α/2} det(1 − uu^{∗})^{α/2}}{det(1 − zu^{∗})^{α}}  (1.12)

on B_{p,q}(ℝ) is positive definite. A simple calculation show that the kernel M_{α} is O(p, q)-invariant. Hence, we obtain an unitary representation of the group O(p, q) in the hilbert space H[M_{α}] ≃ H^{∘}[M_{α}] (see Subsection 1.6). The group O(p, q) acts in H^{∘}[M_{α}] by substitutions

f(z) ↦ f((a + zc)^{−1}(b + zd))  (1.13)

The marked vector Ξ in this model is the element of the supercomplete basis corresponding to the point 0 ∈ B_{p,q}(ℝ).

Let us define the canonical unitary O(p, q)-intertwining operator

A : H^{∘}[L_{α}] → H^{∘}[M_{α}]

Let f ∈ H^{∘}[L_{α}], let z ∈ B_{p,q}(ℝ). Then

Af(z) = f(z) det(1 − zz^{∗})^{α/2}

This map transforms elements of the supercomplete basis in H^{∘}[L_{α}] to elements of the supercomplete basis in H^{∘}[M_{α}].

1.13. Limit as α → ∞. Let λ be the O(p, q)-invariant measure on B_{p,q}(ℝ) (see (1.8)). Denote by C_{0} the space of continuous functions on B_{p,q}(ℝ) with a compact support. If φ ∈ C_{0}, then φ(z)λ(z) is a complex valued measure on B_{p,q}. Hence (see Subsection 1.2), we obtain the scalar product in the space C_{0} given by

< φ, ψ >= A_{α} ∫_{B_{p,q}(ℝ)×B_{p,q}(ℝ)} M_{α}(z, u)φ(z)\overline{φ(u)} dλ(z)dλ(u)  (1.14)

Let us define the normalization constant A_{α} by the condition

A_{α} = (∫_{B_{p,q}(ℝ)}(1 − zz^{∗})^{α}dz)^{−1}

(it is a Hua Loo Keng integral, see (3.5)). Obviously M_{α}(z, z) = 1 and M_{α}(z, u) < 1 if z ≠ u. It is easy to see that the sequence A_{α}M_{α}(z, u) approximates the distribution δ(z − u). Thus, the limit of scalar products (1.14) as α → ∞ is

< φ, ψ >= ∫_{B_{p,q}(ℝ)} φ(z)\overline{ψ(z)} dλ(z)

In this sense the limit of kernel representations as α → ∞ is the space L^{2}(G/K).

We emphasis that the action of O(p, q) in L^{2}(O(p, q)/O(p)×O(q)) and in all spaces H^{∘}[M_{α}] is given by the same formula (1.13) and only scalar product in the space of functions varies. We will see that the spectrum of the representation T_{α} and the structure of Plancherel formula essentially depends on α.

1.14. Preliminary remarks on the Plancherel formula. Our purpose is to obtain a decomposition of the kernel representation T_{α} on irreducible representations.

An irreducible representation of G = O(p, q) is called spherical if it contains a K-fixed vector. This vector is called spherical vector. Remind that the space of K-fixed vectors for G has dimension 0 or 1 (Gelfand theorem, see for instance [17], Theorem 4.3.1 and Lemma 4.3.6). Denote the set of all unitary spherical representations of O(p, q) by Ĝ_{sph}

Remark. The explicit description of this set is not known. Parametrization of all (generally speaking nonunitary) spherical representations of O(p, q) is simple and it is given below in Subsection 1.17).

By H_{ρ} we denote the space of a spherical representation ρ, by ξ(ρ) we denote the spherical vector in H_{ρ} whose length is 1.

Lemma 1.10. Decomposition of the kernel-representation T_{α} has the form

T_{α}(g) = ∫_{ρ∈Ĝ_{sph}} ρ(g)dν_{α}(g)  (1.15)

where ν_{α} is a Borel measure on Ĝ_{sph}.

Remark. For the definition of direct integrals of representations and the abstract Plancherel formula see, for instance, [23], 8.4.

Proof. By Lemma 1.9, the decomposition contains only spherical representations. Hence, by the abstract Plancherel theorem the representation T_{α}(g) has the form

T_{α}(g) = ⨁_{j=1}^{κ} R_{j}

where

R_{j} = ∫_{ρ∈Ĝ_{sph}} ρ(g)dν_{α}^{j}(g)

and the measure ν_{α}^{j+1} is absolutely continuous with respect to ν_{α}^{j} for all j. The number κ can be 1,2, …, ∞. We must prove that κ = 1.

All K-fixed vectors in R_{j} are functions having the form φ_{j}(ρ)ξ(ρ) where φ_{j}(ρ) is a ν_{α}^{1}-measurable function on Ĝ_{sph}.

Consider the projection Ξ^{(1,2)} of the marked vector Ξ to R_{1} ⊕ R_{2}. Since the vector Ξ is cyclic in whole space, its projection must be cyclic in R_{1} ⊕ R_{2}. The vector Ξ^{(1,2)} has the form

(φ_{1}(ρ)ξ(ρ), φ_{2}(ρ)ξ(ρ)) ∈ R_{1} ⊕ R_{2}

Obviously the cyclic span of Ξ^{(1,2)} in R_{1} ⊕ R_{2} contains only vectors

(q_{1}(ρ)ξ(ρ), q_{2}(ρ)ξ(ρ))

satisfying the condition

φ_{2}(ρ)q_{1}(ρ) = φ_{1}(ρ)q_{2}(ρ)

If ν_{α}^{2} ≠ 0 we obtain a contradiction, since the cyclic span of Ξ^{(1,2)} is a proper subspace in R_{1} ⊕ R_{2}. ⊠

1.15. Normalization of the Plancherel measure. The measure ν_{α} in (1.15) is defined up to equivalence of measures^{15}.

The image of the marked vector Ξ in the direct integral (1.15) is some function φ(ρ)ξ(ρ) where ξ(ρ) is an unit K-fixed vector in H_{ρ}. It is convenient to assume

φ(ρ) = 1  (1.16)

This assumption uniquely defines the measure ν_{α}.

Remark. Assumption (1.16) is not restrictive. Indeed, let us assume that the image of Ξ in (1.15) is a function γ(ρ)ξ(ρ). Then the Plancherel measure is completely defined by this assumption and it equals to \frac{1}{\sqrt{|γ|}}ν_{α}.

After normalization (1.16) we obtain the following equality of matrix elements

< T_{α}(g)Ξ, Ξ >_{H_{α}}=< [∫Ĝ_{sph} ρ(g)dν_{α}(ρ)] · 1, 1 >  (1.17)

or

< T_{α}(g)Ξ, Ξ >_{H_{α}}= ∫Ĝ_{sph} < ρ(g)ξ(ρ), ξ(ρ) >_{H_{ρ}} dν_{α}(ρ)  (1.18)

Conversely, assume that we know a measure ν_{α} on Ĝ_{sph} satisfying condition (1.18). Then it satisfies condition (1.17). Hence, the representations T_{α} and ∫Ĝ_{sph} ρ(g)dν_{α} have the same matrix elements, and therefore they are canonically equivalent (see Subsection 1.7).

The marked vector Ξ is K-invariant, therefore (see Subsection 1.6) we can consider the matrix element

ℬ_{α}(g) :=< T_{α}(g)Ξ, Ξ >

as a function on G/K or a function on K \ G/K. Vectors Ξ and T_{α}(g)Ξ are elements of the supercomplete basis in H[M_{α}], therefore the function ℬ_{α} can be easily evaluated.

15measures μ, ν are equivalent if there exists a function χ such that χ ≠ 0 almost everywhere (in sense of ν) and μ = χν.

In the matrix ball model of G/K the function ℬ_{α} is given by the formula

ℬ_{α}(z) = det(1 − zz^{∗})^{α/2};  z ∈ B_{p,q}  (1.19)

Let us obtain the formula for ℬ_{α} as function on K \ G/K. Denote by a_{t} the element of O(p, q) given by the matrix

a_{t} = ( \begin{matrix}cosh t_{1} ; ... ; cosh t_{p} & sinh t_{1} ; ... ; sinh t_{p} & \begin{matrix}0 & ⋯ & 0 \\ . & . & . \\ 0 & ⋯ & 0 \end{matrix} \\ sinh t_{1} ; ... ; sinh t_{p} & cosh t_{1} ; ... ; cosh t_{p} & \begin{matrix}0 & ⋯ & 0 \\ . & . & . \\ 0 & ⋯ & 0 \end{matrix} \\ \begin{matrix}0 & ⋯ & 0 \\ . & . & . \\ 0 & ⋯ & 0 \end{matrix} & \begin{matrix}0 & ⋯ & 0 \\ . & . & . \\ 0 & ⋯ & 0 \end{matrix} & \begin{aligned}1 & ⋯ \\ . & . & . \\ & ⋯ & 1 \end{aligned} \end{matrix} )

(1.20)

It is easy to show that arbitrary element g of G = O(p, q) can be represented in the form

g = k_{1}a_{t}k_{2};  where  k_{1}, k_{2} ∈ K

The collection of parameters t = (t_{1}, …, t_{p}) is uniquely defined up to permutations of t_{j} and reflections

(t_{1}, …, t_{p}) ↦ (σ_{1}t_{1}, …, σ_{p}t_{p})  (1.21)

where σ_{j} = ±1.

We denote by 𝒜 the subgroup in O(p, q) consisting of all elements a_{t}. We denote by D_{p} the group of transformations of ℝ^{p} generated by permutations of coordinates and reflections (1.21).

We identify the set K \ G/K with the set of D_{p}-orbits on 𝒜.

In coordinates (t_{1}, …, t_{p}) the matrix element ℬ_{α} is given by the formula

ℬ_{α}(t_{1}, …, t_{p}) = ∏_{k=1}^{p} cosh^{−α} t_{k}  (1.22)

Hence, we must obtain the expansion (1.18) of the function ℬ_{α} given by formula (1.19) or (1.22) in positive definite spherical functions.

Our purpose in Section C is to give an expression for spherical functions.

C. Spherical representations and spherical transform

1.16. Parabolic subgroup. Consider the space ℝ^{p} ⊕ℝ^{q} equipped with the indefinite symmetric form J defined by formula (1.5). A subspace V ⊂ ℝ^{p} ⊕ ℝ^{q} is called isotropic if the form J is zero on V.

An isotropic flag 𝒱 in ℝ^{p} ⊕ ℝ^{q} is a family of isotropic subspaces

𝒱 : V_{1} ⊂ V_{2} ⊂ ⋯ ⊂ V_{p}; where dim V_{j} = j

The flag manifold ℱ is the space of all isotropic flags in ℝ^{p} ⊕ ℝ^{q}.

The space ℱ is an O(p, q)-homogeneous space. A minimal parabolic subgroup is the stabilizer of a point in ℱ. Let us give more explicit description of the minimal parabolic subgroup.

For this let us consider the basis v_{1}, …, v_{p}, w_{1}, …, w_{q−p}, v′_{1}, …, v′_{p} in ℝ^{p}⊕ℝ^{q} defined by

v_{j} = \frac{1}{\sqrt{2}}(e_{j} + e_{q+j});  v′_{j} = \frac{1}{\sqrt{2}}(e_{j} − e_{q+j}); w_{k} = e_{p+k}  (1.23)

Then

J(e_{k}, e′_{k}) = 1; J(f_{k}, f_{k}) = 1

and the scalar products of all other pairs of basic vectors are zero.

Denote by L_{k} the subspace in ℝ^{p} ⊕ ℝ^{q} generated by the basic vectors v_{1}, …, v_{k}. We denote by P ⊂ O(p, q) the stabilizer in O(p, q) of the isotropic flag

ℒ : L_{1} ⊂ ⋯ ⊂ L_{p}  (1.24)

The subgroup P is a minimal parabolic subgroup in O(p, q) and

ℱ ≃ O(p, q)/P

Elements of the parabolic subgroup P in the basis (1.23) have the form

(\begin{matrix}A & ∗ & ∗ \\ 0 & C & ∗ \\ 0 & 0 & A^{t−1} \end{matrix})  (1.25)

where A is an upper triangular matrix and C ∈ O(q − p).

Elements of the subgroup 𝒜 (see Subsection 1.15) in new basis are diagonal matrices with eigenvalues

e^{t_{1}}, …, e^{t_{p}}, 1, …, 1, e^{−t_{1}}, …, e^{−t_{p}}

Remark. Let us change the order of basic elements (1.23) to v_{1}, …, v_{p} ,w_{1}, …, w_{q−p}, v′_{p}, …, v′_{1}. Then elements of the parabolic subgroup P will be upper triangular matrices.

1.17. Spherical representations. Denote by Gr_{k} the space of all kdimensional isotropic subspaces in ℝ^{p}⊕ℝ^{q}. Consider the tautological embedding of the flag space ℱ to the product of the Grassmannians ×_{k=1}^{p}Gr_{k} ( to each point 𝒱 : V_{1} ⊂ ⋯ ⊂ V_{p} we assign the point (V_{1}, …, V_{p}) ∈ ×_{k=1}^{p}Gr_{k}).

Consider the natural action of O(p, q) on Gr_{k}. For g ∈ G we denote by j_{k}(g, V) the Jacobian of the transformation g at the point V ∈ Gr_{k}. By J(g, 𝒱) we denote the Jacobian of the transformation g on the flag space ℱ.

Fix s_{1}, …, s_{p} ∈ ℂ. Assume s_{0} = s_{p+1} = 0. We define the representation π̃_{s} of the group O(p, q) in the space of functions on ℱ by the formula

π̃_{s}(g)f(V_{1}, …, V_{p}) = f(gV_{1}, …, gV_{p})J(g, 𝒱)^{1/2} ∏_{k=1}^{p} j_{k}(g, V_{k})^{(s_{j−1}−2s_{j}+s_{j+1})/2}

Remark. The representation π̃_{s} is a Harish-Chandra module^{16} and hence a topology in the space of functions on ℱ is not essential. For instance, we can consider the space L^{2}(ℱ), the space of smooth functions C^{∞}(ℱ), the space of distributions 𝒟(ℱ), the space of hyperfunctions etc.

Remark. Consider the δ-function δ_{ℒ} supported at the point ℒ ∈ ℱ (see (1.24)). It is easy to observe that the function δ_{ℒ} is an eigenfunction of P and for a matrix g ∈ P given by (1.25) we have

π̃_{s}(g)δ_{ℒ} (𝒱) = exp{∑_{j=1}^{p} t_{j}(s_{j} − (q + p)/2 + j)}δ_{ℒ} (𝒱)  (1.26)

where e^{t_{j}} are the absolute values of the eigenvalues of the matrix A (see (1.25)). We want to define a canonical irreducible subquotient π_{s} in π̃_{s}. Remark. For generic s ∈ ℂ^{n} the representation π̃_{s} is irreducible and hence

π_{s} ≃ π̃_{s}.

Consider the function f_{0}(𝒱) = 1 on the space ℱ, it is the unique K-invariant function on ℱ (since ℱ is K-homogeneous). Denote by S the cyclic span of f_{0}. Denote by R the sum of all proper O(p, q)-submodules in S.

Lemma 1.11. R ≠ S.

Proof. Indeed there is the unique K-fixed vector in S and this vector is cyclic. Hence, it can’t be element of a proper submodule. Hence, a proper submodule in S hasn’t K-fixed vector. Hence, R also hasn’t K-fixed vectors and hence f_{0} ∉ R. ⊠

We define the O(p, q)-module π_{s} by

π_{s} = S/R

Theorem 1.12.^{17} The representations π_{s} are precisely all spherical representations of O(p, q). Moreover

π_{s} ≃ π_{s′} iff there exists γ ∈ D_{p} such that γs = s′

16This means (for instance, see [24]) that the spectrum of the maximal compact subgroup K in the space of functions on ℱ has finite multiplicities.

17for instance, see [17], Theorem 4.4.3

Hence, we can consider our Plancherel measure ν_{α} as a measure on ℂ^{p}/D_{p}. It will be more convenient for us to consider the Plancherel measure as a D_{p}- invariant measure on ℂ^{p} or any measure on on ℂ^{p} whose D_{p}-average is ν_{α}.

1.18. Unitary spherical representations. Lemma 1.13. Assume the representation π_{s} be unitary. Then for any j

Re s_{j} = 0  or  Im s_{j} = 0  (1.27)

Proof. The representation dual to π_{s} is π_{−s}. The complex conjugate representation to π_{s} is π\overline{s}. If π_{s} is unitary, then the dual representation is equivalent to the complex conjugate representation. Hence, −s = γ\overline{s} for some γ ∈ D_{p}. ⊠

If s_{1}, …, s_{p} are pure imaginary, then the representation π_{s} is unitary in L^{2}(ℱ). These representations are called representations of the principal nondegenerate series.

For some other values of s representations π_{s} also are unitary, but scalar product in these cases is more complicated.

Theorem 1.14. (see [17], 4.8.1) Denote by ρ the vector

((q + p)/2 − 1, (q + p)/2 − 2, …, (q − p)/2)) ∈ ℝ^{p}

Denote by Q the convex polyhedron in ℝ^{p} with vertices γρ where γ ∈ D_{p}. Then for each unitary representation π_{s}

(Re s_{1}, …, Re s_{p}) ∈ Q  (1.28)

Moreover, spherical function of a spherical representation π_{s} is bounded if and only if condition (1.28) holds.

Our next purpose is to give the integral formula for the spherical functions in an explicit form. For this we must give another realization of G/K.

1.19. Matrix wedges. First, consider the case p = q. Consider the matrix ball B_{q,q}(ℝ). Consider the Cayley transform

Cay : z ↦ \frac{1 − z}{1 + z}  (1.29)

Then the map Cay transfers the matrix ball B_{q,q}(ℝ) to the wedge W_{q} consisting of matrices R satisfying the condition^{18}

R + R^{t} > 0

(where the notation Q > 0 means that a matrix Q is positive definite). It is convenient to wright R in the form

R = T + S  where  T = T^{t} > 0; S = −S^{t}

18A matrix satisfying the condition R + R^{t} > 0 is called dissipative.

The group O(q, q) acts on B_{q,q}(R) and hence it acts on W_{q,q}. For description of the last action we consider the basis (1.23) in ℝ^{q} ⊕ ℝ^{q}. In our case p = q, and hence the basic elements w_{j} are lacking. Hence, O(q, q) becomes the group

of real (q + q) × (q + q)-matrices having the form g = (\begin{matrix}a & b \\ c & d \end{matrix}) and satisfying

the condition

(\begin{matrix}a & b \\ c & d \end{matrix}) (\begin{matrix}0 & 1 \\ 1 & 0 \end{matrix}) (\begin{matrix}a & b \\ c & d \end{matrix})^{t} = (\begin{matrix}0 & 1 \\ 1 & 0 \end{matrix})

The group O(q, q) acts on W_{q} by fractional linear transformations

R ↦ R^{[g]} := (a + Rc)^{−1}(b + Rd)

In this model, the parabolic subgroup P ⊂ O(q, q) becomes the group of real matrices having the form

(\begin{matrix}a & b \\ 0 & a^{t−1} \end{matrix});  where  d is upper triangular  (1.30)

Hence, the parabolic subgroup acts on W_{q} by affine transformations

R ↦ a^{−1}Ra^{t} + a^{−1}b

We emphasis that a^{−1}b is a skew-symmetric matrix.

We can easily wright the eigenfunctions of the group P on the wedge W_{q}

Ψ_{s_{1},...,s_{q}}(R) = ∏_{j=1}^{q} det[T]_{j}^{(−θ_{j}+s_{j}−s_{j+1})/2} where θ_{1} = ⋯ = θ_{p−1} = 1, θ_{p} = 0

(1.31)

Here the symbol [T]_{j} denotes the left upper j × j block of the matrix T.

Consider g ∈ P given by formula (1.30). Let e^{t_{1}}, …, e^{t_{p}} be the absolute values of the diagonal elements of the block a. Then

Ψ_{s_{1},...,s_{q}}(R^{[g]}) = exp{∑_{j=1}^{q}(s_{j} − q + j)t_{j}}Ψ_{s_{1},...,s_{q}}(R)  (1.32)

Remark. Compare (1.32) and (1.26).

1.20. Sections of wedges. Consider arbitrary group O(p, q). Let us represent a point z ∈ B_{p,q}(ℝ) as block p × (p + (q − p))-matrix z = (z_{1} z_{2}). Consider block (p + (q − p)) × (p + (q − p)) matrix

z̃ = (\begin{matrix}0 & 0 \\ z_{1} & z_{2} \end{matrix}) ∈ B_{q,q}(ℝ)

Thus, we realized the matrix ball B_{p,q}(ℝ) as a submanifold of B_{q,q}(ℝ). The image SW_{p,q} of B_{p,q}(ℝ) under the Cayley transform (1.29) is the set of (p+(q − p)) × (p + (q − p))-matrices R ∈ W_{q} having block structure

R = (\begin{matrix}1 & 0 \\ Q & H \end{matrix})  (1.33)

The condition R + R^{t} > 0 for matrix (1.33) is equivalent to the condition

\frac{1}{2}(H + H^{t}) − LL^{t} > 0  (1.34)

(spaces SW_{p,q} are real sections of so-called Siegel domains of the second type, see [44])

We will wright matrices R ∈ SW_{p,q} in the form

R = (\begin{matrix}1 & 0 \\ 2L & M + N \end{matrix}); M = M^{t}, N = −N^{t}

The condition (1.34) can be represented in the form

M − LL^{t} > 0  or  (\begin{matrix}1 & L^{t} \\ L & M \end{matrix}) > 0  (1.35)

The eigenfunctions of the parabolic subgroup P ⊂ O(p, q) in this model are given by the formula

Ψ_{s_{1},...,s_{p}}(R) = ∏_{j=1}^{p} det[(\begin{matrix}1 & L^{t} \\ L & M \end{matrix})]_{q−p+j}^{(−θ_{j}+s_{j}−s_{j+1})/2} = ∏_{j=1}^{p} det[M − LL^{t}]_{j}^{(−θ_{j}+s_{j}−s_{j+1})/2} (1.36)

where θ_{1} = ⋯ = θ_{p−1} = 1, θ_{p} = \frac{1}{2}(q − p) (see simple calculations in [35])

The Berezin kernel L_{α} (see Subsection 1.10) in the models W_{q}, SW_{p,q} is given by the formula

L_{α}(R_{1}, R_{2}) = [\frac{det(R_{1} + R_{2}^{t})}{det(1 + R_{1}) det(1 + R_{2})}]^{−α}

This gives the following expression for the function ℬ_{α}(R) in our coordinates

ℬ_{α}(R) = \frac{det 2^{α} (\begin{matrix}1 & L^{t} \\ L & M \end{matrix})^{α/2}}{det(1 + M + N)^{α}}  (1.37)

The O(p, q)-invariant measure on SW_{p,q} is

det (\begin{matrix}1 & L^{t} \\ L & M \end{matrix})^{−(p+q)/2} dL dM dN

where dL, dM, dN are Lebesgue measures on the spaces of matrices.

1.21. Canonical embedding of the spherical G-module π_{s} to the space C^{∞}(G/K). First, we define the canonical intertwining operator J_{s} from π̃_{s} to C^{∞}(G/K). This operator is uniquely defined by the following property

J_{s} : δ_{ℒ} ↦ Ψ_{s}

where P -eigenfunctions δ_{ℒ} , Ψ_{s} were defined in Subsections 1.17, 1.19–1.20. By the intertwining property we obtain

J_{s}δ_{gℒ} (R) = Ψ_{s}(R^{[g]})

and this defines the operator J_{s} on all δ-functions. Then we extend J_{s} by linearity and continuity to the whole space of distributions on ℱ.

Lemma 1.15. The operator J_{s} induces an embedding of the subquotient π_{s} to C^{∞}(G/K).

Le us denote by dk the Haar measure on K = O(p) × O(q). We assume that the measure of the whole group is 1.

Proof. Let R, S be the same as in Subsection 1.17. Let Q ⊂ C^{∞}(G/K) be a G-invariant closed subspace. Then for any function f ∈ R, its average

f^{K}(R) = ∫_{k∈K} f([R]^{k})dk

is contained in Q. Hence, Q contains a K-invariant function.

By this reason, J_{s} maps the submodule R to 0 (since R hasn’t K-invariants). Assume that J_{s} is zero on S. Then J_{s} is an operator from π̃_{s}/S to C^{∞}(G/K). But the module π̃_{s}/S hasn’t K-invariant vectors. Hence, J_{s} is identical zero and this contradicts to its definition. ⊠

Obviously the K-fixed function f_{0} = 1 on ℱ can be represented in the form

f_{0}(𝒱) = ∫_{k∈K} δ_{ℒ} (k𝒱)dk

Hence, its image under J_{s} is the K-average of Ψ_{s}. This gives the integral formula for spherical function given in the next Subsection.

1.22. Integral formula for spherical functions. Spherical functions are K-averages of P -eigenfunctions on G/K

Φ_{s_{1},...,s_{p}}(R) = ∫_{k∈O(p)×O(q)} Ψ_{s_{1},...,s_{p}}(R^{[k]})dπ(k)  (1.38)

Lemma 1.16.

|Φ_{s_{1},...,s_{p}}(t)| ⩽ Φ_{Re s_{1},...,Re s_{p}}(t)  (1.39)

Proof is obvious. ⊠ 1.23. Spherical transform. Let f(z) be a K-invariant function on G/K.

Then the spherical transform of f is defined by the formula

f̂(s) = ∫_{G/K} Φ_{s}(z)f(z)dλ(z)  (1.40)

where λ is the G-invariant measure on G/K.

If f ∈ L^{2} ∩ L^{1}(G/K), then the Gindikin–Karpelevich inversion formula (see [13], [14], [17], [9]) is valid

f(z) = C · ∫_{iℝ^{p}} f̂(s)Φ_{s}(z)ℜ(s)ds  (1.41)

where C is a known constant (see [17], formula (4.6.40)) and ℜ(s) is the Gindikin– Karpelevich density. For G = O(p, q) it is given by the formula

· ℜ(s) = ∏_{k=1}^{p} \frac{Γ((q − p)/2 + s_{k})Γ((q − p)/2 − s_{k})}{Γ(s_{k})Γ(−s_{k})} ×  (1.42)

× ∏_{1⩽k<l⩽p} \frac{Γ(\frac{1}{2}(1 + s_{l} + s_{k}))Γ(\frac{1}{2}(1 + s_{l} − s_{k}))Γ(\frac{1}{2}(1 − s_{l} + s_{k}))Γ(\frac{1}{2}(1 − s_{l} − s_{k}))}{Γ(\frac{1}{2}(s_{l} + s_{k}))Γ(\frac{1}{2}(s_{l} − s_{k}))Γ(\frac{1}{2}(−s_{l} + s_{k}))Γ(\frac{1}{2}(−s_{l} − s_{k}))} (1.43)

Remark. This expression is an elementary function. For instance, using the complement formula for Γ, we reduce factor (1.43) to the form

∏_{1⩽k<l⩽p}(s_{k}^{2} − s_{l}^{2}) tan π(s_{k} + s_{l}) tan π(s_{k} − s_{l})  (1.44)

If q − p is even, then (1.42) equals to

∏_{k=1}^{p}{ ∏_{τ=0}^{(q−p)/2−1}(τ^{2} − s_{k}^{2})}  (1.45)

If (q − p) is odd, then (1.42) equals to

∏_{k=1}^{p}{s_{k} tan πs_{k} ∏_{τ=0}^{(q−p−3)/2}((τ + 1/2)^{2} − s_{j}^{2})}  (1.46)

For pure imaginary s we can replace (1.42)-(1.43) by

∏_{k=1}^{p}|\frac{Γ((q − p)/2 + s_{k})}{Γ(is_{k})}|^{2} ∏_{1⩽k<l⩽p}|\frac{Γ(\frac{1}{2}(1 + s_{l} + s_{k}))Γ(\frac{1}{2}(1 + s_{l} − s_{k}))}{Γ(\frac{1}{2}(s_{l} + s_{k}))Γ(\frac{1}{2}(s_{l} − s_{k}))}|^{2}

since s_{k} is imaginary. Nevertheless long expression (1.42)–(1.43) is more convenient for our calculations.

1.24. Another formula for spherical transform. By integral formula for spherical functions (1.38) we can wright spherical transform (1.40) in the following form

f̂(s) = ∫_{G/K} f(z)Ψ_{s}(z)dλ(z)  (1.47)

1.25. Further structure of the paper. We want to obtain an expansion of the function ℬ_{α}(z) in spherical functions. If ℬ_{α} ∈ L^{1} ∩ L^{2}(B_{p,q}(ℝ)) (or α > p + q − 1), then it is sufficient to evaluate the spherical transform of the function ℬ_{α}(z), and the Gindikin-Karpelevich inversion formula gives required expansion.

In Section 3 we evaluate the spherical transform of ℬ_{α}(z) using formula (1.47). The final result is given in Theorem 2.1. Then in Section 4 we construct the analytic continuation of our formula to arbitrary α. As result, we obtain an expansion of ℬ_{α} in spherical functions. In Section 5 we prove positive definiteness of these spherical functions.

D. Deformation of L^{2} on Riemannian compact symmetric space and kernel representations of O(p + q).

This subject is a supplement to the main topic of the paper.

1.26. The symmetric spaces U(p + q)/U(p) × U(q) and O(p + q)/O(p) × O(q). Consider the group U(p+q) consisting of all complex block (p+q)×(p+q)-

matrices (\begin{matrix}a & b \\ c & d \end{matrix}) satisfying the condition

(\begin{matrix}a & b \\ c & d \end{matrix}) (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}) (\begin{matrix}a & b \\ c & d \end{matrix})^{∗} = (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix})

Consider the subgroup O(p + q) ⊂ U(p + q) consisting of real matrices.

Consider the Grassmannians Gr_{p,q}(ℂ) and Gr_{p,q}(ℝ) consisting of p-dimensional subspaces in ℂ^{p+q} and ℝ^{p+q} respectively. Obviously, these Grassmannians are symmetric spaces

Gr_{p,q}(ℂ) = U(p + q)/U(p) × U(q)
Gr_{p,q}(ℝ) = O(p + q)/O(p) × O(q)

Denote by Mat_{p,q}(ℂ) (resp. Mat_{p,q}(ℝ)) the space of all p × q-matrices over ℂ (resp. over ℝ). For any z ∈ Mat_{p,q} we define its graph GRAPH_{z} ⊂ Gr_{p,q}.

Obviously the map z ↦ GRAPH_{z} is an embedding of Mat_{p,q} → Gr_{p,q} and the image of the embedding is dense in Gr_{p,q}.

In the coordinate z ∈ Mat_{p,q}, the action of the group U(p+q) on Grassmannian is given by the formula

z ↦ z^{[g]} = (a + zc)^{−1}(b + zd)  (1.48)

coinciding with formula (1.7).

1.27. Representations T̃_{−n}. Fix n = 0, 1, 2, . . .. Denote by φ_{a}(z) the polynomial on Mat_{p,q} given by the formula

φ_{a}(z) = det(1 + a^{∗}z)^{n}; where a ∈ Mat_{p,q}

Denote by H_{−n} the linear span of all polynomials φ_{a}(z). Obviously the space H_{−n} is finite dimensional (since degree of the polynomial φ_{a}(z) is pn).

Consider the action of the group U(p, q) in the space H_{−n} given by the formula

T̃_{−n} (\begin{matrix}a & b \\ c & d \end{matrix}) f(z) = f((a + zc)^{−1}(b + zd)) det(a + zc)^{n}  (1.49)

coinciding with formula (1.10). It is easy to check, that the transformations T̃_{−n}(g) preserve the space H_{−n}.

Consider the scalar product in H_{−n} given by

< f_{1}(z), f_{2}(z) >_{−n}= C_{n} ∫_{Mat_{p,q}} f_{1}(z)\overline{f_{2}(z)} det(1 + z^{∗}z)^{−n−p−q}dz

where the normalization constant C_{n} is defined by the condition < 1, 1 >_{−n}= 1.

It is easy to check that the operators T̃_{−n}(g) are unitary with respect to this scalar product and

< φ_{a}, φ_{b} >_{−n}= det(1 + a^{∗}b)^{n}  (1.50)

Hence, the (finite dimensional) hilbert space H_{−n} is the hilbert space H^{∘} defined by the positive definite kernel

L_{−n}(a, b) = det(1 + a^{∗}b)^{n};  a, b ∈ Mat_{p,q}  (1.51)

1.28. Kernel representations of O(p + q). The kernel representation T_{−n} of the group O(p + q) is the restriction of the representation T̃_{−n} to the subgroup O(p + q).

The also define the marked vector Ξ

Ξ : f(z) = 1

1.29. Limit as n → ∞. Let us consider the kernel

M_{−n}(z, u) = \frac{det(1 + z^{t}u)^{n}}{det(1 + z^{t}z)^{n/2} det(1 + u^{t}u)^{n/2}}

on Mat_{p,q}(ℝ). By Lemma 1.5.e) the kernel M_{−n} is positive definite. A simple calculation shows, that the kernel is O(p + q)-invariant. Consider the hilbert space H^{∘}[M_{−n}]. The operator

Af(z) = det(1 + z^{∗}z)^{n/2}f(z)

defines the canonical unitary O(p+q)-intertwining operator H^{∘}[L_{α}] → H^{∘}[M_{α}].

The arguments given in Subsection 1.13 show that a natural limit of the spaces H_{−n} as n → ∞ is

L^{2}(O(p + q)/O(p) × O(q))

1.30. Preliminary remarks on the Plancherel formula. By Subsection 1.7 the matrix element

ℬ_{−n}(g) =< T_{−n}(g)Ξ, Ξ >_{H_{−n}}

is a function on O(p + q)/O(p) × O(q) ≃ Gr_{p,q}(ℝ). In the coordinate z ∈ Mat_{p,q}(ℝ) it is given by

ℬ_{−n}(z) = det(1 + zz^{t})^{−n/2}

Denote by { \atop O(p + q)}_{sph} the set of all irreducible representations of O(p + q) having an O(p) × O(q)-invariant vector (spherical vector); description of this (countable) set is given by Helgason theorem, [17], Theorem 5.4.1. By H_{ρ} we denote the space of a spherical representation ρ ∈ { \atop O(p + q)}_{sph}. Denote by ξ_{ρ} the spherical vector in H_{ρ} having unit length.

Arguments given in Subsections 1.13-1.14 show, that the decomposition of T_{−n} in irreducible representations has the form

T_{−n}(g) = ⨁_{ρ∈Δ_{n}} ρ(g)

where Δ_{n} is a finite subset in { \atop O(p + q)}_{sph}.

The scalar product in ⨁_{ρ∈Δ_{n}} H_{ρ} has the form

< ⨁_{ρ∈Δ_{n}} v_{ρ}, ⨁_{ρ∈Δ_{n}} w_{ρ} >= ∑_{ρ∈Δ_{n}} ν_{ρ}^{n} < v_{ρ}, w_{ρ} >_{H_{ρ}}  (1.52)

where v_{ρ}, w_{ρ} ∈ H_{ρ} and ν_{ρ}^{n} are positive constants. The formula (1.52) is called the Plancherel formula.

We normalize the constants ν_{ρ}^{n} by the assumption

the image of Ξ in ⨁_{ρ∈Δ_{n}} H_{ρ} is ⨁_{ρ∈Δ_{n}} ξ_{ρ}

The constants ν_{ρ}^{n} are evaluated in Section 2 as a corollary of the Plancherel formula for kernel-representations of O(p, q).

E. An interpolation between

L^{2}(O(p, q))/O(p) × O(q)) and L^{2}(O(p + q))/O(p) × O(q)) ?

The purpose of the Section is a formulation of a strange problem.

1.31. General representations T̃_{α}. Denote by Hol(B_{p,q}) the space of holomorphic functions in B_{p,q}(ℂ) equipped with topology of the uniform convergence on compacts.

Consider arbitrary α ∈ ℂ and consider the action of U(p, q) in Hol(B_{p,q}) given by the formula

T̂_{α} (\begin{matrix}a & b \\ c & d \end{matrix}) f(z) = f((a + zc)^{−1}(b + zd)) det(a + zc)^{−α}  (1.53)

Denote by ℋ_{α}^{∘} the cyclic span of the function f(z) = 1. Denote by T̃_{α} the restriction of T̂_{α} to ℋ_{α}^{∘}. It is easy to observe that T̃_{α} is an irreducible Harish-Chandra module.

Let us denote by H_{α}^{fin} the space of polynomials contained in ℋ_{α}^{∘}. Consider the action of the Lie algebra u(p, q) in H_{α}^{fin}. The space H_{α}^{fin} is an irreducible u(p, q)-module with a highest weight. For α ∈ ℝ there exists the unique u(p, q)- invariant hermitian form in H_{α}^{fin} (it is called Shapovalov form^{19}). In general this form is indefinite.

If α ∈ ℝ satisfies Berezin conditions (1.9), then the Shapovalov form is positive definite. It coincides with the Berezin scalar product, and representations T̃_{α} coincides with representations T̃_{α} constructed in Subsection 1.10. If α is a negative integer, then T̃_{α} is finite dimensional. By the unitary Weyl trick, there is no difference between finite dimensional representations of U(p, q), holomorphic finite dimensional representations of GL(p + q, ℂ) and finite dimensional representations of U(p+q). The representations T̃_{α} for negative integer α differs from the representations T̃_{−n} from Subsection 1.27 by a nonessential change of notations.

19Its definition for highest weight modules of various group is uniform, see for instance [32].

1.32. Nonunitary kernel representations of O(p, q)? Consider the restriction T_{α} of T̃_{α} to the subgroup O(p, q). It is a well-defined representation of the group O(p, q) in the space ℋ_{α}^{∘}.

We have seen that

lim_{α→+∞} T_{α} ≃ L^{2}(O(p, q))/O(p) × O(q))
lim_{n→−∞} T_{n} ≃ L^{2}(O(p + q))/O(p) × O(q))

It seems that the Plancherel formula (2.5)-(2.15) gives the decomposition of the kernel representation T_{α} for any complex α. Unfortunately it is a result of ’mathematical physics level’ . This is the solution of a problem which hasn’t a satisfactory formulation (since the definition of the abstract Plancherel formula doesn’t exist for nonunitary representations).

Remark. For the case p = 1 the space ℋ_{α}^{∘} equipped with the Shapovalov form is a Pontryagin space^{20} and in this case our Plancherel formula is really the Plancherel formula for arbitrary real α.

The questions of this type are discussed for a long time and they arise to deep Molchanov work [27](1980) containing the Plancherel decomposition of tensor products of unitary representations of SL(2, ℝ) (see also [8] and references in this paper). It was clear, that Molchanov formula give formal interpolation between tensor products of unitary representations of SL(2, ℝ) and tensor products of finite dimensional representations. Unfortunately before our time a quite satisfactory group-theoretical interpretation of this interpolation doesn’t exist.

2. Formulation of results.

2.1. Large α. Theorem 2.1.Let α > (p + q)/2 − 1. Then the spectrum of the kernel-representation T_{α} is supported by nondegenerate principal series and the Plancherel decomposition is given by the formula

20This means that negative inertia index of the Shapovalov form is finite.

∏_{k=1}^{p} cosh^{−α} t_{j} = C · 2^{α} \frac{1}{∏_{j=1}^{p} Γ(α − j + 1)} ×  (2.1)

∫_{iℝ^{p}} ∏_{k=1}^{p} \{Γ(\frac{1}{2}(α − (p + q)/2 + 1 + s_{k}))Γ(\frac{1}{2}(α − (p + q)/2 + 1 − s_{k}))\} × (2.2)

× ∏_{k=1}^{p} \frac{Γ((q − p)/2 + s_{k})Γ((q − p)/2 − s_{k})}{Γ(s_{k})Γ(−s_{k})} ×  (2.3)

∏_{1⩽k<l⩽p} \frac{Γ(\frac{1}{2}(1 + s_{l} + s_{k}))Γ(\frac{1}{2}(1 + s_{l} − s_{k}))Γ(\frac{1}{2}(1 − s_{l} + s_{k}))Γ(\frac{1}{2}(1 − s_{l} − s_{k}))}{Γ(\frac{1}{2}(s_{l} + s_{k}))Γ(\frac{1}{2}(s_{l} − s_{k}))Γ(\frac{1}{2}(−s_{l} + s_{k}))Γ(\frac{1}{2}(−s_{l} − s_{k}))} ×

(2.4)

× Φ_{s_{1},...,s_{p}}(t_{1}, …, t_{p}) ds_{1}ds_{2} … ds_{p}  (2.5)

where C is a constant.

Remark. Factor (2.3)–(2.4) is the Gindikin–Karpelevich density, it is an elementary function, see (1.44)–(1.46).

2.2. Analytic formula for arbitrary α. Fix m = 0, 1, …, p. Consider nonnegative integers

u_{1} ⩽ u_{2} ⩽ … ⩽ u_{m}

satisfying the condition

α + 2u_{m} + m < \frac{1}{2}(p + q)

(if m = 0, then a collection {u} is empty).

Theorem 2.2. Let p ≠ q and α be arbitrary, or p = q and α ∈ ℝ \ {1, 2, …, p − 1}. Then

∏_{k=1}^{p} cosh^{−α} t_{j} = C · ∑ ; \begin{matrix}m; \\ u_{1} ⩽ … ⩽ u_{m} < \frac{1}{4}(p + q) − \frac{m}{2} − \frac{1}{2}α \end{matrix} E_{m}(α, u) ×

× ∫_{iℝ^{p−m}} Y_{m}(α, u; s)ℜ_{m}(s)Φ_{α−(p+q)/2+1+2u_{1},...,α−(p+q)/2+m+2u_{m},s_{m+1},...,s_{m}}(t_{1}, …, t_{p})×

× ds_{m+1} … ds_{p} (2.6)

where C is the same as above,

E_{m}(α, u) = (2π)^{m} \frac{p!}{m!}2^{α} ∏_{j=1}^{p} \frac{1}{Γ(α − j + 1)} ×  (2.7)

× ∏_{τ=1}^{m} \frac{(−α + \frac{1}{2}(p + q) − 2u_{τ} − τ)Γ(α − p + τ + 2u_{τ})Γ(−α + q − τ − 2u_{τ})}{(u_{τ} − u_{τ−1})!Γ(−α + \frac{1}{2}(p + q) − τ + 1 − u_{τ} − u_{τ−1})} × (2.8)

× ∏_{1⩽σ<τ⩽m}\{(−α + \frac{1}{2}(p + q) − \frac{1}{2}(τ + σ) − u_{σ} − u_{τ})(\frac{1}{2}(τ − σ) + u_{τ} − u_{σ}) × (2.9)

× \frac{Γ(\frac{1}{2}(τ − σ + 1) + u_{τ} − u_{σ})Γ(−α + \frac{1}{2}(p + q) − \frac{1}{2}(τ + σ) − u_{τ} − u_{σ} + \frac{1}{2})}{Γ(\frac{1}{2}(τ − σ) + u_{τ} − u_{σ−1})Γ(−α + \frac{1}{2}(p + q) − \frac{1}{2}(σ + τ) − u_{σ} − u_{τ})}\} (2.10)

Y_{m}(α, u; s) =

= ∏_{k=m+1}^{p}\{Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + m + 1 + s_{k}))Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + m + 1 − s_{k}))\} × (2.11)

× ∏_{τ⩽m; k>m}\{(\frac{1}{2}(−α + \frac{1}{2}(p + q) − τ − 2w_{τ} + s_{k}))(\frac{1}{2}(−α + \frac{1}{2}(p + q) − τ − 2w_{τ} − s_{k})) × (2.12)

× \frac{Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − (τ − 1) − 2u_{τ} + s_{k})Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − (τ − 1) − 2u_{τ} − s_{k})}{Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − τ + 2 + 2w_{τ−1} + s_{k})Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − τ + 2 + 2w_{τ−1} − s_{k})}\} (2.13)

and

ℜ_{m}(s) = ∏_{k=m+1}^{p} \frac{Γ((q − p)/2 + s_{k})Γ((q − p)/2 − s_{k})}{Γ(s_{k})Γ(−s_{k})} ×  (2.14)

× ∏_{m+1⩽k<l⩽p} \frac{Γ(\frac{1}{2}(1 + s_{l} + s_{k}))Γ(\frac{1}{2}(1 + s_{l} − s_{k}))Γ(\frac{1}{2}(1 − s_{l} + s_{k}))Γ(\frac{1}{2}(1 − s_{l} − s_{k}))}{Γ(\frac{1}{2}(s_{l} + s_{k}))Γ(\frac{1}{2}(s_{l} − s_{k}))Γ(\frac{1}{2}(−s_{l} + s_{k}))Γ(\frac{1}{2}(−s_{l} − s_{k}))}

(2.15)

Remarks. a) The factor ℜ_{m}(s) is an elementary function.

b) More convenient notations are used in Section 4 (see 4.13).

c) The formula, which is not so explicit, but short is given in Section 6.

Remark. The summand corresponding m = 0 coincides with integral (2.1)– (2.5). For summands corresponding m = p, the integration is given by one point set and hence these summands are spherical functions Φ_{...} with some coefficients.

2.3. The case α = p−1, p−2, …, 1. In this case some summands disappear. Proposition 2.3. Let α = p − h where h ⩽ p. Then the factor E_{m}(α, u) is

nonzero if and only if

m ⩾ h; u_{1} = u_{2} = ⋯ = u_{h} = 0

Proof. Vanishing of E_{m}(α, u) is completely defined by a behavior of the factor

\frac{∏_{τ=1}^{m} Γ(α − p + τ + 2u_{τ})}{∏_{j=1}^{p} Γ(α − j + 1)}  (2.16)

The denominator has a pole of order h at α = p − h. If the fraction is nonvanishing, then the numerator has a pole of order h at the same point. boxtimes

2.4. The case α = −1, −2, −3, . . .. Assume E_{m}(α, u) ≠ 0. The denominator of (2.16) has a pole of order p in α. Hence, the numerator also has a pole of order p. Hence,

m = p

This means that all integrals in Plancherel formula (2.6) vanish and we have only finite sum of spherical functions with some coefficients. The coefficient E_{m}(α, u) is nonzero iff

m + 2u_{m} ⩽ −α

2.5. The Plancherel formula for the kernel representations T_{α} of O(p, q).

Theorem 2.4. Let α satisfies Berezin conditions (1.9). Then

a)if E_{m}(α, u) ≠ 0 (see Subsection 2.3), then all spherical functions

Φ_{α−(p+q)/2+1+2u_{1},...,α−(p+q)/2+m+u_{m},s_{m+1},...,s_{m}}

are positive definite.

b)formula (2.6)-(2.15) is really the Plancherel formula

2.6. The Plancherel formula for kernel-representations of O(p + q). For a negative integer α = −n (see Subsection 2.4 above) formula (2.6) gives the expansion of det(1−z^{∗}z)^{n} in O(p)×O(q)-spherical functions of O(p, q) and this is equivalent to the Plancherel formula for the kernel representations of O(p+q).

2.7. The case of indefinite Shapovalov form. For noninteger α < p − 1 we obtain the problem discussed in Section 1.E.

3. B-function of the space O(p, q)/O(p) × O(q)

In this section we construct a matrix imitation of the B-integral

B(x, y) = ∫_{0}^{∞} \frac{t^{x−1}}{(1 + t)^{x+y}} dt

for the symmetric spaces O(p, q)/O(p)×O(q). For symmetric spaces GL(n, 𝕂)/U(n, 𝕂) the B-integrals were defined by Gindikin [11] (see also exposition in [9]), for other symmetric spaces B-integrals were derived in [35].

3.1. B-integral. Let

λ_{1}, …, λ_{p}, σ_{1}, …, σ_{p} ∈ ℂ

We also assume

λ_{p+1} = σ_{p+1} = 0

Let SW_{p,q} be the section of wedge defined in Subsection 1.20.

Theorem 3.1. Let λ_{k}, σ_{k} satisfy the inequalities

\frac{1}{2}(q + k)/2 + 1 < λ_{k} < σ_{k} − \frac{1}{2}(p − k)  (3.1)

Then

∫_{SW_{p,q}(ℝ)} ∏_{j=1}^{p} \frac{det[M − LL^{t}]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{σ_{j}−σ_{j+1}}} · det(M − LL^{∗})^{−(p+q)/2}dM dN dL =(3.2)

= ∫ ; \begin{matrix}M − LL^{t} > 0 \\ N = −N^{t} \end{matrix} ∏_{j=1}^{n} \frac{det [ \begin{matrix}1 & L^{t} \\ L & M \end{matrix} ]_{q−p+j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{σ_{j}−σ_{j+1}}} det ( \begin{matrix}1 & L^{t} \\ L & M \end{matrix} )^{−(p+q)/2} dL dM dN =

(3.3)

= = ∏_{k=1}^{p} π^{k−(q−p)/2−1} \frac{Γ(λ_{k} − (q + k)/2 + 1)Γ(σ_{k} − λ_{k} − (p − k)/2)}{Γ(σ_{k} − p + k)}  (3.4)

The proof of the Theorem is given in Subsections 3.2-3.6.

Remark. For p = q we have L = 0 and integral (3.2)–(3.3) has more simple form, see (0.5). In this case the calculation given below also is simpler. The main simplification is the expression for matrix (3.13): the first block row and the first block column are lacked.

Remark. We have M = M^{t} > 0, N = −N^{t}. Hence,

det(1 + M + N) > 0

Indeed, for any v ∈ ℂ^{p} we have Re v(M + N)v^{∗} = vMv^{∗} > 0. Hence, the eigenvalues λ_{j} of M +N satisfy the condition Re λ_{j} > 0. Hence, the eigenvalues of 1 + M + N are nonzero.

Remark. Hua Loo Keng in [21] evaluated the integrals^{21}

∫_{B_{p,q}(ℝ)} det(1 − zz^{∗})^{τ} dz  (3.5)

Cayley transform reduces the Hua integral to the following partial case of our integral

const · ∫_{SW_{q}(ℝ)} \frac{det (\begin{matrix}1 & L^{t} \\ L & M \end{matrix})^{τ}}{det(1 + M + N)^{2τ}} dL dM dN

Our calculation in this case is not homotopic to Hua calculations.

3.2. Replacement of notations. Firstly, we call to mind the standard formula (see [10]) for determinant of block (m + n) × (m + n)-matrix

det (\begin{matrix}A & B \\ C & D \end{matrix}) = det A · det(D − BA^{−1}C)  (3.6)

Let us represent M, N as block ((p − 1) + 1) × ((p − 1) + 1) matrices, and L as a block ((p − 1) + 1) × (q − p) matrix:

M = ( \begin{matrix}P & q^{t} \\ q & r \end{matrix} );  N = ( \begin{matrix}A & −b^{t} \\ b & 0 \end{matrix} );  L = ( \begin{matrix}H \\ l \end{matrix} )

Then for j ⩽ p − 1

[ \begin{matrix}1 & L^{t} \\ L & M \end{matrix} ]_{q−p+j} coincides with [ \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ]_{q−p+j}

[1 + M + N]_{j}  coincides with  [1 + P + A]_{j}

and by (3.6)

det ( \begin{matrix}1 & L^{t} \\ L & M \end{matrix} ) = det ( \begin{matrix}1 & H^{t} & l^{t} \\ H & P & q^{t} \\ l & q & r \end{matrix} ) =

= det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ) · [r − ( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} )]

det(1 + M + N) = det(1+P +A) · (1 + r − (q + b)(1+P +A)^{−1}(q^{t} − b^{t}))

21Hua integrals also can be reduced to the Selberg B-integrals by integration over K \ G/K.

By the Sylvester criterion the condition (\begin{matrix}1 & L^{t} \\ L & M \end{matrix}) > 0 (see (1.35)) in new

notations has the form

( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ) > 0;  r − ( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} ) > 0  (3.7)

By the remark given in Subsection 3.1

det(1 + P + A) > 0

3.3. Substitution. Let us replace the variable r to the variable

u = r − ( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} )

(all other variables are the same). By (3.7) we have u > 0. The Jacobian of the substitution is 1. Our integral converts to the form

∫ dP dA dH(Ξ(A, P, H) ×  (3.8)

×  ∫_{u>0, q,b∈ℝ^{p−1},l∈ℝ^{q−p}} u^{λ_{p}−(p+q)/2}\{1+u+( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} ) + (3.9)

+ ( q b ) ( \begin{matrix}−(1+P +A)^{−1} & −(1+P +A)^{−1} \\ (1+P +A)^{−1} & (1+P +A)^{−1} \end{matrix} ) ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )\}^{−σ_{p}}du dl dq db) (3.10)

where

Ξ(A, P, H) = ∏_{j=1}^{p−2} \frac{det [ \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ]_{q−p+j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{λ_{n−1}−(p+q)/2}}{det(1 + P + A)^{σ_{n−1}}}

(3.11)

is an expression independent on u, b, l, q.

Firstly, we want to evaluate interior integral (3.9)–(3.10)

3.4. Transformation of the integrand. Denote by S the expression

S = 1 + P + A

Let us represent the expression in the curly brackets in (3.10) in the form

\{1 + u + ( l q b ) X ( \begin{matrix}l^{t} \\ q^{t} \\ b^{t} \end{matrix} )\}  (3.12)

where

X = ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} + ( \begin{matrix}0 & 0 \\ 0 & −S^{−1} \end{matrix} ) & ( \begin{matrix}0 \\ −S^{−1} \end{matrix} ) \\ ( 0 S^{−1} ) & S^{−1} \end{matrix} )  (3.13)

(we wright a block matrix whose elements are block matrices itself). The last summand in the curly brackets is a quadratic form in the variables b, q, l. But the matrix X is not symmetric and it is more natural to re-wright expression (3.12) in the form

\{1 + u + ( l q b ) \frac{1}{2}(X + X^{t}) ( \begin{matrix}l^{t} \\ q^{t} \\ b^{t} \end{matrix} )\}  (3.14)

3.5. Separation of variables.

Lemma 3.2.

det (\frac{1}{2}(X + X^{t})) = det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} · det(1 + P + A)^{−2}

Proof. det (\frac{1}{2}(X + X^{t})) =

= det ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} + ( \begin{matrix}0 & 0 \\ 0 & −\frac{1}{2}S^{t−1} − \frac{1}{2}S^{−1} \end{matrix} ) & ( \begin{matrix}0 \\ −\frac{1}{2}S^{−1} + \frac{1}{2}S^{t−1} \end{matrix} ) \\ ( 0 \frac{1}{2}S^{−1} − \frac{1}{2}S^{t−1} ) & \frac{1}{2}S^{−1} + \frac{1}{2}S^{t−1} \end{matrix} )

Adding the third row to the second row and the third column to the second column, we obtain

det ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} & ( \begin{matrix}0 \\ (1 + P − A)^{−1} \end{matrix} ) \\ ( 0 (1 + P + A)^{−1} ) & \frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} \end{matrix} )

Formula (3.6) reduces the determinant to the form

det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} · det(\frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} −

− ( 0 (1 + P + A)^{−1} ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ) ( \begin{matrix}0 \\ (1 + P − A)^{−1} \end{matrix} )) =

= det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} det(1 + P + A)^{−1} det(1 + P − A)^{−1} ×

× det(\frac{1}{2}(1 + P − A) + \frac{1}{2}(1 + P + A) − P )

The last factor is 1. We also observe

(1 + P + A)^{t} = 1 + P − A

and hence their determinants coincides. ⊠ Lemma 3.3. X + X^{t} > 0. Proof. In the identity

det(1 + M + N) = det(1+P +A) · [1 + r − (q + b)(1+P +A)^{−1}(q^{t} − b^{t})]

we have det(1 + M + N) > 0, det(1 + P + A) > 0. Hence, the factor in the square brackets is positive. Hence, expression (3.12) is positive for all u > 0, and all q, b, l. Quantity (3.12) coincides with quantity (3.14). Hence, the matrix X + X^{t} is nonnegative defined. By Lemma 3.2 its determinant is nonzero and we obtain the required statement. ⊠

Consider the linear substitution

( l q b ) \sqrt{\frac{1}{2}(X + X^{t})} = h ∈ ℝ^{q−p} ⊕ ℝ^{p−1} ⊕ ℝ^{p−1}

to interior integral (3.9)–(3.10). Its Jacobian is

det (\begin{matrix}1 & H^{t} \\ H & P \end{matrix})^{1/2} · det(1 + P + A)

and hence the interior integral coverts to the form

det (\begin{matrix}1 & H^{t} \\ H & P \end{matrix})^{1/2} · det(1 + P + A) ×  (3.15)
×  ∫_{u>0, h∈ℝ^{q+p−2}} u^{λ_{p}−(p+q)/2}{1 + u + |h|^{2}}^{−σ_{p}}du dh  (3.16)

The first factor (3.15) adds to the product Ξ(A, P, H) (see (3.11)) and we reduce our B-integral (3.3) to the product of the integrals

∫_{P −HtH>0, A=−A^{t}} Ξ(A, P, H) det (\begin{matrix}1 & H^{t} \\ H & P \end{matrix})^{1/2} · det(1 + P + A)dA dP dH ×

× ∫_{u>0, h∈ℝ^{q+p−2}} u^{λ_{p}−(p+q)/2}{1 + u + |h|^{2}}^{−σ_{p}}du dh

Let us denote B-integral (3.3) by

I_{p,q}(α_{1}, …, α_{p}; σ_{1}, …, σ_{p})  (3.17)

and let us denote factor (3.16) by J_{p,q}(α_{p}; σ_{p}). We obtain the following recurrent identity

I_{p,q}(α_{1}, …, α_{p}; σ_{1}, …, σ_{p}) =

= I_{p−1,q−1}(α_{1}, …, α_{p−2}, α_{p−1} − \frac{1}{2}; σ_{1}, …, σ_{p−2}, σ_{p−1} − 1)J_{p,q}(α_{p}; σ_{p})

3.6. Evaluation of J_{p,q}(α_{p}; σ_{p}). This problem is trivial. Firstly, we consider spherical coordinates in ℝ^{p+q−2} in the variable h. Then J_{p,q}(α_{p}; σ_{p}) converts to the form

\frac{2π^{(p+q)/2−1}}{Γ((p + q)/2 − 1)} ∫_{u>0} ∫_{r>0} u^{λ_{p}−(p+q)/2}r^{p+q−3}\{1 + u + r^{2}\}^{−σ_{p}}dr du

The substitution v = r^{2} reduces our integral to a special case of the Dirichlet B-integral

∫_{u>0,v>0} \frac{u^{a−1}v^{b−1}}{(1 + u + v)^{a+b+c}} du dv = \frac{Γ(a)Γ(b)Γ(c)}{Γ(a + b + c)}

This completes the proof of Theorem 3.1.

3.7. Spherical transform of ℬ_{α}.

Corollary 3.4. Let α > p + q − 1. Then spherical transform of ℬ_{α} is

\frac{2^{α}}{∏_{1⩽j⩽p} Γ(α − j + 1)} ∏_{k=1}^{p} Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 + s_{k}))Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 − s_{k})) (3.18)

Proof. The function ℬ_{α} is given by the formula (1.37). By Subsection 1.24 we must evaluate the integral

∫_{G/K} ℬ_{α}(z)Ψ_{s}(z) dλ(z)

But the integral is a special case of our B-integral.

3.8. Proof of Theorem 2.1. By the Gindikin-Karpelevich inversion formula and Corollary 3.4 we obtain the statement of the theorem for α > (p+q)−1.

For α > (p + q)/2 − 1 the statement of the Theorem follows from trivial Lemma 4.1 proved below.

4. Formal analytic continuation.

We proved the Plancherel formula (2.1)-(2.5) for large values of the parameter α. Its left part ∏ cosh^{−α}(t_{j}) depends analytically on α ∈ ℂ. The integrand in the right part has singularities on the lines

Re α = \frac{1}{2}(p + q) − 1 − 2κ;  where κ = 0, 1, 2, …  (4.1)

Thus, the right part of formula (2.1)–(2.5) may be nonalytic for these values of α.

Our next purpose is to construct the analytic continuation of the right part to arbitrary complex α.

4.1. Analyticity. Let us denote the right part of the formula (2.1)–(2.5) by

𝔉(α) := 𝔉(α; t) = E(α) ∫_{iℝ^{p}} Y (α; s)ℜ(s)Φ_{s}(t) ds  (4.2)

where the meromorphic factor E(α) is given by formula (2.1), the factor Y (α; s) is defined by (2.2) and ℜ(s) is Gindikin–Karpelevich density (2.3)–(2.4). In this Section we fix the variable t and we omit the argument t from the notation 𝔉(α; t).

Consider domains Π_{0}, Π_{1}, … in ℂ defined by

Π_{0} : Re α > \frac{1}{2}(p + q) − 1

Π_{k} : \frac{1}{2}(p + q) − 1 − 2k < Re α < \frac{1}{2}(p + q) − 1 − 2(k − 1) where k > 0

Lemma 4.1. The function 𝔉(α) is an analytical function on Π_{κ} for all κ = 0, 1, 2 . . ..

Proof. a) Convergence of integral (4.2). First, the Gindikin-Karpelevich factor ℜ(s) has a polynomial growth in s, see formulas (1.44)–(1.46).

By the formula (see [19],1.18.6)

|Γ(a + iy)| = (2π)^{1/2}|y|^{a−1/2} exp\{\frac{1}{2}π|y|\}(1 + o(1));  |y| → ∞  (4.3)

the factor Y (α; s) exponentially decreases.

A spherical function Φ_{s}(t) is a spherical function of an unitary representation and hence we have |Φ_{s}(t)| ⩽ 1^{22}.

Hence, the integrand exponentially decreases and the integral absolutely converges.

b)Existence of \frac{∂}{∂α}𝔉(α). It is sufficient to prove uniform convergence of the integral

∫_{iℝ^{p}} \frac{∂}{∂α}Y (α; s)ℜ(s)Φ_{s}(t)ds  (4.4)

in small neighborhood of a fixed point α̃. For this we needs in uniformity by a of o(1) in (4.3). In fact, the asymptotics is really uniform but formally we have no possibility to refer to [19]. Formula (4.3) is derived from the Binet formula (see [19],(1.9.4))

ln Γ(z) = (z − \frac{1}{2}) ln z − z + \frac{1}{2} ln(2π) + ∫_{0}^{∞} [\frac{1}{e^{t} − 1} − \frac{1}{t} + \frac{1}{2}] t^{−1}e^{−tz}dt

22For following inductive steps this arguments must be replaced by inequality (1.39)

This formula easily implies an uniform estimate of the form

|Γ(a + iy)| ⩽ const · exp(−(\frac{1}{2}π − ε)|y|); |a − ã| < δ

The Cauchy integral for derivative

−2πif′(z) = ∫_{L} \frac{f(z)}{(z − u)^{2}} dz

implies the same estimate for derivative of Γ-function.

We observe that integrand (4.4) is dominated by some function having the form

P (s) exp(−b ∑ |s_{j}|)

where P (s) is a polynomial and b > 0.

Thus, the function 𝔉(α) has a derivative in the complex variable α and this complets the proof. ⊠

Lemma 4.2 Let p ≠ q. Then the function 𝔉(α) is continuous on the line α ∈ ℝ.

Proof. Let h = \frac{1}{2}(p + q) − 1 − 2κ be one of our singular points. The singularity of the integrand near this point has the form

const · ∏_{k=1}^{p} \frac{s_{k}^{2}}{(α − h)^{2} − s_{k}^{2}} ∏_{1⩽k<l⩽p}(s_{k}^{2} − s_{l}^{2})^{2}(1 + o(1))

Remind that s_{k} are pure imaginary. Hence, the integrand is bounded in a neighborhood of the point α = h, s = 0. As we have seen in the previous proof, the integrand has an integrable majorant in a domain |s_{k}| > A, |α − h| < ε (α ∈ ℝ). By Lebesgue theorem about dominant convergence, expression (4.2) is continuous at the point α = h. ⊠

Remark. The function 𝔉(α) is continuous at the real points α = \frac{1}{2}(p + q) − 1 − 2κ but it is not smooth at these points.

We denote the restriction of the function 𝔉(α) to the domain Π_{κ} by

𝔉_{κ}(α)

4.2. Analytic continuation of 𝔉_{κ}(α) through a point of a line Re α = \frac{1}{2}(p + q) − 1 − 2κ. The following lemma is main in this Section. Its proof is given in Subsections 4.2-4.5.

Lemma 4.3. Let α_{0} satisfies the condition

Re α_{0} = \frac{1}{2}(p + q) − 1 − 2κ; Im α_{0} ≠ 0

Then

a) the function 𝔉_{κ}(α) admits the analytic continuation to some small neighborhood

𝒪_{δ} :  |α − α_{0}| < δ  where δ < min{1/1000, | Im α|/1000}  (4.5)

of the point α_{0}

b) for any

α ∈ 𝒪 ∩ Π_{κ+1}

we have

𝔉_{κ+1}(α) − 𝔉_{κ}(α) = \frac{πp}{κ!} ∏_{j=1}^{p} \frac{1}{Γ(α − j + 1)} ×  (4.6)

× \frac{(−α + \frac{1}{2}(p + q) − 2κ − 1)Γ(α − p + 1 + 2κ)Γ(−α + q − 1 − 2κ)}{Γ(−α + \frac{1}{2}(p + q) − κ)} × (4.7)
× ∫_{iℝ^{p−1}} ∏_{2⩽k⩽p,±} Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 2 + 2κ ± s_{k})) ×  (4.8)

× ∏_{2⩽k⩽p,±} \frac{(\frac{1}{2}(−α + \frac{1}{2}(p + q) − 1 ± s_{k}))Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − 2κ ± s_{k}))}{Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) + 1 ± s_{k}))} × (4.9)

× ∏_{2⩽k⩽p,±} \frac{Γ(\frac{1}{2}(q − p) ± s_{k})}{Γ(±s_{k})} ×  (4.10)

× ∏_{2⩽k<l⩽p,±} \frac{Γ(\frac{1}{2}(1 + s_{k} ± s_{l}))Γ(\frac{1}{2}(1 − s_{k} ± s_{l})}{Γ(\frac{1}{2}(s_{k} ± s_{l})Γ(\frac{1}{2}(−s_{k} ± s_{l})} ×  (4.11)
× Φ_{α−(p+q)/2+1+2κ,s_{2},...,s_{p}}(t)ds_{2} … ds_{p}  (4.12)

In the last formula we use the following notation

∏_{±} Γ(a ± s) := Γ(a + s)Γ(a − s)  (4.13)

4.3. Existence of the analytic continuations. Let us represent expression (4.2) (or (2.1)–(2.5)) for 𝔉_{κ}(α) in the form

E(α) ∫ ∏_{k,±} Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)ds

Let ε_{1} > ε_{2} ⋯ > ⋯ > ε_{p} ⩾ 0 be very small (for instance ε_{1} < δ/10 where δ was defined in (4.5)). Consider the function

[figure: (p + q)/2 − 1; Π_{0}; Π_{1}; Π_{2}; 𝒪_{δ}; α_{0}]

Picture 1. The complex plane α. The lines α = (p + q)/2 − 1 − 2κ and the domains Π_{k}

𝔉_{κ}(α; ε) = E(α) ∫_{iℝ^{p}} ∏_{k,±} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)ds (4.14)

in the domain

Π_{κ}^{ε} : −2κ < Re(α − \frac{1}{2}(p + q) + 1) < −2(κ − 1) − ε_{1}

Lemma 4.4. a) The function 𝔉_{κ}(α; ε) admits the holomorphic continuation to the domain 𝒪_{δ} (see (4.5)).

b)The functions |𝔉_{κ}(α; ε)| in 𝒪_{δ} are bounded by a constant independent on ε.

Proof. a) The factor ℜ(s) is holomorphic in the domain | Re(s_{j})| < 1/4 and its poles are very far from the contour ℒ which is described below.

Consider

α ∈ 𝒪_{δ} ∩ Π_{κ}^{ε}  (4.15)

Then integrand (4.14) has poles on hyperplanes

s_{k} = ±(\frac{1}{2}(p + q) − 1 − α − 2u − ε_{k}); where u = 0, 1, …

If u = κ, then the poles are lying near points ± Im α_{0}, on Picture 2 the poles are marked as black circles. The arrows show the direction of their motion if Re α decreases. The white circles show rough position of the poles than α ∈ 𝒪_{δ}∩Π_{κ+1}^{ε}

Consider the contour L_{k} on the complex plane s_{k} ∈ ℂ given by Picture 2. Let ℒ ⊂ ℂ^{p} be the product of the contours L_{k}. Obviously, for α ∈ 𝒪_{δ}, we can replace the integration over iℝ^{p} in the formula (4.14) by the integration over ℒ. But the integral

∫_{ℒ} = ∫_{ℒ} ∏_{k,±} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)ds

[figure: S_{1}; R_{1}; Q; S_{2}; R_{2}]

Picture 2. The contour L_{k} = R_{1} ∪ S_{1} ∪ Q ∪ S_{2} ∪ R_{2} on the complex plane s_{k}. The centers of the semicircles S_{1}, S_{2} are ± Im α_{0}, the radius of the semicircles is 10δ. obviously is holomorphic with respect to α in the domain 𝒪_{δ} (indeed, the surface ℒ doesn’t intersect the singularities and the integrand exponentially decreases as |s| → ∞).

b) Consider the parameter θ_{k} := Im s_{k} on the contour L_{k}.

Lemma 4.5. There exist constants A = A(t), N such that

| ∏_{k,±} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)| ⩽ A ∏_{j=1}^{p} ˙(1 + |θ_{j}|)^{N} exp(−π|θ_{j}|)

for all (s_{1}, …, s_{p}) ∈ ℒ.

Proof of Lemma 4.5. Let us estimate all factors in the left part of the inequality.

a) The Gindikin–Karpelevich factor ℜ(s). By the formulas (1.45)–(1.46) for any imaginary s_{k} we have

|\frac{Γ(\frac{1}{2}(q − p) + s_{k})Γ(\frac{1}{2}(q − p) − s_{k})}{Γ(s_{k})Γ(−s_{k})}| ⩽ const · (1 + |θ_{k}|)^{2(q−p)}

The same expression is bounded on the semi-circles S_{1}, S_{2}.

We also must estimate the factor (1.44). Firstly,

| ∏_{k=1}^{p}(s_{k}^{2} − s_{l}^{2})| ⩽ const ∏_{k}(1 + |θ_{k}|)^{2(p−1)}

Secondly, let us estimate the factors

tan(π(s_{k} ± s_{l}))

of (1.44). If s_{k}, s_{l} are imaginary, then | tan(π(s_{k} ± s_{l}))| < 1. If s_{k}, s_{l} ∈ S_{1}, S_{2}, then this expression is bounded (since S_{1}, S_{2} are compact sets). Let s_{k} be imaginary and s_{l} ∈ S_{1}, S_{2}. Then we obtain a value having the form | tan(x+iy)| where x, y ∈ ℝ, |x| < 10πδ. Then

| tan(x+iy)| = |\frac{tan x + tan iy}{1 + tan x tan iy} | ⩽ | tan x+tan iy| ⩽ | tan x|+1 ⩽ tan(10πδ)+1

b) The Γ-factor Y (α; s). By formula (4.3) for imaginary s_{k} we have

|Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 + s_{k}))Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 − s_{k})| ⩽

⩽ const · (1 + |θ_{k}|)^{Re α+ε_{k}− \frac{1}{2} (p+q)+1} exp(−π|θ_{k}|)

For s_{k} ∈ S_{1}, S_{2} the same expression is bounded (but very large).

c)Spherical functions Φ_{s}(t). By estimation (1.39) we have

|Φ_{s}(t)| ⩽ Φ_{Re s}(t) ⩽ max Φ_{r_{1},...,r_{p}}(t)

where maximum is given over all real vectors (r_{1}, …, r_{p}) satisfying the condition |r_{j}| ⩽ 10δ. Hence, for a fixed t the spherical function in integrand is dominated by a constant.

This completes the proof of the Lemma 4.5. ⊠ Now we can complete the proof of Lemma 4.4.b). By Lemma 4.5 we have

|𝔉_{κ}(α; ε)| ⩽ A(t) ∫_{ℝ^{p}} ∏_{j=1}^{p}[(1 + |θ_{j}|)^{N} exp(−π|θ_{j}|)] ∏_{j=1}^{p} χ(θ_{j})dθ_{1} … dθ_{p}

there the function χ(θ) is given by the formula

χ(θ) := \frac{ds}{dθ} = \{ \begin{matrix}1 & if |Im α_{0} − |θ|| ⩾ 10δ \\ (100δ^{2} − (Im α_{0} − θ)^{2})^{−1/2} & if |Im α_{0} − |θ|| ⩽ 10δ \end{matrix}

Hence

|𝔉_{κ}(α; ε)| ⩽ A(t)^{p} (∫_{−∞}^{∞}(1 + |θ|)^{N} exp(−π|θ|)χ(θ)dθ)^{p}

This completes the proof of uniform boundedness of the functions 𝔉_{κ}(α; ε) for a fixed t (Lemma 4.4b). ⊠

Now we are ready to prove existence of the analytic continuation of the function 𝔉_{κ}.

Proof of Lemma 4.3.b. Let us denote by ε/n the vector (ε_{1}/n, …, ε_{p}/n). Consider the sequence of functions

g_{n}(α) = 𝔉_{κ}(α; ε/n)

in the circle 𝒪_{δ}. Since the functions g_{n}(α) are uniformly bounded, by Montel theorem there exists a subsequence g_{n_{j}} which is uniformly convergent on each smaller circle. Let g(α) be its limit. By Weierstrass theorem g(α) is holomorphic in 𝒪_{δ}. It remains to notice that

lim_{n→∞} 𝔉_{κ}(α; ε/n) = 𝔉_{κ}(α)

for α ∈ Π_{κ} ∩ 𝒪_{δ}. Hence, g(α) is the analytic continuation of 𝔉_{κ}(α) to the circle 𝒪_{δ}.

4.4. Forcing of poles. First, we want to obtain an explicit formula for the analytic continuation of 𝔉_{κ}(α; ε) to the domain Π_{κ+1}^{ε}.

Let the contours L_{k} be the same as above. Let iℝ_{k} be the imaginary axis on the complex plane s_{k}. Consider the surface

ℒ_{k} = iℝ_{1} × ⋯ × iℝ_{k−1} × L_{k} × ⋯ × L_{p} ⊂ ℂ^{p}

We have ℒ_{1} = ℒ, ℒ_{p} = iℝ^{p}.

Consider α ∈ 𝒪_{δ} ∩ Π_{κ+1}. Then

𝔉_{κ+1}(α; ε) = E(α) ∫_{iℝ^{p}} ∏_{k,±} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)ds(4.16)

𝔉_{κ}(α; ε) = E(α) ∫_{ℒ} ∏_{k,±} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)ds (4.17)

Hence,

𝔉_{κ+1}(α; ε) − 𝔉_{κ}(α; ε) = ∫_{iℝ^{p}} − ∫_{ℒ} = ∑_{σ}[ ∫_{ℒ_{σ+1}} − ∫_{ℒ_{σ}} ]  (4.18)

Looking to Picture 2 we observe

∫_{ℒ_{σ+1}} − ∫_{ℒ_{σ}} =  (4.19)
= 2πi ∫ ; \begin{matrix}s_{1} ∈ iℝ_{1}, …, s_{σ−1} ∈ iℝ_{σ−1} \\ s_{σ+1} ∈ L_{σ+1}, …, σ_{p} ∈ L_{p} \end{matrix} [  Res_{s_{σ}=α+ε_{σ}− \frac{1}{2} (p+q)+1+2κ} −  (4.20)

− Res_{s_{σ}=−α−ε_{σ}+ \frac{1}{2} (p+q)−1−2κ}]ds_{1} … ds_{σ−1}ds_{σ+1} … ds_{p}  (4.21)

The integrand in (4.16)–(4.17) is an even function in s_{σ} and hence two residues in (4.20)–(4.21) differ only by sign. The order of poles

s_{σ} = ±(α + ε_{σ} − \frac{1}{2}(p + q) + 1 + 2κ)

of the integrand is 1 and hence the residues can be evaluated by a simple substitution

Res_{s_{σ}=α+ε_{σ}− \frac{1}{2} (p+q)+1+2κ} = H_{σ}(α, ε, s) :=

= E(α)[\frac{Γ(\frac{1}{2}(α + ε_{σ} − \frac{1}{2}(p + q) + 1 − s_{σ}))}{s_{σ} − α − ε_{σ} + \frac{1}{2}(p + q) − 1 − 2κ}Γ(\frac{1}{2}(α + ε_{σ} − \frac{1}{2}(p + q) + 1 + s_{σ})) ×

× ∏_{±;k≠σ} Γ(\frac{1}{2}(α + ε_{k} − \frac{1}{2}(p + q) + 1 ± s_{k})) · ℜ(s)Φ_{s}(t)]|_{s_{σ}=α+ε_{σ}− \frac{1}{2} (p+q)+1+2κ} (4.22)

In this way, we reduce sum (4.18) to

2 ∑_{σ=1}^{p} ∫_{iℝ_{1}×···×iℝ_{σ−1}×L_{σ+1}×···×L_{p}} H_{σ}(α, ε, s)ds_{1} … ds_{σ−1}ds_{σ+1} … ds_{p}

We obtain an expression for 𝔉_{κ+1}(α; ε) − 𝔉_{κ}(α; ε). Unfortunately the domains of integration is yet complicated. By this reason we apply transformation (4.18) to each summand in the last expression. We obtain p(p − 1)/2 additional summands which are integrals over (p − 2)-dimensional surfaces. Each integral can be easily evaluated by residues. After this we apply our arguments again, again, again.

It is possible to wright the final expression (it is slightly long). Fortunately, this is not necessary. The only goal of our interest is

lim_{ε→0}(𝔉_{κ+1}(α; ε) − 𝔉_{κ}(α; ε))  (4.23)

For instance, consider the summand obtained by the substitution

s_{σ} = α + ε_{σ} − \frac{1}{2}(p + q) + 1 + 2κ

s_{υ} = α + ε_{υ} − \frac{1}{2}(p + q) + 1 + 2κ

Then the integrand contains the factors

\frac{1}{Γ(±\frac{1}{2}(s_{σ} − s_{υ}))}| \begin{matrix}s_{σ} = α + ε_{σ} − \frac{1}{2}(p + q) + 1 + 2κ \\ s_{υ} = α + ε_{υ} − \frac{1}{2}(p + q) + 1 + 2κ \end{matrix} = \frac{1}{Γ(±\frac{1}{2}(ε_{σ} − ε_{υ}))}

This factors tend to 0 if ε → 0. Of course, it is necessary to check lacking of poles of the numerator in dangerous for us domain.

Therefore a nonzero contribution to the limit (4.23) can be given only by the terms

2 ∑_{σ=1}^{p} ∫_{iℝ^{p}} H_{σ}(α, ε, s)ds_{1} … ds_{σ−1}ds_{σ+1} … ds_{p}  (4.24)

Our expression is symmetric under permutations of s_{j} and hence all summands of (4.28) give the same contribution to the limit. Thus, deleting ε in (4.24) we obtain the formula

𝔉_{κ+1}(α) − 𝔉_{κ}(α) =

\frac{2πi · 2p · 2^{α}}{∏ Γ(α − j + 1)} ∫_{ℝ^{p−1}}[\frac{Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 − s_{1}))}{s_{1} − α + \frac{1}{2}(p + q) − 1 − 2κ}Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 + s_{1})) ×

× ∏_{2⩽k⩽p,±} Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 ± s_{k})) · ∏_{2⩽k⩽p,±} \frac{Γ(\frac{1}{2}(q − p) ± s_{k})}{Γ(±s_{k})} ×

× ∏_{2⩽k<l⩽p,±} \frac{Γ(\frac{1}{2}(1 + s_{k} ± s_{l}))Γ(\frac{1}{2}(1 − s_{k} ± s_{l}))}{Γ(\frac{1}{2}(s_{k} ± s_{l}))Γ(\frac{1}{2}(−s_{k} ± s_{l}))}Φ_{s}(t)]|_{s_{1}=α− \frac{1}{2} (p+q)+1+2κ}ds_{2} … ds_{p}

where α ∈ Π_{κ+1} ∩ 𝒪_{δ}.

4.5. Calculations. Lemma 4.3.b is an obvious corollary of the last formula. Nevertheless we present some elements of the calculation, since it is essential for understanding Subsection 4.7.

1)  [\frac{Γ(\frac{1}{2}(α + \frac{1}{2}(p + q) − 1 ± s_{k}))}{Γ(\frac{1}{2}(s_{1} ± s_{k}))Γ(\frac{1}{2}(−s_{1} ± s_{k}))}]|_{s_{1}=α− \frac{1}{2} (p+q)+2+2κ} =

= \frac{\frac{1}{2}(−α + \frac{1}{2}(p + q) − 1 ± s_{k}) − κ}{Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) + 1 ± s_{k}))}

We observe that the factor Y (α; s) (see (2.2)) is canceled. This factor was the origin of singularities in our integral (2.1)–(2.5).

2) Γ(\frac{1}{2}(1 + s_{1} ± s_{k}))|_{s_{1}=α− \frac{1}{2} (p+q)+1+2κ} = Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 + 2κ ± s_{k}))

We observe appearance of factor (4.8) which is very similar to the factor Y (α; s). Later it will be an origin of new singularities.

3)  [\frac{Γ(\frac{1}{2}(α − \frac{1}{2}(p + q) + 1 + s_{1}))}{Γ(s_{1})Γ(−s_{1})}]|_{s_{1}=α− \frac{1}{2} (p+q)+1+2κ} =

= \frac{(−α + \frac{1}{2}(p + q) − 2κ − 1)}{Γ(−α + \frac{1}{2}(p + q) − κ)}

This gives formula (4.6)-(4.12), and completes the proof of Lemma 4.3. 4.6. Analytic continuation through the line Re α < \frac{1}{2}(p + q) − 1 − 2κ.

Lemma 4.3 gives the analytic continuation of 𝔉_{κ} to 𝒪_{δ} ∩ Π_{κ+1}. Evidentely the expression for the analytic continuation is analytic in the strip

−2κ − 1 < α − \frac{1}{2}(p + q) + 1 < −2(κ − 1)  (4.25)

and hence we obtain the analytic continuation of 𝔉_{κ} to the whole strip (4.25).

4.7. Proof of Theorem 2.2. The Plancherel formula (2.1)–(2.5) is correct if Re α > \frac{1}{2}(p + q) − 1. We want to construct the analytic continuation of its right part to the domain Re α < \frac{1}{2}(p + q) − 1. Let us move α to the left side.

Firstly we pass across the line α = \frac{1}{2}(p+q)−1. Then we obtain the additional summand 𝔉^{0}(α) := 𝔉_{1}(α) − 𝔉_{0}(α) given by formula (4.6)-(4.12) for κ = 0. This is the summand of the Plancherel formula corresponding m = 1, u_{1}=0.

Let us compare the formula (4.6)–(4.12) for 𝔉^{0}(α) and (2.1)–(2.5). First, we have in (4.6)–(4.12) additional factor (4.9). This factor has singularities but all these singularities are lying in the domain α > \frac{1}{2}(p + q) − 1. The factors (2.2) and (4.8) are very similar (α is changed to α + 1). The factors (2.14)–(2.15) and (4.10)–(4.11) also are very similar. In fact (4.10)–(4.11) is the Gindikin– Karpelevich density for O(p − 1, q − 1).

Hence, we can construct the analytical continuation of 𝔉^{0}(α) in the same way as above. The first singularity of 𝔉^{0}(α) on our way is the line Re α = \frac{1}{2}(p+q)−2. After passing across the line we obtain one more summand 𝔉^{00}(α) corresponding m = 2, u_{1} = u_{2} = 0.

The line Re α = \frac{1}{2}(p + q) − 3 contains singularities of the integral 𝔉(α) and also singularities of 𝔉^{00}(α). Hence, we obtain two additional summands 𝔉^{1}(α) and 𝔉^{000}(α) corresponding m = 1, u_{1} = 1 and m = 3, u_{1} = u_{2} = u_{3} = 0 etc. etc. etc.

Formally, we must give complete description of the inductive step but it literally repeats the arguments of Subsections 4.1–4.6.

5. Positive definiteness of spherical functions.

We obtained the expansion of ℬ_{α}(s) in spherical functions having the form

ℬ_{α}(z) = ∫_{ℂ^{p}} Φ_{s}(z) dμ_{our}(s)  (5.1)

where the positive D_{p}-invariant measure μ_{our}(z) is described in Theorem 2.2. Our purpose is to prove positive definiteness of spherical functions Φ_{s}(z) which are contained in the support of the measure μ_{our}.

By the abstract Plancherel theorem, there exists the unique expansion

ℬ_{α}(z) = ∫_{ℂ^{p}} Φ_{s}(z) dμ_{truth}(s)  (5.2)

where μ_{truth} is a positive D_{p}-invariant measure on ℂ^{n} supported by the space Ĝ_{sph} of positive definite spherical functions.

Substitute z = 0 to (5.2). Then ℬ_{α}(0) = 1, Φ_{s}(0) = 1 and hence

∫_{ℂ^{p}} dμ_{truth} = 1  (5.3)

We denote by supp μ_{truth} and supp μ_{our} the supports of the measures μ_{truth} and μ_{our}.

5.1. Preliminary remarks on the supports of the measures. Consider the bounded polyhedron Q ⊂ ℝ^{p} described in Theorem 1.14. Consider the tube Q̃ ⊂ ℂ^{p} defined by the condition

s ∈ Q̃  iff  Re s ∈ Q

. Then

supp μ_{truth} ⊂ Q̃;  supp μ_{our} ⊂ Q̃  (5.4)

(the first is corollary of Theorem 1.14, the second is corollary of Theorem 2.2). Denote by ℝ ∪ iℝ the union of the real and imaginary axises in ℂ. Then

supp μ_{our} ⊂ (ℝ ∪ iℝ) × ⋯ × (ℝ ∪ iℝ); supp μ_{truth} ⊂ (ℝ ∪ iℝ) × ⋯ × (ℝ ∪ iℝ) (5.5)

(the first is the corollary of Theorem 2.2 and the second is the Corollary of Lemma 1.13).

5.2. Heat kernel. Let Δ_{1}, …, Δ_{p} be Laplace operators (see [17], Section 2.5) on the symmetric space G/K. The operator Δ_{j} is some G-invariant partial differential operator of order 2j on G/K = B_{p,q} with rational coefficients. The operator Δ_{1} is the usual Laplace-Beltrami operator on G/K (see [17], Section 2.2.4).

Spherical functions are joint eigenfunctions of the operators Δ_{j} (see [17], Section 4.2). We have equalities

Δ_{j}Φ_{s}(z) = a_{j}(s)Φ_{s}(z)

where a_{j} are some polynomials invariant with respect to the group D_{p} (see Subsection 1.15) consisting of permutations and changing of signs. If Φ_{s} ≠ Φ_{s′}, then a_{j}(s) ≠ a_{j}(s′) for some j.

In particular

Δ_{1}Φ_{s}(z) = (λ + s^{2})Φ_{s}(z)

where λ is a constant and

s^{2} := s_{1}^{2} + ⋯ + s_{p}^{2}

By conditions (5.4)–(5.5) the eigenvalues (λ+s^{2}) are real and they are uniformly bounded above on the supports of the measures μ_{our}, μ_{truth}.

Consider the Cauchy problem for the heat equation

(\frac{∂}{∂τ} − Δ_{1})F (z, τ) = 0; F (z, 0) = f(z)

on G/K. Let R_{τ} (z, u) be the heat kernel. This means that the solution of the Cauchy problem for the heat equation is given by the formula

F (z, τ) = A_{τ} f(z) := ∫_{G/K} R_{τ} (z, u)f(u) dλ(u)

where λ is the G-invariant measure on G/K.

Lemma 5.1. For each τ > 0 and N there exists a constant C(τ, N) independent on z, u such that

R_{τ} (z, u) ⩽ C(τ, N)(1 + dist(z, u))^{−n}

where dist(·, ·) is the distance in G/K associated with Riemannian metric. Proof. Since the heat kernel is G-invariant, we can assume u = 0. Then

R_{τ} (z, 0) = ∫_{s∈iℝ} exp{τ(λ + s^{2})}Φ_{s}(z)ds

By integral formula for spherical functions (1.38)

R_{τ} (z, 0) = ∫_{s∈iℝ} ∫_{k∈K} exp{τ(λ + s^{2})}Ψ_{s}(z^{[k]})dk ds

Rapid decreasing of the last expression is more or less obvious. ⊠ Similar estimates are valid for partial derivatives of R_{τ} (z, u) of any order. For spherical functions we have the equality

A_{τ} Φ_{s}(z) = exp{τ(λ + s^{2})}Φ_{s}(z)

Lemma 5.2. Let μ_{our}, μ_{truth} be the same as above. Then

a)A_{τ} ℬ_{α}(z) = ∫_{G/K} exp{τ(λ + s^{2})}Φ_{s}(z) dμ_{our}(s)  (5.6)
b)A_{τ} ℬ_{α}(z) = ∫_{G/K} exp{τ(λ + s^{2})}Φ_{s}(z) dμ_{truth}(s)  (5.7)

Proof. We must prove a possibility to change the order of the integration. It is sufficient to show absolute convergence of the integrals

∫_{G/K} ∫_{ℂ^{p}} R_{τ} (z, u)Φ_{s}(u) dμ_{our}(s)dλ(u);  ∫_{G/K} ∫_{ℂ^{p}} R_{τ} (z, u)Φ_{s}(u) dμ_{truth}(s)dλ(u) (5.8)

a) For the first integral (5.8) we use estimate |Φ_{s}(u)| ⩽ Φ_{Re s}(u) (see (1.39)). The measure μ_{our} is supported by a finite family of planes P^{(j)} having the form

P_{j} : Re s_{1} = θ_{1}^{(j)}, …, Re s_{m} = θ_{m}^{(j)}

On each plane P_{j} the integrand is dominated by some expression

∑ R_{τ} (z, u)Φ_{σ_{j}}(u)

where σ_{j} are real vectors. The last expression doesn’t depend on s. It remains to notice that the heat kernel rapidly decrease in u for fixed z, spherical functions Φ_{σ_{j}}(u) are bounded (see Theorem 1.14), and the density of μ_{our}(s) exponentially decreases if |s| → ∞.

b) For the second integral we use the inequality |Φ_{s}(z)| ⩽ 1 and this completes the proof (see (5.3)). ⊠

Lemma 5.3. For each polynomial r(x_{1}, …, x_{p}) we have

r(Δ_{1}, …, Δ_{p})A_{τ} ℬ_{α}(z) = ∫_{G/K} r(a_{1}(s), …, a_{p}(s)) exp{τ(λ + s^{2})}Φ_{s}(z) dμ_{our}(s)

r(Δ_{1}, …, Δ_{p})A_{τ} ℬ_{α}(z) = ∫_{G/K} r(a_{1}(s), …, a_{p}(s)) exp{τ(λ + s^{2})}Φ_{s}(z) dμ_{truth}(s)

Proof. It is sufficient to prove that all partial derivatives by z of integrals (5.6), (5.7) absolutely converges. It is obvious by the following reasons.

1. The integrand rapidly decreases in the variable s.

2. For a given z partial derivatives of the heat kernel by u rapidly decrease.

3. Spherical functions are bounded. ⊠ 5.3. Proof of positive definiteness. Consider σ ∈ supp μ_{our}. Consider

the function

η(s) = exp{τ(λ + s^{2})} ∑_{j=1}^{p}(a_{j}(s) − a_{j}(σ))^{2}

Let M be the maximum of η on (ℝ ∪ iℝ) × ⋯ × (ℝ ∪ iℝ). Then the function

ζ(s) = M − η(s)

satisfies conditions

ζ(σ) = M; ζ(s) < M if s ≠ wσ for all w ∈ D_{p}

Consider the sequence of functions

ξ_{k}(s) = C_{k}(M − η(s))^{k} exp{τ(λ + s^{2})}

where C_{k} is determined by the condition ∫ ξ_{k}(s) dν_{our} = 1. Obviously the sequence ξ_{k}(s) converges to distribution ∑_{w∈D_{p}} δ(s − wσ).

The function ξ_{k}(s) is a polynomial expression

ξ_{k}(s) = P_{k}(a_{1}(s), …, a_{p}(s), exp{τ(λ + s^{2})})

Consider the operator

Ξ_{k} := P_{k}(Δ_{1}, …, Δ_{p}, A_{τ})

By Lemma 5.3 we have

Ξ_{k}ℬ_{α}(z) = ∫_{G/K} ξ_{k}(s)Φ_{s}(z)dν_{our}(s)  (5.9)
Ξ_{k}ℬ_{α}(z) = ∫_{G/K} ξ_{k}(s)Φ_{s}(z)dν_{truth}(s)  (5.10)

We have ξ_{k}(s) ⩾ 0. Hence, by Lemma 1.5d and (5.10) the function Ξ_{k}ℬ_{α}(s) is positive definite. By (5.9) the sequence Ξ_{k}ℬ_{α}(s) converges to Φ_{σ}(z). Thus, Φ_{σ}(z) is a point-wise limit of positive definite functions and hence it is positive definite.

6. Other series

6.1. Hermitizations. Below we present the list of hermitizations (see Subsection 0.1)^{23}.

Each Riemannian noncompact classical symmetric space G/K can be realized as a matrix ball^{24}. A matrix ball is a space of all matrices of a given size over 𝕂 = ℝ, ℂ, ℍ with norm < 1 satisfying (or not satisfying) some symmetry condition. The list of matrix balls is given in the Table 1.

Table 1

G/K  𝕂 size condition G̃/K̃
1.GL(n,ℝ)/O(n,ℝ)  ℝ  n×n  z=z^{t}  Sp(2n,ℝ)/U(n)
2.O(p,q)/O(p)×O(q)  ℝ  p×q  U(p,q)/U(p)×U(q)
3.Sp(2n,ℝ)/U(n)  ℂ  n×n  z=z^{t}  Sp(2n,ℝ)/U(n)×Sp(2n,ℝ)/U(n)
4.GL(n,ℂ)/U(n)  ℂ  n×n  z=z^{∗}  U(2n)/U(n)×U(n)
5.O(n,ℂ)/O(n)  ℝ  n×n  z=−z^{t}  SO^{∗}(2n)/U(n)
6.Sp(2n,ℂ)/Sp(n)  ℍ  n×n  z=−z^{∗}  Sp(4n,ℝ)/U(2n)
7.U(p,q)/U(p)×U(q)  ℍ  p×q  [U(p,q)/U(p)×U(q)]×[U(p,q)/U(p)×U(q)]
8.GL(n,ℍ)/Sp(n)  ℍ  n×n  z=z^{∗}  SO^{∗}(2n)/U(n)
9.Sp(p,q)/Sp(p)×Sp(q)  ℂ  p×q  U(2p,2q)/U(2p)×U(2q)
10.SO^{∗}(2n)/U(n)  ℂ  n×n  z=z^{t}  SO^{∗}(2n)/U(n)×SO^{∗}(2n)/U(n)

The last column contains the hermitization G̃/K̃ of G/K. The embedding G/K → G̃/K̃ in all cases is obvious (we must omit nonholomorphic condition to a matrix z, after this we obtain an hermitian matrix ball).

Remark. Some spaces of small dimension are present in the left column two times (for instance Lobachevskii plane O(2, 1)/O(2) × O(1)). Two associated hermitizations are different.

23Classical part of the list is contained in H.Jaffee paper [22]. Olshanskii in [42], [41] observed that all cases (including exceptional cases) can be easily reduced to Nagano work [29]. The list 1-18 is in one-to-one correspondence with the list of compressive semigroups of symmetric spaces and with the list of causal symmetric spaces, see [42], [41]. The list 1-10 is in one-to-one correspondence with the list of real classical categories, see [32], Addendum A.

24This observation is present in [30]

Table 2 contains hermitizations related to future tubes Table 2

G/K  G̃/K̃
11. SO(2,n)/O(n)×O(2)  [SO(2,n)/O(n)×O(2)]×[SO(2,n)/O(n)×O(2)]
12. SO(1,p)×O(1,q)  SO(2,p+q)

Exceptional hermitizations^{25} ^{26} are given in the Table 3 Table 3

13. EIII/SO(10)×SO(2)  [EIII/SO(10)×SO(2)]×[EIII/SO(10)×SO(2)]
14. EVII/EIII×SO(2)  [EVII/EIII×SO(2)]×[EVII/EIII×SO(2)]
15. F II/Spin(9)  EIII/SO(10)×SO(2)
16. Sp(2,2)/Sp(2)×Sp(2)  EIII/SO(10)×SO(2)
17. GL(4,ℍ)/Sp(2)  EVII/EIII×SO(2)
18. EIV×ℝ/F_{4}  EVII/EIII×SO(2)

6.2. Kernel representations. A kernel representation^{27} ρ of G is a restriction of a highest weight representation ρ̃ of G̃ to G. The constructive description of the scalar-valued kernel representations of O(p, q) given in Subsections 1.11-1.12 is valid for all series 1-10.

Below we discuss only scalar-valued kernel-representations.

6.3. Plancherel formula for large α for the series 1-10 was obtained in [35]^{28}

Consider for simplicity nonhermitian case or hermitian case^{29} α = β. In all

25EIII and EIV are real form of E_{6}, EVII is a real forms of E_{7}, and F II is a real form of F_{4}, see [54]

26Here we observe one of the cases when a phenomenon existing for classical groups doesn’t exist for all exceptional groups

27This definition was proposed in [33]

28The case of hermitian spaces 3,7,10,11 was considered by Berezin [4] for α = β, see notations in Subsection 0.2 (the proof is published in [52]), For hermitian case α ≠ β the Plancherel formula independently on [35] was obtained by Zhang [56]. Future tube case is simple exercise. The hermitian cases 13, 14 are covered by [52]; the case 15 is reduced to one of Gindikin (1964) integrals [11]. Probably only for the exceptional cases 16, 17, 18 the formula is not known. Hence, the possibility to obtain the solution in ”general case” (i.e. 16–18) is yet preserved.

29For hermitian case α ≠ β we have the formula

C(α)C(β) ∏_{k=1}^{p} Γ( \frac{1}{2} (α − h_{1} + s_{k}))Γ( \frac{1}{2} (β − h_{2} + s_{k}))d𝔔(s)

where d𝔔 is the Shimeno measure [49].

these cases the Plancherel formula has the form

∏_{k=1}^{p} cosh^{−α} t_{k} = E(α) ∫_{ıℝ^{p}} ∏_{k⩽p,±} Γ(\frac{1}{2}(α − h + s_{k}))ℜ(s)Φ_{s}(t)ds  (6.1)

where m is the rank of G, ℜ(s) is the Gindikin-Karpelevich density, E(α) is a meromorphic factor and h is a constant. In fact, h is the last point of square integrability. This means that ∫_{G/K} |ℬ_{α}(z)|^{2}dz is finite for α > h and infinite for α = h.

6.4. The analytic continuation of the Plancherel formula. Our arguments from Sections 4–5 don’t depend on series and they are valid for all series 1–10.

In fact considerations of Section 4 prove that the following formal procedure gives correct result. We fix m = 0, 1, …, p := rank G and the collection of numbers u_{1} ⩽ u_{2} ⩽ … u_{m} such that α + 2u_{m} + (m − 1) dim 𝕂 < h

Let

Q_{0}^{m;u}(s) = ∏_{k⩽p,±} Γ(\frac{1}{2}(α − h + s_{k})) · ℜ(s)Φ_{s}(t)

Q_{k}^{m;u}(s) = \frac{Q_{k−1}^{m;u}(s)}{s_{k} − α + h − 2u_{k} − k dim 𝕂}|_{s_{k}=α−h+2u_{k}+k dim 𝕂}  (6.2)

Then the analytic continuation of (6.1) has the form

∏_{k=1}^{p} cosh^{−α} t_{k} = ∑_{m;u_{1},...,u_{p}} E(α)\frac{(2π)^{m}p!}{(p − m)!} ∫_{iℝ^{p−m}} Q_{m}^{m;u}(s)ds_{m+1} … ds_{m} (6.3)

In the cases 𝕂 = ℂ, ℍ this formula can be considered as a final result in a closed form.

In the case 𝕂 = ℝ (G = O(p, q), Sp(2n, ℝ), GL(n, ℝ)) substitutions (6.3) are impossible without cancellations and in this case formula (6.3) gives algorithmic procedure of calculation of Plancherel measure.

Remark. For the groups U(p, q) it is easy to obtain the Plancherel formula (see the explicit final expression in [33]) using Berezin–Karpelevich formula for spherical functions (see [5],[20]) and Molev unitarizability results [28]; partially this idea was also realized in [18].

6.5. Kernel representations of compact groups. Hermitization procedure also is valid for compact Riemannian symmetric spaces. To obtain the list of hermitizations we must replace the groups G, G̃ in Tables 1-3 to their compact forms. For instance the table 1 transforms to the following table 1’

Table 1’

G/K  𝕂 size condition G̃/K̃
1′.U(n)/O(n,ℝ)  ℝ  n×n  z=z^{t}  Sp(n)/U(n)
2′.O(p+q)/O(p)×O(q)  ℝ  p×q  U(p+q)/U(p)×U(q)
3′.Sp(n)/U(n)  ℂ  n×n  z=z^{t}  Sp(n)/U(n)×Sp(n)/U(n)
4′.U(n)×U(n)/U(n)  ℂ  n×n  z=z^{∗}  U(2n)/U(n)×U(n)
5′.O(n)×O(n)/O(n)  ℝ  n×n  z=−z^{t}  O(2n)/U(n)
6′.Sp(n)×Sp(n)/Sp(n)  ℍ  n×n  z=−z^{∗}  Sp(2n)/U(2n)
7′.U(p+q)/U(p)×U(q)  ℍ  p×q  [U(p+q)/U(p)×U(q)]×[U(p+q)/U(p)×U(q)]
8′.U(2n)/Sp(n)  ℍ  n×n  z=z^{∗}  O(2n)/U(n)
9′.Sp(p+q)/Sp(p)×Sp(q)  ℂ  p×q  U(2p+2q)/U(2p)×U(2q)
10′.O(2n)/U(n)  ℂ  n×n  z=z^{t}  O(2n)/U(n)×O(2n)/U(n)

The construction of the kernel representation of O(p+q) given in Subsection 1.28 can be literally translated to all series 1–10. For this purpose we must replace the space Mat_{p,q}(ℝ) by the space Mat of all matrices over 𝕂 (see the second column) having the size given in the third column and satisfying the condition in the forth column. The group G acts on Mat by fractional linear transformations^{30}.

Integrals evaluated in [35] easily give Plancherel formulas for all kernelrepresentations in the cases 1’-10’.

The case of hermitian symmetric spaces was earlier considered by Zhang [56].

6.6. Our terminology. a) The term ”hermitization”. The variant G/K is a real form of G̃/K̃ (see [22]) contradicts to the generally accepted usage of the term ”real form”, since G̃/K̃ is not the complexification G_{ℂ}/K_{ℂ} of G/K.

b) The term ”kernel representation” or ”Berezin kernel representation”. There is no common term for this object. One possible term is ”Berezin transform. It is not suitable in our situation, since there is no ”transform” in this paper.

Another term is “canonical representation”. The term “canonical representation” can be used in many other senses. For instance in [53] and several successive papers of Vershik, Gelfand and Graev this term was used for multiplicative integral and also for some infinite divisible representations. The complex of phenomena related to infinite divisibility has his own interest (see, for instance [32], Chapter 10) but it has small relation with kernel representations.

30More details on these models of Riemannian compact symmetric spaces are contained in [37]

There is also the term ”Berezin quantization”. The ”quantization” in Berezin sence is an operation on the space of functions. In our picture this operation can be defined only for hermitian case.

References

[1] Arazy J., Upmeier H. Invariant inner products in spaces of holomorphic functions on bounded symmetric domains. Documenta Math., v.2 (1997), 213–261.

[2] Arazy J., Zhang Genkaj, Invariant mean value and harmonicity in Cartan and Siegel domains. In Interaction between Functional Analysis, Harmonic analysis and probability. Lecture Notes in Pure Appl. Math., 175 (1995), 19-41.

[3] Berezin F.A. Quantization in complex symmetric domains. Izv. Akad. Nauk SSSR, Ser. math., 39, 2, 1362–1402 (1975); English translation: Math USSR Izv. 9 (1975), No 2, 341–379(1976)

[4] Berezin F.A. On relations between covariant and contravariant symbols of operators for complex classical domains. Dokl. Akad Nauk SSSR, 241, No 1 (1978); English translation: Sov. Math. Dokl. 19 (1978), 786–789

[5] Berezin F.A., Karpelevich F.I. Zonal spherical functions and Laplace operators on some symmetric spaces. Dokl. Akad. Nauk SSSR, 118 (1958), 9–12

[6] Bergmann S. Sur les functions orthogonales de plusiers variables complexes avec les applications a la theorie des fonctions analytiques. Paris, Gauthier-Villars, 1947.

[7] van Dijk H., Hille S.C. Canonical representations related to hyperbolic spaces, J.Funct.Anal., 147, 109–139 (1997).

[8] van Dijk G., Molchanov V.F. The Berezin form for rank 1 para-hermitian symmetric spaces. J.Math.Pure.Appl.,IX ser, 78, N1, 99-119 (1999)

[9] Faraut J., Koranyi A. Analysis in symmetric cones. Oxford Univ.Press, (1994)

[10] Gantmaher F.R. Theory of matrices,4-th.ed., Moscow, Nauka(1988); English translation: Chelsea Publ. Corpor., NY (1959)

[11] Gindikin S.G. Analysis on homogeneous spaces. Uspehi mat. nauk,19, No 4, 3–92(1964); Englis translation in Russian Math.Survey, 19.

[12] Gindikin S.G. Invariant distributions in homogeneous domains Funkt.Anal i Prilozh. 9 (1975), No 1, 56–58; English translation: Funct. Anal. Appl. 9 (1975), No.1, 50–52.

[13] Gindikin S.G., Karpelevich F.I. Plancherel measure for Riemannian symmetric spaces of non-positive curvature. Dokl. Akad Nauk SSSR, 145, 252– 255(1962)

[14] Gindikin S.G., Karpelevich F.I. On an integral connected with symmetric Riemannian space of non-positive curvature. Izv. Akad. Nauk SSSR, Ser Mat., 30, 1147–1156(1966) (Russian); English translation in Transl.Amer.Math.Soc., 85, 249–258(1969)

[15] Gutkin E. Coefficients of Clebsch-Gordon for holomorphic discrete series. Lett.Math.Phys. 3(1979), 185–192

[16] Heckman G.I., Opdam E.M. Root systems and hypergeometric functions. Compos.Math, 64(1987), 329–352.

[17] Helgason S. Groups and geometric analysis. Acad. Press (1984).

[18] Hille S.C., Canonical representations. Thesis, Leiden University, June 1999.

[19] Higher transcendental functions, v.1, Grow-Hill book company, 1953

[20] Hoogenboom B. Spherical functions and invariant differential operators on complex Grassmann manifolds. Ark. for Math., 20(1982), 69–58.

[21] Hua Loo Keng, Harmonic analysis of functions of several complex variables in classical domains . Beijing, 1958(Chinese); Russian translation: Moscow, Inostrannaja literatura (1959); English translation: Amer. Math. Soc., Providence (1963)

[22] Jaffee H.A. Real forms of hermitian symmetric spaces. Bull Amer. Math. Soc. 81, 456–458(1975)

[23] Kirillov A.A. Elements of representation theory. Nauka, Moskow, 1972; English translation: Springer.

[24] Knapp A. Representation theory of real semisimple groups. Princeton Univ. Press, 1986.

[25] Kobayachi T. Singular unitary representations and indefinite Stiefel manifolds. Mem. Amer. Math. Soc., 95 (1992).

[26] Krein M.G. Hermitian positive definite kernels on homogeneous spaces, I, II. Ukrain. Math. J., 1 (1949), 4, 64–98; 2(1950), 1, 10-59 (Russian); English translation in Amer.Math.Soc.Transl.

[27] Molchanov V.F. Tensor products for unitary representations of threedimensional Lorentz group. Math. USSR Izv., 15, 113-143, (1980)

[28] Molev A.I. Unitarizability of some Enright–Varadarajan U(p, q)-modules. in Topics in representation theory, ed. A.A.Kirillov (1991), 199–220, Adv. Sov. Math., v.2., Amer. Math. Soc. Translations

[29] T. Nagano, Transformation groups on compact symmetric spaces. Trans. Amer. Math. Soc. 118 (1965) 428–453.

[30] Neretin Yu.A. On discrete occurrence of complementary series representations in tensor products of unitary representations. Funct. Anal. Appl.., 20, 79-80 (1986)(Russian); English translation in Funct.Anal.Appl.,20,68–70

[31] Neretin Yu.A. Extension of representations of classical groups to representations of categories . Algebra i analiz, t.3, No 1, 176–202 (1991); English translation in St.Petersburg Math.J.,Vol3(1992),No 1.

[32] Neretin Yu.A. Categories of symmetries and infinite-dimensional groups Oxford University Press (1996); Russian edition: Moscow, URSS(1998).

[33] Neretin Yu.A. Boundary values of holomorphic functions and some spectral problems for unitary representations. In collection Positivity in Lie groups: open problems, Hilgert J., J.D.Lawson, K.-H.Need, Vinberg E.B. (eds.), Walter de Gruyter, Berlin (1998).

[34] Neretin Yu.A. Restriction of a function holomorphic in a domain to a curves lying on the boundary and discrete SL_{2}(ℝ)-spectra. Izv. Ross. Akad. Nauk, Ser. mat., 62, 3(1998), 67–86; English translation in Izv. Math., 62 (1998), 493–513.

[35] Neretin Yu.A. Matrix analogs of B-function and Plancherel formula for Berezin kernels. Preprint http://xxx.lanl.gov/math.RT/9905045

[36] Neretin Yu.A. On separation of spectra in analysis of Berezin kernels Preprint http://xxx.lanl.gov/math.RT/9906075

[37] Neretin Yu.A. Conformal geometry of symmetric spaces and Krein-Shmulyan functor. Mat. Sbornik., 1999 (Russian); English translation to appear in Russ. Math. Sbornik

[38] Neretin Yu.A., Olshanskii G.I., Boundary values of holomorphic functions , singular unitary representations of groups O(p, q) and their limits as q → ∞. Zapiski nauchn. semin. POMI RAN 223, 9–91(1995); English translation: J.Math.Sci. 87, 6 (1997), 3983–4035.

[39] Olafsson, G., Orsted, B., Bargmann transform for symmetric spaces, in Lie theory and its applications in physics., eds. Doebner H., Dobrev V.K., Hilgert J., World Scientific (1996), 3–15.

[40] Orsted B., Zhang G. Tensor products of analytic continuations of discrete series. Can. J. Math.,49, N 6, 1224–1241(1997)

[41] Olshanskii G.I. Convex cones in symmetric Lie algebras, Lie semigroups and invariant causal (order) structures on pseudoriemannian symmetric spaces. Dokl. Acad Nauk SSSR, 265, 537 – 541(1982) (Russian); English translation in Sov.Math.Dokl.,26, 97–101(1982)

[42] Olshanskii G.I. Invariant cones in Lie algebras, Lie semigroups and holomorphic discrete series. Funct.Anal.Appl.15,275–285 (1982)

[43] Olshanskii G.I. Irreducible unitary representations of the groups U(p, q) which admits the pass to the limit as q → ∞. Zap. Nauchn. Semin. LOMI., 172, 114–120 (1989) (Russian); English translation in J.Sov.Math., 59, 1102– 1107(1992).

[44] Piateckij-Shapiro I.I. Geometry of classical domains and theory of automorphic functions , Moscow, Fizmatlit (1961); English translation: Automorphic functions and the geometry of classical domains, Gordon and Breach, NY,1969

[45] Pukanszky L. On the Kronecker products of irreducible representations of 2 × 2 real unimodular group. Trans. Amer. Math. Soc., 100 (1961), 116–152

[46] Repka J. Tensor products of holomorphic discrete series. Canad. J. Math., 31(1979), 836–844

[47] Rossi.H, Vergne M. Analytic continuations of holomorphic discrete series of semisimple Lie groups. Acta Math., 136, N1-2, 1-59 (1976)

[48] Siegel C.L. Uber die analytische Theorie der quadratischen Formen, Ann. Math. 36 (1935), 527-606

[49] Shimeno N. The Plancherel formula for spherical functions with a onedimensional K-type on a simply connected simple Lie group of Hermitian type. J.Funct.Anal. 121, 330-388(1994)

[50] Schlichtkrull H. A series of unitary irreducible representations induced from the symmetric subgroup of semisimple Lie group. Invent. Math., 68 (1982), 497–516.

[51] Schoenberg I.J. Metric spaces and positive definite functions. Trans. Amer. Math. Soc., 44 (1938), 522–536

[52] Unterberger A., Upmeier H., The Berezin transform and invariant differential operators. Comm.Math.Phys.,164, 563–597(1994)

[53] Vershik A.M., Gelfand I.M., Graev M.I. Representations of SL_{2}(R) where R is function ring. Uspehi Mat. Nauk 28 (1973), No 5,83–128(Russian); English translation: Russian Math. Surveys 28 (1973), No 5, 87–132

[54] Vinberg E.B., Onishchik A.L., Linear algebraic groups.

[55] Wallach N.R. Analytic continuation of discrete series. Trans. Amer. Math. Soc., 251, 19-37 (1979)

[56] Zhang, Genkaj Berezin transform on line bundles over bounded symmetric domains Preprint, 1999.

Moscow State Institute of Electronics and Mathematics (MIEM) Bolshoi Triohsviatitelskii per., 3/12, Moscow – 109 028, Russia

neretin@main.mccme.rssi.ru