Matrix analogs of the integral

∫_{0}^{∞} \frac{x^{α−1}dx}{(1 + x)^{ρ}} = B(α, ρ − α)

and Plancherel formula for Berezin kernel representations

Neretin Yu.A.^{1} neretin@main.mccme.rssi.ru

The purpose of this paper is to obtain Plancherel formula for Berezin kernel representations. We obtain an explicit expression for density of the Plancherel measure for all 10 series of classical Lie groups for sufficiently large values of parameter of representation (precisely for the case when representation is square integrable). A statement of the problem is contained in Subsection 1.4 (see also recent works on this subject [15], [3], [16], [17], [13], [14], [20]). A solution is contained in §9.

An evaluation of the Plancherel measure is reduced to an evaluation of some integrals over symmetric spaces. The main matter of the paper is this evaluation. In fact we evaluate integrals, which are more general than it is necessary for the Plancherel formula. Nevertheless neither calculations, nor final formulas doesn’t become more complicated

We give an explicit Plancherel formula only for the groups O(p, q). For all other series the Plancherel formula can be easily obtained in the same way by the a combination of integrals (0.1)–(0.11), (0.14)–(0.15), (0.17)–(0.18) with classical Gindikin – Karpelevich formula.

I thanks V.F.Molchanov for discussion of this subject.

The structure of the paper is following. Section 0 contains a list of integrals, which we have evaluated. Evaluations themself are contained in §§4-8. The simplest cases are discussed in §4. The basic case is contained in Subsection 5.1. The main obstacle is surmounted in Subsection 6.1. All other Subsections of §5–8 contain some technical (but not obvious) variations of calculations of 5.1 and 6.1.

In §§1–3 we discuss different models of Riemann noncompact symmetric spaces (matrix balls, matrics cones, matrix wedges, and sections of wedges). This information have no relation to calculations themself. We only explain, what domains of integration in §§4–8 are symmetric spaces.

In §9 we obtaine the Plancherel formula itself (formula 9.2).

§0. Formula for Integrals

0.0. Notations. By 𝕂 we denote ℝ, ℂ, or the algebra of quaternions ℍ. Denote the quaternionic imaginary units by i, j, k. We denote by ^{t} the operation of transposition of a matrix. Denote by A^{∗} the matrix adjoint to a matrix A (i.e. A is the matrix obtained from A by conjugation and (complex or quaternionic) conjugation of matrix elements). For real matrices A^{∗} := A^{t}. The symbol z^{[g]} is introduced by formula (1.1).

By [g]_{p} we denote p × p left upper block of a matrix g.

The symbol det(g) denote determinant. Let g be an operator in ℍ^{n}. Then we consider g as an operator g_{ℝ} : ℝ^{4n} → ℝ^{4n} and assume det(g) = \sqrt[4]{det g_{ℝ}}.

The symbol T ≫ 0 means that a matrix T is hermitian (i.e. T = T^{∗}) positive definite matrix (i.e. qT q^{∗} > 0 for any row q).

A matrix R is named dissipative if R + R^{∗} ≫ 0. For any dissipative matrix R and for any complex τ we define quantity det R^{τ} = det(R^{τ}). Let λ_{1}, …, λ_{n} be eigenvalues of the matrix R. Then real parts of the complex numbers λ_{j} are positive. We assume

det R^{τ} := exp{τ ∑ ln λ_{j}}

In the quaternionic case, we define powers of determinant by the formula

det R^{τ} = det R_{ℝ}^{τ/4}.

We also use nonstandard notation

a^{\{σ∥τ\}} := a^{σ}\overline{a}^{τ} , a, λ, τ ∈ ℂ

and the similar notation for complex dissipative matrices R:

det R^{\{σ∥τ\}} := det R^{σ}\overline{det R}^{τ}

1Supported by grants RFBR 98-01-00303 and RFBR 96-01-96249

The symbol dT, where T is matrix, denote the Lebesgue measure on space of matrices. We also fix following normalization of Lesbegue measure. Let for instance T pass the space of n × n hermitian matrices. Denote by t_{kk} diagonal matrix elements of the matrix T, by t_{km} = p_{km} + iq_{km} we denote non-diagonal elements. Then we assume

dT = ∏_{k=1}^{n} dt_{kk} ∏_{1⩽k<m⩽n} dp_{km}dq_{km}

In the same way, we define a normalization for Lebesgue measure for spaces of matrices of other types (symmetric, antisymmetric etc.)

We start the list of integrals over symmetric spaces evaluated in the paper (§§4–8). Discussion of models of symmetric spaces is contained below in §§1–3.

In all cases λ_{j}, σ_{j}, τ_{j} ∈ ℂ, we also assume

λ_{n+1} = 0;  σ_{n+1} = 0;  τ_{n+1} = 0

We don’t formulate explicitely conditions of convergence since they can be easyly observed from explicit formula.

0.1. Integrals over spaces GL(n, ℝ)/O(n), GL(n, ℂ)/U(n), GL(n, ℍ)/Sp(n). Denote by C_{n}(𝕂) the space of positive defined n × n hermitian matrices over 𝕂. In §4 we establish the following formula:

∫_{T ∈C_{n}(ℝ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [T]_{j})^{σ_{j}−σ_{j+1}}} det T^{−(n+1)/2}dT =  (0.1)

= ∏_{k=1}^{n} π^{(k−1)/2} \frac{Γ(λ_{k} − (k − 1)/2)Γ(σ_{k} − λ_{k} − (n − k)/2)}{Γ(σ_{k} − (n − k)/2)}

∫_{T ∈C_{n}(ℂ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [T]_{j})^{σ_{j}−σ_{j+1}}} det T^{−n}dT =  (0.2)

= ∏_{k=1}^{n} π^{k−1} \frac{Γ(λ_{k} − k + 1)Γ(σ_{k} − λ_{k} − n + k)}{Γ(σ_{k} − n + k)}

∫_{T ∈C_{n}(ℍ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [T]_{j})^{σ_{j}−σ_{j+1}}} det T^{−(2n−1)}dT =  (0.3)

= ∏_{k=1}^{n} π^{2k−2} \frac{Γ(λ_{k} − 2k + 2)Γ(σ_{k} − λ_{k} − 2n + 2k)}{Γ(σ_{k} − 2n + 2k)}

0.2. Integrals over the spaces O(n, n)/O(n) × O(n), U(n, n)/U(n) × U(n), Sp(n, n)/Sp(n) × Sp(n). We denote by W_{n}(𝕂) the space of n×n dissipative matrices R over 𝕂. We represent a dissipative matrix R in the form R = T +S where T = T^{∗} ≫ 0, S^{∗} = −S. In §5 we evaluate the following integrals:

∫_{T +S ∈W_{n}(ℝ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + T + S]_{j}^{σ_{j}−σ_{j+1}}} det T^{−n}dT dS =  (0.4)

= ∏_{k=1}^{n} \frac{π^{k−1}Γ(λ_{k} − (n + k)/2 + 1)Γ(σ_{k} − λ_{k} − (n − k)/2)}{Γ(σ_{k} − n + k)}

∫_{T +S ∈W_{n}(ℂ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + T + S]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} det T^{−2n}dT dS =  (0.5)

= ∏_{k=1}^{n} 2^{2(n−k+1)−σ_{k}−τ_{k}}π^{2k−1} \frac{Γ(λ_{k} − n − k + 1)Γ(σ_{k} + τ_{k} − λ_{k} − n + k)}{Γ(σ_{k} − n + k)Γ(τ_{k} − n + k)}

∫_{T +S ∈W_{n}(ℍ)} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + T + S]_{j}^{σ_{j}−σ_{j+1}}} det T^{−4n}dT dS =  (0.6)

= ∏_{k=1}^{n} 2^{σ_{k}−4(n−k+1)}π^{4k−3} \frac{Γ(λ_{k} − 2(n + k) + 1)Γ(σ_{k} − λ_{k} − 2(n − k))}{Γ(σ_{n}/2 − 2(n − k))Γ(σ_{n}/2 − 2(n − k) − 1)}

0.3. Integrals over the spaces O(p, q)/O(p) × O(q), U(p, q)/U(p) × U(q), Sp(p, q)/Sp(p) × Sp(q). Let p ⩽ q. Let us define a space SW_{p,q}(𝕂). Its points are pairs (K, L), where K is a p × p matrix, L is a p × (q − p) matrix, and

\frac{1}{2}(K + K^{∗}) − LL^{∗} ≫ 0

We represent K in the form K = M + N, where M = M^{∗}, N = − N^{∗}. In §6 we evaluate the following integrals

∫_{SW_{p,q}(ℝ)} ∏_{j=1}^{p} \frac{det[M − LL^{∗}]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{σ_{j}−σ_{j+1}}} · det(M − LL^{∗})^{−(p+q)/2}dM dN dL =  (0.7)

= ∏_{k=1}^{p} π^{k−(q−p)/2−1} \frac{Γ(λ_{k} − (q + k)/2 + 1)Γ(σ_{k} − λ_{k} − (p − k)/2)}{Γ(σ_{k} − p + k)}

∫_{SW_{p,q}(ℂ)} ∏_{j=1}^{p} \frac{det[M − LL^{∗}]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} ×

× det(M − LL^{∗})^{−(p+q)}dM dN dL =  (0.8)

= ∏_{k=1}^{p} 2^{2(p−k)−σ_{k}−τ_{k}}π^{2k−1} \frac{Γ(λ_{k} − (q + k) + 1)Γ(σ_{k} + τ_{k} − λ_{k} − p + k)}{Γ(σ_{k} − p + k)Γ(τ_{k} − p + k)}

∫_{SW_{p,q}(ℍ)} ∏_{j=1}^{p} \frac{det[M − LL^{∗}]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{σ_{j}−σ_{j+1}}} · det(M − LL^{∗})^{−2(p+q)}dM dN dL =  (0.9)

= ∏_{k=1}^{p} 2^{4(p−k+1)−σ_{k}}π^{2(q+k)−3} \frac{Γ(λ_{k} − 2(q + k) + 1)Γ(σ_{k} − λ_{k} − 2(p − k))}{Γ(σ_{k}/2 − 2(p − k))Γ(σ_{k}/2 − 2(p − k) − 1)}

(we assume λ_{p+1} = σ_{p+1} = τ_{p+1} = 0).

0.4. Integrals over the spaces Sp(2n, ℝ)/U(n) and Sp(2n, ℂ)/Sp(n). We realize the symmetric space Sp(2n, ℝ)/U(n) as the space of all symmetric n × n matrices R with a positive definite real part. Let us represent R in the form R = T + iS, where matrices T, S are real. In §7 we evaluate the following integral

∫_{T =T^{t}≫0, S=S^{t}} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + T + iS]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} det T^{−(n+1)}dT dS =  (0.10)

= ∏_{k=1}^{n} \frac{2^{2−σ_{k}−τ_{k}+n−k}π^{k}Γ(λ_{k} − (n + k)/2)Γ(σ_{k} + τ_{k} − λ_{k} − (n − k)/2)}{Γ(σ_{k} − (n − k)/2)Γ(τ_{k} − (n − k)/2)}

We realize the space Sp(2n, ℂ)/Sp(n) as the space of pairs (T, S) of complex n×n matrices satisfying the conditions

T = T^{∗} ≫ 0, S = S^{t}

Let j be the quaternionic imaginary unit. Then

∫_{T =T^{∗}≫0, S=S^{t}} ∏_{s=1}^{n} \frac{det[T]_{s}^{λ_{s}−λ_{s+1}}}{det[1 + T + Sj]_{s}^{σ_{s}−σ_{s+1}}} det T^{−(2n+1)}dT dS =  (0.11)

= ∏_{k=1}^{n} 2π^{2k−1} \frac{Γ(λ_{k} − (n + k))Γ(σ_{k} − λ_{k} − (n − k))}{Γ(σ_{k} − 2(n − k) − 1)}

0.5. Integrals over the spaces O(n, ℂ)/O(n), SO^{∗}(2n)/U(n). Let

J = (\begin{matrix}0 & −1 \\ 1 & 0 \\ & & 0 & −1 \\ & & 1 & 0 \\ & & & & . & . & . \end{matrix})  (0.12)

We realize the space O(2n, ℂ)/O(2n) (respectively SO^{∗}(4n)/U(2n)) as the space of 2n × 2n matrices R over ℝ (respectively over ℂ), satisfying the conditions

J^{−1}R^{t}J = R;  R + R^{∗} ≫ 0  (0.13)

Assume R = T + S, where T = T^{∗}, S = −S^{∗}. Then, in the case O(2n, ℂ)/O(2n), we have

∫ ∏_{j=1}^{n} \frac{det[T]_{2j}^{(λ_{j}−λ_{j+1})/2}}{det[1 + T + S]_{2j}^{(σ_{j}−σ_{j+1})/2}} det T^{−(2n−1)/2}dT dS =  (0.14)

= ∏_{k=1}^{n} π^{2k−2} \frac{Γ(λ_{k} − (n + k) + 2)Γ(σ_{k} − λ_{k} − (n − k))}{Γ(σ_{k} − (n − k))}

In the case SO^{∗}(4n)/U(2n), we obtain

∫ ∏_{j=1}^{n} \frac{det[T]_{2j}^{(λ_{j}−λ_{j+1})/2}}{det[1 + T + S]_{2j}^{\{(σ_{j}−σ_{j+1})/2∥(τ_{j}−τ_{j+1})/2\}}} det T^{−(2n−1)}dT dS =  (0.15)

= ∏_{k=1}^{n} 2^{2+4(n−k)−σ_{k}−τ_{k}}π^{4k−3} \frac{Γ(λ_{k} − 2(n + k) + 3)Γ(σ_{k} + τ_{k} − λ_{k} − 2(n − k))}{Γ(σ_{k} − 2(n − k))Γ(τ_{k} − 2(n − k))}

We realize spaces O(2n + 1, ℂ)/O(2n + 1) and SO^{∗}(4n + 2)/U(2n + 1) as spaces of (1 + 1 + 2n) × (1 + 1 + 2n) matrices R, having block structure

R = (\begin{matrix}1 & 0 & ∗ \\ 0 & 1 & 0 \\ 0 & ∗ & ∗ \end{matrix}) ,  (0.16)

and satisfying condition (0.13). Then in the case O(2n + 1, ℂ)/O(2n + 1) we have

∫ ∏_{j=1}^{n} \frac{det[R + R^{t}]_{2j+2}^{(λ_{j}−λ_{j+1})/2}}{det[1 + R]_{2j+2}^{(σ_{j}−σ_{j+1})/2}} det(R + R^{∗})^{−n}dR =  (0.17)

= C_{1} ∏_{k=1}^{n} π^{2k−1} \frac{Γ(λ_{k} − (n + k) + 1)Γ(σ_{k} − λ_{k} − (n − k))}{Γ(σ_{k} − (n − k))}

and in the case SO^{∗}(4n + 2)/U(2n + 1) we have

∫ ∏_{j=1}^{n} \frac{det[R + R^{∗}]_{2j+2}^{(λ_{j}−λ_{j+1})/2}}{det[1 + R]_{2j+2}^{\{(σ_{j}−σ_{j+1})/2∥(τ_{j}−τ_{j+1})/2\}}} det(R + R^{∗})^{−2n}dR =  (0.18)

= C_{2} ∏_{k=1}^{n} π^{4k−1} \frac{Γ(λ_{k} − 2(n + k) + 2)Γ(σ_{k} + τ_{k} − λ_{k} − 2(n − k))}{Γ(σ_{k} − 2(n − k))Γ(τ_{k} − 2(n − k))}

where C_{1}, C_{2} are some powers of 2.

0.5. Known partial cases. a) Hua Loo Keng integrals ([10]) are integrals over matrix balls (see below §1) having type

∫ det(1 − zz^{∗})^{α}dz

They can be reduced to very partial cases of our integrals by means of the Cayley transform z = (1 + R)^{−1}(1 − R).

b) Integrals (0.1)–(0.3) for λ_{1} = ⋯ = λ_{n} can be reduced by substitution T + 1 = (H − 1)^{−1} to a partial case of Gindikin B-function of cone, see [6], see also [4], Theorem VII.1.7.

c) Integrals (0.1)–(0.3) for σ_{1} = ⋯ = σ_{n} were evaluated by Faraut and Koranyi (see. [4], XIV.4).

d)Berezin in one of the last his papers ([2]) announced the Plancherel formula for kernel representations of the groups U(p, q), Sp(2n, ℝ), SO^{∗}(2n). In fact, it is equivalent to an evalution of integrals (0.5), (0.8), (0.10), (0.15), (0.18) for σ_{1} = τ_{1} = σ_{2} = τ_{2} = … (up to meromorphic multiplier depending on σ)^{2}. Proofs were published by Upmeier and Unterberger [20]. Our method differs from [20] and (as far as I know) it differs from Berezin method.

e) Plancherel formula is known for rank 1 classical groups, i.e. O(n, 1), Sp(n, 1), see. [3], This case is essentially more simple than the case of rank ⩾ 2. (our calculations in this case are trivial).

§1. Models of Symmetric Spaces: Matrix Balls

1.1. Matrix balls. By a norm of a matrix we always mean the norm of a operator from a coordinate Euclidean space 𝕂^{p} to 𝕂^{q}. By a matrix ball we mean

the set of all p × p matrices over 𝕂 having a norm < 1 (where 𝕂 = ℝ, ℂ, ℍ) or set of all n × n matrices over 𝕂, satisfying one of symmetry conditions: symmetric (z = z^{t}), antisymmetric

(z = −z^{t}), hermitian (z = z^{∗}), anti-hermitian (z = −z^{∗}).

Let G be one of classical groups GL(n, ℝ), GL(n, ℂ), GL(n, ℍ), O(p, q), U(p, q), Sp(p, q), Sp(2n, ℝ), Sp(2n, ℂ), O(n, ℂ), SO^{∗}(2n). Let K be a maximal compact subgroup in G. Any non-compact classical Riemann symmetric space G/K has matrix ball type realization (see [11]). Let us enumerate these realizations. Emphasis that in all cases we consider a space of matrices with norm < 1.

1. O(p, q)/O(p) × O(q) is the space of p × q matrices over ℝ.

2. GL(n, ℝ)/O(n) is the space of symmetric matrices n × n matrices over ℝ.

3. O(n, ℂ)/O(n) – is the space of antisymmetric n × n matrices over ℝ.

4. U(p, q)/U(p) × U(q) is the space of p × q matrices over ℂ.

5. Sp(2n, ℝ)/U(n) is the space of symmetric n × n matrices over ℂ

6. SO^{∗}(2n)/U(n) is the space of antisymmetric n × n matrices over ℂ

7. GL(n, ℂ)/U(n) is the space of hermitian n × n matrices over ℂ

8. Sp(p, q)/Sp(p) × Sp(q) is the space p × q matrices over ℍ.

9. GL(n, ℍ)/Sp(n) is the space of hermitian n × n matrices ℍ

10. Sp(2n, ℂ)/Sp(n) is the space of n × n anti-hermitian matrices over ℍ In all cases, the group G acts on the matrix ball by fractional linear transformations having the form

g : z ↦ z^{[g]} := (α + zγ)^{−1}(β + zδ)  (1.1)

Moreover, the group G consist of all fractional linear transformations, mapping bijectively matrix ball to itself. Let us describe the group G explicitly.

In the cases 1,4,8 (i.e. in the case G = U(p, q, 𝕂)) we consider the space 𝕂^{p} ⊕𝕂^{q} equipped with indefinite hermitian form M = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ). We realize the pseudo-unitary group U(p, q, 𝕂) over 𝕂 (i.e. one of the groups O(p, q), U(p, q),

Sp(p, q)) as the group of block matrices g = ( \begin{matrix}α & β \\ γ & δ \end{matrix} ), having the size (p + q) × (p + q) and preserving the hermitian form M.

In all other cases, we consider the space 𝕂^{n} ⊕ 𝕂^{n} equipped with indefinite hermitian form M = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ). As above elements of the group G preserve this form and also they preserve once more form Λ depending on type of the matrix space:

a) in the case of a space of symmetric matrices ( G/K = GL(n, ℝ)/O(n), Sp(2n, ℝ)/U(n)), the form Λ is an antisymmetric bilinear form with the matrix ( \begin{matrix}0 & 1 \\ −1 & 0 \end{matrix} ).

b) in the case of a space of antisymmetric matrices (G/H = O(n, ℂ)/O(n, ℝ), SO^{∗}(2n)/U(n)), the form Λ is a symmetric bilinear form with the matrix ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ).

2after submitting of the paper I received interesting preprint of G.Zhang [21] generalizing [20]

c) in the case of a space of hermitian matrices ( G/K = GL(n, ℝ)/O(n), GL(n, ℂ)/U(n), GL(n, ℍ)/Sp(n)), the form Λ is an anti-hermitian form with the matrix ( \begin{matrix}0 & 1 \\ −1 & 0 \end{matrix} ); remind that a sesquilinear form Λ is named anti-hermitian if Λ(v, w) = −\overline{Λ(w, v)} for all v, w.

d) in the case of a space of anti-hermitian matrices ( G/H = Sp(2n, ℂ)/Sp(n)), the form Λ is a hermitian form with the matrix ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ).

In all cases, a stabilizer of the point z = 0 consists of matrices ( \begin{matrix}α & 0 \\ 0 & δ \end{matrix} ) ∈ G, where matrices α and δ are unitary over 𝕂; note also that in all cases, exept G = U(p, q, 𝕂)/U(p, 𝕂) × U(q, K), we have δ = α^{∗−1}.

1.2. Jacobian. Lemma 1.1. A differential of a map z ↦ z^{[g]} in a point z is

(α + zγ)^{−1}dz(−γz^{[g]} + δ)

Proof. We have to calculate the following expression up to o(ε)

(α + (z + εu)γ)^{−1}(b + (z + εu)δ) =

= (α + zγ)^{−1}(1 + εuγ(α + zγ)^{−1})^{−1}(β + zδ + εuδ) =

= (α + zγ)^{−1}(1 − εuγ(α + zγ)^{−1} + o(ε))(β + zδ + εuδ) =
= (α + zγ)^{−1}(β + zδ) + ε(α + zγ)^{−1}u [−γ(α + zγ)^{−1}(β + zδ) + δ] + o(ε) =
= z^{[g]} + ε(α + zγ)^{−1}u(−γz^{[g]} + δ) + o(ε) ■

Recall a formula for the determinant of block matrix (see [5],§II.5). Below it is used very intensively

det ( \begin{matrix}A & B \\ C & D \end{matrix} ) = det A · det(D − CA^{−1}B)  (1.2)

Lemma 1.2.

det(−γz^{[g]} + δ) = (det g) det(α + zγ)^{−1}  (1.3)

Proof.

det(−γz^{[g]} + δ) = det(−γ(α + zγ)^{−1}(β + zδ) + δ)

By formula (1.2), we reduce this expression to the form

det(α + zγ)^{−1} det ( \begin{matrix}α + zγ & β + zδ \\ γ & δ \end{matrix} ) =

= det(α + zγ)^{−1} det ( \begin{matrix}α & β \\ γ & δ \end{matrix} ) ■

Lemma 1.1, 1.2 impliy the following statement Lemma 1.3. A Jacobian of a transformation g ↦ z^{[g]} in a point z is

| det(α + zγ)|^{−h}

where h = (p + q) dim 𝕂 in the cases 1,4,8, and h = 2n dim G/K in other cases.

1.3. Invariant measure. Lemma 1.4.

1 − z^{[g]}(z^{[g]})^{∗} = (α + zγ)^{−1}(1 − zz^{∗})(α + zγ)^{∗−1}  (1.4)

Proof.

1 − z^{[g]}(z^{[g]})^{∗} = 1 − (α + zγ)^{−1}(β + zδ)(β^{∗} + δ^{∗}z^{∗})(α^{∗} + γ^{∗}z^{∗})^{−1} =
= (α + zγ)^{−1}{(α + zγ)(α^{∗} + γ^{∗}z^{∗}) − (β + zδ)(β^{∗} + δ^{∗}z^{∗})}(α^{∗} + γ^{∗}z^{∗})^{−1}  (1.5)

By

( \begin{matrix}α & β \\ γ & δ \end{matrix} ) ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) ( \begin{matrix}α & β \\ γ & δ \end{matrix} )^{∗} = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} )

we obtain that the factor in (1.5) in the curly brackets has required form. ■

Corollary 1.5.

det(1 − z^{[g]}(z^{[g]})^{∗}) = | det(α + zγ)|^{−2} det(1 − zz^{∗})

Proposition 1.6. A G-invariant measure on a matrix ball is given by the formula

| det(1 − zz^{∗})|^{−h/2}dz

where h is the same as in Lemma 1.3, and dz denote Lebesgue measure on the space of matrices.

Proof. It is consequence of Lemma 1.3 and Corollary 1.5.

1.4. Kernel representations. Consider a function on the matrix ball G/K given by the formula

ℬ_{α}(z) = det(1 − zz^{∗})^{−α}

We can consider the function ℬ as a function on the the group G

ℬ_{α}(g) := ℬ_{α}(0^{[g]})

Kernel representation T_{α} of the group G is the unitary representation containing a cyclic vector v such that

⟨T_{α}(g)v, v⟩ = ℬ_{α}(g)

.

Representation T_{α} exists iff the function ℬ_{α}(g) on the group G is positive definite. Conditions of positive definiteness for ℬ_{α}(g) were obtained by Berezin in [1] (some additions were given by Gindikin [7]). For instance, Berezin condition for the groups O(p, q) is

2α = 0, 1, 2, …, p − 1  or  2α > p − 1

(we assume p ⩽ q). For other series conditions are similar.

Kernel representation also can be defined as restrictions of highest weight representation of some ’overgroup’ G^{∗} ⊃ G to G. They were introduced by Berezin [2] for hermitian symmetric spaces, in [16], [13] there were observed that the problem is general for all Riemannian noncompact symmetric spaces (besides several exceptional spaces). Detail discussion and more ’material’ definitions see [13], [14], see also recent works [15], [16], [20], [3].

§2. Models of Symmetric Spaces: Symmetric Cones and Symmetric Wedges

Matrix ball models are not convenient for our calculations. Our purpose in §2-§3 is to obtain realizations at which parabolic subgroups acts by affine transformations. Our realizations are similar to realizations from [19].

2.1. Cayley transform. The Cayley transform is the transform of the space of square matrices given by the formula

z = \frac{1 − R}{1 + R} = −1 + (1 + R)^{−1}  (2.1)
R = \frac{1 − z}{1 + z} = −1 + (1 + z)^{−1}  (2.2)

By Lemma 1.1, the differential of the Cayley transform is given by the formula

dR = 2(1 + z)^{−1}dz (1 + z)^{−1}

Proposition 2.1. a) zz^{∗} = 1 ⟺ R = −R^{∗}

b) ∥z∥ < 1 ⟺ R + R^{∗} ≫ 0

c) z = z^{t} ⟺ R = R^{t} d) z = z^{∗} ⟺ R = R^{∗} (Statements a),b) are well-known, and c),d) are obvious.)

2.2. Symmetric cones. Proposition 2.1b),d) implies Corollary 2.2. The Cayley transform maps the matrix ball

GL(n, 𝕂)/U(n, 𝕂) =

= GL(n, ℝ)/O(n), GL(n, ℂ)/U(n), GL(n, ℍ)/Sp(n)

to the cone of positive definite hermitian matrices over 𝕂 = ℝ, ℂ, ℍ respectively.

Thus we obtaine another realization for symmetric spaces enumerated in the Corollary. The group G = GL(n, 𝕂) acts on the cone of positive definite matrices by the formula

A : T ↦ AT A^{∗}.

2.2. Wedge of dissipative matrices. Proposition 2.1b) also implies Corollary 2.3. The Cayley transform maps matrix ball

U(n, n, 𝕂)/U(n, 𝕂) × U(n, 𝕂) =

= O(n, n)/O(n) × O(n), U(n, n)/U(n) × U(n), Sp(n, n)/Sp(n) × Sp(n)

to the space of all dissipative matrices over 𝕂 = ℝ, ℂ, ℍ respectively.

In this case, it is convenient to realize the group G = U(n, n, 𝕂) as the group of matrices ( \begin{matrix}a & b \\ s & d \end{matrix} ) preserving the

hermitian form ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ).

The group G acts on the space of dissipative matrices by fractional linear transformations

R ↦ R^{[g]} := (a + Rc)^{−1}(b + Rd)  (2.3)

2.4. Siegel upper half-plane. Recall that the matrix ball Sp(2n, ℝ)/U(n) consists of complex symmetric matrices. By Proposition 2.1 b),c) its image under the Cayley transform is the space of complex symmetric matrices with positive definite real part.

2.5. The space Sp(2n, ℂ)/Sp(n). In this case the matrix ball consists of quaternionic anti-hermitian matrices. Consider the modified Cayley transform

R = −1 + 2(1 + iz)^{−1}

The image of this transform consist of quaternionic matrices R satisfying the conditions

R^{∗} = i^{−1}Ri;  R + R^{∗} ≫ 0.  (2.4)

Represent a matrix R in the form R = S + T j, where R, S are complex matrices. Condition (2.4) is equivalent to

T = T^{∗};  S = S^{t};  T ≫ 0.

2.6. Spaces O(2n, ℂ)/O(2n, ℝ), SO^{∗}(4n)/U(2n). Recall that these matrix balls consist of 2n × 2n antisymmetric matrices over ℝ, ℂ respectively. Define the matrix

J = (\begin{matrix}0 & −1 \\ 1 & 0 \\ & & 0 & −1 \\ & & 1 & 0 \\ & & & & . & . & . \end{matrix})  (2.5)

Consider the modified Cayley transform

R = −1 + (1 + JZ)^{−1}

The image of the matrix ball consists of matrices R satisfying the conditions

R^{t} = JRJ^{−1};  R + R^{∗} ≫ 0  (2.6)

We will continue a discussion of these spaces in §8.

2.7. Invariant measure. Lemma 2.4. Let z and R are linked by the Cayley transform (2.1). Then

1 − zz^{∗} = 2(1 + R)^{−1}(R + R^{∗})(1 + R^{∗})^{−1}

Proof.1 − zz^{∗} = 1 − (1 + R)^{−1}(1 − R)(1 − R^{∗})(1 + R^{∗})^{−1} =

= (1 + R)^{−1} {(1 + R)(1 + R^{∗}) − (1 − R)(1 − R^{∗})} (1 + R^{∗})^{−1}

and we remove brackets in the middle factor. ■

The Lemma implies two corollaries. In the thirst place, it allows to rewrite functions ℬ_{α} (see Subsection 1.4) in new coordinates. In all cases (see Subsections 2.1–2.6) we obtain

ℬ_{α}(R) = (\frac{det(2(R + R^{∗}))}{| det(1 + R)|^{2}})^{α}  (2.7)

In the second place, Lemma 2.4 and Proposition 1.6 give formula for G-invariant measure on a cone or a wedge. In all enumerated cases the invariant measure is given by the formula

det(R + R^{∗})^{\frac{dim(G/K)}{m}} dR  (2.8)

where m is the size of matrices, which we consider.

2.8. Action of a parabolic subgroup. In the case of symmetric cones GL(n, 𝕂)/U(n, 𝕂), we consider the minimal parabolic subgroup 𝒫 ⊂ GL(n, 𝕂) consisting of lower-triangular matrices. Obviously, functions

∏_{j=1}^{n} det[T]_{j}^{λ_{j}−λ_{j+1}}  (2.9)

are eigenfunctions of the subgroup P.

Consider the case when a space G/K is a wedge of dissipative matrices (i.e G/K = U(n, n, 𝕂)/U(n, K)×U(n, K)), Let us define the group 𝒫 consisting of all transformations

R ↦ QRQ^{∗} + S

of the wedge, where Q ∈ GL(n, 𝕂) is a lower-triangular matrix and S = −S^{∗}. It is easy to see, that 𝒫 is a minimal parabolic subgroup in G = U(n, n, 𝕂).

Eigenfunctions of the parabolic subgroup 𝒫 on the wedge have the form (2.9) where T = (R + R^{∗})/2.

For all other wedges the picture is similar.

2.9. Comments to exterior of integrals §0. Now we are ready to explain a sense of some factors in integrands (0.1)–(0.18). The last factor is density of the invariant measure, see (2.8), it is possible to include this factor to numerator of the fraction). The numerator of the fraction ∏_{j=1}^{n} det[T]_{j}^{λ_{j}−λ_{j+1}} is an eigenfunction of the minimal parabolic subgroup.

I don’t understand the sense of the product in denominator. In any case (see (2.7),(2.8)) all expressions of the form ℬ_{α}(R) · ∏_{j=1}^{n} det[T]_{j}^{λ_{j}−λ_{j+1}} are contained in the set of our integrands.

§3. Models of Symmetric Spaces: Sections of Wedges

In this section we discuss symmetric spaces G/K which have no matrix cone and matrix wedge realizations(it is the case when the group G is not split).

3.1. The case G = U(p, q)/U(p) × U(q). Denote by B_{α,β} = B_{α,β}(𝕂) the space of all α × β matrices over 𝕂 with norm < 1. Recal that we identifies B_{α,β}(𝕂) with symmetric space U(α, β, 𝕂)/U(α, 𝕂) × U(β, 𝕂), see Subsection 1.1.

Assume p < q. Let us wright Z ∈ B_{p,q} as block p × ((q − p)) + p) matrix:

Z = ( X Y )

Complete the matrix Z to the matrix Z̃ of the size ((q − p)) + p) × ((q − p)) + p) by the formula

Z̃ = ( \begin{matrix}0 & 0 \\ X & Y \end{matrix} )  (3.1)

The Cayley transform (2.1) maps this matrix Z̃ to a dissipative matrix R having the form

R = ( \begin{matrix}1 & 0 \\ 2L & K \end{matrix} )  (3.2)

where

K = −1 + 2(1 + Y)^{−1}; L = −(1 + Y)^{−1}X

Conversely, it is easy to check, that any dissipative matrix having form (3.2) is a Cayley transform of some matrix having form (3.1).

Thus we realized the space U(p, q, 𝕂)/U(p, 𝕂) × U(q, 𝕂) as the space SW_{p,q} = SW_{p,q}(𝕂) of dissipative matrices having block structure (3.2).

Let us represent the matrix (3.2) in the form

R = ( \begin{matrix}1 & 0 \\ 2L & M + N \end{matrix} )  (3.3)

where M = M^{∗} ≫ 0, N = −N^{∗}. Then

\frac{1}{2}(R + R^{∗}) = ( \begin{matrix}1 & L^{∗} \\ L & M \end{matrix} )  (3.4)

Formula (1.2) implies the identities

det [\begin{matrix}1 & L^{∗} \\ L & M \end{matrix}]_{q−p+j} = det[M − LL^{∗}]_{j}  (3.5)
det (\begin{matrix}1 & L^{∗} \\ L & M \end{matrix}) = det(M − LL^{∗})  (3.6)

Lemma 3.1. Dissipativity of the matrix (3.3) is equivalent to positive definiteness of the matrix M − LL^{∗}.

Proof. It is enough to apply Sylvester criterion, see (3.5). ■

3.2. Minimal parabolic subgroup.

Lemma 3.2. The group of all fractional linear maps of the wedge W_{q} mapping the set SW_{p,q} to itself is isomorphic to the group U(q − p, 𝕂) × U(p, q, 𝕂). Moreover elements of the first factor U(q − p) fix all points of SW_{p,q}.

Proof. The statement is equivalent to the following fact. The group H of all fractional linear transformations of the matrix ball, B_{q,q} mapping the set of matrices (3.1) to itself, is U(q − p, 𝕂) × U(p, q, 𝕂). First the inclusion H ⊃ U(q − p, 𝕂) × U(p, q, 𝕂) is obvious. Secondly the group U(q − p, 𝕂) × U(p, q, 𝕂) is a maximal subgroup in U(q, q). ■

Let us consider the group U(q, q, 𝕂) acting on the wedge W_{q} and maximal parabolic subgroup 𝒬 ⊂ U(,) consisting of transformations

R ↦ URU^{∗} + H  (3.7)

where U ∈ GL(n, 𝕂), a H = −H^{∗}. Consider the group

𝒮 = 𝒬 ∩ (U(q − p) × U(p, q, 𝕂))

which consist of all transformations (3.7) mapping the set SW_{p,q} to itself. It is easy show that these transformations have the form

(\begin{matrix}1 & 0 \\ 2L & K \end{matrix}) ↦ (\begin{matrix}A & 0 \\ C & D \end{matrix}) (\begin{matrix}1 & 0 \\ 2L & K \end{matrix}) (\begin{matrix}A^{∗} & C^{∗} \\ 0 & D^{∗} \end{matrix}) + (\begin{matrix}0 & −AC^{∗} \\ CA^{∗} & Z \end{matrix}) =

= (\begin{matrix}1 & 0 \\ 2(DLA^{∗} + CA^{∗}) & DKD^{∗} + CC^{∗} + 2DLC^{∗} + Z \end{matrix}) (3.8)

where AA^{∗} = 1; Z = −Z^{∗}

By construction, 𝒮 ⊂ U(q − p, 𝕂) × U(p, q, 𝕂). But the factor U(q − p, 𝕂) fix all points of the set SW_{p,q} Hence we can assume 𝒮 ⊂ U(p, q, 𝕂).

Formula (3.8) seems cumbersome. Let us give more pleasant description. The transformation group 𝒮 is generated by the following transformations

1. transformations (L, K) ↦ (LA^{∗}, K) forming the group U(q − p, 𝕂),

2. transformations (L, K) ↦ (DL, DKD^{∗}) forming the group GL(p, 𝕂),

3. transformations (L, K) ↦ (L, K + Z) forming an Abel group.

4. transformations (L, K) ↦ (L + C, K + CC^{∗} + 2LC^{∗}).

The transformations of the type 4 don’t form group. Nevertheless the transformations of types 3 and 4 generate a metabelian group of high 2.

It is readily seen, that 𝒮 is a maximal parabolic subgroup in U(p, q, 𝕂). We extract a minimal parabolic subgroup 𝒫 from 𝒮 by the condition: the matrix D is lower-triangular.

Obviously, eigenfunctions of the minimal parabolic subgroup 𝒮 on the space SW_{p,q} have the form

∏_{j=1}^{n} det [\begin{matrix}1 & L^{∗} \\ L & M \end{matrix}]_{q−p+j}^{λ_{j}−λ_{j+1}} = det[M − LL^{∗}]_{j}^{λ_{j}−λ_{j+1}}

These functions are restrictions to SW_{q} of 𝒬-eigenfunctions on W_{q}.

3.3. Functions ℬ_{α} in our coordinates on SW_{p,q} have the form

ℬ_{α}(R) = (\frac{det(2(R + R^{∗}))}{| det(1 + R)|^{2}})^{α} = (\frac{det(4(M − LL^{∗})}{| det(1 + M + N)|^{2}} )^{α}  (3.9)

(they also are restrictions to SW_{q} of functions ℬ_{α} on W_{q}).

3.4. The spaces O(2n + 1, ℂ)/O(2n + 1), SO^{∗}(4n + 2)/U(2n + 1). In Subsection 1.1 these spaces were realized as spaces of anticymmetric (2n + 1) × (2n + 1) matrices over ℝ, ℂ with norm < 1. Let us embed the space of (2n + 1) × (2n + 1)-matrices to the space of (2n + 2) × (2n + 2)-matrices. For this purpose we wright zero column on the left and zeros row above. Then we apply the construction of Subsection 2.6 and obtain the models described above in Subsection 0.5.

§4. Calculations in Matrix Cones.

In these section we evaluate integrals (0.1)–(0.2) over symmetric cones GL(n, ℝ)/O(n), GL(n, ℂ)/U(n), GL(n, ℍ)/Sp(n). In generally, these 3 series are essentially more simple than the others, see for instance [4]). Our calculations in these case also are essentially simplified.

4.1. Separation of variables in the cone of real symmetric matrices. Thus we have the integral

∫_{T ≫0} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [T]_{j})^{σ_{j}−σ_{j+1}}} det T^{−(n+1)/2}dT  (4.1)

Represent a matrix T as block (1 + (n − 1)) × (1 + (n − 1)) matrix:

T = ( \begin{matrix}P & q^{t} \\ q & r \end{matrix} )

By formula (1.2),

det T = det P · (r − qP^{−1}q^{t})  (4.2)
det(1 + T) = det(1 + P)(1 + r − q(1 + P)^{−1}q^{t})  (4.3)

In new coordinates, the domain of integration T ≫ 0 is replaced to

P ≫ 0; r − qP^{−1}q^{t} > 0

(it is consequence of Sylvester criterion and equality (4.2)). Using these notations we wright the original integral in the form

∫ ∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [P]_{j})^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−(n+1)/2}}{det(1 + P)^{σ_{n−1}}} ×

× (r − qP^{−1}q^{t})^{λ_{n}−(n+1)/2}{1 + r − q(1 + P)^{−1}q^{t}}^{−σ_{n}}dq dr dP

Let us change variable u = r − qP^{−1}q^{t} (we change r to u, other variables are the same ). We obtain

∫_{P ≫0}(∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [P]_{j})^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−(n+1)/2}}{det(1 + P)^{σ_{n−1}}} ×

× ∫_{u>0,q∈ℝ^{n−1}} u^{λ_{n}−(n+1)/2}{1 + u + qP^{−1}q^{t} − q(1 + P)^{−1}q^{t}}^{−σ_{n}}dq du)dP

Next we reduce the expression in the curly brackets to the form

{1 + u + q(P (1 + P))^{−1}q^{t}}

Then we change the variable q = (q_{1}, …, q_{n−1}) ∈ ℝ^{n−1} to the variable

h = (h_{1}, …, h_{n−1}) = q(P (1 + P))^{−1/2} ∈ ℝ^{n−1},

leaving the variables P, u the same. The Jacobian of the substitution is

det P^{1/2} det(1 + P)^{1/2}.  (4.4)

Thus integral can be decomposed to a product

∫_{P ≫0} ∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det(1 + [P]_{j})^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−n/2}}{det(1 + P)^{σ_{n−1}−1/2}} dP ×  (4.5)

× ∫_{u>0,h∈ℝ^{n−1}} u^{λ_{n}−(n+1)/2}{1 + u + hh^{t}}^{−σ_{n}}dh du  (4.6)

The first fuctor (4.5) has a form (4.1) with other parameters (the parameter n is replaced to n − 1, and also σ are changed). Hence we can apply to integral (4.5) the same arguments. It remains to calculate the second factor.

4.2. Several lemmas. Now we will give without proof several standard propositions which will be intensively used below.

Lemma 4.1. The area of an unit sphere in ℝ^{k} is

S = \frac{2π^{k/2}}{Γ(k/2)}  (4.7)

Lemma 4.2. (see. [18], 3.1.2.1)

∫_{u>0} ∫_{v>0} f(u + v)u^{μ−1}v^{ν−1}du dv = B(μ, ν) ∫_{w>0} f(w)w^{μ+ν−1}dw  (4.8)

Lemma 4.3.

∫_{x>0} ∫_{y>0} (1 + x + y)^{−a}x^{b−1}y^{c−1}dx dy = \frac{Γ(b)Γ(c)Γ(a − b − c)}{Γ(a)}  (4.9)

(We substitute z = u, w = u + v to the integral (4.8) and integrate in z; integral (4.9) can be easily reduced to (4.8)).

4.3. Calculation of separated factor. The factor (4.6) is a partial case of the following integral.

Lemma 4.4.

∫_{u>0, h∈ℝ^{k}} u^{λ−k/2−1}(1  +  u  +  |h|^{2})^{−σ}du dh  =  \frac{π^{k/2}Γ(λ − k/2)Γ(σ − λ)}{Γ(σ)} (4.10)

Proof. Passing to spherical coordinates (see (4.7)) in the variable h = (h_{1}, …, h_{n−1}), we obtain

\frac{2π^{k/2}}{Γ(k/2)} ∫_{u>0} ∫_{ρ>0} u^{λ−k/2−1}(1 + u + ρ^{2})^{−σ}ρ^{k−1}dρ du =

= \frac{π^{k/2}}{Γ(k/2)} ∫_{u>0} ∫_{v>0} u^{λ−k/2−1}(1 + u + v)^{−σ}v^{k/2−1}dv du (4.11)

The last expression is a partial case of integral (4.9). ■

4.4. Complex and quaternionic cases are very similar to real case . Formula (4.4) for a Jacobian is replaced to

det P^{dim 𝕂/2} det(1 + P)^{dim 𝕂/2}

Then all is reduced to Lemma 4.4.

§5. Integrals over Wedges of Dissipative Matrices.

In Subsection 2.3 we realized the spaces O(n, n)/O(n) × O(n), U(n, n)/U(n) × U(n), Sp(n, n)/Sp(n) × Sp(n) as spaces of dissipative matrices. Let us represent a matrix R in the form R = T + S, where T = T^{∗}, S = −S^{∗}.

5.1. Evaluation for the spaces O(n, n)/O(n) × O(n). Consider the integral

∫_{T =T^{t}≫0 S=−S^{t}} ∏_{j=1}^{n} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + T + S]_{j}^{σ_{j}−σ_{j+1}}} det T^{−n}dT dS  (5.1)

Let us represent the matrices T, S as block ((n − 1) + 1) × ((n − 1) + 1) matrices:

T = ( \begin{matrix}P & q^{t} \\ q & r \end{matrix} ); S = ( \begin{matrix}A & −b^{t} \\ b & 0 \end{matrix} )

By formula (1.2), we obtain

det T = det P · (r − qP^{−1}q^{t});
det(1 + T + S) = det(1 + P + A) · (1 + r − (q + b)(1 + P + A)^{−1}(q^{t} − b^{t}))  (5.2)

Substituting u = r − qP^{−1}q^{t}, we obtain

∫_{P ≫0,A=A^{t}} dP dA(∏_{j=1}^{n−2} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−n}}{det(1 + P + A)^{σ_{n−1}}} ×

× ∫_{u>0, q,b∈ℝ^{n−1}} u^{λ_{n}−n}{1 + u + qP^{−1}q^{t} − (q + b)(1 + P + A)^{−1}(q^{t} − b^{t})}^{−σ_{n}}du dq db) (5.3)

Let us wright the expression in the curly brackets as

\{1 + u + ( q b ) X ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )\}  (5.4)

where

X = ( \begin{matrix}P^{−1} − (1 + P + A)^{−1} & (1 + P + A)^{−1} \\ −(1 + P + A)^{−1} & (1 + P + A)^{−1} \end{matrix} )  (5.5)

Unfortunately, the matrix X is not symmetric, hence we wright (5.4) in the form

\{1 + u + \frac{1}{2} ( q b ) (X + X^{t}) ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )\}

Lemma 5.1. a) The matrix \frac{1}{2}(X + X^{t}) is positive definite.

b) det(\frac{1}{2}(X + X^{t})) = det P^{−1} det(1 + P + A)^{−2}

Proof. b) \frac{1}{2}(X + X^{t}) =

( \begin{matrix}P^{−1} − \frac{1}{2}(1 + P + A)^{−1} − \frac{1}{2}(1 + P − A)^{−1} & \frac{1}{2}(1 + P + A)^{−1} − \frac{1}{2}(1 + P − A)^{−1} \\ \frac{1}{2}(1 + P − A)^{−1} − \frac{1}{2}(1 + P + A)^{−1} & \frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} \end{matrix} )

Adding the second column to the first row, and the second row to the first row, we obtain

det(\frac{1}{2}(X + X^{t})) = ( \begin{matrix}P^{−1} & (1 + P + A)^{−1} \\ (1 + P − A)^{−1} & \frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} \end{matrix} )

Applying formula (1.2), we obtain that the original determinant equals to

det P^{−1} ×

× det(\frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} − (1 + P − A)^{−1}P (1 + P + A)^{−1}) =

= det P^{−1} det(1 + P + A)^{−1} det(1 + P − A)^{−1} ×

× det(\frac{1}{2}(1 + P − A) + \frac{1}{2}(1 + P + A) − P )

It remains to notice, that the expression in the big brackets equals 1 and (1 + P + A)^{t} = 1 + P − A. This complete our evaluation.

a) The expression det(1 + T + S) doesn’t vanish. By formula (5.2), u and (5.4) also don’t vanish. Thus the

quadratic summand ( q b ) X ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} ) is nonnegative. Since the evaluated determinant doesn’t vanish, we obtain

required statement. ■ Then we change the variables q ∈ ℝ^{n−1}, b ∈ ℝ^{n−1} to the variable h ∈ ℝ^{2n−2} in integral (5.3) by the formula

h = ( q b ) \sqrt{\frac{1}{2}(X + X^{∗})}  (5.6)

All other variables remain the same. The Jacobian of the substitution is det P^{1/2} det(1 + P + A). This yields that our integral decomposes to a product of two integrals

∫_{P ≫0,A=−A^{t}} ∏_{j=1}^{n−2} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−n+1/2}}{det(1 + P + A)^{σ_{n−1}−1}} dP dA ×  (5.7)

× ∫_{u>0,h∈ℝ^{2n−2}} u^{λ_{n}−n}(1 + u + |h|^{2})^{−σ_{n}}du dh  (5.8)

The second factor can be easily evaluated by Lemma 4.4. The first factor itself has form (5.1).

5.2. Separation of variables for the series U(n, n)/U(n) × U(n). Consider integral (0.5) over a wedge of complex dissipative matrices. Let us represent T, S (recall that T = T^{∗}, S = −S^{∗}) as block ((n−1)+1)×((n−1)+1) matrices:

T = ( \begin{matrix}P & q^{∗} \\ q & r \end{matrix} ); S = ( \begin{matrix}A & −b^{∗} \\ b & ic \end{matrix} )

Repeating considerations of Subsection 5.1, we transform our integral to the form

∫_{P ≫0,A=−A^{∗}} dP dA(∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det[1+P+A]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} · \frac{det P^{λ_{n−1}−2n}}{det(1+P+A)^{\{σ_{n−1}∥τ_{n−1}\}}} ×

× ∫_{u>0,b∈ℂ^{n−1},q∈ℂ^{n−1},c∈ℝ} u^{λ_{n}−2n} ×

× {1 + u + ic + qP^{−1}q^{∗} − (q + b)(1 + P + A)^{−1}(q^{∗} − b^{∗})}^{{−σ_{n}∥−τ_{n}}}du db dq dc)

The expression in curly brackets has the form

\{1 + u + ic + ( q b ) X ( \begin{matrix}q^{∗} \\ b^{∗} \end{matrix} )\}

where the matrix X is the same as above (5.5). Let us change the variable c to s by the formula

s = c − \frac{1}{2i} ( q b ) (X^{∗} − X) ( \begin{matrix}q^{∗} \\ b^{∗} \end{matrix} )

The Jacobian of the substitution is 1. The expression in the curly brackets transforms to

\{1 + u + is + \frac{1}{2} ( q b ) (X + X^{∗}) ( \begin{matrix}q^{∗} \\ b^{∗} \end{matrix} )\}

Consider the substitution

h = det(\frac{1}{2}(X + X^{∗}))^{1/2}

Its Jacobian is det P | det(1 + P + A)|^{2} (see. Lemma 5.1). Then our integral decomposes to a product of the integrals

∫_{P ≫0,A=−A^{∗}} ∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det[1+P +A]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} · \frac{det P^{λ_{n−1}−2n+1}}{det(1+P+A)^{\{σ_{n−1}−1∥τ_{n−1}−1\}}} dP dA ×

×  ∫_{u>0,h∈ℂ^{2n−2},s∈ℝ} u^{λ_{n}−2n}{1 + u + is + |h|^{2}}^{{−σ_{n}∥−τ_{n}}}du dh dc  (5.9)

It remains to calculate multiplier (5.9).

5.3. Calculation of separated factor. Passing to spherical coordinates in the variable h ∈ ℂ^{2n−2} ≃ ℝ^{4n−4}, we transform (5.9) to the form

\frac{2π^{2n−2}}{Γ(2n − 2)} ∫_{u>0,ρ>0,c∈ℝ} u^{λ_{n}−2n}(1 + u + ic + ρ^{2})^{−σ_{n}}(1 + u − ic + ρ^{2})^{−τ_{n}}ρ^{4n−5}dρ du dc =

= \frac{π^{2n−2}}{Γ(2n − 2)} ∫_{u>0,v>0,c∈ℝ} u^{λ_{n}−2n}(1 + u + ic + v)^{−σ_{n}}(1 + u − ic + v)^{−τ_{n}}v^{2n−3}dv du dc

Applying Lemma 4.2 we obtain

\frac{π^{2n−2}Γ(λ_{n} − 2n + 1)}{Γ(λ_{n} − 1)} ∫_{0}^{∞}dw ∫_{−∞}^{∞} dc \{(1 + w + ic)^{−σ_{n}}(1 + w − ic)^{−τ_{n}}w^{λ_{n}−2}\}  (5.10)

For integration in the variable c, we use the formula (see. [18], 2.2.5.33)

∫_{−∞}^{∞} \frac{dx}{(a + ix)^{μ}(b − ix)^{ν}} = 2π(a + b)^{1−μ−ν} \frac{Γ(μ + ν − 1)}{Γ(μ)Γ(ν)}

(where a > 0, b > 0) and obtain

\frac{2π^{2n−1}Γ(λ_{n} − 2n + 1)}{Γ(λ_{n} − 1)} ∫_{0}^{∞}(2 + 2w)^{1−σ_{n}−τ_{n}}w^{λ_{n}−2}dw · \frac{Γ(σ_{n} + τ_{n} − 1)}{Γ(σ_{n})Γ(τ_{n})}

Last integral reduces to B-function in the standard way, and finally we obtain

\frac{2^{2−σ_{n}−τ_{n}}π^{2n−1}Γ(λ_{n} − 2n + 1)Γ(σ_{n} + τ_{n} − λ_{n})}{Γ(σ_{n})Γ(τ_{n})}

5.4. The case G/K = Sp(n, n)/Sp(n) × Sp(n). Consider integral (0.6) over the wedge of dissipative quaternionic matrices R = T + S. Recall that T = T^{∗}, S = −S^{∗}. As above, we represent matrices T, S in the form

T = ( \begin{matrix}P & q^{∗} \\ q & r \end{matrix} ); S = ( \begin{matrix}A & −b^{∗} \\ b & c \end{matrix} )

where c = c_{1}i + c_{2}j + c_{3}k is a purely imaginary quaternion. Repeating considerations of Subsection 5.2, we reduce integral (0.6) to the form

∫_{P ≫0,A=−A^{∗}} ∏_{j=1}^{n−2} \frac{det[T]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det P^{λ_{n−1}−4n+2}}{det(1 + P + A)^{σ_{n−1}−4}} dP dA ×

×  ∫_{u>0,h∈ℝ^{8n−8},c=−c̄∈ℍ} u^{λ_{n}−4n}|1 + u + s + |h|^{2}|^{−σ_{n}}du dh dc  (5.11)

It remains to evaluate separated factor (5.11).

Passing to spherical coordinates in variable h ∈ ℝ^{8n−8}, we obtain

\frac{2π^{4k−4}}{Γ(4k − 4)} ∫_{u>0,ρ>0,c=−c̄∈ℍ} u^{λ_{n}−4n}|1 + u + ρ^{2} + c|^{−σ_{n}}ρ^{8n−9}du dρ dc =

= \frac{π^{4k−4}}{Γ(4k − 4)} ∫_{u>0,v>0,c=−c̄∈ℍ} u^{λ_{n}−4n}|1 + u + v + c|^{−σ_{n}}v^{4n−5}du dv dc =

= \frac{π^{4k−4}}{Γ(4k − 4)} ∫_{u>0,v>0,c=−c̄∈ℍ} u^{λ_{n}−4n}((1 + u + v)^{2} + |c|^{2})^{−σ_{n}/2}v^{4n−5}du dv dc

Then we pass to spherical coordinates in variable c ∈ ℝ^{3}, and get

\frac{4π · π^{4k−4}}{Γ(4k − 4)} ∫_{u>0,v>0,z>0} u^{λ_{n}−4n}((1 + u + v)^{2} + z^{2})^{−σ_{n}/2}v^{4n−5}z^{2}du dv dz =

= \frac{2π^{4k−3}}{Γ(4k − 4)} ∫_{u>0,v>0,x>0} u^{λ_{n}−4n}((1 + u + v)^{2} + x)^{−σ_{n}/2}v^{4n−5}x^{1/2}du dv dx

Integration in x gives

\frac{2π^{4k−3}}{Γ(4k − 4)}B(3/2, σ_{n}/2 − 3/2) ∫_{u>0,v>0,} u^{λ_{n}−4n}v^{4n−5}(1 + u + v)^{3−σ_{n}}du dv

Applying Lemma 4.4, we get the final formula

\frac{2π^{4k−3}}{Γ(4k − 4)} · \frac{Γ(3/2)Γ(σ_{n}/2 − 3/2)}{Γ(σ_{n}/2)} · \frac{Γ(4k − 4)Γ(λ_{n} − 4n + 1)Γ(σ_{n} − λ_{n})}{Γ(σ_{n} − 3)}

Applying Duplication Formula to Γ(σ_{n} − 3), we reduce the last expression to the form

2^{σ_{n}−4}π^{4k−3} \frac{Γ(λ_{n} − 4n + 1)Γ(σ_{n} − λ_{n})}{Γ(σ_{n}/2)Γ(σ_{n}/2 − 1)}

§6. Integrals over the Spaces U(p, q, 𝕂)/U(p, 𝕂) × U(q, 𝕂).

In the Section we evaluate integrals (0.7)–(0.9) over the spaces O(p, q)/O(p)×O(q), U(p, q)/U(p)×U(q), Sp(p, q)/Sp(p)× Sp(q). For definitenes, we assume p ⩽ q.

6.1. The case G/K = O(p, q)/O(p) × O(q). Identities (3.5)–(3.6) allow to wright integral (0.7) in the form

∫_{M−LL^{t}≫0,N=−N^{t}} ∏_{j=1}^{n} \frac{det [ \begin{matrix}1 & L^{t} \\ L & M \end{matrix} ]_{q−p+j}^{λ_{j}−λ_{j+1}}}{det[1 + M + N]_{j}^{σ_{j}−σ_{j+1}}} det ( \begin{matrix}1 & L^{t} \\ L & M \end{matrix} )^{−(p+q)/2} dL dM dN  (6.1)

Let us represent M, N as block ((p − 1) + 1) × ((p − 1) + 1) matrices, and L as a block ((p − 1) + 1) × (q − p) matrix:

M = ( \begin{matrix}P & q^{t} \\ q & r \end{matrix} );  N = ( \begin{matrix}A & −b^{t} \\ b & 0 \end{matrix} );  L = ( \begin{matrix}H \\ l \end{matrix} )

By (1.2),

det ( \begin{matrix}1 & L^{t} \\ L & M \end{matrix} ) = det ( \begin{matrix}1 & H^{t} & l^{t} \\ H & P & q^{t} \\ l & q & r \end{matrix} ) =

= det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ) · \{r − ( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} )\}

det(1 + M + N) = det(1+P +A) · (1 + r − (q + b)(1+P +A)^{−1}(q^{t} − b^{t}))

Denote the expression in curly brackets by u. The integral converts to the form

∫ dP dA dH(∏_{j=1}^{n−2} \frac{det [ \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ]_{q−p+j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{λ_{n−1}−(p+q)/2}}{det(1 + P + A)^{σ_{n−1}}} ×

×  ∫_{u>0, q,b∈ℝ^{p−1},l∈ℝ^{q−p}} u^{λ_{p}−(p+q)/2}\{1+u+( l q ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} ( \begin{matrix}l^{t} \\ q^{t} \end{matrix} ) +  (6.2)

+ ( q b ) ( \begin{matrix}−(1+P +A)^{−1} & −(1+P +A)^{−1} \\ (1+P +A)^{−1} & (1+P +A)^{−1} \end{matrix} ) ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )\}^{−σ_{p}}du dl dq db)  (6.3)

The expression in the curly brackets in (6.2)–(6.3) has form

\{1 + u + ( l q b ) X ( \begin{matrix}l^{t} \\ q^{t} \\ b^{t} \end{matrix} )\}  (6.4)

where

X = ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} + ( \begin{matrix}0 & 0 \\ 0 & −S^{−1} \end{matrix} ) & ( \begin{matrix}0 \\ −S^{−1} \end{matrix} ) \\ ( 0 S^{−1} ) & S^{−1} \end{matrix} )

and S denote

S = 1 + P + A

As in Subsection 5.1, we wright (6.4) in the form

\{1 + u + ( l q b ) \frac{1}{2}(X + X^{t}) ( \begin{matrix}l^{t} \\ q^{t} \\ b^{t} \end{matrix} )\}  (6.5)

Lemma 6.1.

det (\frac{1}{2}(X + X^{t})) = det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} · det(1 + P + A)^{−2}

Proof. det (\frac{1}{2}(X + X^{t})) =

= det ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} + ( \begin{matrix}0 & 0 \\ 0 & −\frac{1}{2}S^{t−1} − \frac{1}{2}S^{−1} \end{matrix} ) & ( \begin{matrix}0 \\ −\frac{1}{2}S^{−1} + \frac{1}{2}S^{t−1} \end{matrix} ) \\ ( 0 \frac{1}{2}S^{−1} − \frac{1}{2}S^{t−1} ) & \frac{1}{2}S^{−1} + \frac{1}{2}S^{t−1} \end{matrix} )

Then we add the third row to the second row and the third column to the second column. We obtain

det ( \begin{matrix}( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} & ( \begin{matrix}0 \\ (1 + P − A)^{−1} \end{matrix} ) \\ ( 0 (1 + P + A)^{−1} ) & \frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} \end{matrix} )

Formula (1.2) reduces the determinant to the form

det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} · det(\frac{1}{2}(1 + P + A)^{−1} + \frac{1}{2}(1 + P − A)^{−1} −

− ( 0 (1 + P + A)^{−1} ) ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ) ( \begin{matrix}0 \\ (1 + P − A)^{−1} \end{matrix} )) =

= det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{−1} det(1 + P + A)^{−1} det(1 + P − A)^{−1} ×

× det(\frac{1}{2}(1 + P − A) + \frac{1}{2}(1 + P + A) − P ) ■

After the substitution

( l q b ) \sqrt{\frac{1}{2}(X + X^{t})} = h ∈ ℝ^{q−p} ⊕ ℝ^{p−1} ⊕ ℝ^{p−1}

to integral (6.2)–(6.3), we get

∫ ∏_{j=1}^{n−2} \frac{det [ \begin{matrix}1 & H^{t} \\ H & P \end{matrix} ]_{q−p+j}^{λ_{j}−λ_{j+1}}}{det[1 + P + A]_{j}^{σ_{j}−σ_{j+1}}} · \frac{det ( \begin{matrix}1 & H^{t} \\ H & P \end{matrix} )^{λ_{n−1}−(p+q)/2+1/2}}{det(1 + P + A)^{σ_{p−1}−1}} dP dA dH ×

× ∫_{u>0, h∈ℝ^{q+p−2}} u^{λ_{p}−(p+q)/2}{1 + u + |h|^{2}}^{−σ_{p}}du dh

The first factor itself has form (6.1), and the second factor can be easily evaluated by Lemma 4.4.

6.2. The cases G/K = U(p, q)/U(p) × U(q) and Sp(n, n)/Sp(n) × Sp(n). A separation of variables in these cases is the same as in the case G/K = O(p, q)/O(p) × O(q). Separated factors coincide with integrals (5.9), (5.11) evaluated above.

§7. Integrals over the Spaces Sp(2n, ℝ)/U(n)

and Sp(2n, ℂ)/Sp(n).

7.1. Integrals over Sp(2n, ℝ)/U(n). We realize this space as in Subsection 2.4. Let R be a complex symmetric matrix, let R = T + iS, where T, S are symmetric. Consider integral (0.10). Let us represent matrices T, S in the form

T = ( \begin{matrix}P & q^{t} \\ q & r \end{matrix} ) , S = ( \begin{matrix}A & b^{t} \\ b & c \end{matrix} )

Then

det T = det P · (r − qP^{−1}q^{t})
det(1+T +iS) = det(1+P +iA){1 + r + ic − (q + ib)(1 + P + iA)^{−1}(q^{t} + ib^{t})}

Substituting

u = r − qP^{−1}q^{t}

we convert (0.10) to the form

∫ dT dS(∏_{j=1}^{n−2} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + P + iA]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} \frac{det P^{λ_{n−1}−(n+1)}}{det(1 + P + iA)^{\{σ_{n−1}∥τ_{n−1}\}}} ×

× ∫ u^{λ_{n}−(n+1)}\{1 + u + ic + ( q b ) X ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )\}^{\{−σ_{n}∥−τ_{n}\}}du dp db)

where

X = ( \begin{matrix}P^{−1} − (1 + P + iA)^{−1} & −i(1 + P + iA)^{−1} \\ −i(1 + P + iA)^{−1} & (1 + P + iA)^{−1} \end{matrix} )

Then we repeat the arguments of Subsections 5.1–5.2. We consider the substitution

s =c + Im ( q b ) X ( \begin{matrix}q^{t} \\ b^{t} \end{matrix} )
h = ( q b ) \sqrt{ReX}

Repeating proof of Lemma 5.1, we obtain

det(ReX) = det P^{−1}|det(1 + P + iA|^{−2}

Hence the original integral equals to

∫ ∏_{j=1}^{n} \frac{det[P]_{j}^{λ_{j}−λ_{j+1}}}{det[1 + P + iA]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} ×

× \frac{det[P]_{j}^{λ_{n−1}−(n+1)+1/2}}{det[1 + P + iA]_{j}^{\{σ_{n−1}−1/2∥τ_{n−1}−1/2\}}} dP dA ×

× ∫ u^{λ_{n}−2n}{1 + u + is + |h|^{2}}^{{−σ_{n}∥−τ_{n}}}du ds dh (7.1)

Separated factor (7.1) can be easily evaluated by Lemma 4.4.

7.2. Spaces Sp(2n, ℂ)/Sp(n). Consider the realization of this space described in Subsection 2.5. Represent the matrices T, S in the form

T = (\begin{matrix}P & q^{∗} \\ q & r \end{matrix}) S = (\begin{matrix}A & b^{t} \\ b & c \end{matrix}) where q, b ∈ ℂ^{n}; c ∈ ℂ, r ∈ ℝ

Repeating our calculations, we reduce integral (0.11) to the form

∫ dP dA(∏_{s=1}^{n} \frac{det[P]_{s}^{λ_{s}−λ_{s+1}}}{det[1 + P + Aj]_{s}^{σ_{s}−σ_{s+1}}} \frac{det P^{λ_{n−1}−(2n+1)}}{det(1 + P + Aj)^{σ_{n−1}}} ×

× ∫ u^{λ_{n}−(2n+1)}|1 + u + cj + Ψ(q, b)|^{−σ_{n}}du dp db dc) (7.2)

where

Ψ(q, b) = qP^{−1}q^{∗} − (q + bj)(1 + P + Aj)^{−1}(q^{∗} + jb^{∗})

Notice that the expression

\frac{1}{2}(Ψ(q, b) − i^{−1}Ψ(q, b)i)

is a quaternion having the form αj + βk. Hence we can change the variable cj = (c_{1} + c_{2}i)j to the variable y_{1}j + y_{2}k by the formula

y_{1}j + y_{2}k = cj + \frac{1}{2}(Ψ(q, b) − i^{−1}Ψ(q, b)i)

in integral (7.2). Then the expression under symbol of module in (7.2) changes to

|1 + u + y_{1}j + y_{2}k + \frac{1}{2}(Ψ(q, b) + i^{−1}Ψ(q, b)i)| =

= |1 + u + y_{1}j + y_{2}k + qP^{−1}q^{∗} +

+ \frac{1}{2}\{(q + bj)(1 + P + Aj)^{−1}(q^{∗} + jb^{∗}) + (q − bj)(1 + P − Aj)^{−1}(q^{∗} − jb^{∗})\}| (7.3)

The expression in curly brackets has purely complex values (in spite of the fact that the expression itself contains quaternionic unit j), moreover this expression is hermitian form depending of complex vector (q, b). The determinant of this form equals

det P^{−1}|det(1 + P + Aj)|^{−2}

Thus our integral decomposes to the product

∫ ∏_{s=1}^{n} \frac{det[P]_{s}^{λ_{s}−λ_{s+1}}}{det[1 + P + Aj]_{s}^{σ_{s}−σ_{s+1}}} \frac{det P^{λ_{n−1}−(2n−1)+1}}{det(1 + P + Aj)^{σ_{n−1}−2}} dP dA ×

× ∫ u^{λ_{n}−(2n+1)}|1 + u + y_{1}j + y_{2}k + |h|^{2}|^{−σ_{n}}du dy_{1}dy_{2}dh (7.4)

where h ∈ ℂ^{2n−2}. Factor (7.3) can be easily evaluated in the same way as integral (5.11), only integration in c ∈ ℝ^{3} is replaced by integration in y ∈ ℝ^{2}).

§8 Integrals over O(2n, ℂ)/O(n), SO^{∗}(2n)/U(n).

8.1. The case G/K = O(2n, ℂ)/O(2n). Denote by ℳ(ℝ) the algebra of real 2 × 2 matrices. Consider the following elements in ℳ(ℝ)

J = (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}); θ = (\begin{matrix}0 & 1 \\ 1 & 0 \end{matrix})

Obviously,

J^{2} = −1,  θ^{2} = 1;  Jθ = −θJ  (8.1)

To any z ∈ ℂ we assign

z^{∘} = α + βJ = (\begin{matrix}α & β \\ −β & α \end{matrix}) ∈ ℳ(ℝ)  (8.2)

Matrices z^{∘} form a subalgebra in ℳ(ℝ), which is isomorphic to the algebra ℂ. Let us denote this subalgebra by ℂ^{∘}. Note that the imaginary unite i ∈ ℂ corresponds to the matrix J ∈ ℳ(ℝ). Let us define in ℳ(ℝ) also the subalgebra ℝ^{∘} ⊂ ℂ^{∘} consisting of all matrices of the form α · 1, where α ∈ ℝ. It is easy to see that the algebra ℳ(ℝ) is generated by the subalgebra ℂ^{∘} and the element θ. For each z ∈ ℂ we have

θz^{∘} = (\overline{z})^{∘}θ  (8.3)

Define the involution Ψ ↦ Ψ^{⋆} in ℳ(ℝ) by the formula

Ψ^{⋆} = JΨ^{t}J^{−1}

Equivalently for all z, w ∈ ℂ

(z^{∘} + w^{∘}θ)^{⋆} = \overline{z}^{∘} − w^{∘}θ = \overline{z}^{∘} − θ\overline{w}^{∘}

Then (pq)^{⋆} = q^{⋆}p^{⋆}. We stress that the set of fixed elements of the involution ^{⋆} coincides with the subalgebra ℝ^{∘}.

Let A be an element of the algebra of k × k matrices over ℳ(ℝ). Let A_{ℝ} be the corresponding 2k × 2k matrix over ℝ. We define a determinant DET(A) as square root from | det(A_{ℝ})|. For matrices over subalgebra ℂ^{∘}, this determinant coincides with module of the usual determinant of a complex matrix.

Consider the model of the space O(2n, ℂ)/O(2n) defined in Subsection 2.6 (we preserve all notations of Subsection 2.6) . Represent a matrix R in the form R = T + H, where R = R^{t}, H = −H^{t}. Then we wright H as H = Sθ. Thus we represent R in the form

R = T + Sθ where T = T^{t} ≫ 0, S^{t} = −θSθ, JT = T J, JS = SJ

The matrix R has the size 2n × 2n. We can consider R as n × n matrix consisting of 2 × 2 matrices. In other words, we can consider R as matrix over ℳ(ℝ). The condition

R^{t} = J^{−1}RJ  (8.4)

(see (2.6)) on this language means that R is ⋆-hermitian (i.e. R as matrix over ℳ(ℝ) coincides with matrix obtained by transposition and element-wise involution ^{⋆}).

The conditions JT = T J, JS = SJ implies that 2 × 2 blocks of T and S have form (8.1), i.e. they are elements of ℂ^{∘}. Hence we can consider T , S as n × n matrices over ℂ^{∘}. Moreover, the matrix T is a hermitian matrix over ℂ^{∘}, and S is an antisymmetric matrix over ℂ^{∘}.

In these notations integral (0.14) has the form

∫ ∏_{j=1}^{n} \frac{DET[T]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + T + Sθ]_{j}^{σ_{j}−σ_{j+1}}} DET T^{−(2n−1)}dT dS

Then we wright ℂ^{∘}-matrices T, S in the form

T = (\begin{matrix}P & q^{∘∗} \\ q & r \end{matrix})  S = (\begin{matrix}A & −b^{∘t} \\ b & 0 \end{matrix}) ,  where r ∈ ℝ^{∘}

where symbols ^{∘∗}, ^{∘t} emphasis that we consider transposition and conjugation over ℂ^{∘}.

Repeating our calculations, we reduce the integral to the form

∫ dP dA(∏_{j=1}^{n−2} \frac{DET[P]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + P + Aθ]_{j}^{σ_{j}−σ_{j+1}}} · \frac{DET P^{λ_{n−1}−(2n−1)}}{DET(1 + P + Aθ)^{σ_{n−1}}} ×

× ∫ u^{λ_{n}−(2n−1)} DET(1 + u + Ψ(q, b))^{−σ_{n}}du dq db)

where

Ψ(q, b) = qP^{−1}q^{∘∗} − (q + bθ)(1 + P + Aθ)^{−1}(q^{∘∗} − b^{∘t}θ))

We wright the last expression in two following ways

Ψ(q, b) = qP^{−1}q^{⋆} − (q + bθ)(1 + P + Aθ)^{−1}(q + bθ)^{⋆}  (8.5)
Ψ(q, b) = (q b) X (\begin{matrix}q^{∘∗} \\ b^{∘∗} \end{matrix})  (8.6)

where

X = (\begin{matrix}P^{−1} − (1 + P + Aθ)^{−1} & −(1 + P + Aθ)^{−1}θ \\ θ(1 + P + Aθ)^{−1} & θ(1 + P + Aθ)^{−1}θ \end{matrix})

It is clear from (8.5), that Ψ(q, b) ∈ ℝ^{∘} (since P and (1 + P + Aθ) are ⋆-hermitian; this implies that Ψ(q, b) ∈ ℳ(ℝ) is a stable element of involution ⋆ ).

Remark. The summand qP^{−1}q^{⋆} is not linear over ℳ(ℝ). Hence the expression Ψ(q, b) is not ⋆-hermitian form on module ℳ(ℝ)^{n−1} over algebra ℳ(ℝ).

We have

Ψ(q, b) = \frac{1}{2}(Ψ(q, b) + J^{−1}Ψ(q, b)J)

Applying this equality to (8.6), we obtain

Ψ(q, b) = (q b) Y (\begin{matrix}q^{∘∗} \\ b^{∘∗} \end{matrix})

where

Y = (\begin{matrix}P^{−1}− \frac{1}{2} (1+P +Aθ)^{−1}− \frac{1}{2} (1+P −Aθ)^{−1} & − \frac{1}{2} (1+P +Aθ)^{−1}θ+ \frac{1}{2} (1+P −Aθ)^{−1}θ \\ \frac{1}{2} θ(1+P +Aθ)^{−1}− \frac{1}{2} θ(1+P −Aθ)^{−1} & \frac{1}{2} θ(1+P +Aθ)^{−1}θ+ \frac{1}{2} θ(1+P −Aθ)^{−1}θ \end{matrix})

First notice that the matrix Y doesn’t depend on θ, in spite of expression for Y contains θ. Formally speaking Y is a matrix over ℂ^{∘}. Secondly the matrix Y is ⋆-hermitian (since X, J^{−1}XJ are ⋆-hermitian). Hence Y is a hermitian matrix over ℂ^{∘}. It is easy to show that its determinant equals

DET(P)^{−1} DET(1 + P + Aθ)^{−2}

We obtain

∏_{j=1}^{n−2} \frac{DET[P]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + P + Aθ]_{j}^{σ_{j}−σ_{j+1}}} \frac{DET P^{λ_{n−1}−(2n−1)+1}}{DET(1 + P + Aθ)^{σ_{n−1}−2}} ∫ dP dA ×

× ∫_{u>0,h∈ℂ^{2n−2}} u^{λ_{n}−(2n−1)} DET(1 + u + |h|^{2})^{−σ_{n}}du dh (8.7)

Factor (8.7) can be easily evaluated by Lemma 4.4.

8.2. The case SO^{∗}(2n)/U(n). Denote by ℳ(ℂ) the algebra of complex 2 × 2 matrices over ℂ. The algebra ℳ(ℂ) contains the subalgebra ℍ^{∘}, which consists of matrices having the form

(\begin{matrix}α & β \\ \overline{β} & \overline{α} \end{matrix})  (8.8)

Obviously, the subalgebra ℍ^{∘} is isomorphic to the algebra of quaternionsℍ.

For a k × k matrix X over ℳ(ℂ), we define the determinant DET(X) as square root from determinant of the same matrix considered as 2k × 2k matrix over ℂ (we will use this definition only for dissipative matrices, hence square root is well-defined).

We preserve notations of Subsection 2.6. Let us represent the matrix R (having the size 2n × 2n and satisfying the conditions R^{t} = J^{−1}RJ) as a sum R = T + iS, where T = T^{∗}, S = S^{∗}. Then

T = J^{−1}\overline{T} J  S = J^{−1}\overline{S}J  (8.9)

As in preceding Subsection, we can consider the matrix R as n × n matrix over ℳ(ℂ). Conditions (8.9) mean that T, S are matrices over ℍ^{∘}.

Let us wright integral (0.15) in the form

∫ ∏_{j=1}^{n} \frac{DET[T]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + T + iS]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} DET T^{−2(2n−1)}dT dS

where i means multiplication by the scalar i in algebra of complex matrices having the size 2n × 2n. Then we wright the matrices T, S over ℍ^{∘} in the form

T = (\begin{matrix}P & q^{∘∗} \\ q & r \end{matrix})  S = (\begin{matrix}A & b^{∘∗} \\ b & c \end{matrix}) , where  b, q ∈ ℍ^{∘n−1} r, c ∈ ℝ^{∘}

We get

∫ dP dA(∏_{j=1}^{n−2} \frac{DET[T]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + T + iA]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} · \frac{DET T^{λ_{n−1}−2(2n−1)}}{DET(1 + T + iA)^{\{σ_{n−1}∥τ_{n−1}\}}} ×

× ∫ u^{λ_{n}−2(2n−1)} DET(1 + u + ic + Ψ(q, b))^{{−σ_{n}∥−τ_{n}}}du dc db dq) (8.10)

where

Ψ(b, q) = qP^{−1}q^{∘∗} − (q + ib)(1 + P + iA)^{−1}(q^{∘∗} + ib^{∘∗})

This expression satisfies to the identity

Ψ(b, q)^{t} = J^{−1}Ψ(b, q)J

where the symbol ^{t} means transposition in the algebra ℳ(ℂ) of 2 × 2-matrices. Hence Ψ(b, q) has the form (\begin{matrix}μ & 0 \\ 0 & μ \end{matrix}),

where μ ∈ ℂ.

Consider the substitution

y = c + Im(q + ib)(1 + P + iA)^{−1}(q^{∘∗} + ib^{∘∗})

(the symbol Im here means an imaginary part of a complex 2 × 2 matrix). Then multiplier (8.10) converts to the form

∫ u^{λ_{n}−2(2n−1)} DET(1 + u + iy + Re Ψ(q, b))^{{−σ_{n}∥−τ_{n}}}du dc db dq

The expression Re Ψ(q, b) is a hermitian form over ℍ^{∘}, and its determinant is

DET P^{−1}| DET(1 + P + iA)|^{−2}

Hence the integral transforms to the form

∫ dP dA(∏_{j=1}^{n−2} \frac{DET[P]_{j}^{λ_{j}−λ_{j+1}}}{DET[1 + P + iA]_{j}^{\{σ_{j}−σ_{j+1}∥τ_{j}−τ_{j+1}\}}} ×

× \frac{DET P^{λ_{n−1}−2(2n−1)+2}}{DET(1 + P + iA)^{\{σ_{n−1}−2∥τ_{n−1}−2\}}} ×

× ∫_{u>0,h∈ℍ^{2n−2}} u^{λ_{n}−2(2n−1)} DET(1 + u + ic + |h|^{2})^{{−σ_{n}∥−τ_{n}}}du dc db dq) (8.11)

Factor (8.11) can be easily evaluated by Lemma 4.4.

§9. Plancherel Formula for Kernel Representations.

Here we discuss only the case of groups G = O(p, q), considerations for all other series are the same.

For the spaces G/K = O(p, q)/O(p) × O(q) we consider ’section of wedge’ model SW_{p,q}(ℝ), see 3.1. We realize a point of the space G/K as a ((q − p) + p) × ((q − p) + p) matrix R having the form

R = (\begin{matrix}1 & 0 \\ 2L & M + N \end{matrix}) , where M = M^{t}, N = −N^{t}, M − LL^{t} ≫ 0

Let s_{1}, …, s_{p} be real numbers. Consider the function u_{s} on SW_{p,q}(ℝ) given by the formula

u_{s}(R) = ∏ det[M − LL^{t}]_{j}^{−1/2+i(s_{j}−s_{j+1})}

Recall that (see 3.2) that these functions are eigenfunctions of a minimal parabolic subgroup 𝒫 ⊂ G. Denote by φ_{s} the average of the function u_{s} under the action of the maximal compact subgroup K = O(p) × O(q) ⊂ O(p, q):

φ_{s}(R) = ∫_{K} u_{s}(R^{[k]})dk

where dk denote Haar measure on K(normalized by the condition: the total measure of the group is 1) and R^{[g]} is a fractional linear transformation defined by the formula (2.3).

It is easy to see that φ_{s} are exactly spherical functions on the symmetric space G/K(see for instance [9], [4]). Berezin function ℬ_{α} in our coordinates is given by the formula

ℬ_{α}(R) = \frac{4^{α(p+q)} det(M − LL^{t})^{α}}{det(1 + M + N)^{2α}}

(see Subsection 3.3). Its spherical transform (see [9],[4]) equals to

A_{α}(s) = ∫_{G/K} ℬ_{α}(R)φ_{−s}(R)dL dM dN

By K-invariance of the function ℬ_{α}(R), this integral equals to the integral

∫_{G/K} ℬ_{α}(R)u_{−s}(R)dL dM dN

The last integral is a partial case of integral (0.7). Thus

A_{α}(s) = const · ∏_{j=1}^{p} \frac{Γ(α + is_{j} − (q + p)/4 + 1/2)Γ(α − is_{j} − (q + p)/4 + 1/2)}{Γ(2α_{j} − (p − j))}

By Gindikin–Karpelevich inversion formula (see. [8], [9])

ℬ_{α}(R) = const ∫_{s_{1}⩾s_{2}⩾...⩾s_{p}⩾0} A_{α}(s)φ_{s}(R) dμ(s)  (9.1)

where dμ(s) is Gindikin–Karpelevich measure.

Wrighting (9.1) in the explicit form we obtain the following theorem

Theorem 9.1. Let α > (q + p)/4 − 1/2. Then

ℬ_{α}(R) = ∏_{k=1}^{p} \frac{1}{Γ(2α − (p − k))} ×

× ∫_{s_{1}⩾s_{2}⩾...⩾s_{p}⩾0} ∏_{k=1}^{p}|\frac{Γ(α + is_{k} − (q + p)/4 + 1/2)Γ((q − p)/2 + 2is_{k})}{Γ(2is_{k})}|^{2} ×

× ∏_{1⩽k<l⩽p}|\frac{Γ(1/2 + i(s_{k} − s_{l}))Γ(1/2 + i(s_{k} + s_{l}))}{Γ(i(s_{k} − s_{l}))Γ(i(s_{k} + s_{l}))}|^{2}φ_{s}(R)ds_{1} ds_{2} … ds_{p} (9.2)

Proof. Formula (9.2) is certainly correct if

∫ ℬ_{α}(R) < ∞, ∫ |A_{α}(s)| dμ(s) < ∞

(see [9], Theorem IV.9.5., [4], Theorem XIV.5.3). The both conditions are satisfied for sufficiently large α. By analytical continuation considerations, formula (9.2) is correct for all α > (q + p)/4 − 1/2. ■

Remark. At the point α = (q + p)/4 − 1/2 in the right part of (9.2) there appears a pole in numerator. At the same point ℬ_{α} became not square integrable on the group G = O(p, q). As it is shown in [14], for α < (q + p)/4 − 1/2 the spectrum of a kernel representation is not contained in principle series. By this reason, Theorem 9.1 for α < (q + p)/4 − 1/2 is not correct.

References

[1] Berezin F.A. Quantization in complex symmetric domains. Izv. Akad. Nauk SSSR, Ser. math., 39, 2, 1362–1402 (1975); English translation: Math USSR Izv. 9 (1975), No 2, 341–379(1976)

[2] Berezin F.A. On relations between covariant and contravariant symbols of operators v for complex classical domains. Dokl. Akad Nauk SSSR, 241, No 1 (1978); English translation: Sov. Math. Dokl. 19 (1978), 786–789

[3] van Dijk H., Hille S.C. Canonical representations related to hyperbolic spaces, J.Funct.Anal., 147, 109–139 (1997).

[4] Faraut J., Koranyi A. Analysis in symmetric cones. Oxford Univ.Press, (1994)

[5] Gantmaher F.R. Theory of matrices,4-th.ed., Moscow, Nauka(1988); English translation: Chelsea Publ. Corpor., NY (1959)

[6] Gindikin S.G. Analysis on homogeneous spaces. Uspehi mat. nauk,19, No 4, 3–92(1964).

[7] Gindikin S.G. it Invariant distributions in homogeneous domains Funkt.Anal i Prilozh. 9 (1975), No 1, 56–58; English translation: Funct. Anal. Appl. 9 (1975), No.1, 50–52.

[8] Gindikin S.G., Karpelevich F.I. Plancherel measure for Riemann symmetric spaces of non-positive curvature. Dokl. Akad Nauk SSSR, 145, 252–255(1962)

[9] Helgason S. Groups and geometric analysis. Acad. Press (1984).

[10] Hua Loo Keng, Harmonic analysis of functions of several complex variables in classical domains . Beijing, 1958(Chinese); Russian translation: Moscow, Inostrannaja literatura (1959); English translation: Amer. Math. Soc., Providence (1963)

[11] Neretin Yu.A. Extension of representations of classical groups to representations of categories . Algebra i analiz, t.3, No 1, 176–202 (1991); English translation in St.Petersburg Math.J.,Vol3(1992),No 1.

[12] Neretin Yu.A. Categories of symmetries and infinite-dimensional groups Oxford University Press (1996);Russion edition: Moscow, URSS(1998).

[13] Neretin Yu.A. Boundary values of holomorphic functions and some spectral problems for unitary representations. In collection Positivity in Lie groups: open problems, Hilgert J., J.D.Lawson, K.-H.Need, Vinberg E.B. (eds.), Walter de Gruyter, Berlin (1998).

[14] Neretin Yu.A. On separation of spectra in analysis of Berezin kernels (to appear)

[15] Neretin Yu.A., Olshanskii G.I., Boundary values of holomorphic functions , singular unitary representations of groups O(p, q) and their limits as q → ∞. Zapiski nauchn. semin. POMI RAN 223, 9–91(1995); English translation: J.Math.Sci. (1997).

[16] Olafsson, G., Orsted, B., Bargmann transform for symmetric spaces, (Odense university, preprint,1996).

[17] Orsted B., Zhang G. Tensor products of analytic continuations of discrete series. Can. J. Math.,49, N 6, 1224– 1241(1997)

[18] Prudnikov A.P., Brychkov Yu.A., Marichev O.I. Integrals and series. V.1. Elementary functions. Moscow, Nauka(1984) English translation: Gordon and Breach, 1986

[19] Piateckij-Shapiro I.I. Geometry of classical domains and theory of automorphic functions , Moscow, Fizmatlit (1961); English translation: Automorphic functions and the geometry of classical domains, Gordon and Breach, NY,1969

[20] Unterberger A., Upmeier H., The Berezin transform and invariant differential operators. Comm.Matn.Phys.,164, 563–597(1994)

[21] Zhang Genkai Berezin transform on line bundles over bounded symmetric domains, preprint (to appear in J.Lie theory)