Separation of spectra in analysis of Berezin kernels

Neretin Yu.A.^{1}

neretin@main.mccme.rssi.ru

Let G be a classical real group, let K be a maximal compact subgroup in G. There exists a canonical embedding of a symmetric space G/K to some noncompact hermitian symmetric space G^{∘}/K^{∘} such that dim_{ℝ} G/K = dim_{ℂ} G^{∘}/K^{∘}, (see Addendum A). Consider a problem about restriction of a highest weight representation of the group G^{∘} to the subgroup G. We use the term ”kernelrepresentations” (by analogy with Bergman kernel-function) for restrictions of this type.

A studying of kernel-representations was initiated by F.A.Berezin [2] shortly before his death (proofs were published only in 1994 by Upmeier and Unterberger [31]). In last years an interest to this subject appeared again. Partially it is related to continuation of Berezin work, partially this interest has independent reasons ([31], [23], [24], [28],[3], [19], [4], [33], [21]). It appeared that spectra of kernel representations are very intricate. For instance (see [20]) they include all representations which can appear in spectra of Howe dual pairs; a decomposition of Howe dual pairs is known as extremely rich spectral problem. G.I.Olshanskii and myself used kernel-representations for construction of exotic unitary representations of the groups O(p, q), U(p, q), Sp(p, q), see [23].

Emphasis that in the case of spaces of scalar-valued functions (see §1) a limit of kernel representations as a parameter α tends +∞ is the usual representation of the group G in the space L^{2}(G/K), i.e. the kernel-representations can be considered as natural ”deformation” of the spaces L^{2}(G/K)^{2}.

The main purpose of this paper is a decomposition of kernel-representations to ”blocks” having more or less uniform spectra ( in fact we consider only the case G = O(p, q); the cases G = U(p, q), Sp(p, q) are similar and all information necessary for repetition of our construction for U(p, q), Sp(p, q) is contained in the paper). The idea to construct decompositions of this type in various problems of harmonic analysis is old (see Gelfand–Gindikin [8]). G.I.Olshanskii [26] proposed a way (noncommutative Hardy spaces) for separation of highest weight increments (see survey [6] on this subject). There are also papers of Gindikin [9] and Molchanov [16] containing solution of the problem for hyperboloids. Our method is based on theorems about restrictions of discontinuous functions to submanifolds. In [17], [27], [23], [3],[19] this method was used for separation of the discrete part of spectrum).

1Partially supported by grant RFBR 98-01-00303 and Russian program of support of scientific schools (grant RFBR 96-01-96249).

2In a vector-valued case, a kernel-representation can be obtained by a deformation of a space of L^{2}-sections of a vector bundle over G/K).

In §1 we define kernel representations for groups O(p, q) (in the spaces of scalar-valued functions). In §2 we evaluate some integrals over orthogonal groups, which seems pleasant itself. In §3 we prove existence of restriction operators. In §4 we prove similar statements for general kernel representations (in spaces of vector-valued functions).

Addendum A contains brief discussion of definition of kernel representations Addendum B is devoted to analytical continuation of Plancherel formula for

kernel representations.

I thanks G.I.Olshanskii, V.F.Molchanov, and B.Orsted for numerous discussion of this subject. I also thanks H.Schlichtrull and G. van Dijk for discussions and references.

§1. Kernel-representations

1.0. Positive definite kernels. Let X be a set. Recall that a function K(x, y) on X × X is called by positive definite kernel if for all x_{1}, …, x_{n} ∈ X the matrix

( \begin{matrix}K(x_{1}, x_{1}) & … & K(x_{1}, x_{n}) \\ ⋮ & ... & ⋮ \\ K(x_{n}, x_{1}) & … & K(x_{n}, x_{n}) \end{matrix} )

is positive definite. A positive definite kernel defines an unique (up to a natural equivalence) Hilbert space H and a system of vectors θ_{x} ∈ H (supercomplete basis or system of coherent states), enumerated by points x ∈ X such that

1. ⟨θ_{x}, θ_{y}⟩_{H} = K(x, y) for all y, x ∈ X

2. The linear span of vectors θ_{x} is dense in H.

For each vector v ∈ H we define a function f_{v}(x) on X by the rule

f_{v}(x) = ⟨v, θ_{x}⟩_{H}

We denote by H^{∘} the space of all function obtained in this way.

1.1. Matrix balls. Let p ⩽ q. By B_{p,q} we denote the space of p × q matrices with norm < 1 over ℝ (a norm of a matrix is the norm of operator in Euclidean space). By \overline{B}_{p,q} we denote the closure of B_{p,q} in ℝ^{pq}, i.e. the set of matrices with norm ⩽ 1.

By O(p, q) we denote a group of real (p+q)×(p+q) matrices g = ( \begin{matrix}a & b \\ c & d \end{matrix} )

preserving bilinear form with the matrix ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ). The group O(p, q) acts

on the matrix ball B_{p,q} (and on \overline{B}_{p,q}) by fractional linear transformations

z ↦ z^{[g]} := (a + zc)^{−1}(b + zd)

It is easy to check that the action of O(p, q) on B_{p,q} is transitive, and the

stabilizer of the point z = 0 consists of matrices ( \begin{matrix}a & 0 \\ 0 & d \end{matrix} ) where a ∈ O(p), d ∈

O(q). Hence

B_{p,q} ≃ O(p, q)/O(p) × O(q)

.

By M_{h} we denote the space of matrices z ∈ \overline{B}_{p,q} such that rank of (1 − zz^{t}) is h. Obviously, the sets M_{h} are orbits of the group O(p, q) on the boundary of the matrix ball \overline{B}_{p,q}. By \overline{M}_{h} we denote the closure of the orbit M_{h}:

\overline{M}_{h} = ∪_{r⩽h}M_{r}

1.2. Berezin spaces. Let

α = 0, 1, 2, …, p − 1 or α > p − 1

The Berezin kernel K_{α} = K_{α}(z, u) is the function on B_{p,q} × B_{p,q} given by the formula

K_{α}(z, u) = det(1 − zu^{t})^{−α} = det(1 − uz^{t})^{−α}

By well-known Berezin theorem [1] for α indicated above the Berezin kernel is positive definite^{3}. Hence we obtain the space of functions on the matrix ball. Let us describe this space.

By D′(B_{p,q}) we denote the space of distributions supported in the open matrix ball B_{p,q}. Let us define the scalar product in D′(B_{p,q}) by the formula

⟨ϕ, ψ⟩_{α} := {ϕ(z)ψ(u), det(1 − zu^{t})^{−α}}

The Berezin space H_{α} = H_{α}(B_{p,q}) is the completion of pre-Hilbert space D′(B_{p,q}) (in our case, a super-complete basis θ_{x} consists of δ-functions).

1.3. Dual realization of Berezin spaces and kernel-representations. By δ_{a} we denote the delta-function supported at the point a ∈ B_{p,q}. To each element v ∈ H_{α}(B_{p,q}) we assign the function f_{v} on B_{p,q} defined by the formula

f_{v}(z) = ⟨v, δ_{z}⟩_{H_{α}}

Hence we obtain some space H_{α}^{∘}(B_{p,q}) ≃ H_{α}(B_{p,q}) consisting of functions on B_{p,q}

It is readily seen that elements of the space H_{α}^{∘}(B_{p,q}) are real analytical functions on B_{p,q}. Then we define the unitary representation of the group O(p, q) in H_{α}^{∘}(B_{p,q}) by the formula

T_{α} ( \begin{matrix}a & b \\ c & d \end{matrix} ) f(z) = (a + zc)^{−α}f ((a + zc)^{−1}(b + zd))  (1.1)

3There were also papers of Gindikin(1975), Vergne–Rossi(1976) and Wallach(1979) on this subject, see references in [23] or [5].

We name the representations T_{α} by term kernel-representations

Remark. Berezin [2] in the case of hermitian symmetric spaces used the term canonical representation. He gave this term from the paper iz [32]. This term can have many different senses ; in fact A.M.Vershik, I.M.Gelfand, M.I.Graev themself in paper [32] and subsequent works use the term ”canonical representation” for multiplicative integrals and and infinitely divisible representations. In fact the class of infinitely divisible representations have very small intersection with the class of ”kernel-representations”, this small intersection is a chance.

1.4. Holomorphic continuation. By B_{p,q}(ℂ) we denote the set of all complex p × q matrices with norm < 1. The kernel K_{α} extends to the function det(1 − zu^{∗})^{−α} on B_{p,q}(ℂ) × B_{p,q}(ℂ), which is holomorphic in z and antiholomorphic in u; this function is positive definite on B_{p,q}(ℂ) ([1]). Then we deduce that functions f ∈ H_{α}^{∘}(B_{p,q}) extend to holomorphic functions on B_{p,q}(ℂ).

§2. Some integrals over orthogonal groups

Consider a matrix A. By [A]_{p} we denote its left upper corner having a size p × p. We denote the Haar measure on SO(n) by dσ_{n}(g).

2.1. The maps Υ^{m}. Let us wright a matrix g ∈ SO(n) as a (m + (n −

m)) × (m + (n − m)) block matrix ( \begin{matrix}P & Q \\ R & T \end{matrix} ). Consider the map

Υ^{m} : ( \begin{matrix}P & Q \\ R & T \end{matrix} ) ↦ T − R(1 + P)^{−1}Q

defined almost everywhere Proposition 2.1.a) Υ^{m} is a map SO(n) to SO(n − m).

b)  Υ^{k} ∘ Υ^{m} = Υ^{k+m}.
c) S = \frac{g−1}{g+1} implies \{S\}_{p} = \frac{Υ^{n−p}(g)−1}{Υ^{n−p}(g)+1}

where {S}_{p} denotes the right lower p × p corner of the matrix S.

Remark. The map Υ is a value of Livshic characteristic function χ(z) = T + zR(1 − zP)^{−1}Q at the point z = −1 (see [14]).

Proof. Statement c) can be easily checked by direct calculation using Frobenius formula for inversion of block matrix (see [7], § II.4) ,

( \begin{matrix}A & B \\ C & D \end{matrix} )^{−1} = ( \begin{matrix}A^{−1} + A^{−1}B(D − CA^{−1}B)^{−1}CA^{−1} & −A^{−1}B(D − CA^{−1}B)^{−1} \\ −(D − CA^{−1}B)^{−1}CA^{−1} & (D − CA^{−1}B)^{−1} \end{matrix} )

Statements a),b) are consequences of c).

Proposition 2.2. Let A, B, ∈ SO(n − m). Then

Υ^{m} [( \begin{matrix}1 & 0 \\ 0 & A \end{matrix} ) g ( \begin{matrix}1 & 0 \\ 0 & B \end{matrix} )] = AΥ^{m}(g)B

Proof is obvious.

2.2. Another proof of Proposition 2.1. Consider the space V_{n} := ℝ^{n} ⊕ ℝ^{n} equipped with indefinite bilinear form

L_{n}((x, y), (x′, y′)) := ∑_{k=1}^{n} x_{k}x′_{k} − ∑_{k=1}^{n} y_{k}y′_{k}

A graph of an unitary operator ℝ^{n} → ℝ^{n} is a L_{n}-isotropic subspace in V_{n} = ℝ^{n} ⊕ ℝ^{n}.

For each linear relation M : V_{n} ⇉ V_{n−m} (recall that a linear relation is a subspace M ⊂ V_{n} ⊕ V_{n−m}, for details see [18]) and each subspace H ⊂ V_{n} it is possible to define a subspace MH ⊂ V_{n−m} in the following way. A vector y ∈ V_{n−m} is an element of MH iff there exists x ∈ H such that (x, y) ∈ M.

Consider the linear relation Z_{n}^{n−m} : V_{n} ⇉ V_{n−m} consisting of all vectors having the form

({(v, h), (w, −h)}, {v, w}) ∈ {ℝ^{n} ⊕ ℝ^{n}} ⊕ {ℝ^{n−m} ⊕ ℝ^{n−m}}

The condition (u, v), (u′, v′) ∈ Z_{n}^{n−m} implies L_{n}(u, u′) = L_{n−m}(v, v′). Hence the transformation H ↦ Z_{n}^{n−m}H maps isotropic subspaces to isotropic subspaces.

It is readily seen that the map Υ^{m} coincides with the transformation H ↦ Z_{n}^{n−m}H of isotropic grassmannians. Now Proposition 2.1 becomes obvious.

2.3. A projection of the Haar measure. Denote by I the segment [−1, 1]. Consider the map

Ξ : SO(n) → SO(n − 1) × I

given by the formula (see a notation in the beginning of the section)

Ξ(g) = (Υ^{1}(g), [g]_{1})

Proposition 2.3. The image of the Haar measure dσ_{n} under the map Ξ is

const · (1 − x^{2})^{(n−3)/2}dx dσ_{n−1}

Proof. By SO(n − 1)- equivariance of the map Υ^{1} (Proposition 2.2), the image of the Haar measure has the form ϕ(x)dx dσ_{n−1}. On another side the projection of the Haar measure to I is const · (1 − x^{2})^{(n−3)/2} and we obtain required form of the function ϕ(x).

2.4. The map of the orthogonal group to the cub. Consider the maps

SO(n) → SO(n − 1) × [−1, 1] → SO(n − 2) × [−1, 1] × [−1, 1] → … As result we obtain the map of the group SO(n) to the cube [−1, 1]^{n−1}, given by the formula

(x_{1}, x_{2}, …, x_{n−1}) = ([Υ^{n−2}(g)]_{1}, …, [Υ^{1}(g)]_{1}, [g]_{1})

By Proposition 2.3, the image of the Haar measure under this map is

dμ(x_{1}, x_{2}, …, x_{n−1}) = const · ∏_{j=1}^{n−1}(1 − x_{j}^{2})^{(j−2)/2}dν(x_{1}) ∏_{j=1}^{n−1} dx_{j}

Hence for each function f depending on n − 1 variables x_{1}, …, x_{n−1} ∈ [−1, 1] we have

∫_{SO(n)} f([Υ^{n−1}(g)]_{1}, …, [Υ^{1}(g)]_{1}, [g]_{1}) dσ_{n}(g) =
= ∫_{[−1,1]^{n−1}} f(x_{1}, x_{2}, …, x_{n−1})dμ(x_{1}, x_{2}, …, x_{n−1})  (2.1)

2.5. Multiplicativity. Proposition 2.4.Let g ∈ O(n). Let m < p ⩽ n. Then

det(1 + [g]_{p}) = det(1 + [g]_{m}) det(1 + [Υ^{m}(g)]_{p−m})

Proof. Let us wright g as block (m+(p−m)+(n−p))×(m+(p−m)+(n−p)) matrix:

g = ( \begin{matrix}P & Q_{1} & Q_{2} \\ R_{1} & T_{11} & T_{12} \\ R_{2} & T_{21} & T_{22} \end{matrix} )

By the usual formula for determinant of a block matrix, we obtain

det [1 + ( \begin{matrix}P & Q_{1} \\ R_{1} & T_{11} \end{matrix} )] = det(1 + P) det(1 + T_{11} − R_{1}(1 + P)^{−1}Q_{1})

On another side

Υ^{m}(g) = ( \begin{matrix}T_{11} − R_{1}(1 + P)^{−1}Q_{1} & T_{12} − R_{1}(1 + P)^{−1}Q_{2} \\ T_{21} − R_{2}(1 + P)^{−1}Q_{1} & T_{22} − R_{2}(1 + P)^{−1}Q_{2} \end{matrix} )

and now the statement becomes obvious.

2.6. Some integrals. Theorem 2.5. Assume that the Haar measure of the whole group SO(n) is 1. Let λ_{1}, …, λ_{n} ∈ ℂ, λ_{n+1} = 0, and Re λ_{k} > −(n − k)/2 for all k = 0, 1, …, n. Then

∫_{SO(n)} ∏_{k=1}^{n−1} det(1 + [g]_{k})^{λ_{k}−λ_{k+1}} dσ_{n}(g) = ∏_{k=1}^{n−1} \frac{Γ(n − k)Γ(λ_{k} + (n − k)/2)}{Γ((n − k)/2)Γ(λ_{k} + n − k)} (2.2)

Proof. Let us apply formula (2.1) and Proposition 2.4. Then our integral splits to the product of integrals having the form

∫_{−1}^{1}(1 − x^{2})^{(n−k−2)/2}(1 − x)^{λ_{k}}dx =  (2.3)

= 2^{λ_{k}+n−3}B (λ_{k} + \frac{n−k}{2} , \frac{n−k}{2} ) = 2^{λ_{k}+n−k−3} \frac{Γ(λ_{k}+(n−k)/2)Γ((n−k)/2)}{Γ(λ_{k}+n−k)}

2.7. Some integrals over unitary and symplectic groups. The following integrals can be evaluated in the same way, the only the obvious integral (2.3) is replaced by more complicated integrals over ball |x| < 1 in ℂ or in H.

∫_{U(n)} ∏_{k=1}^{n} det(1 + [g]_{k})^{λ_{k}−λ_{k+1}} \overline{det(1 + [g]_{k})}^{μ_{k}−μ_{k+1}} dσ_{n}(g) =

= ∏_{k=1}^{n} \frac{Γ(n − k + 1)Γ(n − k + 1 + λ_{k} + μ_{k})}{Γ(n − k + 1 + λ_{k})Γ(n − k + 1 + μ_{k})}  (2.4)

∫_{Sp(n)} ∏_{k=1}^{n} | det(1 + [g]_{k})|^{λ_{k}−λ_{k+1}} dσ_{n}(g) =

= ∏_{k=1}^{n} \frac{Γ(2(n − k + 1))Γ(2(n − k + 1) + λ_{k} + 1)}{Γ(2(n − k + 1) + λ_{k}/2)Γ(2(n − k + 1) + λ_{k}/2 + 1)}  (2.5)

(we define the compact symplectic group Sp(n) as the group of unitary operators in n-dimensional quaternionic space ℍ^{n}; a quaternionic operator A can be considered as an operator A_{ℝ} : ℝ^{4n} → ℝ^{4n}; we define a determinant of the quaternionic operator A as \sqrt[4]{det A_{ℝ}}).

2.8. Other ways of evaluation of integrals (2.2), (2.4), (2.5). We obtain a verbal evaluation of these integrals, nevertheless our way is not quite usual. There exist another way of evaluation of the same integrals. The Cayley transform reduces (2.2) to the form

∫ ∏_{k=2}^{n} det(1 − [T]_{k})^{μ_{k}} dT  (2.6)

where integration is given over the space of all skew-symmetric matrices. Last integral can be easily evaluated by the method described in [21] (in fact (2.6) is simpler than integrals from [21]). Similar way is valid for (2.4),(2.5).

Integral (2.4) reduces to the form

∫ ∏_{k=1}^{n} det(1 + [T]_{k})^{μ_{k}} det(1 − [T]_{k})^{ν_{k}} dT

where integration is given over the space of all anti-hermitian matrices (T = −T^{∗}). In this case where are two additional possibilities: we can use Laplace transform or we can use Gindikin method of separation of variables (see [5], chapter 7).

Remark. If λ_{1} = λ_{2} = ⋯ = λ_{n}, then integral (2.2) is equivalent to one of Hua Loo Keng integrals [12], chapter 2

§3. Operators of restriction to the boundary

In this section we assume p < q.

3.1. Operators of restriction to M_{r}. Denote by H_{α}^{∘} the subspace in H_{α}^{∘}(B_{p,q}) consisting of functions analytical in some neighborhood of closed matrix ball \overline{B}_{p,q}.

Lemma 3.1.The subspace H_{α}^{∘} is dense in H_{α}^{∘}.

Proof. There exists a basis in H_{α}^{∘} consisting of orthogonal homogeneous polynomials (see for instance[23], §1). These functions are eigenfunctions of the operator A_{ε}f(z) = f((1 − ε)z). The eigenvalues are (1 − ε)^{k}. Hence for each ε > 0 the operator A_{ε} is a continuous operator H_{α}^{∘} → H_{α}^{∘}. Obviously the image of the operator A_{ε} is dense in H_{α}^{∘} and also it is contained in H_{α}^{∘}. ■

By L_{loc}^{1}(M_{r}) we denote the space of locally integrable functions on manifold M_{r} (see Subsection 1.1; if r ≠ 0 then M_{r} is not compact).

The group O(p, q) acts in the space L_{loc}^{1}(M_{r}) by same formula (1.1), as in H_{α}^{∘}(B_{p,q}) (we only have to think that a matrix z is an element of M_{r}).

By J_{r} we denote the operator of restriction of a function f ∈ H_{α}^{∘} to the submanifold M_{r}.

Theorem 3.2. Let α < (q − p + 2r)/2. Then the operator J_{r} extends to a continuous operator H_{α}^{∘} → L_{loc}^{1}(M_{r}).

Remark. Let α ≠ 0 (if α = 0 then the space H_{α}^{∘} is one-dimensional). In general, functions f ∈ H_{α}^{∘}, are discontinuous in point of boundary of the matrix ball. In other words, for points a ∈ \overline{B}_{p,q} \ B_{p,q} the lineal functional f ↦ f(a) on H_{α}^{∘} is not well-defined. Hence existence of the restriction operator is not obvious.

Evidently, the operator J_{r} : H_{α}^{∘} → L_{loc}^{1}(M_{r}) is O(p, q)-intertwining. Hence its kernel ker J_{r} is a subrepresentation. Hence the orthogonal complement (ker J_{r})^{⊥} also is a subrepresentation. The operator J_{r} is a bijection from (ker J_{r})^{⊥} to the image im J_{r} of the operator J_{r}. Hence we constructed a subrepresentation in H_{α}^{∘}, and this subrepresentation has a natural realization in a space of functions on M_{r}.

3.2. Proof of the theorem 3.2. In [23] (theorems 2.2–2.2) there was obtained one general statement on boundary values of holomorphic functions. In our case, it implies the following statement

Theorem 3.3. Let M be a compact subset in the boundary of the matrix ball B_{p,q}. Let μ be a measure supported in M. Assume that

i) For almost all (with respect to the measure μ × μ) points (z, u) ∈ M × M there exists a limit as ε tends to 0 from the right

L(z, u) = lim_{ε→+0} det(1 − (1 − ε)zu^{t})^{−α}

ii) This limit is dominated, i.e. there exists a function h ∈ L^{1}(μ × μ) such that

det(1 − (1 − ε)zu^{t})^{−α} ⩽ h(z, u)

Then the restriction operator of a function f ∈ H_{α}^{∘} to the subset M is a continuous operator H_{α}^{∘} → L^{1}(M).

First we will explain briefly a way of application of the theorem in our case. We want to apply it to the orbit M_{r}. Obviously the limit exists. In fact it is sufficient to check the convergence of integral of the function L(z, u) = det(1 − zu^{∗})^{−α}. Nevertheless I don’t see a natural measure on M_{r}. Hence there is a small hope to evaluate integral if we don’t know natural integrand.

An estimation of convergence also seems heavy since the singularity of the integral is very unpleasant. In fact we construct non-directly a calculable integral with given singularities. w

To apply Theorem 3.3, we consider arbitrary compact subset M on the manifold M_{r}, we also assume that M is a closure of its interior. Let μ be the surface Lebesgue measure on M. It is sufficient to check that the integral

∫_{M} ∫_{M} det(1 − zu^{t})^{−α}dμ(z) dμ(u)  (3.1)

is convergent. The condition i) is a consequence of the convergence. Condition ii) follows from the obvious inequality ([23],3.8):

det(1 − c^{2}zu^{t})^{−α} ⩽ 2^{pα} det(1 − zu^{t})^{−α}.

It is sufficient to check that the integral

∫_{M} det(1 − zu^{t})^{−α}dμ(z)  (3.2)

is convergent for any u and is dominated by a constant that don’t depend on u. As it will be shown below, it is sufficient to prove convergence of the integral for some u.

First we consider u = u^{(0)} given by block ((p−r)+r)×((p−r)+(q −p+r))

matrix ( \begin{matrix}−1 & 0 \\ 0 & 0 \end{matrix} ). Then integral (3.2) transforms to the form

∫_{M} det[1 + z]_{p−r}^{−α}dμ(z)  (3.3)

Lemma 3.4. Integral (3.3) converges.

Lemma 3.5. Let us represent g ∈ SO(q+r) as block matrix g = ( \begin{matrix}g_{11} & g_{12} \\ g_{21} & g_{22} \end{matrix} )

having the size (p + (q + r − p)) × (q + r). Then g_{11} ∈ \overline{M}_{r}. Conversely each matrices z ∈ \overline{M}_{r} is a left upper corner of some matrix g ∈ SO(q + r).

Proof of Lemma 3.5 is obvious.

Proof of Lemma 3.4. By Lemma 3.5 the map τ : g ↦ g_{11} takes SO(q + r) to \overline{M}_{r}. Obviously, the image of the Haar measure on SO(q + r) under the map τ is a measure equivalent to a surface Lebesgue measure on M_{r}. Hence it is sufficient to check convergence of the integral

∫_{SO(q+r)} det[1 + g]_{p−r}^{−α}dσ_{q+r}(g)

But this integral was evaluated in §2 ( we substitute λ_{1} = ⋯ = λ_{p−r−1} = −α; λ_{p−r} = ⋯ = 0 to formula (2.2)). ■

Lemma 3.6. The kernel det(1 − zu^{t})^{−α} satisfies to the equality

det(1 − z^{[g]}(u^{[g]})^{t})^{−α} = det(1 − zu^{t})^{−α} det(a + zc)^{α} det(a + zu)^{α}  (3.4)

Proof is obvious.

Consider a compact subset L ⊂ O(p, q) such that for each v ∈ M there exists g_{v} ∈ L satisfying the condition v^{[g_{v}]} = u^{(0)}. Denote by K the set of all points in M_{r} having the form u^{[g]} where u ∈ M and g ∈ L. Obviously, K is compact.

Substitute w = z^{[g]} to the integral (3.2). By formula (3.4), we obtain

∫_{M^{[gu]}} det(1 − u^{(0)}w^{t})^{−α}H(w, u) dμ(w),  (3.5)

where M^{[g_{u}]} denote the image of the set M under the map g_{u}, and H(w, u) is some function (a product of Jacobian and additional factor which appears from (3.4)). Obviously, the function H(w, u) is bounded by some constant C. Hence integral (3.5) is dominated by quantity

C ∫_{K} det(1 − u^{[0]}w^{t})^{−α}dμ(w)

This quantity doesn’t depend on u and it is finite by Lemma 3.4. Theorem is proved.

3.3. Restriction of partial derivatives to the boundary. In Subsection 3.2 we restricted functions to M_{r}. Also it is possible to restrict partial derivatives of the function f ∈ H_{α}^{∘}. The existence of restriction operator for partial derivative \frac{∂}{∂z_{ij}} depends on convergence of integral

∫_{M} ∫_{M} |\frac{∂}{∂z_{ij}} \frac{∂}{∂u_{ij}} det(1 − zu^{t})^{−α}| dz du,

where M ⊂ M_{r} is a compact subset. The integrand has the form |P (z, u)| det(1− zu^{t})^{−α−2} where P (z, u) is a polynomial. Hence we can apply the same arguments. As a result, we obtain the following theorem.

Theorem 3.7. Let α + 2k < (q − p + 2r)/2. Let ∂^{ν} be a partial derivative of order k. Then the operator of restriction of a function ∂^{ν}f(z) to M_{r} is a continuous operator from H_{α}^{∘} to L_{loc}^{1}(M_{r}).

Now we will slightly formalize the picture.

Denote by N (M_{r}) the conormal bundle to M_{r} in ℝ^{pq}. By S^{k}N (M_{r}) we denote k-th symmetric power of the conormal bundle. By F_{r}^{k} we denote the set of all f ∈ H_{α}^{∘} whose partial derivatives of order ⩽ k vanish on M_{r}. Then elements of quotient space F_{r}^{k−1}/F_{r}^{k} are identified with sections of the bundle S^{k}N (M_{r}).

Theorem 3.7 implies the following statement:

Let α + 2k < (q − p + 2r)/2. Then the operator of restriction of partial derivatives of order k to M_{r} is a well-defined operator from the space H_{α}^{∘} to the space of L_{loc}^{1}-sections of the bundle S^{k}N (M_{r}).^{4}

3.4. Some comments. Thus we obtain the intertwining operator J_{r,k} from H_{α}^{∘} to the space of sections of bundle S^{k}N (M_{r}). Its image imJ_{r,k} is a quotient space of a Hilbert space. Hence imJ_{r,k} has a natural structure of a Hilbert space. (we emphasis that the Hilbert topology in imJ_{r,k} is stronger than topology induced from L_{loc}^{1}). We denote the representation of the group O(p, q) in imJ_{r,k} by

Ξ_{r,k}^{α} = Ξ_{r,k}^{α}(p, q)  (3.6)

The quotient space of a Hilbert space H by arbitrary closed subspace L can be canonically identified with the orthogonal complement L^{⊥} to L. Hence the representation Ξ_{r,k}^{α} is a subrepresentation in the kernel-representation T_{α}.

Hence we obtained a canonical family of subspaces (3.6) in H_{α}^{∘}(B_{p,q}), we name these subspaces by the term blocks.

If α is sufficiently large (i.e. α > (p + q)/2 − 1), then the operators of restriction to submanifolds M_{r} are not defined. In this case blocks Ξ_{r,k}^{α}(p, q) don’t exist. It is known that for α > (p + q)/2 − 1 the representation of O(p, q) in H_{α}^{∘}(B_{p,q}) is equivalent to the representation of O(p, q) v in L^{2} on symmetric space B_{p,q}^{5}.

For all α > p − 1 the summand (”block”), that equivalent to L^{2}(B_{p,q}), is present in decomposition of the kernel-representation.

For α < (p + q)/2 − 2 the block Ξ_{p−1,0}^{α} appears. This summand corresponds to the operator of restriction of function to the maximal boundary orbit M_{p−1}.

4the algebraic version of this construction arises to [13]

5this statement for arbitrary scalar-valued kernel-representations is proved in cite [24]; earlier similar statement for kernel representations of some groups were obtained (in different degree of generality) in [2], [30], [10]).

For α < (p + q)/2 − 3 block Ξ_{p−2,0}^{α} appears. After point α = (p + q)/2 − 4 two additional blocks Ξ_{p−1,1}^{α} and Ξ_{p−3,0}^{α} appears etc.

At the point α = p − 1 (it is the end of continuous series, see 1.2) the largest block L^{2}(B_{p,q}) disappears from decomposition. Also at this point all blocks Ξ_{p−1,1}^{p−1}, Ξ_{p−1,2}^{p−1}, Ξ_{p−1,3}^{p−1} etc. disappear. But the block Ξ_{p−1,0}^{p−1} at this point still alive.

If we continue decreasing of α (after point α = p − 1 it is a discrete process, see Subsection 1.2) ”new” blocks continue their appearance. Nevertheless ”old” blocks quickly disappear (this process can be checked using Plancherel formula, see Addendum B).

At the point α = 0 only the block Ξ_{0,0}^{0} is still alive, and this block is onedimensional representation.

Let us briefly discuss spectra of representations Ξ_{r,k}^{α}.

Proposition 3.8. Each subrepresentation in the kernel-representation T_{α} contains an O(p) × O(q)-invariant vector.

Proof. The function f(z) = 1 is an O(p, q)-cyclic vector in H_{α}^{∘} (since its orbit is exactly super-complete basis). Hence the projection of the function f(z) = 1 to any invariant subspace is a cyclic vector in this subspace. Hence the projection is non zero.

Proposition 3.9. a) The representation of the group O(p, q) in Ξ_{0,0}^{α} is irreducible

b) The representation of O(p, q) in Ξ_{0,k}^{α} is a finite sum of irreducible representations.

Proof. a) Indeed the constant is the unique O(p) × O(q)-invariant function on M_{0}.

b) We have to check that the space of O(p) × O(q)-invariant sections of the bundle S^{k}N (M_{0}) is finite-dimensional. But dimension of this space can’t exceed the dimension of fiber.

Remark. ”Block” Ξ_{0,0}^{α} which corresponds to operator of restriction to the compact orbit M_{0} was discovered in the paper of G.I.Olshanskii and myself [23], §3 (under more rigid conditions to parameters of representations) α < (q − 2p + 1)/2.

Remark. Spectrum in blocks Ξ_{r,k}^{α} (where 0 < r < p) is purely continuous. It consist of representations induced from unitary representations of groups O(p−r, q −r). The last representations are contained in spectrum of Ξ_{0,k}^{α+2k}(p− r, q − r).

Remark. ”Additional” block with continuous spectrum in kernel-representations was discovered earlier only for the case G = U(2, 2) in the paper [28]. In [20] I conjectured that the spectrum of kernel-representations for small values of α contains many various series. The proof of this conjecture is the subject of this paper.

§4. General kernel-representations.

4.1. Definition. By ρ_{Λ} we denote finite dimensional holomorphic representation of the group GL(n, ℂ) defined by a signature Λ = (λ_{1}, …, λ_{n}), where λ_{1} ⩾ … ⩾ λ_{n}. By V_{Λ} we denote the representation ρ_{Λ}. Recall that there exists a canonical U(n)-invariant scalar product in V_{Λ}.

Consider the representations ρ_{Λ′}, ρ_{Λ′′} of the groups GL(p, ℂ), GL(q, ℂ) defined by signatures Λ′ = (λ′_{1}, …, λ′_{p}), Λ′′ = (λ′′_{1}, …, λ′′_{q}) respectively. We will use notations Λ̃ = (λ′_{1}, …, λ′_{p}; λ′′_{1}, …, λ′′_{q}) and also VΛ̃ = V_{Λ′} ⊗ V_{Λ′′}.

Consider the function

KΛ̃(z, u) = ρ_{Λ′}(1 − zu^{t}) ⊗ ρ_{Λ′′}(1 − uz^{t}); z, u ∈ B_{p,q}

taking values in the space End(VΛ̃) of linear operators in VΛ̃.

Assume that KΛ̃(z, u) is a matrix-valued positive defined kernel. Recall that this means positive definiteness of the kernel

L ((z, ξ); (u, η)) = ⟨KΛ̃(z, u)ξ, η⟩_{VΛ̃}; x, y ∈ B_{p,q}; ξ, η ∈ VΛ̃

on the space X = B_{p,q} × VΛ̃ (for the condition of positive definiteness of the kernel Λ̃(z, u) we obtained in [25], see also any text on classification of highest weight unitary representations, see also [23].

The kernel L ((z, ξ); (u, η)) defines a Hilbert space consisting of functions on the space X = B_{p,q} × VΛ̃. But these functions are linear in the variable ξ ∈ VΛ̃. Hence we can consider them as VΛ̃-valued functions on B_{p,q}.

As a result we obtain the space H∘ ; Λ̃, which consists of VΛ̃-valued functions on B_{p,q}. The group O(p, q) acts on H∘ ; Λ̃ by unitary operators given by the formula

TΛ̃(g)f(z) = [ρ_{Λ′}(a + zc) ⊗ ρ_{Λ′′}(−d + cz^{[g]})] f(z^{[g]})

4.2. Operators of restriction to the boundary. Theorem 4.1.a) Assume λ′′_{1} − λ′_{p} < (q − p + 2r)/2. Then the operator of restriction to an orbit M_{r} is a well-defined operator in H∘ ; Λ̃.

b) Assume 2k + λ′′_{1} − λp^{′} < (q − p + 2r)/2. Then the operator of restriction of partial derivatives of order k to the orbit M_{r} is a well-defined operator in H∘ ; Λ̃.

Proof is similar to the proof of Theorem 3.2, it is necessary also use simple considerations from [23],4.5.

4.3. Comments. As above, we obtain decomposition of kernel-representation to ”blocks” Ξ_{r,k}^{Λ}, where r is a parameter enumerating orbits, and k is an order of partial derivatives.

Existence of operator of restriction to M_{0} was discovered in [23],§4 (under more rigid restriction to the signature Λ). In particular this construction gives a simple way for obtaining of ”exotic” unitary representations of the groups O(p, q) (see [23],§§4-5). Emphasis also that this representations admit a limit as q → ∞ ([23], §6).

The representations of O(p, q) in Ξ_{0,0}^{Λ} are reducible; their decomposition is discussed in [23] , §5. As above, the representation of O(p, q) in all spaces Ξ_{0,k}^{Λ} has a purely discrete spectrum. Nevertheless the spectrum of blocks Ξ_{r,k}^{Λ} for r > 0 is still sophisticated and contains representations of different types. Here I don’t have likely conjectures.

Addendum A. Hermitization of symmetric spaces.

Consider the following list.

1. G = GL(n, ℝ) G^{∘} = Sp(2n, ℝ)

2. G = O(p, q) G^{∘} = U(p, q)

3. G = Sp(2n, ℝ) G^{∘} = Sp(2n, ℝ) × Sp(2n, ℝ)

4. G = GL(n, ℂ) G^{∘} = U(n, n)

5. G = SO(n, ℂ) G^{∘} = SO^{∗}(2n)

6. G = Sp(2n, ℂ) G^{∘} = Sp(4n, ℝ)

7. G = U(p, q) G^{∘} = U(p, q) × U(p, q)

8. G = GL(n, ℍ) G^{∘} = SO^{∗}(2n)

9. G = Sp(p, q) G^{∘} = U(2p, 2q) 10.G = SO^{∗}(2n) G^{∘} = SO^{∗}(2n) × SO^{∗}(2n) 11.G = SO(2, n) G^{∘} = SO(2, n) × SO(2, n) 12.G = SO(1, p) × SO(1, q) G^{∘} = SO(2, p + q)

Denote by K, K^{∘} the maximal compact subgroup in G, G^{∘} respectively. Then G^{∘}/K^{∘} is a hermitian symmetric space and G/K ⊂ G^{∘}/K^{∘} is a totally real subspace, and

dim_{ℝ} G/K = dim_{ℂ} G^{∘}/K^{∘}

We say that G^{∘}/K^{∘} is hermitization of G/K. If G/K is hermitian then its hermitization is G/K × G/K. We see that all classical riemann symmetric spaces have hermitization^{6}

We define kernel-representation as restriction of an unitary highest weight representation^{7} ρ of the group G^{∘} to the subgroup G^{8}

6Unfortunately I don’t know the author of this observation. First time I heard this 20 years ago.

7A highest weight representation of the group G^{∘} is a representation in a space of holomorphic vector-valued functions on G^{∘}/K^{∘}

8references: [2] (the cases G = U(p, q), Sp(2n, ℝ), SO^{∗}(2n), SO(2, n)), [23] (G = O(p, q), U(p, q), Sp(p, q)), [24], [20] (some discussion of other cases); there exists an idea to extend a definition of kernel representation to pseudoriemann symmetric spaces, see [4].

Addendum B. Analytic continuation of the Plancherel formula

B.1. Plancherel formula. Plancherel formula for all classical series 1-10 of kernel-representations for large values of the parameter α there was obtained in [21] (this method is valid also for future tubes).^{9} In all cases the Plancherel measure has the form

V (α) ∏_{k=1}^{r} |Γ(\frac{1}{2}(α − h + is_{k}))|^{2} \frac{ds}{c(is)c(−is)} where \frac{1}{i}s_{k} ∈ ℝ  (B.1)

Here r is the rank of the group; is_{1}, is_{2},...,is_{r} are the standard parameters of spherical functions, spherical functions of unitary representations of non degenerate principal series corresponds to real s_{k}; \frac{ds}{c(is)c(−is)} is Gindikin–Karpelevich measure (see [11]), and V (α) is a holomorphic factor. The parameter h is the least value of α for which the kernel-representation is square integrable.

In this formula, the Plancherel measure is supported on unitary principal series. We have seen that for small α spectrum of kernel-representation is complicated and hence Plancherel formula (B.1) is not correct for small α.

B.2. Analytic continuation of the Plancherel formula. Plancherel formula for α < h can be obtained by the analytic continuation of formula (B.1). We will give the final formula for the cases G = O(p, q), U(p, q) (for other cases see [22]).

Denote by g_{t} an element of O(p, q) having eigenvalues e^{±t_{k}}. Let θ_{0} be the same as in Subsection 1.2. Then

< T_{α}(g_{t})θ_{0}, θ_{0} >= ∏_{k=1}^{p} cosh^{−α} t_{k}

Plancherel formula is equivalent to expansion of matrix element < T_{α}θ_{0}, θ_{0} > by spherical functions of O(p, q). Let us denote spherical functions of O(p, q) by Φ_{is_{1},...,is_{r}}. By Φ_{is_{1},...,is_{r}}(t_{1}, t_{2}, … t_{p}) we denote a value of spherical function on g_{t}.

9For the groups U(p, q), Sp(2n, ℝ), SO^{∗}(2n), SO(n, 2) and large values of parameter α the Plancherel formula was obtained by Berezin in 1978 ([2]), Shortly after Berezin died and proof was not published until [31]. For the groups O(p, 1), Sp(p, 1) the Plancherel formula was obtained in [3]

In these notations the Plancherel formula for α > (p + q)/2 − 1 has the form

∏_{k=1}^{p} cosh^{−α}t_{k} =

= A·∏_{k=1}^{p} \frac{1}{Γ(α − k + 1)} ∫_{s∈ℝ^{n}} ∏_{k=1}^{p} Γ((α−\frac{1}{2}(p+q)+1+is_{k})/2)Γ((α−\frac{1}{2}(p+q)+1−is_{k})/2)×

× ∏_{k=1}^{p} \frac{Γ((q − p)/2 + is_{k})Γ((q − p)/2 − is_{k})}{Γ(is_{k})Γ(−is_{k})} ×

∏_{1⩽k<l⩽p}(s_{k}^{2} − s_{l}^{2}) tanh \frac{π}{2} (s_{k} − s_{l}) tanh \frac{π}{2} (s_{k} + s_{l}) ×

× Φ_{is_{1},...,is_{r}}(t_{1}, t_{2}, …, t_{r}) ds_{1} … ds_{p} (B.2)

where A is a constant.

Now we will wright the analytic continuation of formula (B.2). Let r = 0, 1, …, p. Let u_{1}, u_{2}, …, u_{s} be nonnegative integers. Let

w_{j} = u_{1} + ⋯ + u_{j} + j/2

Then

∏_{m=1}^{p} cosh^{−α}(t_{m}) =

= ∑_{r=0}^{p} ∑_{u_{1},...,u_{r}:w_{r}<h−α} C_{u} · V_{u}(α) ×

× ∫ Q_{u}(α|s_{r+1}, …, s_{p})Φ_{α−(p+q)/2+2w_{1},...,α−(p+q)/2+2w_{r},is_{r+1},...,is_{p}}(t_{1}, …, t_{p}) ×

ds_{r+1} … ds_{p} (B.3)

where

C = A\frac{2^{p−r}p!(2π)^{r}}{(p − r)!} \frac{(−1)^{u_{1}+···+u_{s}}}{∏ u_{j}!} · ∏_{k<m⩽s} \frac{Γ(\frac{1}{2} + w_{m} − w_{k})}{Γ(w_{m} − w_{k})(\frac{1}{2} + w_{k−1} − w_{m})_{u_{k}}}

and A is a constant,

V_{u}(α) = \frac{1}{∏_{m=1}^{p} Γ(α − k + 1)} ×

∏_{k=1}^{r} \frac{Γ(α − p + 1 + +2w_{k})Γ(−α + q − 1 − 2w_{k})}{Γ(−α + \frac{1}{2}(p + q) − 2w_{k})(α − \frac{1}{2}(p + q) + w_{k} + w_{k−1} + \frac{1}{2})_{u_{k}}} ×

∏_{k<m⩽r} \frac{Γ(\frac{1}{2} − α + \frac{1}{2}(p + q) − w_{k} − w_{m})}{Γ(−α + \frac{1}{2}(p + q) − w_{k} − w_{m})(α − \frac{1}{2}(p + q) + w_{m} + w_{k−1} + \frac{1}{2})_{u_{k}}}

and

Q_{u}(α|s_{r+1}, …, s_{p}) =

= ∏_{n=r+1}^{p} |Γ(\frac{1}{2}(α − (p + q)/2 + 1 + is_{n}) + w_{n−1})|^{2} ×

∏_{k⩽r,l>r} |\frac{Γ(\frac{1}{2}(1 − α + \frac{1}{2}(p + q) − 2w_{k} + is_{l})}{Γ(\frac{1}{2}(−α + \frac{1}{2}(p + q) − 2w_{k} + is_{l}))(\frac{1}{2}(α − \frac{1}{2}(p + q) + w_{k} + is_{l})_{u_{k}}} |^{2} ×

∏_{l>n>r}(s_{l}^{2} − s_{n}^{2}) tanh(\frac{π}{2} (s_{l} − s_{n})) tanh(\frac{π}{2} (s_{l} + s_{n}))

In the case G = U(p, q) we obtain

∏_{m=1}^{p} cosh^{−α}(t_{m}) =

= ∑_{r=0}^{p}  ∑_{w_{1},...,w_{r}} C_{w} · V_{w}(α) ×

×∫ Q_{w}(α|s_{r+1}, …, s_{p})Φ_{α−(p+q)+1+2w_{1},...,α−(p+q)+1+2w_{r},is_{r+1},...,is_{p}}(t_{1}, …, t_{p})×

ds_{r+1} … ds_{p} (B.4)

where

C_{w} = A · \frac{2^{p−r}p!(2π)^{r}}{(p − r)!} · ∏_{k=1}^{r} \frac{(−1)^{w_{k}}}{w_{k}!} ∏_{k<l⩽r}(w_{k} − w_{l})^{2}

A is a constant, V_{w}(α) =

= ∏_{m=1}^{p} \frac{1}{Γ(α/2 − m + 1)^{2}} ∏_{k=1}^{r} \frac{Γ(\frac{1}{2}α − p + 1 + w_{k})^{2}Γ(−\frac{1}{2}α + q − w_{k})^{2}}{Γ(−α + (p + q) − 1 + 2w_{k})(α − (p + q) + 1 + w_{k})_{w_{k}}} ×

∏_{k<l⩽r}(α − (p + q) + 1 + w_{k} + w_{l})^{2}

and

Q_{w}(α|s_{r+1}, …, s_{p}) =

= ∏_{k⩽r,n>r}(\frac{1}{4}(α − (p + q) + 1 + 2w_{k})^{2} + s_{n}^{2})^{2} ×

∏_{n=r+1}^{p} \frac{|Γ(\frac{1}{2}((q − p + 1 + is_{n}))|^{4}|Γ(\frac{1}{2}(α − p − q + 1 + is_{n})|^{2}}{|Γ(is_{n})|^{2}} ×

∏_{n>m>r}(s_{n}^{2} − s_{m}^{2})^{2}

B.3. Negative integer α. Analytic formula (B.3),(B.4) are valid for all values of the parameter α ∈ ℝ. Nevertheless it has visible sense only for

α = 0, 1, 2, …, p − 1 or α > p − 1

when representation T_{α} is unitary, and for

α = 0, −1, −2, …

when the representation T_{α} is finite-dimensional. In the last case the right parts of identities contain only finite number of summands with r = p ( Γ-factors in denominator annihilate all summands with r < p). In the case G = U(p, p) we obtain Pickrell decomposition , [29]. In this place our work also has an intersection with a recent preprint of G.Zhang [33].

For over values of parameter α Plancherel formula (B.3), (B.4) are elements of mysterious nonunitary harmonic analysis (see, for instance [15],[4]).

References

[1] Berezin, F. A., Quantization in complex symmetric spaces, Izv. Akad. Nauk SSSR, Ser. Mat. 39 (1975) (Russian); English translation Math. in USSR Izv. 9 (1975)

[2] Berezin F.A A connection between the co- and contravariant symbols of operators on classical symmetric spaces. Dokl. Akad. Nauk SSSR 241 (1978), N1(Russian); English translation in Sov. Math. Dokl. 19 (1978), 786–789.

[3] van Dijk H., Hille S.C. Canonical representations related to hyperbolic spaces, J.Funct.Anal., 147, 109–139 (1997).

[4] van Dijk G., Molchanov V.F.The Berezin form for rank 1 para-hermitian symmetric spaces. J. Math. Pure Appl., 77, 1998, 749–799

[5] Faraut J., Koranyi A. Analysis in symmetric cones. Oxford Univ.Press, (1994)

[6] Faraut J., Olafsson G. Causal semisimple symmetric spaces, the geometry and harmonic analysis in Semigroups in algebra, geometry and analysis. eds K.-H. Hofmann, J.D.Lawson, E.B.Vinberg, Walter de Gruyter pp. 3–32.

[7] Gantmaher F.R. Theory of matrices,4-th ed., Moskva, Nauka(1988); English translation; Chelsea Publ. Corp., NY(1959)

[8] I. M. Gelfand and S. G. Gindikin,Complex manifolds whose skeletons are semi–simple Lie groups, and analytic discrete series of representations, Funkts. Analiz i Pril. 11, No. 4 (1977), 19–27 (Russian); English translation: Funct. Anal. Appl. 11 (1977), 258–265.

[9] Gindikin S. Conformal analysis on hyperboloids. J.Geom.Phys.,10, 175–184 (1993)

[10] Gutkin E. Coeffitients of Clebsch–Gordon for holomorphic discrete series, Lett. Math. Phys. 3, 185–192(1979)

[11] Helgason S.Group and geometric analysis Acad. Press, 1984

[12] Hua Loo Keng, Harmonic analysis of functions of several complex variables in classical domains, 1958 (Chinese); Russian translation: Moscow, Inostrannaja literatura (1959); English translation: Amer. Math. Soc., Providence,(1963)

[13] Jackobsen H., Vergne M. Restrictions and extensions of holomorphic representations. J. Funct.Anal, 34, 29-53 (1979).

[14] Livshic M.S. On spectral decomposition of non selfadjoint operators Mat. Sbornik. 34, 145–199(1954); English translation: Translations of Amer.Math.Soc

[15] Molchanov V.F. Indefinite Plancherel formula, folklore, Tambov-Moscow, 1980-1999.

[16] Molchanov V.F. Separation of spectra for hyperboloids Funkts. Anal. i Prilozh., 31:3, 35–43 (1997); English translation: Funct. Anal. Appl.,(1997)

[17] Neretin Yu.A.Representations of complementary series entering discretely in tensor products of unitary representations. Funct.Anal.i Prilozh, 20 (1986) N4, 79-80; English translation in Funct.Anal.Appl.,(1986), 20, 68– 70.

[18] Neretin Yu.A. Categories of symmetries and infinite-dimensional groups Oxford University Press (1996); Moscow, URSS(1998).

[19] Neretin Yu.A. Restriction of a function holomorphic in a domain to curves lying on the boundary and discreteSL_{2}(ℝ)-spectra. Izv.Ross.Akad,Nauk, ser. matem., (1998), 62, 3, 67-86; English translation in Izv. Math. 62(1998), 493-513

[20] Neretin Yu.A. Boundary values of holomorphic functions and some spectral problems for unitary representations. In collection Positivity in Lie groups: open problems, Hilgert J., J.D.Lawson, K.-H.Need, Vinberg E.B. (eds.), Walter de Gruyter, Berlin (1998).

[21] Neretin Yu.A. Matrix analogues of B-functions and Plancherel formula for Berezin kernel-representations. Preprint xxx.lanl.gov math-9905045

[22] Neretin Yu.A. On decomposition of Berezin kernels. (in preparation).

[23] Neretin Yu.A., Olshanskii G.I., Boundary values of holomorphic functions, singular unitary representations of the groups O(p, q_{)} and their limits as q → ∞. Zapiski nauchn. semin. POMI RAN 223, 9–91(1995); English translation: J.Math.Sci., 87, No. 6, 3983-4035 (1997).

[24] Ólafsson, G., Orsted, B., Generalization of Bargmann transform. in Lie theory and its applications in physics eds H.Doebner, V.K.Dobrev, J.Hilgert, World scientific (1996), pp. 3–15

[25] Olshanskii G.I. Description of the highest weight unitary representations for groups U(p, q) Funkt.Anal. i Prilozh., 14; No 3(1980); English translation in Funct.Anal.Appl.(1980)

[26] Olshanskii G.I. Complex Lie semigroups Hardy spaces and Gelfand-Gindikin programm. Group theory and cohomological algebra, Edition of Yaroslavl’ State Universitety, 85–98(1982); English translation in: Differential geometry and its applications, 1(1991)1, 297–308

[27] Olshanskii G.I. Irreducible unitary representations of the groups U(p,q) which admit the pass to limit as q → ∞. Zapiski Nauchn. Semin. LOMI, 172 (1989), 114-120 (1989); English translation in J. Sov. Math., 199.

[28] Orsted B., Zhang G. Tensor products of analytic continuations of discrete series. Can. J. Math.,49, N 6, 1224–1241(1997)

[29] Pickrell D. Measures on infinite dimensional Grassmann manifold. J. Funct. Anal., 70(1990), 1–26.

[30] Repka J. On tensor products of holomorphic discrete series Canad. J. Math., 31, 638–844 (1979)

[31] Unterberger A., Upmeier H., The Berezin transform and invariant differential operators. Comm.Matn.Phys.,164, 563–597(1994)

[32] Vershik A.M., Gelfand I.M., Graev M.I. Representations of SL_{2}(R), where R is ring of functions. Uspehi Mat. Nauk, 28, 3, 3-41 (1973); English translation in Russian Math. Surveys(1975)

[33] Zhang G. Berezin transform on line bundles over bounded symmetric domains., to appear.