Boundary values of holomorphic functions and some spectral problems for unitary representations

Yurii A. Neretin^{∗} Moscow State Institute of Electronics and mathematics Max-Plank-Institute-fur-Mathematik, Bonn baksan@dionis.iasnet.ru neretin@main.mccme.rssi.ru

May 26, 1997

Consider a (reducible) unitary representation of a semisimple Lie group G in a hilbert space H (for instance we can consider an action of the group G in some functional space on some homogeneous space, a tensor product of irreducible unitary representations, a restriction of a irreducible unitary representation to a subgroup etc.). There arises a natural problem: to decompose our representation to irreducible representations or (it is a synonym) to find spectrum of the representation. Problems of this type were widely investigated for last 50 years (since [Kre]) and now many such spectral problems are solved completly or partialy.

In several simplest cases the spectra are continuous and more or less ’uniform’. But sometimes such spectra contains strange discrete increments (or equivalently there exist minimal G-invariant subspaces in the hilbert space H). First examples of discrete increments were observed in [Nai], [Puk], [H-Ch],

[Mol1] (1961 - 1966) and now there exist large literature devoted to discrete spectra, see [Boy], [Ism], [Mol3-4], [Str], [Far], [Pat], [F-J1], [F-J2], [Sch], [Kob1-2], [RSW], [Tsu], [How], [Ada], [Li], [BO] etc.

Quite often ’discrete increments’ are very singular (’exotic’) representations and it is very difficult to construct such unitary representations by some other way. Even Harish-Chandra discrete series now don’t seems quite hand objects. Remarkable paper [F-J1] (1980, see also [F-J2]) contained serious generalization of Harish-Chandra discrete series and this paper put origin to systematic search of ’exotic’ unitary representations in discrete spectra.

∗supported by RFRF (grant 95-01-00814) and Russian program of support of scientific schools

In [Ner1] (1986), [Ols3],[Ols4], [NO], [Ner2] it was observed that very often discrete increments to spectra are related to some functional-theoretical phenomena, namely to so-called ’trace theorems’ (i.e theorems about restrictions of discontinuos functions to submanifolds). On various analytical trace theorems see [RS], IX.3,IX.9, [ESh], 2.3, [Beu], [Bar],5.12, [NR], [Rud], [Vla], [VS], [Ner2].

This paper is some kind of addendum to Olshanskii and my papers [NO], [Ner2]. Its purpose is to discuss some open problems related to applications of trace theorems in noncommutative harmonic analysis.

I am grateful to G.I.Olshanskii for collaboration. I also thanks V.F.Molchanov, H.Schlichtkrull, G.Zuckerman, B.Ørsted, M.Flensted-Jensen, T.Kobayshi, A.G.Sergeev, G.Olafson, B.Kostant, A.Dvorsky, R.S.Ismagilov, R.Howe, V.M.Gichev, V.V.Lebedev for discussions, comments and references. These notes are mainly based on my lectures [Ner5] on the school ’Analysis on homogeneous spaces’ (Tambov, august 1996) and I thanks organizers of this school.

1 Tensor product of two representations of SL(2, ℝ) of complementary series

1.1. Complementary series. Consider the sircle T^{1} with coordinates z = e^{iϕ} (where |z| = 1, ϕ ∈ [0, 2π]). Let −1 < s < 1, s ≠ 0. Consider a scalar product

< f_{1}, f_{2} >= ∫_{0}^{2π} ∫_{0}^{2π} \frac{f_{1}(ϕ_{1})\overline{f_{2}(ϕ_{2})}dϕ_{1}dϕ_{2}}{| sin(ϕ_{1} − ϕ_{2})/2|^{(1+s)/2}}  (1)

in the space of C^{∞}-functions on the circle. The functions e^{inϕ} are pairwise orthogonal with respect to this scalar product and

< e^{inϕ}, e^{inϕ} >= \frac{Γ(n + (1 + s)/2)}{Γ(n + (1 − s)/2)}  (2)

If s > 0 then the integral (1) diverges; neverless the formula (2) has sence and we can consider (2) as the definition of divergent integral (1) . Denote by H_{s} the completion of the space C^{∞}(T^{1}) with respect to the scalar product (1). We have

f(ϕ) = ∑ c_{n}e^{inϕ} ∈ H_{s} ⇔ ∑ n^{s}|c_{n}|^{2} < ∞

i.e. H_{s} is a Sobolev space on the sircle T^{1}.

We realize the group SL(2, ℝ) as the group of matrices g = ( \begin{matrix}a & b \\ \overline{b} & \overline{a} \end{matrix} )

satisfying the condition |a|^{2} − |b|^{2} = 1. Define a representation of the group SL(2, ℝ) in the space H by the formula

T_{s} ( \begin{matrix}a & b \\ \overline{b} & \overline{a} \end{matrix} ) f(z) = f (\frac{az + b}{\overline{b}z + \overline{a}}) |\overline{b}z + \overline{a}|^{s−1}

The representations T_{s} are unitary in the hilbert space H_{s} (this fact is simple but it is not trivial). These representations are called by representations of SL(2, ℝ) of complementary series. Recall that representations T_{s} and T_{−s} are equivalent.

Consider the tensor product T_{s_{1}} ⊗ T_{s_{2}} of two representations of the compementary series. The space H_{s_{1}}⊗H_{s_{2}} is the space of functions on two-dimensional torus T^{2} : z_{1} = e^{iϕ_{1}}, z_{2} = e^{iϕ_{2}} equipped with the scalar product

< f_{1}, f_{2} >= ∫_{0}^{2π} ∫_{0}^{2π} ∫_{0}^{2π} ∫_{0}^{2π} \frac{f_{1}(ϕ_{1}, ϕ_{2})\overline{f_{2}(ψ_{1}, ψ_{2})}dϕ_{1}dϕ_{2}dψ_{1}dψ_{2}}{| sin(ϕ_{1} − ψ_{1})/2|^{(1+s_{1})/2}| sin(ϕ_{2} − ψ_{2})/2|^{(1+s_{2})/2}}

The group SL_{2}(ℝ) acts in this space by the formula

(T_{s_{1}} ⊗T_{s_{2}}) ( \begin{matrix}a & b \\ \overline{b} & \overline{a} \end{matrix} ) f(z_{1}, z_{2}) = f (\frac{az_{1} + b}{\overline{b}z_{1} + \overline{a}}, \frac{az_{2} + b}{\overline{b}z_{2} + \overline{a}}) |\overline{b}z_{1} +\overline{a}|^{s_{1}−1}|\overline{b}z_{2} +\overline{a}|^{s_{2}−1}

1.2. Restriction to the diagonal.

Denote be Δ the diagonal ϕ_{1} = ϕ_{2} of the torus T^{2}.

Theorem 1.1.Let s_{1} + s_{2} > 1 . Then operator of restriction of function f ∈ C^{∞}(T^{2}) to the diagonal Δ extends to a bounded operator from the space H_{s_{1}} ⊗ H_{s_{2}} to the space L^{2}(Δ).

We emphasis that functions f ∈ H_{s_{1}} ⊗ H_{s_{2}} are discontinuous and hence a function f ∈ H_{s} have no value in a individual point of the torus. Neverless the operator R of the restriction of a function f to the diagonal is well defined .

Proof of this fact is very simple. The space H_{s_{1}} ⊗ H_{s_{2}} consist of functions

f(ϕ_{1}, ϕ_{2}) = ∑ c_{n,m}e^{inϕ_{1}}e^{imϕ_{2}}

satisfying the condition

∑ n^{s_{1}}m^{s_{2}}|c_{n,m}|^{2} < ∞

The restriction of f to the diagonal Δ has the form

∑_{k}( ∑_{m+n=k} c_{m,n})e^{ikϕ}

and the convergence of the series ∑ c_{m,n} easily can be deduced from the Cauchy-Bunyakovsky unequality. □

The restriction operator is a bounded SL(2, ℝ)-interwinning operator from T_{s_{1}} ⊗T_{s_{2}} to T_{s_{1}+s_{2}−1}. The representation T_{s_{1}} ⊗T_{s_{2}} is unitary and hence T_{s_{1}+s_{2}−1} is a subrepresentation in T_{s_{1}} ⊗ T_{s_{2}}.

1.3. Dual picture: disributions supported on the diagonal. Now let s_{1} + s_{2} < −1 (we remind that T_{−s} = T_{s}). It turns out to be that in this case the hilbert space H_{s_{1}} ⊗ H_{s_{2}} contains distributions supported on the diagonal Δ. Obviously the space of distributions supported on the diagonal is a invariant subspace. It is easy to show that the representation of SL(2, ℝ) in this subspace is equivalent to T_{s_{1}+s_{2}+1} and we again obtain a embedding of representation of complementary series to the tensor product.

1.4. Separation of discrete increment. Let s_{1} = s_{2} = s < −1/2. Consider the operator

Jf(ϕ_{1}, ϕ_{2}) = |sin (\frac{ϕ_{1} − ϕ_{2}}{2} )|^{2s} f(ϕ_{1}, ϕ_{2})

Operator J interwines the representation T_{s} ⊗ T_{s} and representation

(T_{0} ⊗ T_{0}) ( \begin{matrix}a & b \\ \overline{b} & \overline{a} \end{matrix} ) f(z_{1}, z_{2}) = f(\frac{az_{1} + b}{\overline{b}z_{1} + \overline{a}}, \frac{az_{2} + b}{\overline{b}z_{2} + \overline{a}})|\overline{b}z_{1} + \overline{a}|^{−1}|\overline{b}z_{2} + \overline{a}|^{−1}

The last representation is the standard representation of SL(2, ℝ) in L^{2}(T^{2}). The kernel of J consist of distributions supported on the diagonal and the image is dence in L^{2}. Hence the representation T_{s} ⊗ T_{s} is equivalent to the direct sum of the representation T_{1+2s} and the representation of SL(2, ℝ) in L^{2} on the torus.

Remark. The operator J is unbounded and hence it is nessesary to be careful, see [NO], $ 7 for details and for the case s_{1} ≠ s_{2}.

Remark. The existence of embedding T_{s_{1}+s_{2}−1} to T_{s_{1}} ⊗T_{s_{2}} was obtained by Pukanzky [Puk] in 1961 , analogical rezult for SL(2, ℂ) was obtained by Naimark [Nai] (1961). The explanation of this phenomenon using the restriction operator wasn’t observed before [Ner1] (1986).

2 Hilbert spaces defined by positive defined kernels.

2.1. Positive definite kernels. Let Ξ be a set. A function K(x, y) on Ξ × Ξ is named by a positive definite kernel if for each finite collection of points x_{1}, …, x_{n} ∈ Ξ we have

det ( \begin{matrix}K(x_{1}, x_{1}) & K(x_{1}, x_{2}) & … & K(x_{1}, x_{n}) \\ K(x_{2}, x_{1}) & K(x_{2}, x_{2}) & … & K(x_{2}, x_{n}) \\ ⋮ & ⋮ & ... & ⋮ \\ K(x_{n}, x_{1}) & K(x_{n}, x_{2}) & … & K(x_{n}, x_{n}) \end{matrix} ) ≥ 0

For each positive definite kernel K(x, y) we associate a Hilbert space H and fixed system of vectors Ψ_{x} ∈ H enumerated by x ∈ Ξ such that

1. The linear span of the system Ψ_{x} is dence in H

2. For all x, y ∈ Ξ we have

< Ψ_{x}, Ψ_{y} >_{H}= K(x, y)  (3)

The system Ψ_{x} is named by supercompete basis or overfilled system.

Let h ∈ H. We consider a function f_{h}(x) = Jh(x) on B_{p,q} given by formula

JF (x) = f_{h}(x) =< h, Ψ_{x} >  (4)

where x ∈ Ξ.

We will identify the space H and the image of the operator J. In other words we consider the hilbert space H as a space of functions on Ξ.

2.2. Hilbert spaces of holomorphic functions. Let Ω be a open domain in ℂ^{n}. A function K(z, u) on Ω × Ω is called by a reproducing kernel on Ω if

1. K(z, u) is a positive definite kernel

2. Function (z, u) is holomorphic with respect to variables z and antiholomorphic with respect to varables u.

Consider associated hilbert space H and let us realize it as a space of functions on Ω by the formula (4) . It is easy to see that functions f_{h} are holomorphic, i.e. hilbert space H can be considered as a hilbert space of holomorphic functions on Ω.

2.3. Example: Berezin spaces on Cartan domains. Let p ≤ q. Denote by B_{p,q} the space of all p × q-matrices z satisfying the condition ∥z∥ < 1, i.e B_{p,q} is a Cartan domain of the first type.

A pseudounitary group U(p, q) consists of (p + q) × (p + q)-matrices g =

( \begin{matrix}a & b \\ c & d \end{matrix} ) satisfying the condition

g · ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) · g^{∗} = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} )  (5)

The group U(p, q) acts on B_{p,q} by the fractional-linear transformations

z ↦ z^{[g]} := (a + zc)^{−1}(b + zd)  (6)

The stabilizer of the point z = 0 consists of matrices having the form ( \begin{matrix}a & 0 \\ 0 & d \end{matrix} )

where a ∈ U(p), d ∈ U(q). Hence B_{p,q} is the riemann symmetric space

U(p, q)/(U(p) × U(q))

Theorem 2.1. (see [Ber1], see also [Wal]) Let s = 0, 1, 2, …, p − 1 or s > p − 1. Then the kernel

K_{s}(z, u) = det^{−s}(1 − zu^{∗})

is positive defined on B_{p,q}.

Denote by V_{s} the hilbert space of holomorphic functions on B_{p,q} defined by the kernel K_{s}(z, u) (for discussion of this space see for instance [Ber1], [NO]). The group U(p, q) acts in V_{s} by a unitary operators

T_{s} ( \begin{matrix}a & b \\ c & d \end{matrix} ) = f(z^{[g]})det^{−s}(a + zc)  (7)

where z^{[g]} is given by the formula (6).

The representations T_{s} are called by spherical highest weight representations of U(p, q).

Remark. If s is integer then T_{s} is a representation of the group U(p, q) itself. If s is not integer then the function det^{−s} is multivalued and hence T_{s} is a representation of the universal covering group of U(p, q) (in other words the representation T_{s} is a projective representation of the group U(p, q) ).

Remark. If s > p − 1 then the space V_{s} contains all polynomial functions on Cartan domain B_{p,q}. If s = 0, 1, …, p − 1 when all functions f ∈ V_{s} satisfies to the following system of partial differential equations

[det ( \begin{matrix}\frac{∂}{∂z_{i_{1}j_{1}}} & … & \frac{∂}{∂z_{iαj_{1}}} \\ ⋮ & ... & ⋮ \\ \frac{∂}{∂z_{i_{1}jα}} & … & \frac{∂}{∂z_{iαjα}} \end{matrix} )] f(z) = 0

for all α = 2, 3, …, s + 1 and all i_{μ}, j_{ν} satisfying unequalities 0 ≤ i_{μ} ≤ q, O ≤ j_{ν} ≤ p.

2.4. Cartan domains of second and third types. A Cartan domain C_{n} of the third type is the set of all n × n symmetric matrices z with complex coeffitients satisfying the condition ∥z∥ < 1. The group G of biholomorphic transformations (automorphisms) of the domain C_{n} consists of transformations

having form (6) where a matrix g = ( \begin{matrix}a & b \\ c & d \end{matrix} ) satisfies the condition

( \begin{matrix}a & b \\ c & d \end{matrix} ) ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) ( \begin{matrix}a & b \\ c & d \end{matrix} )^{t} = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ); d = \overline{a}, c = \overline{b} (8)

It is easy to see that the group G is isomorphic to a real symplectic group Sp(2n, ℝ) and the domain C_{n} is a riemann symmetric space

C_{n} = Sp(2n, ℝ)/U(n)

A kernel

K_{s}(z, u) := det^{−s}(1 − z\overline{u})

is positive definite if and only if

{s = 0, 1/2, …, (n − 1)/2}or{s > (n − 1)/2}

A Cartan domain D_{n} of the second type is a set of all n × n skew-symmetric matrices z satisfying the condition ∥z∥ < 1. Group G of automorphisms of the

domain D_{n} is a group of transformations (6) where a matrix g = ( \begin{matrix}a & b \\ c & d \end{matrix} )

satisfies the condition

( \begin{matrix}a & b \\ c & d \end{matrix} ) ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ) ( \begin{matrix}a & b \\ c & d \end{matrix} )^{t} = ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ); d = \overline{a}, c = \overline{b}  (9)

Remark. Condition (9) implies (5) and also (9) implies (5).

The group G is isomorphic to a real classical group SO^{∗}(2n) and the domain D_{n} is the symmetric space

D_{n} = SO^{∗}(2n)/U(n)

The kernel K(z, u) = det^{−s}(1 − zu^{∗}) on D_{n} is positive defined if s = 0, 1, 2, …, n − 1 or s > n − 1.

Spherical highest weight representations of Sp(2n, ℝ) and SO^{∗}(2n) act in hilbert spaces associated to the kernels K_{s} by the formula (7).

2.5. Matrix-valued positive definite kernels. Consider a set Ξ and a finite dimensional complex euclidian space Y. Denote by GL(Y) the group of invertible linear operators in Y. We say that a function

L : Ξ × Ξ → GL(ℂ, Y)

is a matrix-valued positive definite kernel if a function

L̃((z, ξ); (u, η)) :=< L(z, u)ξ, η > ; (z, ξ), (u, η) ∈ Ξ × Y

is a positive defined kernel on Ξ × Y.

Let we have a matrix-valued positive defined kernel on Ξ. Then there exists a unique hilbert space H and a map Ψ : Ξ × Y → H such that

a) The map Ψ is linear on each fibre z × Y ⊂ Ξ × Y.

b) < Ψ(z, ξ), Ψ(u, η) >_{H}=< L(z, u)ξ, η >_{Y}

c) The image of map Ψ is dence in H.

Again there exists a canonical map J from the hilbert space H to space of Y-valued functions on Ξ. Let h ∈ H. Then the function f_{h} = Jf is defined from the equation

< f_{h}, ξ >_{Y}=< h, Ψ(z, ξ) >_{H}

2.6. Highest weight representations of the groups G = U(p, q), Sp(2n, ℝ), SO^{∗}(2n).

Consider the case G = U(p, q). Consider a irreducible finite-dimensional holomorphic representation ρ = ρ_{1} ⊗ ρ_{2} of the group GL(p, ℂ) × GL(q, ℂ) or of its universal covering. Let W = W_{1} ⊗ W_{2} be the space of the representation ρ. Assume that function

K_{ρ}(z, u) := ρ_{1}(1 − zu^{∗}) ⊗ ρ_{2}(1 − u^{∗}z)

is a matrix-valued positive-definite kernel (see [Ols2] for conditions of positive definiteness, see also [NO]). Consider associated hilbert space H_{ρ} of W_{1} ⊗ W_{2}- valued holomorphic functions on B_{p,q}. Then unitary highest weight representation T_{ρ} of the group U(p, q) (or of its universal covering) acts in the space H_{ρ} by the formula

T_{ρ} ( \begin{matrix}a & b \\ c & d \end{matrix} ) F (z) = (ρ_{1}(a + zc) ⊗ (b − dz^{[g]}))F (z^{[g]})  (10)

Let G = Sp(2n, ℝ) or SO^{∗}(2n). Consider a finite dimensional holomorphic representation ρ of group GL(n, ℂ) or of its universal covering group. Assume matrix-valued kernel K_{ρ}(z, u) = det^{−s}(1 − zu^{∗}) be positive-definite on the Cartan domain C_{n} or D_{n}. Denote by H_{ρ} associated hilbert space of holomorphic vector-valued functions. Then unitary highest weight representation T_{ρ} of the group G is given by the formula

T_{ρ} ( \begin{matrix}a & b \\ c & d \end{matrix} ) f(z) = ρ(a + zc)F (z^{[g]})

2.7. Bibliographical comments.

a) There are also Cartan domains of 4-th type SO(n, 2)/(SO(n) × O(2)) and two exeptional Cartan domains. There exists associated highest weight representation theory. Cartan domains SO(n, 2)/(O(n) × O(2)) are popular object of compex analysis ( tube of future ), see [SV]. These domains often are called by Lie spheres or Lie balls, but this term is a result of philological mistake (it is a result of the following sequence of translations: german-chinese-russian-english).

b) Conditions of positive-definiteness of standard scalar-valued kernels on Cartan domains of 1-4 typpes were discovered by Berezin [Ber] (with some additions given by S.G.Gindikin; Berezin and Gindikin didn’t consider exeptional domains; later conditions of positive-defineteness in general case (including exeptional) were obtained by Wallach [Wal] and Rossi - Vergne).

c) Conditions of positive-defineteness of standard kernels for B_{p,q} where obtained in [Ols2]. For other domains this problem was discussed by many authors, the final result was obtained in [EHW].

3 Boundary values of holomorphic functions.

3.1. L^{1}-limits

Let Ω ⊂ ℂ^{n} be a open domain, let ∂Ω be its boundary, let \overline{Ω} be the closure of Ω. We say that Ω is a regular circle domain if

a) for all z ∈ Ω and λ ∈ ℂ such that |λ| ≤ 1 we have λz ∈ Ω

b) for all z ∈ ∂Ω and λ ∈ ℂ such that |λ| < 1 we have λz ∈ Ω

Let K(z, u) be a reproducing kernel in Ω satisfying the condition

K(e^{iϕ}z, e^{iϕ}u) = K(z, u)  (11)

Let H be the hilbert space of holomorphic functions associated to this kernel.

Remark. Cartan domains are regular circle domains and Berezin kernels satisfy the condition (11)

Theorem 3.1.(see [NO]) Let M ⊂ ∂Ω be a compact subset. Let μ be a measure supported on M. Let

a)K^{∗}(z, u) := lim_{c→1−0} K(cz, cu) exist almost sure on M × M with respect to the measure μ × μ.

b)K^{∗} ∈ L^{1}(M × M, μ × μ) and lim_{c→1−0} K(cz, cu) is dominated, i.e. there exists a function S(z, u) ∈ L^{1}(M × M, μ × μ) such that |K(z, u)| < S(z, u) almost sure on M × M .

Then the operator of restriction of a function f ∈ H on the set M is welldefined operator

H → L^{1}(M, μ)

This theorem doesn’t cover all cases then the restriction operator exists (see below subsections 3.2-3.3) . Hence the following natural problem arises.

Problem. Let Ω ⊂ ℂ^{n} be a a open domain. Let K(z, u) be a reproducing kernel and let H be the associated hilbert space. Let M be a submanifold in the Shilov boundary of Ω. Find conditions for existence of restriction operator from H to some functional hilbert space on M.

Remark. The case which is interesting for harmonic analysis is the followind case. Let the domain Ω be a Cartan domain G/K and H = V_{s} be a Berezin space. Consider a subgroup Q ⊂ G and some Q-orbit M in Shilov boundary of domain Ω. Let the operator of restriction of function f ∈ H_{s} to the manifold M exists. Then we obtain a interwinning operator from the Berezin space H_{s} to some hilbert space H(M) of functions on M. The representation of Q in H(M) is equivalent to a subrepresentation in representation of Q in H. If the group Q is semisimple and orbit M is compact then representation of Q in H(M) has purely discrete spectrum.

3.2. Restrictions of functions of polynomial grouth in a ball. Denote by B_{q} the unit ball

|z_{1}|^{2} + … + |z_{q}|^{2} < 1

in ℂ^{q}. Denote by H^{∞}(B_{q}) Hardy space of bounded holomorphic functions on B_{q}. Denote by ⟨·, ·⟩ be standard scalar product in ℂ^{n}.

Let γ(t) be a C^{1}-curve in the sphere ∂B^{q} = S^{2q−1}.

Theorem 3.2(see [NR],[Rud]) Let γ(t) satisfies the condition

∀t : Im⟨γ(t), γ′(t)⟩ ≠ 0  (12)

Then for each f ∈ H^{∞}(B^{q}) the nontangent limit f(z) as z → γ(t) exists almost sure on the curve γ(t).

Remark. Obviously vector γ′(t) is tangent to the sphere S^{2q−1}. Hence we have Re⟨γ(t), γ′(t)⟩ = 0 for all t. Fix z ∈ S^{2q−1}, let v ∈ ℂ^{q}be a vector. The equation ⟨z, v⟩ = 0 defines a subspace of codimension 1 in a tangent space to the sphere S^{2q−1} in a point z ∈ §^{2q−1} ⊂ ℂ^{n}. We obtain a contact distribution on sphere (see [Rud], 5.4, 10.5). The condition (12) means that γ(t) is transversal to this distribution.

Denote by D′ the space of all holomorphic functions of polynomial grouth in B^{q}:

f ∈ D′ ⇔ ∃N : sup |f(z)|(1 − |z|^{2})^{N} < ∞

Remark. The space D′ is a union D′ = ∪_{s>0}V_{s}(B_{q}) of Berezin spaces in balls, see subsection 2.3. The space V_{q}(B_{q}) coincides with Hardy space H^{2}(B_{q}). The space V_{q+1}(B_{q}) is Bergman space (i.e intersection of L^{2}(B_{q}) and the space of holomorphic functions).

Remark. It is well known (and more or less obvious) that each function f ∈ D′ has limit on the boundary in the sence of distributions (see [RS],IX.3 for discussion of such type theorems and references).

Denote by R operator of resctriction of function f holomorphic in the ball B_{q} to the curve γ(t). Obviously function Rf is well-defined if the function f is continuous up to the boundary.

Theorem 3.3.(see [Ner2]) Let γ(t) be a C^{∞}-smooth curve in ∂B^{q} satisfying the condition (12). Then the operator R of restriction of holomorphic function to γ(t) extends to a bounded operator from D′ to the space of distributions on γ(t) Moreover distrubution Rf coincides with the limit

lim_{ϵ→+0} f(1 − ϵ)z

(where z ∈ S^{2q−1}) in the sence of distributions.

3.3. Restriction of functions of polynomial grouth in polydisk. Denote by P^{n} a polydisk |z_{1}| < 1, …, |z_{n}| < 1. Let T^{n} be the torus z_{1} =

e^{iϕ_{1}}, …, z_{n} = e^{iϕ_{n}} . Let γ(t) = (ϕ_{1}(t), …, ϕ_{n}(t)) be a time-like C^{∞}-curve in T^{n} i.e.

∀t : ϕ′_{1}(t) > 0, …, ϕ′_{n}(t) > 0  (13)

Denote by D′ the space of holomorhic functions of polynomial grouth in P^{n}, i.e

f ∈ D′ ⇔ ∃N : sup f(z) ∏_{k=1}^{n}(1 − |z_{k}|)^{N} < ∞

This condition is equivalent to the condition

f(z_{1}, …, z_{n}) = ∑ c_{k_{1},...,k_{n}}z_{1}^{k_{1}} … z_{n}^{k_{n}} ∈ D′ ⇔

⇔ ∃L : ∑ \frac{|c_{k_{1},...,k_{n}}|}{[(k_{1} + 1) … (k_{n} + 1)]^{L}} < ∞

Theorem 3.4.(see [Ner2]) Operator R of restriction of holomorphic function f on P^{n} to the curve γ(t) extends to a bounded operator from the space D′ to the space of distributions on the curve γ(t). Moreover Rf coincides with the radial limit

lim_{ϵ→+0} f((1 − ϵ)ϕ_{1}(t), …, (1 − ϵ)ϕ_{n}(t))

(where z ∈ T^{q}) in the sence of distributions.

Example. Consider a bidisk P^{2} and the hilbert space H_{s_{1},s_{2}} associated to the reproducing kernel

(1 − z_{1}\overline{u}_{1})^{−s_{1}}(1 − z_{2}\overline{u}_{2})^{−s_{2}}

where s_{1}, s_{2} > 0. It is easy to show that D′ = ∪_{s_{1},s_{2}>0}H_{s_{1},s_{2}}. If γ(t) has a form (γ_{1}(t), γ_{2}(t)) = (e^{it}, e^{iσ}) where σ = const, then operator of restriction to γ(t) doesn’t exist for all s_{1}, s_{2}. If γ(t) satisfies the condition γ′_{1}(t) > 0, γ′_{2}(t) < 0 then the restriction operator exists only in the case s_{1} + s_{2} < 1.

3.4. A conjecture It seems to me that Theorems 3.3-3.4 are partial cases of some general fact. Let Ω ⊂ ℂ^{N} be a open domain. Let M be a submanifold in the Shilov boundary of Ω. Denote by T_{m} the tangent space to M in the point m ∈ M. We identify T_{m} with a linear submanifold in ℂ^{n}. Denote by S_{m} = i·T_{m} the linear submanifold which consists of vectors

i · (v − m) + m

where v ∈ T_{m} (and i^{2} = −1). We say submanifold M is time-like if for each point m ∈ M there exists a open cone C_{m} ⊂ S_{m} with the vertex m and ϵ-neighbourhood U_{ϵ}(m) of m such that

C_{m} ∩ Ω ⊃ U_{ϵ}(m) ∩ C_{m}

Conjecture.Each holomorphic function of polynomial grouth in Ω has restriction to time-like manifold M in the Shilov boundary in the sense of distributions.

Theorems 3.3-3.4 are partial cases of this conjecture. It is also similiar to the standard facts on limits of functions of polynomial grouth on the whole Shilov boundary mentioned above (see [RS], IX.3, [Vla], [VS]). Neverless I couldn’t find this fact in the literature.

4 Positive defined kernels on riemann noncompact symmetric spaces

4.1. Spherical kern-representations of the groups U(p, q). Consider a matrix ball B_{p,q}, see subsection 2.3. Let s = 0, 1, 2, …, p − 1 or s > p − 1. Consider a function

L_{s}(z, u) = | det(1 − zu^{∗})|^{−2s}

where z, u ∈ B_{p,q}. With notations of subsection 2.3 we have

L_{s}(z, u) = K_{s}(z, u)\overline{K_{s}(z, u)}

and hence L_{s}(z, u) is a positive defined kernel on B_{p,q}. Denote by H_{s} the hilbert space associated with the kernel L_{s}(z, u). Using the formula (4) we identify this space with some space of real analytic functions on B_{p,q}.

The group U(p, q) acts in H_{s} by the unitary operators

A_{s}(g)f(z) = f(z^{[g]})| det(a + zc)|^{−2s}

where z^{[g]} is defined by the formula (6).

Problem. Decompose the repersentation A_{s}

We will name representations T_{s} by spherical kern-representations of the group U(p, q).

Some partial cases of this question were discussed in the end of 70-ies, see [Ber2],[Rep],[Gut]. In fact there was discussed only the case when s is large. In this case the most interesting phenomena don’t appear. Then such problems were more or less forgotten. In last several years some this problem attracted interest again (see [NO], [UU], [OØ], [OZ],[Dij]).

I would like to try to explain why this problem is interesting and also to discuss some approaches to this problem.

4.2. Limit as s → ∞. We want to show that the limit of kern-representations as s → +∞ is the canonical representation of U(p, q) in the space L^{2} on riemann symmetric space U(p, q)/(U(p) × U(q))

Concider the system of vectors Ψ_{z} ∈ H_{s} (see subsection 2.2). Let χ be a distribution in B_{p,q} with a compact support. Consider the vector Θ(χ) ∈ H_{s} defined by the equality

Θ(χ) = ∫_{B_{p,q}} |det^{s}(1 − zz^{∗})|χ(z)Ψ_{z}dzd\overline{z}

Consider a scalar product {·, ·}_{s} in the spase of distributions on B_{p,q} with compact support given by the formula

{χ_{1}, χ_{2}}_{s} := ⟨Θ(χ_{1}), Θ(χ_{2})⟩ =

= ∫_{B_{p,q}} ∫_{B_{p,q}} |\frac{det(1 − zz^{∗}) det(1 − uu^{∗})}{det^{2}(1 − zu^{∗})}|^{s}χ_{1}(z)χ_{2}(u)dzd\overline{z}dud\overline{u}

We can identify the space H_{s} with the completion of the space of distributions with respect to scalar produt {·, ·}_{s}. The group U(p, q) acts in this space of distributions by the formula

B_{s}(g)f(z) = f(z^{[g]})

(the formula doesn’t depend of s, neverless the scalar product and spectrum of representation depend on s essentially)

It appeared that for some normalizing multiplier ω(s) for all continuous functions ϕ_{1}(z), ϕ_{2}(z) on B_{p,q} with compact support we have

lim_{s→+∞} ω(s)\{ϕ_{1}, ϕ_{2}\}_{s} = ∫_{B_{p,q}} ϕ_{1}(z)ϕ_{2}(z)dzd\overline{z}

4.3. Another formulation of the problem. Consider a spherical highest weight representation T_{s} of the group U(p, q) (see subsection 2.3). Denote by V_{s} the space of representation T_{s}. Denote by T_{s}^{∗} the representation contragradient to T_{s}. Consider the tensor product T_{s} ⊗ T_{s}^{∗} . This representation acts in the space V_{s} ⊗ V_{s} of holomorphic functions on B_{p,q} × B_{p,q} by the operators

(T_{s} ⊗ T_{s}^{∗})(g) = f(z_{1}, z_{2}) = f(z_{1}^{[g]}, z_{2}^{[\overline{g}]})(a + z_{1}c)^{−s}(\overline{a} + z_{2}\overline{c})^{−s}

The U(p, q)-invariant scalar product in the space of holomorphic functions on B_{p,q} × B_{p,q} is defined by a reproducing kernel

M(z_{1}, z_{2}; u_{1}, u_{2}) = det^{−s}(1 − z_{1}u_{1}^{∗})det^{−s}(1 − z_{2}u_{2}^{∗})

Consider a operator

I : V_{s} ⊗ V_{s} → H_{s}

defined by the formula

If(z) = f(z, \overline{z})

Obviously I is a unitary operator interwining representations

T_{s} ⊗ T_{s}^{∗} ↔ A_{s}

Hence we can formulate our problem in the form:

Problem.Decompose the tensor product T_{s} ⊗ T_{s}^{∗}.

4.4. Orbits of the group U(p, q) on the Shilov boundary.

Denote by M_{p,q} Shilov boundary of B_{p,q}. Elements of M_{p,q} are matrices z satisfying the condition

z · z^{∗} = 1

In the other words z is a matrix of a isometric embedding ℂ^{p} → ℂ^{q}. Hence M_{p,q} is a complex Stiefel manifold.

Shilov boundary of B_{p,q} ×B_{p,q} is M_{p,q} ×M_{p,q} . The group U(p, q) has (p+1) orbits on M_{p,q} × M_{p,q}. The unique invariant of a orbit is number

α = rk(z − \overline{u})  (14)

i.e if (z_{1}, u_{1}), (z_{2}, u_{2}) ∈ M_{p,q} × M_{p,q} and rk(z_{1} − \overline{u}_{1}) = rk(z_{2} − \overline{u}_{2}) when points (z_{)}, u_{1}), (z_{2}, u_{2}) are elements of one orbit.

We denote by Ξ_{α} the orbit corresponding to a given invariant α.

Remark. Orbit Ξ_{0} is compact, orbit Ξ_{p} is open (in Shilov boundary). For all α the closure of Ξ_{α} is ∪_{σ≤α}Ξ_{σ}.

4.5. Restriction of holomorphic functions to U(p, q)-orbits in the Shilov boundary. Fix a orbit Ξ_{α} of U(p, q) in the Shilov boundary of B_{p,q} × B_{p,q}. It can happens (and it realy happens) that for small s function f ∈ H_{s} = V_{s} ⊗ V_{s} has well defined restriction to the orbit Ξ_{α}. In this case the restriction operator is a interwinning operator from V_{s} × V_{s} to some hilbert space of functions on Ξ_{α}.

It can also happened (and it realy happens) that for small s all first partial derivatives of function f ∈ H_{s} = V_{s} ⊗ V_{s} have well defined restriction to the orbit Ξ_{α} etc.(see discussion of this phenomenon in [NO],section 7).

Fix s. For each α = 0, 1, …, p − 1 consder the maximal number τ_{α} such that all partial derivatives of order ≤ τ_{α} of functions f ∈ V_{s} ⊗ V_{s} have welldefined restrictions to Ξ_{α}. (these numbers aren’t known, but Theorem 3.1 give a possibility to estimate them; I don’t know are such estimates strict or not). If restrictions of functions f ∈ H_{s} = V_{s}⊗V_{s} to Ξ_{α} don’t exist we suppose τ_{α} = −1.

Remark. For large s restriction operators don’t exist, i.e. we have τ_{α} = −1 for all α < p − 1.

Fix α = 0, 1, … p − 1. Consider some i_{α} = 1, ⋯, τ_{α}. Denote by Q[α, i_{α}] the space of functions f ∈ V_{s} ⊗ V_{s} such that all partial derivatives of f of orders ≤ i equals zero on Ξ_{α}. We obtain a filtration

\begin{matrix}0 ⊂ Q[p − 1, τ_{p−1}] ⊂ Q[p − 1, τ_{p−1} − 1] ⊂ … ⊂ Q[p − 1, 0] ⊂ \\ ⊂ Q[p − 2, τ_{p−2}] ⊂ … ⊂ Q[p − 2, 1] ⊂ Q[p − 2, 0] ⊂ … \\ … ⊂ Q[0, τ_{0}] ⊂ … Q[0, 1] ⊂ Q[0, 0] ⊂ V_{s} ⊗ V_{s} \end{matrix}  (15)

Remark. For large s this filtration is trivial, neverless for small s it is quite long.

Consider representations of U(p, q) in subquotients of this filtration. Obviously A_{s} = T_{s} ⊗ T_{s}^{∗} is equivalent to the direct sum of the subquotients of the filtration (15).

Remark. The representations of U(p, q) in the subquotients have simple interpretations. For instance (V_{s} ⊗ V_{s})/Q[0, 0] is a subspace in the space of functions on the orbit Ξ_{0}. The space Q[0, 1]/Q[0, 0] is a subspace in space of sections of normal bundle to the orbit Ξ_{0} . The space Q[0, 1]/Q[0, 2] is a subspace in space of sections of symmetric square of normal bundle etc., see discussion in [NO],$ 7.

It is natural to hope that spectrum in each subquotient is more or less ’uniform’, i.e. orbit structure of Shilov boundary give separation of quite complicated spectrum of A_{s} to the different types (compaire with [GG]-project )

4.6. Large s. If s is large enough then the restriction operators don’t exist. In this case the representation A_{s} is equivalent to standart representation of the group U(p, q) in L^{2} on riemann symmetric space U(p, q)/(U(p) × U(q)) , see [Ber2], [Rep], [Gut], [OØ]. Sufficient (not nessessary) condition for this is s > p + q − 1 (i.e. T_{s} is a element of Harish-Chandra discrete seriees).

4.6. Restriction to the compact orbit. The part of spectrum which corresponds to the compact orbit Ξ_{0} is purely discrete and it consists of quite exotic representations of U(p, q). This part of spectrum is not empty for small s (for s < (q − 2p − 1)/4), see in [NO] description of analogical situation for compact orbit of O(p, q) on the Shilov boundary of B_{p,q}.

In particular it gives a relatively simple way for constructing singular unitary representations of the groups U(p, q), see [NO].

4.7. Spherical kern-representations of other classical groups All classical riemann noncompact symmetric spaces G/K (up to centrum of G) can be realized as (real, complex or quaternionic) matrix balls (see [Ner3]). Namely the space G/K is the space of matrices z over the field 𝕂 = ℝ, ℂ, ℍ (see below) satisfying the additional condition (see below) such that ∥z∥ < 1.

Now we are enumerate symmetric spaces G/K, fields and additional conditions.

1^{∗}. U(p, q)/(U(p) × U(q)) is the space of p × q matrices over ℂ.

2^{∗} . Sp(2n, ℝ)/U(n) is the space of symmetric n × n-matrices over ℂ.

3^{∗}. SO^{∗}(2n)/U(n) is the space of skew symmetric n × n-matrices over ℂ. 4^{∗}. O(p, q)/(O(p) × O(q)) is the space of p × q-matrices over ℝ. 5^{∗}. GL(n, ℝ)/O(n) is the space of symmetric n × n-matrices over ℝ. 6^{∗}. O(n, ℂ)/O(n) is the space of skew-symmetric n × n-matrices over ℝ. 7^{∗}. GL(n, ℂ)/U(n) is the space of hermitian n × n-matrices over ℂ. 8^{∗}. Sp(p, q)/(Sp(p) × Sp(q)) is the space of p × q-matrices over ℍ. 9^{∗}. GL(n, ℍ)/Sp(p, q) is the space of hermitian n × n-matrices over ℍ. 10^{∗}. Sp(2n, ℂ)/Sp(n) is the space of skew hermitian (i.e. z = −z^{∗}) n × n-

matrices over ℍ.

In all cases enumerated above the group G acts on the matrix ball G/K by fractional-linear transformations

z ↦ z^{[g]} := (a + zc)^{−1}(b + zd)  (16)

Now let us consider positive definite kernels on G/K having a form

L_{s}(z, u) = | det(1 − zu^{∗})|^{−2s}

(conditions for positive-definiteness are different for different spaces). Consider the hilbert space defined by the positive defined kernel L_{s}(z, u). We identify this space with some space of real analytic functions on G/K by the same way as in 2.1. The group G acts in H_{s} by the unitary operators

A_{s}(g)f(z) = f(z^{[g]})| det(a + zc)|^{−2s}

We say A_{s} is a spherical kern-representation of G.

4.8. Nonspherical kern-representations.

Consider a symmetric spaces having the form 2^{∗}, 3^{∗}, 5^{∗}, 6^{∗}, 7^{∗}, 9^{∗}, 10^{∗}. Consider a finite dimensional irreducible representation ρ of the group GL(n, 𝕂) or of its universal covering. Assume that the function

L(z, u) = ρ(1 − zu^{∗})

is a matrix valued positive definite kernel. Then we consider the associated hilbert space H_{ρ} of real-analytic functions G/K → Y and the unitary kernrepresentation of G in H_{ρ} given by the formula

T_{ρ}(g)f(z) = ρ(a + zc)f(z^{[g]})  (17)

Consider the cases 1^{∗}, 4^{∗}, 9^{∗}. Consider a finite-dimensional irreducible representation ρ = ρ_{1} ⊗ ρ_{2} of the group GL(p, 𝕂) × GL(q, 𝕂) or of its universal covering. Assume that the function

L_{ρ}(z, u) := ρ_{1}(1 − zu^{∗}) ⊗ ρ_{2}(1 − u^{∗}z)

is a positive defined matrix-valued kernel on G/K. Then the group G acts in the associated space of real-analytic functions on G/K by the formula

T_{ρ}(g)f(z) = (ρ_{1}(a + zc) ⊗ ρ_{2}(d − cz^{[g]}))f(z^{[g]})

Remark. Our arguments from subsections 4.3-4.4 are valid for general kernrepresentations.

4.9. Another description of kern-representations.(see [OØ]) For G = Sp(2n, ℝ) ,U(p, q), SO^{∗}(2n) a kern representation is a tensor product of irreducible highest weight representation of G and irreducible lowest weight representation of G.

In other cases a kern representation of G is a restriction of a highest weight representation of the group G^{∗} to the symmetric subgroup G:

4^{∗}. G = O(p, q)  G^{∗} = U(p, q) 5^{∗}. G = GL(n, ℝ) G^{∗} = Sp(2n, ℝ)
6^{∗}. G = O(n, ℂ) G^{∗} = SO^{∗}(2n) 7^{∗} G = GL(n, ℂ) G^{∗} = U(n, n)
8^{∗} G = Sp(p, q) G^{∗} = U(2p, 2q) 9^{∗} G = GL(n, ℍ) G^{∗} = SO^{∗}(2n)
10^{∗} G = Sp(2n, ℂ) G^{∗} = Sp(4n, ℝ)

Remark. The cases 1^{∗} − 3^{∗} can be described by the same way. We have G^{∗} = G×G and the embedding G → G^{∗} given by the formula g ↦ (g, g^{θ}) where θ is a outer automorphism of G.

Remark. There are some additional possibilities related to highest weight representations of O(p, 2) and two exeptional groups (see [OØ]).

4.10. Remark: action of Olshanskii semigroup. For each matrix

ball G/K consider the set of invertible matrices g = ( \begin{matrix}a & b \\ c & d \end{matrix} ) such that

the fractional-linear mapping (16) maps the matrix ball to itself. Obviously Γ is a semigroup and the group G is the group of invertible elements of Γ. The formula (17) defines representation of semigroup Γ. This representation is irreducible and all irreducible representations of Γ can be obtaned by this way(see[Ols1]). See [Ols1], [Ner3],[Ner4],appendix A, for the explicit description of semigroups Γ. Moreover kern-representations extends to representations of some categories(see[Ner3],[Ner4],appendix A). I don’t know any applications of these phenomena to harmonic analysis of kern-representations of groups.

4.11. Bibliographical comments.

a) Let T be a highest weight representation of G = Sp(2n, ℝ), SO^{∗}(2n) , U(p, q), … and S be a lowest weight representation of G. Assume that T, S be elements of Harish-Chandra discrete series. Then T ⊗ S is equivalent to a representation of G induced from irreducible representation of maximal compact subgroup K ⊂ G (see [Ber2],[Rep],[Gut]).

b) Spherical kern-representations for large s are equivalent to canonical representation of G in L^{2}(G/K),see [OØ].

c) Discrete spectra associated to the compact orbit in Shilov boundary were investigated in [NO] for the case G = O(p, q). Analogical results are valid for G = U(p, q), Sp(p, q) ([Ols2], [NO], 7.12.). One method of separation of discrete spectrum is discussed in [Ner1], [NO],7.1-7.8. I think that the restriction operator to the compact orbit doesn’t exist for G ≠ U(p, q), Sp(p, q), O(p, q).

d) Spectra for spherical kern-representations of U(2, 2) were obtained in [OZ2] . For 1 < s < 3/2 the spectrum consist of two different pieces. One of pieces coincides with the spectrum of L^{2}(U(2, 2)/(U(2) × U(2)). Another piece is a integral of notrivial representations. It is natural to think (it is not proved) that this piece of spectrum is associated to noncompact U(2, 2)-orbit in the Shilov boundary of B_{2,2} × B_{2,2}. If it is so it is the unique known case when spectrum associated to noncompact orbit is observed. It is natural to think that such spectra exist in various spectral problems (not only for kernrepresentations).

e)Spectrum of spherical kern-representations of SU(1, 1) was obtained in [VGG] and in the case of U(p, 1) it was obtained in [Dij] (construction of discrete increments was observed in [Ner1]).

f) Plancherel formula for tensor product of highest weight and lowest weight (single-valued) representations of SL(2, ℝ) is obtained in [Mol5]. In [Ber2] there was announced theorem which is almost equivalent to the Plancherel formula for spherical kern-representations in the case then s is large and symmetric space is hermitian.

g) Nonspherical kern-representations have discrete spectrum which is not associated to compact orbit in the Shilov boundary. Some possibilities to observe it are contained in the following two sections. A way to observe Harish-Chandra discrete series increments using trace theorems is proposed in [Ner2].

h) Let ρ be the same as in 2.10. Let ρ_{s}(γ) = det^{−s}(γ)ρ(γ). Then limit of T_{ρ_{s}} as s → ∞ is the representation of G induced from finite dimensional representation of K (i.e the representation in sections of vector bundle on G/K). The theory of such representations is more or less equivalent to Harish-Chandra theory of L^{2}(G).

e) Deformation quantization, see [Ber2]. As we have seen in 4.2 H_{s} = V_{s}⊗V_{s}. Hence we can identify the space H_{s} and space of Hilbert-Schmidt operators V_{s} → V_{s}. Multiplication of Hilbert-Schmidt operators induces some associative operation in H_{s}. This operation can be concidered as so-called ”deformation quantization”, see [Ber3].

5 Dual pairs

5.1. Harmonic representation of Sp(2n, ℝ). We remind a definition of harmonic representation W_{2n} (= Weil representation = Segal-Shale-Weil representation = Friedrichs-Segal-Berezin-Shale-Weil representation = oscilator representation ) of the group Sp(2n, ℝ) (see [KV],[Ner4] for discussion of this representation).

Consider a hilbert space F_{n} of holomorpric (entire) functions on ℂ^{n} associated to the reproducing kernel exp(∑ z_{j}\overline{u}_{j}) (it is so-called boson Fock space).

Let ( \begin{matrix}Φ & Ψ \\ \overline{Ψ} & \overline{Φ} \end{matrix} ) ∈ Sp(2n, ℝ) (see the realization of Sp(2n, ℝ) described in sub-

section 2.4) The harmonic representation is a unitary projective representation of the group Sp(2n, ℝ) in the space F_{n} defined by the Berezin formula (see [Ner4]):

W ( \begin{matrix}Φ & Ψ \\ \overline{Ψ} & \overline{Φ} \end{matrix} ) f(z) =

= ∫_{ℂ^{ℕ}} exp\{(z \overline{u}) ( \begin{matrix}\overline{Ψ}Φ^{−1} & Φ^{t−1} \\ Φ^{−1} & −Φ^{−1}Ψ \end{matrix} ) ( z ; \overline{u} ) f(z) exp(−|z|^{2})dzd\overline{z}

5.2. Noncompact dual pairs. Consider the following subgroups in the symplectic group (noncompact Howe dual pairs):

Sp(2k(p + q), ℝ) ⊃ Sp(2k, ℝ) × O(p, q)
Sp(2(k + l)(p + q), ℝ) ⊃ U(k, l) × U(p, q)
Sp(4k(p + q), ℝ) ⊃ SO^{∗}(2k) × Sp(p, q)

Remark. Let us describe the first embedding. Consider a space ℝ^{2n} equipped with a nondegenerate skew-symmetric bilinear form. Consider a space ℝ^{p+q} equipped with symmetric bilinear form with inertia indices p, q. The tensor product of symmetric and skew-symmetric forms is a skew-symmetric form in the space ℝ^{2n} ⊗ ℝ^{p+q} and we obtain required embedding of groups. □

Let us restrict the harmonic representation W_{2N} of large symplectic group to these subgroups and then let us consider restriction to subgroups

O(p, q), Sp(2k, ℝ), U(k, l), U(p, q), SO^{∗}(2k), Sp(p, q)

It was proved in [How], [Ada], [Li] that the spectra of these restrictions have discrete increments (and moreover discrete spectra are exetremely rich). This construction is one of standard way to obtain singular unitary representations of the groups U(p, q), O(p, q), Sp(p, q).

Proposition 3.1. (see [NO]) Each representation of G = O(p, q), Sp(2k, ℝ), U(k, l), U(p, q), SO^{∗}(2k), Sp(p, q) which occurs in spectrum of a dual pair discretly (resp. weakly) occurs in spectrum of some kern-representation discretly (resp. weakly).

Proof. This proposition is more or less obvious. Concider for instance the case Sp(2k, ℝ) × O(p, q). The restriction of W_{2k(p+q)} to the subgroup

Sp(2k, ℝ) ⊂ Sp(2k, ℝ) × O(p, q) ⊂ Sp(2k(p + q), ℝ)

is equivalent to the representation

W_{2k}^{⊗p} ⊗ (W_{2k}^{∗})^{⊗q}  (18)

First tensor factor is a direct sum of highest weight representations and second tensor factor is a direct sum of lowest weight representations. Hence (18) is a direct sum of kern-representations. Consider the following subgroups in Sp(2k(p + q), ℝ):

Sp(2k(p + q), ℝ) ⊃ O(p, q) × Sp(2k, ℝ) ⊃ O(p, q)
∥  ∩  ∪  ∩
Sp(2k(p + q), ℝ) ⊃ U(p, q) ×  U(k)  ⊃ U(p, q)

The restriction of W_{2k(p+q)} to U(p, q) is a direct sum of highest weight representations and hence the restriction of W_{2k(p+q)} to O(p, q) is a direct sum of kern-representations.

5.3. Restriction to orbits.

Consider a Cartan domain C_{N} = Sp(2N, ℝ)/U(N). Consider a reproducing kernel

K(z, u) = det^{−1/2}(1 − zu^{∗})

on C_{N}. Denote by H the associated hilbert space. The group Sp(2N, ℝ) acts in H by the unitary operators

W_{2N}^{+}(g) = f(z^{[g]})det^{−1/2}(Φ + z\overline{Ψ})

The representation W_{2N}^{+} is one of two irreducible components of the representation W_{2N} (another irreducible component can be realized in some space of holomorphic vector-valued functions). Again we have question about restrictions of holomorphic functions to O(p, q) × Sp(2k, ℝ)-orbits in the Shilov boundary of C_{N}. The orbit structure of Shilov boundary in this case is very complicated. In any case there exists a orbit

F = Sp(2k, ℝ)/U(n) × O(p, q)/Q

where Q is stabilizer of maximal isotropic subspace in pseudoeuclidean space ℝ^{p+q}. I can show that operator of restriction to orbit F exists and this observation give a way to observe a part of discrete spectra for dual pair. It is interesting to calculate this part of spectrum.

Another question which seems interesting to me: is it possible to obtain by such way some hand realizations of some Harish-Chandra discrete series representations?

6 Space L^{2} on Stiefel manifolds

.

6.1. Stiefel manifolds. We name by Stiefel manifolds the following 10 types of homogeneous spaces G/Q:

1^{∘}. O(p, q)/O(p − t, q − s) 2^{∘}. U(p, q)/U(p − t, q − s)
3^{∘}. Sp(p, q)/Sp(p − t, q − s) 4^{∘}. Sp(2n, ℝ)/Sp(2(n − t), ℝ)
5^{∘}. Sp(2n, ℂ)/Sp(2(n − t), ℂ) 6^{∘}.  O(n, ℂ)/O(n − t, ℂ)
7^{∘}. SO^{∗}(2n)/SO^{∗}(2(n − t))

8^{∘} − 10^{∘}. The spaces of all linear embeddings

ℝ^{n−t} → ℝ^{n} ℂ^{n−t} → ℂ^{n} ℍ^{n−t} → ℍ^{n}

In the last 3 cases the group G is GL(n, ℝ),GL(n, ℂ),GL(n, ℍ) respectively and Q is the group of matrices having the form

( \begin{matrix}1_{s} & ∗ \\ 0 & ∗ \end{matrix} )  (19)

Remark. The Stiefel manifold Sp(2n, ℝ)/Sp(2(n − t), ℝ) is the space of isometric (form preserving) embeddings of the space ℝ^{2t} equipped with a nondegenerate skew symmetric bilinear form to the space ℝ^{2n} equipped with nondegenerated skew symmetric bilinear form. Other Stiefel manifolds 1^{∘} −7^{∘} have the analogical description.

6.2. Additional symmetries.

Consider the case G/Q = Sp(2n, ℝ)/Sp(2(n−t), ℝ). Then the group Sp(2t, ℝ) acts by the obvious way on the space of symplectic-isometric embeddings ℝ^{2t} → ℝ^{2n}(sinse it acts on the space ℝ^{2t}) . Hence the manifold Sp(2n, ℝ)/Sp(2(n − t), ℝ) is a Sp(2t, ℝ) × Sp(2n, ℝ)-homogeneous space:

Sp(2n, ℝ)/Sp(2(n−t), ℝ) = (Sp(2t, ℝ)×Sp(2n, ℝ))/(Sp(2t, ℝ)×Sp(2(n−t), ℝ))

Analogical additional group of symmetries exists in all cases 1^{∘} − 10^{∘}. These additional symmetries are useful since the spaces L^{2}(G/Q) have G-spectrum of infinite multiplicity.

6.3. Spectra of L^{2} on Stiefel manifolds. A few is known about spectral decomposition of L^{2}(G/Q). Neverless it is known that this problem is interesting. See [Sch] and [Kob1] for Flensted-Jensen type constructions of discrete spectra in L^{2} on

O(p, q)/O(p − r, q) U(p, q)/U(p − r, q) Sp(p, q)/Sp(p − r, q)

For the case r = 1 the Plancerel formula is obtained in [ØZ1]. Some consrtructions for discrete increments to spectra of

L^{2}(U(p, q)/(U(p − t, q − t) × U(p) × U(q))) ⊂ L^{2}(U(p, q)/(U(p) × U(q))) are contained in [RSW].

Remark. The cases G/Q where G = GL(n, ℝ),GL(n, ℂ),GL(n, ℍ) are very simple. We can consider two-step unitary induction from the subgroup Q (see (19)) to parabolic and from parabolic to the group GL(n) itself.

Proposition 6.1. Each representation of G which is contained in spectra L^{2}(G/Q) discretly (resp. weakly) is contained in specrtrum of some kernrepresentation of G discretly (resp. weakly).

Proof. We use arguments from [How],[NO]. Concider for instance the case G = Sp(2n, ℝ). The representation

W_{2n} ⊗ W_{2n}^{∗}

of Sp(2n, ℝ) is equivalent to representation of Sp(2n, ℝ) in L^{2}(ℝ^{2n}). Hence the representation

(W_{2n} ⊗ W_{2n}^{∗})^{⊗2k} = W_{2n}^{⊗2k} ⊗ (W_{2n}^{∗})^{⊗2k}

is equivalent to representation of Sp(2n, ℝ) in L^{2} on the space Mat_{2k,2n} of all 2k × 2n-matrices. A generic orbit of Sp(2n, ℝ) in Mat_{2k,2n} is a Stiefel manifold Sp(2n, ℝ)/Sp(2(n − k), ℝ).

Remark. For other groups G Proposition 6.1 can be proved by the same arguments. The basic observation is

W_{2n}|_{GL(n,ℝ)} ≃ L^{2}(ℝ^{⋉})

(see real model of harmonic representation in [KV]).

6.3. Some pseudoriemann symmetric spaces. By the obvious way we have

\begin{aligned}L^{2}(O(p, q)/(O(p) × O(p − r, q))) & ⊂ & L^{2}(O(p, q)/O(p − r, q)) \\ L^{2}(U(p, q)/(U(r) × U(p − r, q))) & ⊂ & L^{2}(U(p, q)/U(p − r, q)) \\ L^{2}(Sp(p, q)/(Sp(r) × Sp(p − r, q))) & ⊂ & L^{2}(Sp(p, q)/Sp(p − r, q)) \end{aligned}

and spectra of these spaces are contained in spectra of Stifel manifolds. I don’t know such embeddings of spectra for other pseudoriemannian symmetric spaces.

7 Miscellaneous

7.1 Molchanov representations In 1970 Molchanov [Mol2] investigated the most degenerate representations of the groups O(p, q). It is representations induced from maximal-dimensional parabolic subgroup and unitary subquotients of these induced representations. Molchanov representations have very simple K = O(p) × O(q)-spectra (spheric harmonics with multiplicities ≤ 1). By this reason Molchanov representations are very simple and hand objects. On another hand these representations are very singular in different sences: they have small rank in the sence of Howe (on How rank see [Li]) and their matrix coeffitients decreasing very slowly (on decreasing of matrix coeffitients see [CHH]).

Recently these representations attract interest again, see [HT]. The most degenerate Molchanov representations now are popular objects, see [BK], [Kos], [KØ] (they are minimal representations of O(p, q) in the sence of A.Joseph, see in [BK] discussion of minimal representations for O(p, q) and other series of simple groups; the most strange case is ’semi-exeptional’ group O(4, 4), see [Kos].

I only want to mention that Molchanov representations have very interesting spectral properties. Restrictions of Molchanov representations to the subgroup O(p, q − 1) were discussed in [Mol] and [KØ]. The problem is solved for continuous series of Molchanov representations (in this case we have continuous spectrum with discrete increments, see [Mol2]) and for the most degenerate representation (in this case spectrum is discrete for restriction to O(p, q − r), see [KØ]).

Questions. It seems to me that very interesting problem is decomposition of tensor product of two Molchanov representations. Another interesting question is ’hunting’ on discrete spectra in tensor products of several Molchanov representations (paper [Ner2] contains false remark on this subject).

Remark. It is possible to show that spectra of tensor products of Molchanov representations contains spectra of dual pairs. I don’t know relations between spectra of kern-representations and spectra of tensor products of Molchanov representations.

7.2. Representations unitary with respect to idefinite scalar product. There exist folk-lor (originated by Ismagilov and Molchanov) on harmonic analysis of representations unitary in indefinite hermitian scalar products (partially it is published, see [Ism], [Sul]). It is known that these representations often (more often than in definite unitary case) contains discret increments. It was observed in [Ner1] that these discrete increments usualy can be explained by restriction theorems. Here there are very much nonsolved problems (for instance let us consider negative s in our section 4). Neverless the main (well-known) question here in this moment is the following:

Question. Let a representation ρ of a semisimple group G unitary in indefinite metrics is given. When indefinite scalar product defines by some natural way a topology in space of functions?

The question is very dangerous and sophisticated (this is also well-known), neverless in some cases answer is positive (see [Ism], [Sul], []).

7.3. Harish-Chandra discrete series increments. Tools which are proposed in [NO] don’t give possibility to search Harish-Chandra discrete series increnents in unitary representations. Neverless posibilities to expose Harich-Chandra discrete series increments with restriction theorems exist (this is the subject of [Ner2]).

7.4. On totaly discrete spectra. In [Ols4], [Kob2] there were discovered a large classes of restriction problems which have totaly discrete spectra ( relations between these works are not clear to me, probably they have a open nondense intersection). It seems to me that it is interesting to understand are these spectra related to some functional-theoretical phenomena or not?

References

[Ada] Adams J. Discrete spectrum of reductive dual pair O(p, q), Sp(2m). Inv.math., 74(1983),449-475.

[Bar] Bari N.K. Treatise on trigonometric series. Fizmatgiz (1961) (Russian) ; English translation: Pergamon Press (1964)

[Ber1] Berezin F.A. Quantization in compex symmetric spaces Izv. Akad. Nauk SSSR,Ser.Mat.,39(1975) (Russian); English translation Math.in USSR Izv. 9(1975)

[Ber2] Berezin F.A. A connection between the co- and contravariant symbols of operators on classical symmetric spaces Dokl. Akad. Nauk SSSR, 241 (1978), N1(Russian); English translation in Sov.Math.Dokl.,19(1978),786-789.

[Ber3] Berezin F.A. General concept of quantization Comm.math.phys, 40 (1975), 153-174.

[Beu] Beurling A. Ensembles exeptionels. Acta Math., 72(1940), 1-13.

[Boy] Boyer Ch.P. On complementary series of SO(p, 1) J.Math.Phys.,14(1973), 609-617

[BK] Brylinski R., Kostant B. Minimal representations, geometric quantization and unitarity. Proc.Nac.Acad.Sci. USA, v.91(1994), 6026-6029

[BO] Bunke U., Oldrich M. Spectrum of Kleinian manifolds, to appear

[CHH] owling M., Haagerup U., Howe R. Almost L^{2}-coefficients. J. reine angew. math., 387 (1988), 97-110

[Dij] van Dijk G.Canonical representations, to appear.

[ESh] Egorov Yu.V., Shubin M.A. Partial differential equations-1 VINITI, 1987 (Russian) ; English translation: Enciclopaedia of Mathematical sciences, v.30, Partial differential equations-1.

[EHW] Enright T.J., Howe R., Wallach N. Classification of unitary highest weight modules. Representation theory of reductive groups , Birkhauser, 97-143 (1983).

[Far] Faraut J. Distributions spheriques sur les spaces hyperbolique. J.Math.Pure Appl. 58 (1979), 369-444

[Fef] Feferman C. Inequalites for strongly singular convolution operators. Acta Math.,124(1970), 9-36

[F-J1] Flensted-Jensen M. Discrete series for semisimple symmetric spaces, Ann.Math.,111(1980), 253-311

[F-J2] Flensted-Jensen M. Analysis on Non-Riemanian symmetric spaces Regional Conference series in Math.61, AMS, Providence, 1986

[GG] Gelfand I.M., Gindikin S.G.Complex manifolds whose skeletons are semisimple Lie groups, and analytic discrete series of representations. Funkt. Anal. i Prilozh. 11(1977),N4 (Russian) ; English translation: Funct. Anal. Appl. 11,258-265

[Gut] Gutkin E. Coeffitients of Clebsh-Gordon for the holomorphic discrete series. Lett.Math.Phys.3(1979),185-192.

[H-Ch] Harish-Chandra Discrete series for semisimple Lie groups.I,II, Acta Math.,113(1965),241-318; Acta Math, 116(1966), 1-111.

[How] Howe R. On some results of Strishartz and of Rallis and Schiffmann J.Funct.Anal., 32(1979),297-303

[HT] Howe R., Tan Homogeneuos functions on light cones: the infinitesimal structure of some degenerate principal series representations. Bull.Amer.Math.Soc., 28(1993), 1-74.

[Ism] Ismagilov R.S. On the representations of Lorents group which are unitary in indefinite metric.. Trudy MIEM,2(1996), 492-504. (Russian)

[KV] Kashwara M., Vergne M. On Segal-Shale-Weil representation and the harmonic polynomials. Inv.Math.44 (1978) , 1-47

[Kob1] Kobayashi T. Singular unitary representations and discrete series for indefinite Stiefel manifolds. Mem. AMS,95 (1992)

[Kob2] Kobayashi T. Discrete decomposability of the restrictions of A_{𝔮}(λ) with respect to reductive subgroups and its applications. Inv.Math.117(1994), 181-205.

[KØ] Kobayashi T., Ørsted B. , to appear

[Kos] Kostant B. The vanishing of scalar curvature and the minimal representation of O(4, 4) in Operator algebras, unitary representations, enveloping algebras and invariant theory, eds Connes A., Duflo M., Joseph A., Rentschler R. (Birkhauser), 1990, pp.85-124.

[Kre] Krein M.G. Hermitian positive kernels on homogeneous spaces I,II Ukr.mat.zh.,1(1949),4,64-98; 2(1950),1,10-59 (Russian) ; English translation: in Translations of Amer. Math. Soc.

[Li] Li J.S. Singular unitary representations of classical groups. Inv.Math., 97(1989), 235-257.

[Mol1] Molchanov V.F. Analogue of the Plancherel formula for hyperboloids. Dokl.Akad.Nauk SSSR 171(1966) (Russian) ; English translation: Sov.Math.Dokl 7(1968),1382-1385

[Mol2] Molchanov V.F. Representations of pseudoorthogonal group related to cone. Mat.Sbornik, 81 (1970), 3,358-375 (Russian) ; English translation: Math.USSR Sbornik 10(1970), 333-347

[Mol3] Molchanov V.F. Restriction of complementary series representations to pseudoorthogonal group of lower dimension. Dokl.Akad.Nauk SSSR, 237(1977),N4 (Russian) ; English translation: Sov.Math.Dokl.18(1977), 1493-1497

[Mol4] Molchanov V.F. The Plancerel formula for hyperboloids. Trudy MIAN, 147 (1980) 65-85 (Russian) ; English translation: Proc. Steklov Inst. Math. 147, (1981),63-83

[Mol5] Molchanov V.F. Tensor products of unitary representations of 3dimensional Lorents group Izv.Akad.Nauk SSSR,ser.Mat. 43(1979), N4 (Russian) ; English translation: Math USSR Izv., 15 (1980), 113-143

[Nai] Naimark M.A. Decomposition of tensor product of irreducible representations of Lorents group into irreducible components.III Trudy Moscovskogo Matem. Obshchestva 10(1961), 181-216. (Russian)

[NR] Nagel A., Rudin W. Local boundary behavoir of bounded holomorphic functions. Can. J. Math. ,30(1978), 583-592

[Ner1] Neretin Yu.A. Representations of complementary series entering discretely in tensor products of unitary representations. Funkt. Anal. i Prilozh. 20(1986)N4,79-80 (Russian) ; English translation: Funct. Anal. Appl. 20, 68-70

[Ner2] Neretin Yu.A. Restriction of function which is holomorphic in a domain to a curve in boundary, and discrete SL(2, ℝ)-spectra. (to appear in Izv. Russ. Akad. Nauk)

[Ner3] Neretin Yu.A. Extensions of representations of classical groups to representations of categories. Algebra i Analiz 3(1992), N1 (Russian) ; English translation: St-Petersburg Math. J. 3(1992), 147-169 [Ner4] Neretin Yu.A. Categories of symmetries and infinite dimensional groups.

Oxford Univ. Press (1996)

[Ner5] Neretin Yu.A. Boundary values of holomorphic functions and discrete spectra in noncommutative harmonic analysis. Proceedings of school Analysis on homogeneous spaces, Tambov State University, 1997.

[NO] Neretin Yu.A., Olshanskii G.I. Boundary values of holomorphic functions, singular unitary representations of groups O(p, q), and their limits as q → ∞. Zap.Nauchn.Semin.POMIRAN, 223(1995) 9-91 (Russian) ; English translation: J.Math.Sci(1996).

[OØ] Olafsson G., Ørsted B. Bargmann transform for symmetric spaces (to appear)

[Ols1] Olshanskii G.I. Invariant cones in Lie algebra, Lie semigroups, and the holomorphic discrete series. Funkt. Anal. i Prilozh. 14(1980),N3 (Russian) ; English translation: Funct. Anal. Appl. 15(1981)

[Ols2] Olshanskii G.I. Description of highest weight unitary representations for groups U(p, q)^{∼}. Funkt. Anal. i Prilozh. 14 (1980), 3, 32-44 (Russian) ; English translation: , 14.

[Ols3] Olshanski G.I. Singular unitary representations of the groups O(p, q), U(p, q), Sp(p, q)(nonpublished notes)

[Ols4] Olshanskii G.I. Unitary irreducible representations of the groups U(p, q) which admit the pass to the limit as q → ∞ Zap.Nauchn.Semin.LOMI 172(1989), 114-220. (Russian) ; English translation: J.Sov. Math. (199 )

[OZ1] Ørsted B., Zhang G. L^{2}-version of Howe correspondence II. J.Math.Pure.Appl., 74(1995), 165-183.

[OZ2] Ørsted B., Zhang G. Tensor products of analytic continuations of discrete series. Can.J.Math.(to appear)

[Pat] Patterson S.J. The Laplacian operator on a riemann surface. I,II,III Compos.Math. 31(1975), 83-107, Compos.Math. 32(1976), 71-112, Compos.Math. 33(1977) , 227-259.

[Puk] Pukanszky L. On the Kronecker products of irreducible representations of 2 × 2 real unimodular group. Trans. Amer. Math. Soc.,100 (1961), 116-152

[RSW] Rawnsley J, Schmid W., Wolf J. Singular unitary representations and indefinite harmonoc theory J.Funct.Anal.,51(1983),1-114.

[RS] Reed M., Simon B. Methods of modern mathematical physics. II. Fourier analysis. Self-adjointness. Acad.Press (1975).

[Rep] Repka J. Tensor products of holomorphic discrete series. Can.J.Math. 31(1979), 836-844

[Rud] Rudin W. The function theory in the unit ball in ℂ^{n}. Springer(1980).

[Sch] Schlichtkrull H. A series of unitary irreducible representations induced from symmetric subgroup of a semisimple Lie group. Inv.Math.,68(1982), 497-516.

[Str] Strishartz R.S. Harmonic analysis oh hyperboloids J.Funct.Analysis, 12(1973), 341-383.

[Sul] Sultanov Sh.Sh. On tensor products of representations of the group SL_{2}(ℝ). Funkt. Anal. i Prilozh. 11 (1977), 4, 90-92 (Russian) ; English translation: Funct. Anal. Appl. 11

[Tsu] Tsuchikawa The Plancherel transform on SL_{2}(k) and its applications to tensor products of irreducible representations J.Kyoto Univ., 22(1982), 369-473.

[UU] Unterberger A., Upmeier H. The Berezin transform and invariant differential operators. Comm. Math. Phys., 164 (1994), 563-597

[VGG] Vershik A.M. , Gelfand I.M., Graev M.I. Representations of SL(2, ℝ where R is function ring Uspekhi Mat. Nauk, 28 (1973), 5, 83-128 (Russian) ; English translation: Russian Math. Survey, 28

[Vla] Vladimirov V.S. On constructions of envelopes of holomorphy for special domains and its applications. Trudy MIAN (Proceedings of Steklov Institute), 60(1961), 101-144 (Russian)

[VS] Vladimirov V.S., Sergeev A.G. Complex analysis in future tube. Complex analysis: several variables-2. Moscow:VINITI, 1985, 191-260 (Russian) ; English translation: in Enciclopaedia of mathematical sciences, v.8. Complex analysis. Several variables-2. Springer.

[Wal] Wallach N. The analytic continuation of discrete series I,II. Trans.Amer.Math.Soc. 251(1979),1-17,19-37.