DEFORMATION BOUNDARY RIGIDITY FOR SGM-MANIFOLDS

AND INTEGRAL GEOMETRY PROBLEMS FOR NONTRAPPING MANIFOLDS

NURLAN S. DAIRBEKOV

Abstract. We consider the deformation boundary rigidity problem for SGMmanifolds and corresponding integral geometry problems for nontrapping manifolds. We prove that an SGM-manifold with a certain upper bound of curvature is deformation boundary rigid. For nontrapping manifolds with a similar curvature bound, we prove that a smooth symmetric tensor field of rank m, m ≥ 1, on such a manifold, integrating to zero over geodesics with endpoints on the boundary, is potential. For functions and 1-forms, this result is shown to be true for arbitrary nontrapping manifolds having no conjugate points.

1. Introduction

Let (M, g) be a (smooth) compact Riemannian manifold with boundary. The boundary rigidity problem asks whether the Riemannian metric g on M is determined by the boundary distance function d_{g} uniquely up to isometries that are the identity on the boundary. Here d_{g} : ∂M × ∂M → ℝ, with d_{g}(x, y) the distance in M between x and y. If so, (M, g) is called boundary rigid.

Not all compact manifolds with boundary are boundary rigid. Corresponding examples can be found in [1]. There are two natural classes of Riemannian manifolds to consider, the “simple” manifolds and the SGM-manifolds.

(M, g) is said to be simple if the boundary ∂M is strictly convex and there is a unique geodesic between any pair of boundary points.

The SGM condition (formulated in terms of d_{g} only) is harder to define. A precise definition is given in [1], but loosely speaking it means that (M, g) is nontrapping and all nongrazing geodesic segments strongly minimize.

(M, g) is called nontrapping if every geodesic ray hits the boundary ∂M and there is no complete geodesic in M.

By a nongrazing geodesic we mean a segment of a geodesic which lies in the interior of M except possibly the endpoints. A segment is said to strongly minimize if its length equals the distance between the endpoints and it is a unique such path.

Most papers deal with the case of “simple” manifolds, and there are a few papers devoted to the SGM-manifolds.

In this paper we consider the linearized problem for SGM-manifolds. Assume (M, g) to be an SGM-manifold. Let g_{τ}, −ε < τ < ε, be a deformation of g preserving the boundary distances, i.e., g_{τ} depends smoothly on τ, g_{0} = g, and d_{g_{τ}} = d_{g} for all τ. Since the validity of the condition SGM depends only on the boundary distance function, all (M, g_{τ}) will be SGM. Furthermore, they are equivalent as geodesic lenses (see [1]). Therefore, a straightforward calculation shows (cf. [4]) that for any nongrazing g_{0}-geodesic γ : [a, b] → M with endpoints on the boundary, γ(a) ∈ ∂M, γ(b) ∈ ∂M, we have

∫_{a}^{b} f_{ij}(γ(t)γ̇^{i}(t)γ̇^{j}(t) dt = 0,  (1.1)

where f = (f_{ij}) = \frac{dg_{τ}}{dτ} |_{τ=0}. So f integrates to zero over geodesics with endpoints on the boundary.

Let Z(S_{2}M) be the space of all smooth symmetric 2-fields on M integrating to zero over geodesics with endpoints on the boundary. This space is nonzero and contains the subspace of potential fields f = σ∇p, where σ∇p is the symmetrization of the covariant derivative of a smooth covector field p vanishing on the boundary, p|_{∂M} = 0.

(M, g) is called deformation boundary rigid [6] if each field f ∈ Z(S_{2}M) appears in this way from some smooth covector field p on M which vanishes on ∂M. (See [4] for a detailed discussion on the relationship between the boundary rigidity problem and the deformation boundary rigidity problem.)

Introduce one more notation. Let UM denote the unit tangent bundle of M. Given x ∈ UM and a 2-plane σ ⊂ T_{x}M, denote by K(x, σ) the sectional curvature of σ at x. For v ∈ U_{x}M, put

K(v) = sup_{σ, v∈σ} K(x, σ), K^{+}(v) = max{K(v), 0}.

Define the following characteristic:

k^{+}(M, g) = sup_{γ} ∫_{a}^{b} tK^{+}(γ̇(t)) dt,

where γ, γ : [a, b] → M, ranges in the set of all unit speed geodesics in M. We prove the following theorem.

Theorem 1.1. An n-dimensional SGM-manifold (M, g), n ≥ 2, is deformation boundary rigid whenever k^{+}(M, g) < \frac{n+3}{2(n+2)}.

This theorem follows as a special case (m = 2) from a corollary to our general theorem in which we deal with tensor fields of an arbitrary rank m on nontrapping manifolds.

As before, let (M, g) be a compact Riemannian manifold with boundary. For m = 0, 1, 2, …, let C^{∞}(S_{m}M) denote the space of symmetric covariant tensor fields on M of rank m (for m = 0, we abbreviate this to C^{∞}(M)). Given a field f ∈ C^{∞}(S_{m}M) and m tangent vectors v_{j} = (x, ξ_{j}) ∈ T M, x ∈ M, ξ_{j} ∈ T_{x}M, j = 1, …, m, we write

⟨f; v_{1}, …, v_{m}⟩ = f_{i_{1}...i_{m}}(x)ξ_{1}^{i_{1}} … ξ_{m}^{i_{m}},

and also introduce the notation

⟨f, v^{m}⟩ = ⟨f(x), ξ^{m}⟩ = f_{i_{1}...i_{m}}(x)ξ^{i_{1}} … ξ^{i_{m}}

for v = (x, ξ) ∈ T M.

We say that f integrates to zero over geodesics with endpoints on the boundary, and write f ∈ Z(S_{m}M), if the equality

∫_{a}^{b}⟨f, γ̇^{m}(t)⟩ dt = 0  (1.2)

holds for all nongrazing geodesics γ : [a, b] → M with γ(a) ∈ ∂M and γ(b) ∈ ∂M.

Given m ≥ 1 and f ∈ C^{∞}(S_{m}M), we say that f is potential, and write f ∈ P (S_{m}M), if f = dp for some field p ∈ C^{∞}(S_{m−1}M) vanishing on the boundary, p|_{∂M} = 0. Here d = σ∇ is the inner derivative, with ∇ denoting the covariant derivative with respect to g and σ the symmetrization (see details in [4]).

If f = dp then the integrand in (1.2) is a total derivative with respect to t, ⟨f, γ̇^{m}(t)⟩ = \frac{d}{dt}⟨p, γ̇^{m−1}(t)⟩. Therefore, for potential fields we always have (1.2), i.e,

P (S_{m}M) ⊂ Z(S_{m}M).  (1.3)

The problem we address here is to indicate conditions on (M, g) and m in order to have equality in (1.3).

This problem was earlier considered in the case when M is a simple manifold (see [4, 2] and references therein) or, more generally, when M lies in the interior of a simple manifold [5].

To formulate our result, we define the number

s_{n,m} = \frac{m(m + n)}{n + 2m − 1}  (1.4)

for m ≥ 1. Note that s_{n,1} = 1. Following [2], we say that (M, g) has no s-conjugate points if the equation

D_{t}^{2}X + sRγ̇X = 0  (1.5)

has no pair of conjugate points on any geodesic γ in M. Here D_{t} stands for the covariant derivative along γ and Rγ̇ is the curvature operator, Rγ̇X = R(γ̇, X)γ̇.

As remarked by Sharafutdinov [4, p. 120], it is not very hard to justify the following:

Remark 1.2. If k^{+}(M, g) < 1/s, then (M, g) has no s-conjugate points.

Our main result in this paper is the following theorem.

Theorem 1.3. Let n, m be integers such that n ≥ 2, m ≥ 1. Suppose that (M, g) is a nontrapping n-dimensional manifold having no s_{n,m}-conjugate points. Then

P (S_{m}M) = Z(S_{m}M).

In the case of simple manifolds, this result was established in [2].

For s = 1, equation (1.5) is nothing but the Jacobi equation. So (M, g) has no 1-conjugate points if and only if it has no conjugate points in the ordinary sense. Therefore, for m = 1 Theorem 1.3 yields the following corollary.

Corollary 1.4. A smooth 1-form ω on a nontrapping manifold M without conjugate points, integrating to zero over geodesics with endpoints on the boundary, is exact, ω = dϕ, with ϕ a smooth function on M vanishing on ∂M, ϕ|_{∂M} = 0.

In view of Remark 1.2, we also have the following corollary to Theorem 1.3.

Corollary 1.5. Let n, m be integers such that n ≥ 2, m ≥ 1. If (M, g) is a nontrapping n-dimensional manifold such that k^{+}(M, g) < \frac{n+2m−1}{m(m+n)}, then P (S_{m}M) = Z(S_{m}M).

Theorem 1.1 follows now from Corollary 1.5 with m = 2.

For m = 0, i.e., for functions, we establish the following:

Theorem 1.6. A smooth function on a nontrapping manifold without conjugate points, integrating to zero over geodesics with endpoints on the boundary, is itself zero.

The paper is further organized as follows. In Section 2 we perform a reduction of the proofs of Theorems 1.3 and 1.6 to the kinetic equation. This reduction is a crucial step in our argument and it became possible due to Sharafutdinov’s observations of [5] which we formulate below in Lemmas 2.1 and 2.2. Afterwards, the matter reduces to the analysis of the kinetic equation by the scheme of [3, 4, 2]. The main tools in this analysis are Pestov’s identity for semibasic tensor fields and Sharafutdinov’s decomposition of tensor fields into potential and solenoidal parts. Section 3 is devoted to Pestov’s identity and Section 4, to Sharafutdinov’s decomposition of tensor fields. In Section 5 we complete the proof of Theorem 1.3, as well as prove Theorem 1.6. The important idea, used in this section, of introducing

the “tangent” vertical derivative ∇^{v}^{||} and utilizing the extremal properties of Jacobi fields on manifolds having no s_{n,m}-conjugate points is borrowed from [2].

Closing the introduction, the author wants to thank Christopher Croke, who has read the manuscript and suggested a number of improvements. The author is also grateful to Max Planck Institute of Mathematics (Bonn) for hospitality during the preparation of this paper.

2. Reduction to the kinetic equation

Lemma 2.1. Let (M, g) be a compact Riemannian manifold with boundary such that every geodesic ray in M hits the boundary. If f ∈ Z(S_{m}M), then

⟨f, v^{m}⟩ = 0 for all v ∈ T (∂M).  (2.1)

The proof of this lemma is essentially given in [5, Lemma 2.3], although the assumptions there are stronger. Since the proof of Lemma 2.1 repeats that of Lemma 2.3 in [5] almost word for word, we omit it.

For a function f ∈ Z(S_{0}M), the lemma yields f|_{∂M} = 0. The next lemma shows that, for a tensor field f ∈ Z(S_{m}M) of higher rank, a certain correction can be added to f to make it vanish on ∂M.

Lemma 2.2 ([5, Lemma 2.2]). Let (M, g) be a compact Riemannian manifold with boundary and let f ∈ C^{∞}(S_{m}M) be a smooth symmetric tensor field. There exists a field p ∈ C^{∞}(S_{m−1}M), vanishing on the boundary, p|_{∂M} = 0, and a neighborhood W ⊂ M of ∂M, such that the field f̃ = f − dp has the following property: For every geodesic γ : [a, b] → W starting from the boundary at the right angle to ∂M, γ(a) ∈ ∂M, γ̇(a) ⊥ ∂M, the equality

⟨f̃; γ̇(t), v_{1}, …, v_{m−1}⟩ = 0

holds for all t ∈ [a, b] and all v_{i} ∈ T_{γ(t)}M (1 ≤ i ≤ m − 1).

In particular,

⟨f̃; ν, v_{1}, …, v_{m−1}⟩ = 0  (2.2)

for every x ∈ ∂M, all v_{1}, …, v_{m−1} ∈ T_{x}M, and a normal ν to ∂M at x.

We will reduce the proof of Theorems 1.3 and 1.6 to a certain assertion about the kinetic equation. To this end, we fix a nontrapping manifold (M, g) and a field f ∈ Z(S_{m}M), m ≥ 0.

In view of (2.1) and (2.2), we may henceforth assume that the field f itself vanishes on the boundary:

f|_{∂M} = 0.  (2.3)

Further, losing no generality we assume that (M, g) is a smooth submanifold of an n-dimensional compact smooth Riemannian manifold N without boundary (for example, by taking N to be the double of M and extending the Riemannian metric from M to N smoothly). We preserve the notation g for the Riemannian metric on N.

Now, we extend f from M to all of N by the zero field, denoting the resulting field by f again. The boundary condition (2.3) guarantees that the so-obtained field f is continuous on the whole manifold N and belongs to the Sobolev space H^{1}(S_{m}N) of tensor fields with square integrable first-order derivatives.

Given v ∈ UN, let γ_{v} be the complete geodesic in N issuing from v, γ̇_{v}(0) = v. Since (M, g) is nontrapping, there is no complete geodesic which would be contained entirely in M. Therefore, for any v there is a number l_{v} such that γ_{v}(l_{v}) ∉ M. We define a function φ : UN → ℝ by

φ(v) = ∫_{l_{v}}^{0}⟨f, γ̇_{v}^{m}(t)⟩ dt.  (2.4)

Observe that the value of φ(v) is independent of the choice of l_{v} subject to the above requirement. Indeed, the difference between the two integrals (2.4) with different values of l_{v} is the sum of integrals over nongrazing geodesic segments in M with endpoints on the boundary ∂M of M and integrals over geodesic segments disjoint from the interior of M. The integrals over the segments of the first type vanish by condition (1.2), and those over the segments of the second type vanish since f vanishes on ∂M and in the exterior of M.

Call a point v ∈ UM regular if the geodesic γ_{v} intersects ∂M transversally from either side and the open segment of γ_{v} between the basepoint of v and the point of intersection lies entirely in the interior of M. Denote by RM ⊂ UM the set of all regular points. It is clear that RM is open in UM and has full measure in UM.

Lemma 2.3. The function φ : UN → ℝ has the following properties:

(1) φ|_{U(N\M)} = 0.

(2) φ ∈ H^{1}(UN) ∩ C(UN) ∩ C^{∞}(RM).

(3) φ is C^{1} smooth along the lifts of geodesics to UN and satisfies the kinetic

equation on UN:

Hφ(v) = ⟨f, v^{m}⟩,  (2.5)

where H is the geodesic spray.

Proof. Item (1) is immediate from (2.4), since for v ∈ U(N \ M) we can choose l_{v} so that the geodesic γ_{v} will never enter M for the values of the parameter t between l_{v} and 0, but then the integrand in (2.4) vanishes since f vanishes in the exterior of M.

Prove item (2). Take any point v ∈ UN and the corresponding geodesic γ_{v}. Since γ_{v}(l_{v}) ∉ M, we can take a small hypersurface Φ in N transversally intersecting γ_{v} at the point γ_{v}(l_{v}) and disjoint from M. Then there is a neighborhood of v in UN such that for every v′ in this neighborhood the geodesic γ_{v′} will hit Φ at the time l′_{v′} smoothly depending on v′ and such that l′_{v} = l_{v}. For these v′ we can therefore take l′_{v′} as l_{v′} while defining φ by (2.4). So locally the lower limit of integration in (2.4) can be chosen to be a smooth function. Since f is continuous and lies in the Sobolev space H^{1}(S_{m}N), this observation allows us to prove routinely that φ is continuous and belongs to H^{1}(UN).

If v ∈ RM, then there is a neighborhood of v in UM such that for all v′ in this neighborhood the geodesics γ_{v′} intersect ∂M transversally from the same side as γ_{v} so that the interior of the segment between the basepoint of v′ and the point of intersection lies in the interior of M for each of these geodesics, moreover the parameter values τ_{v′} of the intersection points are smooth functions of v′. Obviously, we can calculate the values of φ at these points v′, taking the lower limits in (2.4) to be τ_{v′}. Since f is smooth in M, we conclude that φ is smooth in the chosen neighborhood of v and therefore smooth on RM.

To prove item (3), consider the geodesic flow G^{t} : UN → UN, G^{t}(v) = γ̇_{v}(t) for v ∈ UN. The geodesic spray H is the vector field on UN generating the flow G^{t}. In coordinates, for (x, ξ) ∈ UN, x ∈ N, ξ ∈ U_{x}N,

H = ξ^{i} \frac{∂}{∂x^{i}} − Γ_{jk}^{i}ξ^{j}ξ^{k} \frac{∂}{∂ξ^{i}} ,  (2.6)

where Γ_{jk}^{i} are the Christoffel symbols.

For ψ ∈ C^{∞}(UN) and v ∈ UN, we have

Hψ(v) = \frac{d}{dt}ψ(G^{t}(v))|_{t=0} = \frac{d}{dt}ψ(γ̇_{v}(t))|_{t=0}.  (2.7)

The vector field H on T N is tangent to the submanifold UN of T N. Therefore, it can (and will) be considered also as a vector field on UN. In this case, it is the generating vector field of the geodesic flow on UN, G^{t} : UN → UN.

By way of approximation, it is easy to see that the rule (2.7) works equally well for the functions ψ ∈ H^{1}(UN) ∩ C(UN). We apply it to our function φ.

Take v ∈ UN. For s ∈ ℝ, put v_{s} = G^{s}(v) = γ̇_{v}(s). Then γ_{v_{s}}(t) = γ_{v}(t + s) and we can take l_{v_{s}} = l_{v} − s. Then by (2.4)

φ(G^{s}(v)) = ∫_{l_{vs}}^{0} ⟨f, γ̇_{v_{s}}^{m}(t)⟩ dt = ∫_{l_{v}}^{s}⟨f, γ̇_{v}^{m}(t)⟩ dt.

Taking the derivative at s = 0 gives (2.5) and finishes the proof of the lemma. □

The further argument in the proofs of Theorems 1.3 and 1.6 consists in proving that the validity of the kinetic equation (2.5), with φ having the properties listed in items (1) and (2) of Lemma 2.3, forces the membership f ∈ P (S_{m}M) if m ≥ 1 and forces f to vanish if m = 0.

3. Pestov’s identity

We first recall the notion of a semibasic tensor field and its vertical and horizontal derivatives [4, 2].

Let τ_{s}^{r}N denote the bundle of tensors of degree (r, s) over N. As usual, the presence of the Riemannian metric allows us to identify all bundles τ_{s}^{r}N of the same total degree r + s by fixing a certain rule for raising and lowering indices. Therefore, we will not distinguish between co- and contravariant tensors, preferring to speak of co- and contravariant components of the same tensor.

Let π : T N → N be the natural projection. The bundle β_{s}^{r}N = π^{∗}(τ_{s}^{r}N) over T N is called the bundle of semibasic tensor fields of degree (r, s) and its sections are called semibasic tensor fields. For such a field u, the coordinate representation

u = (u_{j_{1}...j_{s}}^{i_{1}...i_{r}})(x, ξ)

holds in the domain of a local coordinate system on T N. Under a change of a local coordinate system, the components of a semibasic tensor field are transformed by the same formula as those of an ordinary tensor field on N. Like the ordinary tensors on N, we can identify all bundles β_{s}^{r}N having the same total degree r + s. This clearly allows us to identify the sections of these bundles as well.

If F is a notation for a function space (C^{k}, L^{p}, H^{k}, etc.), then we denote by F(β_{s}^{r}N) the space of semibasic tensor fields on T N whose components belong to the corresponding space F in each local coordinate system. In particular, C^{∞}(β_{0}^{0}N) = C^{∞}(T N).

We now define certain differential operators for semibasic tensor fields.

The vertical derivative

∇^{v} : C^{∞}(β_{s}^{r}N) → C^{∞}(β_{s+1}^{r}N)

is defined in coordinates by

∇^{v}_{k}u_{j_{1}...j_{s}}^{i_{1}...i_{r}}(x, ξ) = \frac{∂}{∂ξ^{k}} u_{j_{1}...j_{s}}^{i_{1}...i_{r}}.

The horizontal derivative

∇^{h} : C^{∞}(β_{s}^{r}N) → C^{∞}(β_{s+1}^{r}N)

is defined in coordinates by

∇^{h}_{k}u_{j_{1}...j_{s}}^{i_{1}...i_{r}}(x, ξ) =\frac{∂}{∂x^{k}} u_{j_{1}...j_{s}}^{i_{1}...i_{r}} − Γ_{kq}^{p}ξ^{q} \frac{∂}{∂ξ^{p}} u_{j_{1}...j_{s}}^{i_{1}...i_{r}}

+ ∑_{m=1}^{r} Γ_{kp}^{i_{m}}u_{j_{1}...j_{s}}^{i_{1}...i_{m−1}pi_{m+1}...i_{r}} − ∑_{m=1}^{s} Γ_{kj_{m}}^{p}u_{j_{1}...j_{m−1}pj_{m+1}...j_{s}}^{i_{1}...i_{r}}.

For a function u ∈ C^{∞}(T N), we in particular have

∇^{h}u(x, ξ) = \frac{∂}{∂x^{k}} u − Γ_{kq}^{p}ξ^{q} \frac{∂}{∂ξ^{p}} u.

A look at (2.6) shows that

Hu(x, ξ) = ξ^{i}∇^{h}_{i}u.  (3.1)

In this form H, regarded as a first-order partial differential operator, extends to the semibasic tensor fields u ∈ C^{∞}(β_{s}^{r}N). It is easy to see that for u ∈ C^{∞}(β_{s}^{r}N) the rule (2.7) changes to

Hu(v) = D_{t}u(γ̇_{v}(t))|_{t=0},  (3.2)

where D_{t} is the covariant derivative of the tensor field u(γ̇(t)) along the geodesic γ_{v}.

The vertical divergence, div^{v}, and the horizontal divergence , div^{h}, of a semibasic vector field u are defined by

div^{v} u = ∇^{v}_{i}u^{i}, div^{h} u = ∇^{h}_{i}u^{i}.

As can be easily shown, ∇^{v} and ∇^{h} are well-defined first-order differential operators. In particular, they extend naturally to the spaces H^{1}(β_{s}^{r}N).

With the above notations, Pestov’s identity reads [4]:

2⟨∇^{h}u, ∇^{v}(Hu)⟩ = |∇^{h}u|^{2} + div^{h} V + div^{v} W − ⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩  (3.3)

for every function u ∈ C^{∞}(T N). Here H is the geodesic spay on T N, R is the curvature tensor, and the semibasic vector fields V and W are given by

V = ⟨∇^{h}u, ∇^{v}u⟩ξ − ⟨∇^{h}u, ξ⟩∇^{v}u,  (3.4)
W = ⟨∇^{h}u, ξ⟩∇^{h}u.  (3.5)

Let D be an n-dimensional submanifold of N with boundary ∂D. Given u ∈ C^{∞}(UD) such that u|_{∂(UD)} = 0, extend the function u(x, ξ) to T D \ 0 so that the function becomes homogeneous of degree m − 1 in the second argument. Integrate (3.3) over UD with respect to the natural (local product) measure dΣ on UN. Using the Gauss-Ostrogradskiĭ formulas for horizontal and vertical divergences [4, Theorem 3.6.3], we arrive at the relation

∫_{UD} 2⟨∇^{h}u, ∇^{v}(Hu)⟩ dΣ = ∫_{UD} |∇^{h}u|^{2} dΣ + (−1)^{n} ∫_{∂(UD)}⟨V, ν⟩ dΣ^{2n−2}

+ (n + 2m − 2) ∫_{UD}⟨W, ξ⟩ dΣ − ∫_{UD}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ dΣ, (3.6)

where dΣ^{2n−2} is the natural local product measure on ∂(UD) = {(x, ξ) : x ∈ ∂D, ξ ∈ U_{x}N} and ν is the outward unit normal to ∂D.

For (x, ξ) ∈ ∂(UD), put

∇^{h}^{||}u = ∇^{h}u − ⟨∇^{h}u, ν⟩ν.

Then ∇^{h}^{||} is in fact a differential operator on ∂(UD) and ∇^{h}^{||}u is completely determined by the restriction u|_{∂(UD)} (see [2]).

For (x, ξ) ∈ ∂(UD), by a straightforward calculation we deduce from (3.4) that

⟨V, ν⟩ = ⟨∇^{h}^{||}u, ∇^{v}u⟩⟨ξ, ν⟩ − ⟨∇^{h}^{||}u, ξ⟩⟨∇^{v}u, ν⟩.

In view of the boundary condition u|_{∂(UD)} = 0, we therefore have ⟨V, ν⟩ = 0. Using also the equality ⟨W, ξ⟩ = (Hu)^{2}, from (3.6) we obtain

∫_{UD} 2⟨∇^{h}u, ∇^{v}(Hu)⟩ dΣ = ∫_{UD} |∇^{h}u|^{2} dΣ + (n + 2m − 2) ∫_{UD}(Hu)^{2} dΣ

− ∫_{UD}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ dΣ. (3.7)

Lemma 3.1. Let D be an n-dimensional submanifold of N with boundary ∂D. Let a function u : UD → ℝ be such that u ∈ H^{1}(UD), Hu ∈ H^{1}(UD), u is smooth in some neighborhood of ∂(UD) in UD, and u|_{∂(UD)} = 0. Then the integral identity (3.7) is valid for u.

Proof. We will construct a sequence of smooth functions u_{k} coinciding with u in a neighborhood of ∂(UD) and such that u_{k} → u in H^{1}(UD) and Hu_{k} → Hu in H^{1}(UD) as k → ∞. Applying (3.7) to each function u_{k} and passing to the limit as k → ∞, we will come to the desired conclusion.

The construction of such an approximation is quite standard and we merely sketch it. Call a chart (O, ψ) of the manifold UN straightening the vector field H if ψ_{∗}H coincides with a coordinate vector field on the range ψ(O) ⊂ ℝ^{2n−1}. Cover UD by a finite number of sufficiently small straightening charts O_{i}, so that the charts having nonempty intersection with ∂(UD) lie entirely in that neighborhood of ∂(UD) where u is smooth. Choose a partition of unity {μ_{i}} subordinate to this covering. Then u = ∑_{i} μ_{i}u, where each μ_{i}u has support in O_{i} Choose those indices i for which O_{i} is disjoint from ∂(UD). For such an index i, approximate the function (μ_{i}u) ∘ ψ_{i}^{−1} using the Sobolev mollification whose radius is small enough and lift the result back to O_{i} by means of ψ_{i}. For the other indices, take the functions μ_{i}u themselves. Then the sum of the so-obtained functions gives the sought approximation to u. □

4. Decomposition of tensor fields into potential and solenoidal parts

For m ≥ 1, let δ be the divergence [4],

δ : C^{∞}(S_{m}N) → C^{∞}(S_{m−1}N), (δT)_{i_{1}...i_{m−1}} = g^{jk}∇_{k}T_{ji_{1}...i_{m−1}}.

The operator −δ is the formal dual to the inner derivative d with respect to the natural L^{2}-product on the space of tensor fields. In an obvious way, it extends to the spaces H^{1}(S_{m}N).

A a field T ∈ H^{1}(S_{m}N), m ≥ 1, is called solenoidal [4] if it satisfies the equation

δT = 0.  (4.1)

In the following lemma we recall Sharafutdinov’s decomposition of tensor fields into potential and solenoidal parts.

Lemma 4.1 (see [4, Theorem 3.3.2]). Let (D, g) be an arbitrary smooth compact Riemannian manifold with boundary ∂D. Each field f ∈ H^{k}(S_{m}D), m ≥ 1, can be uniquely decomposed as

f = dp + f̃  (4.2)

with some p ∈ H^{k+1}(S_{m−1}D) and f̃ ∈ H^{k}(S_{m}D) such that p|_{∂D} = 0 and δf̃ = 0. The summands depend continuously on f in the following sense: the estimates

∥p∥_{H^{k+1}(S_{m−1}D)} ≤ C∥f∥_{H^{k}(S_{m}D)}, ∥f̃∥_{H^{k}(S_{m}D)} ≤ C∥f∥_{H^{k}(S_{m}D)}

hold with some constant C independent of f. In particular, if f ∈ C^{∞}(S_{m}D) then p ∈ C^{∞}(S_{m−1}D) and f̃ ∈ C^{∞}(S_{m}D).

We will apply this decomposition over a nested family of subdomains in N whose intersection is our manifold M.

Let W ⊂ N be a collar neighborhood of ∂M in N. This means that there is a diffeomorphism Φ : ∂M × (−1, 1) → W such that the restriction Φ|_{∂M×{0}} is the identity. We also assume that Φ(∂M × (−1, 0)) ⊂ M and Φ(∂M × (0, 1)) ⊂ N \ M.

Put M_{ε} = M ∪Φ(∂M ×[0, ε]), 0 ≤ ε < 1, obtaining a nested family of subdomains in N, with M_{0} = M.

Now, let f be the symmetric field on N that we used in Section 2 while defining the function u of Lemma 2.3. Applying Lemma 4.1 to D = M_{ε} and the field f|_{M_{ε}}, we obtain for every ε ∈ [0, 1) the corresponding fields p_{ε} ∈ H^{2}(S_{m−1}M_{ε}) and f̃_{ε} ∈ H^{1}(S_{m−1}M_{ε}) and the decomposition

f = dp_{ε} + f̃_{ε}.  (4.3)

Lemma 4.2. With the above notation, the following holds:

(1) p := p_{0} ∈ C^{∞}(S_{m−1}M), p|_{∂M} = 0.

(2) f̃:= f̃_{0} ∈ C^{∞}(S_{m}M), δf̃ = 0.
(3) p_{ε} ∈ H^{2}(S_{m−1}M_{ε}) ∩ C^{∞}(S_{m−1}(M_{ε} \ ∂M)), p_{ε}|_{∂M_{ε}} = 0.
(4) f̃_{ε} ∈ H^{1}(S_{m}M_{ε}) ∩ C^{∞}(S_{m}(M_{ε} \ ∂M)), δf̃_{ε} = 0.
(5) p̃_{ε}|_{M} → p in H^{2}(S_{m}M) as ε → 0, and for ε small enough the estimate
∥p̃_{ε}∥_{H^{2}(S_{m}M_{ε})} ≤ C∥f∥_{H^{1}(S_{m}M)}

holds uniformly in f and ε.

(6) f̃_{ε}|_{M} → f̃ in H^{1}(S_{m}M) as ε → 0, and for ε small enough the estimate

∥f̃_{ε}∥_{H^{1}(S_{m}M_{ε})} ≤ C∥f∥_{H^{1}(S_{m}M)}

holds uniformly in f and ε.

Proof. Items (1) and (2) are straightforward from Lemma 4.1.

To prove the other items, let us recall the construction of p_{ε} and f̃_{ε} provided by Lemma 4.1. The field p_{ε} is a solution of the boundary value problem

δdp_{ε} = v, p_{ε}|_{∂M_{ε}} = 0,  (4.4)

with v = δf. Afterwards, f̃_{ε} is defined by f̃_{ε} = f − dp_{ε}.

Denote by H_{0}^{2}(S_{m−1}M_{ε}) the subspace of tensor fields h ∈ H^{2}(S_{m−1}M_{ε}) of zero trace on ∂M_{ε}, h|_{∂M} = 0. It is shown in [4, Section 3.3] that the boundary value problem (4.4) is elliptic and the operator

D_{ε} = δd : H_{0}^{2}(S_{m−1}M_{ε}) → L^{2}(S_{m}M_{ε})

is invertible.

Now, items (3) and (4) are obvious, since the field f is smooth on M_{ε} \ ∂M. Item (6) is a direct consequence of item (5) and the formula f̃_{ε} = f − dp_{ε}. We now prove item (5). Let Ψ_{ε} : M → M_{ε}, 0 ≤ ε < 1, be a smooth family

of diffeomorphisms such that Ψ_{0} is the identity of M. Such a family is easy to construct by using the above diffeomorphism Φ. Define the following operators:

D̃_{ε} : H_{0}^{2}(S_{m−1}M) → L^{2}(S_{m}M),
D̃_{ε}p = Φ_{ε}^{−1} ∘ D_{ε}(p ∘ Φ_{ε}).

It is easy to see that these operators are well-defined and are in fact second-order partial differential operators on M. Furthermore, as ε → 0 they converge in the operator norm to the operator D_{0}. Since D_{0} is invertible, the inverse operators D̃_{ε}^{−1} converge in the operator norm to D_{0}^{−1}. Now, the claims of item (5) follow trivially. □

5. End of the proof of Theorem 1.3

As before, we let φ be the function of Lemma 2.3 and consider the same M_{ε}, f̃_{ε}, and p_{ε} as in the previous section.

Put

u_{ε}(v) = φ(v) − ⟨p_{ε}, v^{m−1}⟩  (5.1)

and

F_{ε}(v) = ⟨f̃_{ε}, v^{m}⟩.  (5.2)

A direct computation in coordinates, using (2.6), yields

H⟨p_{ε}(x), ξ^{m−1}⟩ = ⟨dp_{ε}(x), ξ^{m}⟩.  (5.3)

DEFORMATION BOUNDARY RIGIDITY AND INTEGRAL GEOMETRY PROBLEMS 11

For v ∈ UM_{ε}, (5.3) and (2.5) give

Hu_{ε}(v) = F_{ε}(v).  (5.4)

From the properties of φ listed in Lemma 2.3 and the properties of p_{ε} and f̃_{ε} listed in Lemma 4.2 we deduce the next lemma.

Lemma 5.1. The functions u_{ε} and F_{ε} have the following properties:

(1) u := u_{0} ∈ H^{1}(UM) ∩ C(UM) ∩ C^{∞}(RM), u|_{∂UM} = 0.

(2) F := F_{0} ∈ C^{∞}(UM), div^{h} (∇^{v}F) = 0.

(3) u_{ε} ∈ H^{1}(UM_{ε}) ∩ C^{∞}(M_{ε} \ M), u_{ε}|_{∂(UM_{ε})} = 0.

(4) u_{ε}|_{UM} → u in H^{1}(UM) as ε → 0, and for ε small enough the estimate ∥u_{ε}∥_{H^{1}(UM_{ε})} ≤ C holds uniformly in ε.

(5) F_{ε}|_{UM} → F in H^{1}(UM) as ε → 0, and for ε small enough the estimate ∥F_{ε}∥_{H^{1}(UM_{ε})} ≤ C holds uniformly in ε.

In particular, each function u_{ε} with ε > 0 satisfies the conditions of Lemma 3.1 on M_{ε}. By this lemma and (5.4) we have

∫_{UM_{ε}} 2⟨∇^{h}u_{ε}, ∇^{v}F_{ε}⟩ dΣ = ∫_{UM_{ε}} |∇^{h}u_{ε}|^{2} dΣ + (n + 2m − 2) ∫_{UM_{ε}} F_{ε}^{2} dΣ

− ∫_{UM_{ε}}⟨R(ξ, ∇^{v}u_{ε})ξ, ∇^{v}u_{ε}⟩ dΣ.

Items (4) and (5) of Lemma 5.1 allow us to pass to the limit as ε → 0 in this identity. We then arrive at

∫_{UM} 2⟨∇^{h}u, ∇^{v}F ⟩ dΣ = ∫_{UM} |∇^{h}u|^{2} dΣ + (n + 2m − 2) ∫_{UM} F^{2} dΣ

− ∫_{UM}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ dΣ. (5.5)

Notice that

⟨∇^{h}u, ∇^{v}F ⟩ = div^{h} (u∇^{v}F) − u div^{h} (∇^{v}F) = div^{h} V,  (5.6)

with

V := u∇^{v}F = muf̃_{ji_{2}...i_{m}}ξ^{i_{2}} … ξ^{i_{m}}.  (5.7)

Since V ∈ H^{1}(β_{1}^{0}M) and V |_{∂(UM)} = 0, the Gauss-Ostrogradskiĭ formula for the horizontal divergence yields

∫_{UM} 2⟨∇^{h}u, ∇^{v}F ⟩ dΣ = ∫_{UM} div^{h} V dΣ = ∫_{∂(UM)}⟨V, ν⟩ dΣ^{2n−2} = 0.

Hence, the left-hand side of (5.5) vanishes. We thus come to the identity

∫_{UM}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ = ∫_{UM} |∇^{h}u|^{2} + (n + 2m − 2) ∫_{UM} F^{2}.  (5.8)

We will derive one more identity.

Following [2], for (x, ξ) ∈ UM put

∇^{v}^{||}u = ∇^{v}u − ⟨∇^{v}u, ξ⟩ξ.  (5.9)

It is easy to see that ∇^{v}^{||} is just gradient when restricted to U_{x}M; in particular,

∇^{v}^{||}u is independent of a particular choice of the extension of u from UM to T M. From formula (3.1) for H one can easily deduce the commutation formula

H(∇^{v}u) = ∇^{v}(Hu) − ∇^{h}u.  (5.10)

(For smooth functions u this equality is valid pointwise; for u ∈ H^{1}(UM) such that Hu ∈ H^{1}(UM), the left-hand side should be understood as the distributional

derivative of the field ∇^{v}u; in our case, by item (1) of Lemma 5.1 the function u is smooth on the set RM of full measure in UM, and so we can understand the left-hand side of (5.10) pointwise as well.)

Since Hu = F, we infer that

H(∇^{v}u) = ∇^{v}F − ∇^{h}u.  (5.11)

Since u is positively homogeneous of degree (m − 1) in ξ, Euler’s formula shows

that ⟨∇^{v}u, ξ⟩ = (m − 1)u and therefore from (5.9) and (5.11) we obtain

H(∇^{v}^{||}u) = ∇^{v}F − ∇^{h}u − (m − 1)F ξ.  (5.12)

Now,

|H(∇^{v}^{||}u)|^{2} = |∇^{v}F |^{2} + |∇^{h}u|^{2} + (m − 1)^{2}F^{2}

− 2⟨∇^{v}F, ∇^{h}u⟩ − 2(m − 1)F ⟨∇^{v}F, ξ⟩ + 2(m − 1)F ⟨∇^{h}u, ξ⟩. (5.13)

By (5.6)

⟨∇^{v}F, ∇^{h}u⟩ = div^{h} V,  (5.14)

with V defined by (5.7). Next,

⟨∇^{v}F, ξ⟩ = mF  (5.15)

and

⟨∇^{h}u, ξ⟩ = Hu = F.  (5.16)

Inserting (5.14)–(5.16) in (5.13), we obtain

|H(∇^{v}^{||}u)|^{2} = |∇^{v}F |^{2} + |∇^{h}u|^{2} − (m − 1)^{2}F^{2} − 2 div^{h} V.  (5.17)

Integrating and recalling that the integral of div^{h} V vanishes, we come to the second integral identity

∫_{UM} |H(∇^{v}^{||}u)|^{2} = ∫_{UM} |∇^{v}F |^{2} + ∫_{UM} |∇^{h}u|^{2} − (m − 1)^{2} ∫_{UM} F^{2}.  (5.18)

Multiplying (5.8) by the number s_{n,m} of (1.4), subtracting the result from (5.18),

and using the inequality |∇^{h}u| ≥ |Hu| = |F |, we arrive at the inequality

∫_{UM} |H(∇^{v}^{||}u)|^{2} − s_{n,m} ∫_{UM}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩

≤ ∫_{UM} |∇^{v}F |^{2} − m(n + 2m − 2) ∫_{UM} F^{2}. (5.19)

Lemma 5.2 ([4, Lemma 4.5.3]). The following inequality holds:

∫_{UM} |∇^{v}F |^{2} dΣ ≤ m(n + 2m − 2) ∫_{UM} F^{2} dΣ.

Thus, the right-hand side of (5.19) is nonpositive. Note also that by (5.9) and the properties of the curvature tensor, we have

⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ = ⟨R(ξ, ∇^{v}^{||}u)ξ, ∇^{v}^{||}u⟩.

Therefore, (5.19) leads to the inequality

∫_{UM} (|H(∇^{v}^{||}u)|^{2} − s_{n,m}⟨R(ξ, ∇^{v}^{||}u)ξ, ∇^{v}^{||}u⟩) dΣ ≤ 0.  (5.20)

We will show that this inequality may hold only if ∇^{v}^{||}u = 0.

Let ∂_{+}(UM) be the set of inward unit tangent vectors of M above ∂M:

∂_{+}(UM) = {v ∈ ∂(UM) : ⟨v, ν⟩ > 0}
= {(x, ξ) : x ∈ ∂M, ξ ∈ U_{x}M, ⟨ξ, ν(x)⟩ > 0},

where ν is the inward unit normal to ∂M.

For v ∈ ∂_{+}(UM), let τ_{v} stand for the parameter value at which the geodesic γ_{v} meets ∂M for the first time (since (M, g) is nontrapping, τ_{v} is well defined).

Given an integrable function ψ : UM → ℝ, the Santaló formula holds:

∫_{UM} ψ dΣ = ∫_{∂_{+}(UM)}⟨v, ν⟩( ∫_{0}^{τ_{v}} ψ(γ_{v}(t)) dt) dΣ^{2n−2}(v).

We apply it to the integral in (5.20).

Put

Y_{v}(t) = ∇^{v}^{||}u(γ_{v}(t)).

Using Santaló’s formula, we can rewrite (5.20) as follows:

∫_{∂_{+}(UM)}⟨v, ν⟩( ∫_{0}^{τ_{v}}(|D_{t}Y_{v}|^{2} − s_{n,m}⟨Rγ̇Y_{v}, Y_{v}⟩) dt) dΣ^{2n−2}(v) ≤ 0,  (5.21)

where D_{t}Y_{v} is the covariant derivative along γ. Observe that by item (1) of Lemma 5.1 we have u ∈ C^{∞}(RM) and u|_{∂(UM)} = 0. Therefore, for every v ∈ ∂_{+}(UM) the field Y_{v} in (5.21) is a smooth field along γ_{v} vanishing at its endpoints, Y_{v}(0) = Y_{v}(τ_{v}) = 0.

Lemma 5.3 (see [2, p. 46]). A geodesic γ : [a, b] → M in an arbitrary Riemannian manifold (M, g) has no s-conjugate points if and only if

E(X) := ∫_{a}^{b} (|D_{t}X|^{2} − s⟨R_{γ}X, X⟩) dt > 0

for every nonzero smooth vector field X along γ vanishing at its endpoints, X(a) = X(b) = 0.

This lemma is just an analog of the corresponding extremal properties of Jacobi fields and is a consequence of the fact that (1.5) is the Euler–Lagrange equation for the functional E(X).

Recalling that by assumption M has no s_{n,m}-conjugate points and applying Lemma 5.3, we see that Y_{v}(t) = 0 for all v ∈ ∂_{+}(UM) and t ∈ [0, τ_{v}]. Thus,

∇^{v}^{||}u = 0 a.e. on UM.

Now, (5.8) yeilds F = 0, i.e., f̃ = 0. This means that f = dp with p|_{∂M} = 0, i.e., f ∈ P (S_{m}M).

The proof of Theorem 1.3 is over.

The proof of Theorem 1.6 proceeds along the same lines, but is slightly simpler, because in this case Sharafutdinov’s decomposition of Lemma 4.1 is trivial.

Now, the kinetic equation takes the form Hφ(x, ξ) = f(x). On taking in the above arguments m = 0, u_{ε} = φ, and F_{ε} = f, identities (5.8) and (5.18) acquire the form

∫_{UM}⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ = ∫_{UM} |∇^{h}u|^{2} + (n − 2) ∫_{UM} F^{2},

∫_{UM} |H(∇^{v}^{||}u)|^{2} = ∫_{UM} |∇^{h}u|^{2} − ∫_{UM} F^{2}.

Subtracting the first of them from the second, we come to the identity

∫_{UM} |H(∇^{v}^{||}u)|^{2} − ⟨R(ξ, ∇^{v}u)ξ, ∇^{v}u⟩ = −(n − 1) ∫_{UM} F^{2}.

Now, the application of Lemma 5.3 gives F = 0, i.e., f = 0, finishing the proof of Theorem 1.6.

References

[1] Croke C. B. “Rigidity and distance between boundary points,” J. Differental Geom. 33 (1991), 445–464.

[2] Pestov L. N. Well-Posedness Questions of the X-Ray Tomography Problems [Russian]. Siberian Science Press, Novosibirsk, 2003.

[3] Pestov L. N. and Sharafutdinov V. A. “Integral geometry of tensor fields on a manifold of negative curvature,” Siberian Math. J. 29 (1988), no. 3, 427–441.

[4] Sharafutdinov V. A. Integral Geometry of Tensor Fields. VSP, Utrecht, the Netherlands, 1994.

[5] Sharafutdinov V. A. “An integral geometry problem in a nonconvex domain,” Siberian Math. J. 46 (2002), no. 6, 1159–1168.

[6] Sharafutdionov V. A. and Uhlmann G. “On deformation boundary rigidity and spectral rigidity of Riemannian surfaces with no focal points,” J. Diff. Geom. 56 (2000), no. 1, 93– 110.

Sobolev Institute of Mathematics, Koptyuga 4, 630090, Novosibirsk, Russia Current address: Max Planck Institute of Mathematics, Vivatsgasse 7, D-53111, Bonn, Ger-

many

E-mail address: nurlan@mpim-bonn.mpg.de, dair@math.nsc.ru