1 INTRODUCTION

Consider a control system

$$\dot{x}=f(x,u),$$ (1.1)

where controls are Lebesgue measurable functions $u:\mathbb{R} \rightarrow U$, with the control constraint set $U\subset \mathbb{R} ^{m}$ being compact. The following hypotheses will remain in effect throughout:

(F1)

$f:\mathbb{R}^{n}\times \mathbb{R}^{m}\rightarrow \mathbb{R}^{n}$ is continuous and locally Lipschitz in the state variable $x$; that is, for each bounded subset $\Gamma$ of $\mathbb{R}^{n}$, there exists $K_{\Gamma }>0$ such that

$$\Vert f(x,u)-f(y,u)\Vert \leq K_{\Gamma }\Vert x-y\Vert$$

whenever $(x,u)$ and $(y,u)$ are in $\Gamma \times U$.

(F2)

$f$ satisfies the linear growth condition

$$% latex2html id marker 1268\Vert f(x,u)\Vert \leq c_{1}+c_... ...t \quad \ \forall \,u\in U,\quad \forall \,x\in \mathbb{R}^{n}$$

for some positive constants $c_{1}$ and $c_{2}$.

Under these assumptions, for each control function $u(\cdot )$ and each initial phase $(\tau ,\alpha )\in \mathbb{R}\times \mathbb{R}^{n}$, there exists a unique solution $x(t)=x(t;\tau ,\alpha ,u(\cdot ))$ on $[\tau ,\infty )$ satisfying $x(\tau) = \alpha$. Although the preceding hypotheses on the dynamics can be relaxed in certain results to follow, for ease of exposition we will retain the above assumptions throughout the entire article.

A central issue in control theory is the construction of feedback control laws which achieve a desired behavior of the control system (1.1) in a manner which is both

• universal: the desired behavior achieved by the feedback holds for all initial phases in a prescribed set

and

• robust: the feedback is effective in the presence of external disturbance and state measurement error.

Two general types of feedback control problems will be considered in this article; these can be described in very general terms as follows:

• Stabilizability: Under an asymptotic controllability hypothesis (to be specified below), an autonomous feedback $k:\mathbb{R} ^{n}\rightarrow U$ is sought such that for any initial phase $(\tau ,\alpha )\in (-\infty ,T]\times \mathbb{R}^{n}$, all trajectories of
  

  $$\dot{x}=f(x,k(x))$$ (1.2)

  
  
  asymptotically approach the origin in a stable and uniform manner.

• Fixed duration optimal control with state constraints: One seeks to design a nonautonomous feedback law $k:(-\infty ,T]\times \mathbb{R}^{n}\rightarrow U$ such that for any initial phase $(\tau ,\alpha )\in \lbrack t_{0},T]\times S$, all trajectories of
  

  $$\dot{x}=f(t,x,k(t,x))$$ (1.3)

  
  
  satisfying $x(\tau) = \alpha$ minimize a cost of the form $\ell (x(T))$ while satisfying $x(t)\in S$ on $[\tau ,T]$. Here the state constraint $S$ is a given compact subset of $\mathbb{R}^{n}$ which is weakly invariant with respect to the dynamics (1.1).

Some articles and general references which are relevant to the present work, but which are not directly referenced below as are others, include [36], [48], [39], [40], [54], [2], [3], [21], [25], [28], [29], [30], and [43] .

Our approach in this survey is to use the constructions employed in solving the stabilizability problem, where an infinitesimal decrease property of a nonsmooth control Lyapunov function is utilized, in designing feedbacks in the optimal control problem; there, the infinitesimal decrease of the value function is employed in an analogous way.

Even in very simple cases, one cannot expect the existence of universal feedback laws $k$ which are continuous, this being the minimal condition for the classical existence theory of ordinary differential equations to apply to (1.2) or (1.3). This inadequacy of continuous feedback is well illustrated via the following two examples.

Example 1.1   A famous result of Brockett [7] asserts that when $f(\cdot ,\cdot )$ is assumed to be $C^{1}$-smooth, stabilizability by means of a continuous feedback law implies the covering condition that the image of $f$ contains an open neighborhood of the origin; for an elegant degree-theory-based proof of this fact, attributed to R. D. Nussbaum ca. 1982, see Sontag [50]. Brockett provided the following 3-dimensional example, known as the ``nonholonomic integrator,'' showing that the covering condition can fail for asymptotically controllable bilinear systems:

$$\begin{eqnarray*} \dot{x}_{1} &=& u_{1} \\ \dot{x}_{2}&=& u_{2}\quad \quad \quad... ..._{1},u_{2})\in B_{2}=:U \\ \dot{x}_{3}& = &x_{1}u_{2}-x_{2}u_{1} \end{eqnarray*}$$

The asymptotic controllability is verifiable by an ad hoc argument; the failure of the covering condition is easy to check.

Remark 1.2   By a Filippov solution (see [24]) of $\dot{x}% =f(x,k(x))=:g(x)$ is meant an absolutely continuous function $x$ solving the differential inclusion

$$\dot{x}(t)\in \bigcap_{\delta >0}~\bigcap_{meas({\cal N})=0}co\,g({x+\delta B_{n}}\backslash {\cal N}),$$

where the second intersection is taken over all subsets ${\cal N}$ of $\mathbb{R}^{n}$ with Lebesgue measure zero. The belief that the Filippov solution concept was adequate for using discontinuous feedback laws to achieve stabilizability was negated by a result of Ryan [46]; see also Coron and Rosier [22]. On the other hand, the problem of finding an adequate solution concept for discontinuous feedback under which asymptotic controllability and feedback stabilization are equivalent was first solved by Clarke, Ledyaev, Sontag, and Subbotin [13]. There, as in the present exposition, a discretized solution concept was employed.

The main idea in the next example goes back to Barabanova and Subbotin [4], [5]; see also Krasovskii and Subbotin [35], [34] as well as Clarke, Ledyaev, and Subbotin [16].

Example 1.3   Consider the optimal control problem with dynamics

$$\begin{eqnarray*} \dot{x} = u,\quad t\in [0,1], \\ x(0) = \alpha , \end{eqnarray*}$$

where $x\in \mathbb{R}$, $u(\cdot )$ is valued in $U=[-1,1]$, and where it is desired to minimize the endpoint cost functional $\ell (x(1))=-\vert x(1)\vert$. First consider any continuous feedback $k(t,x)$ which is Lipschitz in $x$ and such that solutions of (1.3) are defined on $[0,1]$ for every $% \alpha \in \lbrack -1,1]$. Then there is a unique solution $x(t)=x(t;\alpha )$ of $\dot{x}=k(t,x)$ on $[0,1]$ for each such $\alpha$. Now note that the function $Q:[-1,1]\rightarrow \lbrack -1,1]$ given by

$$Q(\alpha ):=-\int_{0}^{1}k(t,x(t;\alpha ))dt$$

is continuous and, therefore, has a fixed point $\hat{\alpha}$ by virtue of Brouwer's fixed point theorem. Furthermore, $x(1;\hat{\alpha})=0$. Now consider the discontinuous feedback law

$$k(t,x):=\left\{ \begin{array}{cl} \frac{x}{\vert x\vert} & if~~x\neq 0, \\ 1 & if~~x=0 . \end{array}\right.$$

Upon substituting into the dynamics, we see that for any $\alpha \in \lbrack -1,1]$, the resulting trajectory satisfies $-\Vert x(1)\Vert \leq -1$. This argument can be extended, by approximation, to the case of merely continuous $k$. Therefore, given any continuous feedback $k$, there is at least one initial phase $(0,\hat{\alpha})$ with $\hat{\alpha}\in \lbrack -1,1]$, such that the above discontinuous feedback produces a better outcome than the continuous one. Hence, there does not exist a continuous feedback which is optimal universally for initial phases in the set $\{0\}\times \lbrack -1,1]$. (Of course, it is not always possible to obtain a classical solution to the differential equation

$$\dot{x}=f(t,x,k(t,x))=:g(t,x)$$

when the feedback $k$ is discontinuous, since the existence theory of ordinary differential equations can break down, but in the present example, this difficulty does not occur.)

1.1 Nonsmooth analysis: basic definitions and notation

We now pause to fix our notation from nonsmooth analysis. Our main general references for the basic results we will employ from this subject are Clarke, Ledyaev, Stern, and Wolenski [20]; see also [14], Clarke [9], [10], and Loewen [41].

The Euclidean norm is denoted $\Vert \cdot \Vert$, and $\langle \,,\,\rangle$ is the usual inner product. The open unit ball in $\mathbb{R}^{n}$ is denoted $B_{n}$. For a set $Z\subset \mathbb{R}^{n}$, we denote by ${\rm comp}% (Z)$, ${\rm co}(Z)$, $\overline{Z}$, ${\rm bdry}(Z)$ and ${\rm int}(Z)$ the complement, convex hull, closure, boundary, and interior of $Z$, respectively.

Let $S$ be a nonempty subset of $\mathbb{R}^{n}$. The distance of a point $u$ to $% S$ is given by

$$d_{S}(u):=\inf \{\Vert u-x\Vert :x\in S\}.$$

The metric projection of $u$ on $S$ is denoted

$$% latex2html id marker 1410{\rm proj}_{S}(u):=\{x\in S:\Vert u-x\Vert =d_{S}(u)\}.$$

If $u\notin S$ and $x\in {\rm proj}_{S}(u)$, then the vector $u-x$ is called a perpendicular to $S$ at $x$. The cone consisting of all nonnegative multiples of these perpendiculars is denoted $N_{S}^{P}(x)$ and is referred to as the proximal normal cone (or P-normal cone) to $S$ at $x$. If $% x\in {\rm int}(S)$ or no perpendiculars to $S$ exist at $x$, then we set $% N_{S}^{P}(x)=\{0\}$. Observe that the P-normal cone is a local construction, since as can readily be shown,

$$N_{S}^{P}(x)=N_{S\cap \{x+\delta B_{n}\}}^{P}(x)\quad \forall \,\delta \geq 0.$$

Let $g:\mathbb{R}^n\rightarrow (-\infty,\infty]$ be an extended real valued function which is lower semicontinuous; that is, for each $x\in \mathbb{R}^n$,

$$g(x) \leq \liminf_{y\to x}g(y),$$

a property equivalent to closedness of the epigraph of $g$,

$$% latex2html id marker 1446{\rm epi}(g) := \{(x,y)\in \mathbb{R}^n\times \mathbb{R}: y \geq g(x)\}.$$

We denote the effective domain of $g$ by

$$% latex2html id marker 1450{\rm dom} (g) := \{x\in \mathbb{R}^n:g(x) < \infty\}.$$

A vector $\zeta \in {\rm dom} (g)$ is said to be a proximal subgradient (or P-subgradient) of $g$ at a point $x\in {\rm dom} (g)$ provided that

$$% latex2html id marker 1458(\zeta ,-1)\in N_{{\rm epi}(g)}^{P}(x,g(x)).$$

The set of all such vectors is called the P-subdifferential of $g$ at $% x$, denoted $\partial _{P}g(x)$. One can show that $\zeta \in \partial _{P}g(x)$ iff there exists $\sigma > 0$ such that

$$g(y)-g(x)+\sigma \Vert y-x\Vert ^{2}\geq \langle \zeta,y-x\rangle$$

for all $y$ near $x$, and that $\partial_Pg(x)\neq \phi$ for a dense subset of ${\rm dom}(g)$ (where we adopt the convention that $\partial_P g(x) = \phi$ when $x\not{\in }{\rm dom}(g)$).

The limiting normal cone (or L-normal cone) to $S$ at $x\in S$ is defined to be the set

$$N^L_S(x) := \{\zeta: \zeta_i \to \zeta,~\zeta_i \in N^P_S(x_i),~x_i \rightarrow x\}.$$

The L-normal cone leads to a corresponding L-subdifferential set for $g$:

$$% latex2html id marker 1492\partial_Lg(x) := \{ \zeta: (\zeta,-1) \in N^L_{{\rm epi}(g)}(x,g(x)\},$$

the members of which are called L-subgradients.

The Clarke normal cone (or C-normal cone) to $S$ at $x\in S$ is defined by

$$% latex2html id marker 1498N^C_S(x) := \overline {{\rm co}} [N^L_S(x)],$$

and the corresponding subdifferential for $g$ at $x$ is

$$% latex2html id marker 1504\partial_Cg(x) := \{ \zeta: (\zeta,-1) \in N^C_{{\rm epi}(g)}(x,g(x)\}.$$

The lower Dini derivate of $g$ at $y$ in the direction $w$ is

$$Dg(y;w) := \liminf_{\stackrel{\lambda\downarrow 0}{u\to w}}\frac{g(y+\lambda u)-g(y)}{\lambda}.$$ (1.4)

In case $g$ is locally Lipschitz, one has the simplification

$$Dg(y;w) = \liminf_{\lambda\downarrow 0}\frac{g(y+\lambda w)-g(y)}{\lambda}.$$ (1.5)

1.2 Discretized solutions

The discretized solution concept we will work with includes a state measurement error $p$ and an external disturbance $q$ in a perturbed version of (1.1) under a (generally discontinuous) feedback $k$, modeled by

$$\dot{x}=f(x,k(t,x+p))+q.$$

Here $q:\mathbb{R}\rightarrow \mathbb{R}^{n}$ is a bounded measurable function, but no measurability assumptions are made on $p:\mathbb{R}\rightarrow \mathbb{R}^{n}.$ Let $\pi = \{t_i\}_{i\geq 0}$ be a partition of $[0,\infty)$, by which is meant a countable, strictly increasing sequence $\{t_i\}$ with $t_0=0$ and $t_i \to \infty$ as $i\to \infty$. Let an initial phase $(\tau ,\alpha )\in (-\infty ,T]\times \mathbb{R}^{n}$ be specified. The associated $\pi$-trajectory $% x_{\pi }$ is the curve $x_{\pi }$ satisfying $x_{\pi }(\tau )=\alpha$ and

$$\dot{x}_{\pi }(t)=f(x_{\pi }(t),k(t_{i},x_{\pi }(t_{i})+p(t_{i})))+q(t)\quad \forall \,t\in (t_{i},t_{i+1}),\quad i \geq 0.$$ (1.6)

Note that $x_{\pi }$ is the unique solution on $[0,\infty)$ of the differential equation $\dot{x}=f(t,x,u)+q$ satisfying $x(\tau) = \alpha$, with a piecewise constant control function $u$ determined by the feedback $k$ and the measurement error $p$. This solution procedure, which involves discretizing the feedback control law $k$ via ``closed loop system sampling,'' is the same as the ``step-by-step'' solution concept employed by Krasovskii and Subbotin [34], [35] in differential game theory.