Density Estimation in Inventory Control Systems under a Discounted Optimality Criterion

Abstract

This chapter deals with a class of discrete-time inventory control systems where the demand process D t is formed by independent and identically distributed random variables with unknown density. Our objective is to introduce a suitable density estimation method which, combined with optimal control schemes, defines a procedure to construct optimal policies under a discounted optimality criterion.

1. Introduction

Inventory systems are one of the most studied sequential decision problems in the fields of operation research and operation management. Its origin lies in the problem of determining how much inventory of a certain product should be kept in existence to meet the demand of buyers, at a cost as low as possible. Specifically, the question is: How much should be ordered, or produced, to satisfy the demand that will be presented during a certain period? Clearly, the behavior of the inventory over time depends on the ordered quantities and the demand of the product in successive periods. Indeed, let $I_t$ and $q_t$ be the inventory level and the order quantity at the beginning of period $t,$ respectively, and $D_t$ be the random demand during period $t.$ Then ${\left\{I_t\right\}}_{t\ge 0}$ is a stochastic process whose evolution in time is given as

$$I_{t+1}=\text{max}\left\{0,I_t+q_t-D_t\right\}≕{\left(I_t+q_t-D_t\right)}^{+},t=0,1,\dots$$

Schematically, this process is illustrated in the following figure.

(Standard inventory system)

In this case, the inventory manager (IM) observes the inventory level $I_t$ and then selects the order quantity $q_t$ as a function of $I_t.$ The order quantity process causes costs in the operation of the inventory system. For instance, if the quantity ordered is relatively small, then the items are very likely to be sold out, but there will be unmet demand. In this case the holding cost is reduced, but there is a significant cost due to shortage. Otherwise, if the size of the order is large, there is a risk of having surpluses with a high holding cost. These facts give rise to a stochastic optimization problem, which can be modeled as a Markov decision process (MDP). That is, the inventory system can be analyzed as a stochastic optimal control problem whose objective is to find the optimal ordering policy that minimizes a total expected cost.

The analysis of the control problem associated to inventory systems has been done under several scenarios: discrete-time and continuous-time systems with finite or infinite capacity, inventory systems considering bounded and unbounded one-stage cost, as well as partially observable models, among others (see, e.g., [1, 2, 3, 4, 5, 7]). Moreover, such scenarios have their own methods and techniques to solve the corresponding control problem. However, in most cases, it has been assumed that all the components that define the behavior of the inventory system are known to the IM, which, in certain situations, can be too strong and unrealistic. Hence it is necessary to implement schemes that allow learning or collecting information about the unknown components during the evolution of the system to choose a decision with as much information as possible.

In this chapter we study a class of inventory control systems where the density of the demand is unknown by the IM. In this sense, our objective is to propose a procedure that combines density estimation methods and control schemes to construct optimal policies under a total expected discounted cost criterion. The estimation and control procedure is illustrated in the following figure:

(Estimation and control procedure)

In this case, unlike the standard inventory system, before choosing the order quantity $q_t$ , the IM implements a density estimation method to get an estimate ${\rho }_t,$ and, possibly, combines this with the history of the system $h_t=\left(I_0,q_0,D_0,\dots ,I_{t-1},q_{t-1},D_{t-1},I_t\right)$ to select $q_t=q_t\left(h_t,{\rho }_t\right).$ Specifically, the density of the demand is estimated by the projection of an arbitrary estimator on an appropriate set, and its convergence is stated with respect to a norm which depends on the components of the inventory control model.

In general terms, our approach consists in to show that the inventory system can be studied under the weighted-norm approach, widely studied by several authors in the field of Markov decision processes (see, e.g., [11] and references therein) and in adaptive control (see, e.g. [9, 12, 13, 14]). That is, we prove the existence of a weighted function W which imposes a growth condition on the cost functions. Then, applying the dynamic programming algorithm, the density estimation method is adapted to such a condition to define an estimation and control procedure for the construction of optimal policies.

The chapter is organized as follows. In Section 2 we describe the inventory model and define the corresponding optimal control problem. In Section 3 we introduce the dynamic programming approach under the true density. Next, in Section 4 we present the density estimation method which will be used to state, in Section 5, an estimation and control procedure for the construction of optimal policies. The proofs of the main results are given in Section 6. Finally, in Section 7, we present some concluding remarks.

2. The inventory model

We consider an inventory system evolving according to the difference equation

$$I_{t+1}={\left(I_t+q_t-D_t\right)}^{+},t=0,1,\dots ,$$ E1

where $I_t$ and $q_t$ are the inventory level and the order quantity at the beginning of period $t,$ taking values in $\mathbb{I}≔\left[0,\infty \right)$ and $\mathbb{Q}≔\left[0,\infty \right)$ , respectively, and $D_t$ represents the random demand during period $t.$ We assume that $\left\{D_t\right\}$ is an observable sequence of nonnegative independent and identically distributed (i.i.d.) random variables with a common density $\rho \in L_1\left[0,\infty \right)$ which is unknown by the inventory manager. In addition, we assume finite expectation

$$\overline{D}≔E\left(D_t\right)<\infty .$$ E2

Moreover, there exists a measurable function $\overline{\rho }\in L_1\left[0,\infty \right)$ such that

$$\rho \left(s\right)\le \overline{\rho }\left(s\right)$$ E3

almost everywhere with respect to the Lebesgue measure. In addition

$$\underset{0}{\overset{\infty }{\int }}{s}^2\overline{\rho }\left(s\right)\mathit{ds}<\infty .$$ E4

For example, if $\overline{\rho }\left(s\right)≔K\text{min}\left\{1,1/s^{1+r}\right\},s\in \text{[}0,\infty ),$ for some positive constants $K$ and $r,$ then there are plenty of densities that satisfy (3)–(4).

The one-stage cost function is defined as

$$\tilde{c}\left(I,q,D\right)=\mathit{cq}+h{\left(I+q-D\right)}^{+}+b{\left(D-I-q\right)}^{+},\left(I,q\right)\in \mathbb{I}\times \mathbb{Q},$$ E5

where $h,c,$ and $b$ are, respectively, the holding cost per unit, the ordering cost per unit, and the shortage cost per unit, satisfying $b>c.$

The order quantities applied by the IM are selected according to rules known as ordering control policies defined as follows. Let ${\mathcal{H}}_t$ be the space of histories of the inventory system up to time $t.$ That is, a typical element of ${\mathcal{H}}_t$ is written as

$$h_t=\left(I_0,q_0,D_0,\dots ,I_{t-1},q_{t-1},D_{t-1},I_t\right).$$

An ordering policy (or simply a policy) $\gamma =\left\{{\gamma }_t\right\}$ is a sequence of measurable functions ${\gamma }_t:$ ${\mathcal{H}}_t\to \mathbb{Q},$ such that ${\gamma }_t\left(h_t\right)=q_t,t\ge 0$ . We denote by $\mathrm{\Gamma }$ the set of all policies. A feedback policy or Markov policy is a sequence $\gamma =\left\{g_t\right\}$ of functions $g_t:\mathbb{I}\to \mathbb{Q},$ such that $g_t\left(I_t\right)=q_t.$ A feedback policy $\gamma =\left\{g_t\right\}$ is stationary if there exists a function $g:\mathbb{I}\to \mathbb{Q}$ such that $g_t=g$ for all $t\ge 0.$

When using a policy $\gamma \in \mathrm{\Gamma },$ given the initial inventory level $I_0=I$ , we define the total expected discounted cost as

$$V\left(\gamma ,I\right)≔E\left[\sum _{t=0}^{\infty }{\alpha }^t\tilde{c}\left(I_t,q_t,D_t\right)\right],$$ E6

where $\alpha \in \left(0,1\right)$ is the so-called discount factor. The inventory control problem is then to find an optimal feedback policy ${\gamma }^{\ast }$ such that $V\left({\gamma }^{\ast },I\right)=V^{\ast }\left(I\right)$ for all $I\in \mathbb{I}$ , where

$$V^{\ast }\left(I\right)≔\underset{\gamma \in \mathrm{\Gamma }}{\text{inf}}V\left(\gamma ,I\right),I\in \mathbb{I},$$ E7

is the optimal discounted cost, which we call value function.

We define the mean one-stage cost as

$$\begin{array}{lll}c\left(I,q\right)& =& \mathit{cq}+\mathit{hE}{\left(I+q-D\right)}^{+}+\mathit{bE}{\left(D-I-q\right)}^{+}\\ & =& \mathit{cq}+h\underset{0}{\overset{I+q}{\int }}{\left(I+q-s\right)}^{+}\rho \left(s\right)\mathit{ds}+b\underset{I+q}{\overset{\infty }{\int }}{\left(s-I-q\right)}^{+}\rho \left(s\right)\mathit{ds},\left(I,q\right)\in \mathbb{I}\times \mathbb{Q}.\end{array}$$ E8

Then, by using properties of conditional expectation, we can rewrite the total expected discounted cost (6) as

$$V\left(\gamma ,I\right)=E_I^{\gamma }\left[\sum _{t=0}^{\infty }{\alpha }^{t}c\left(I_t,q_t\right)\right],$$ E9

where $E_I^{\gamma }$ denotes the expectation operator with respect to the probability $P_I^{\gamma }$ induced by the policy $\gamma ,$ given the initial inventory level $I_0=I$ (see, e.g., [8, 10]).

The sequence of events in our model is as follows. Since the density $\rho$ is unknown, the one-stage cost (8) is also unknown by the IM. Then if at stage $t$ the inventory level is $I_t=I\in \mathbb{I},$ the IM implements a suitable density estimation method to get an estimate ${\rho }_t$ of $\rho .$ Next, he/she combines this with the history of the system to select an order quantity $q_t=q={\gamma }_t^{{\rho }_t}\left(h_t\right)\in \mathbb{Q}.$ Then a cost $c\left(I,q\right)$ is incurred, and the system moves to a new inventory level $I_{t+1}=I^{\text{'}}\in \mathbb{I}$ according to the transition law

$$\begin{array}{l}Q\left(B|I,q\right)≔\text{Prob}\left[I_{t+1}\in B|I_t=I,q_t=q\right]\\ ={\int }_0^{\infty }{1}_B\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\end{array}$$ E10

where $1_B\left(.\right)$ denotes the indicator function of the set $B\in \mathcal{B}\left(\mathbb{I}\right),$ and $\mathcal{B}\left(\mathbb{I}\right)$ is the Borel $\sigma -$ algebra on $\mathbb{I}$ . Once the transition to the inventory level $I^{\text{'}}$ occurs, the process is repeated. Furthermore, the costs are accumulated according to the discounted cost criterion (9).

3. Dynamic programming equation under the true density $\mathit{\rho }$

The study of the inventory control problem will be done by means of the well-known dynamic programming (DP) approach, which we now introduce in terms of the unknown density $\rho .$ In order to establish precisely the ideas, we first present some preliminary and useful facts.

The set of order quantities in which we can find the optimal ordering policy should be ${\mathbb{Q}}^{\ast }=\left[0,Q^{\ast }\right]\subset \mathbb{Q},$

where

$$Q^{\ast }=\frac{b\overline{D}}{c\left(1-\alpha \right)}.$$

Thus, we can restrict the range of $q$ so that $q\in {\mathbb{Q}}^{\ast }.$ Specifically we have the following result.

Lemma 3.1 Let ${\gamma }^0\in \mathrm{\Gamma }$ be the policy defined as ${\gamma }^0=\left\{0,0,\dots \right\}$ , and let $\overline{\gamma }=\left\{{\overline{\gamma }}_t\right\}$ be a policy such that ${\overline{\gamma }}_k\left(h_k\right)={\overline{q}}_k>Q^{\ast },$ for at least a $k=0,1,.\dots$ Then

$$V\left({\gamma }^0,I\right)\le V\left(\overline{\gamma },I\right),I\in \mathbb{I}.$$ E11

That is, ${\gamma }^0$ is a better solution than $\overline{\gamma }.$

Proof. Let $I_t^0,t=0,1,\dots ,$ be the inventory levels generated by the application of ${\gamma }^0,$ and $\left({\overline{I}}_t,{\overline{q}}_t\right)$ be the sequence of inventory levels and order quantities generated by $\overline{\gamma },$ where $I_0^0={\overline{I}}_0=I,$ $I_{t+1}^0={\left(I_t^0-D_t\right)}^{+}$ , and ${\overline{I}}_{t+1}={\left({\overline{I}}_t+{\overline{q}}_t-D_t\right)}^{+},$ $t\ge 0.$ Without loss of generality, we suppose that for a $\overline{q}>Q^{\ast }$ we have ${\overline{q}}_0=\overline{q}.$ Note that $I_t^0\le {\overline{I}}_t,$ for all $t\ge 0.$ Then observing that $c\overline{q}>b\overline{D}/\left(1-\alpha \right),$

$$\begin{array}{lll}V\left({\gamma }^0,I\right)& =& E\left[\sum _{t=0}^{\infty }{\alpha }^{t}c\left(I_t^0,0,D_t\right)\right]=E\left[\sum _{t=0}^{\infty }{\alpha }^t\left(h{\left(I_t^0-D_t\right)}^{+}+b{\left(D_t-I_t^0\right)}^{+}\right)\right]\\ & \le & E\sum _{t=0}^{\infty }{\alpha }^{t}h{\left({\overline{I}}_t-D_t\right)}^{+}+b\sum _{t=0}^{\infty }{\alpha }^{t}E\left(D_t\right)\\ & \le & E\left[\sum _{t=0}^{\infty }{\alpha }^t\left(h{\left({\overline{I}}_t+{\overline{q}}_t-D_t\right)}^{+}+b{\left(D_t-{\overline{I}}_t-{\overline{q}}_t\right)}^{+}+\frac{b\overline{D}}{1-\alpha }\right)\right]\\ & \le & E\left[\sum _{t=0}^{\infty }{\alpha }^t\left(h{\left({\overline{I}}_t+{\overline{q}}_t-D_t\right)}^{+}+b{\left(D_t-{\overline{I}}_t-{\overline{q}}_t\right)}^{+}+c\overline{q}\right)\right]\\ & \le & E\left[\sum _{t=0}^{\infty }{\alpha }^t\left(c{\overline{q}}_t+h{\left({\overline{I}}_t+{\overline{q}}_t-D_t\right)}^{+}+b{\left(D_t-{\overline{I}}_t-{\overline{q}}_t\right)}^{+}\right)\right]\\ & =& V\left(\overline{\gamma },I\right),I\in \mathbb{I}.∎\end{array}$$

Remark 3.2 Observe that for $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }$ we have

$$c\left(I,q\right)=\mathit{cq}+L\left(I+q\right),$$

where, by writing $y=I+q,$

$$L\left(y\right)≔\mathit{hE}{\left(y-D\right)}^{+}+\mathit{bE}{\left(D-y\right)}^{+}.$$

In addition, observe that for any fixed $s\in \text{[}0,\infty ),$ the functions $y\to {\left(y-s\right)}^{+}$ and $y\to {\left(s-y\right)}^{+}$ are convex, which implies that $L\left(y\right)$ is convex. Moreover

$$\underset{y\to \infty }{\text{lim}}L\left(y\right)=\infty .$$

The following lemma provides a growth property of the one-stage cost function (8).

Lemma 3.3 There exist a number $\beta$ and a function $W:\mathbb{I}\to \text{[}1,\infty )$ such that $0<\mathit{\alpha \beta }<1,$

$$\underset{\left(I,q,s\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }\times \text{[}0,\infty )}{\text{sup}}\frac{W\left({\left(I+q-s\right)}^{+}\right)}{W\left(I\right)}≔\phi <\infty ,$$ E12

and for all $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }$

$$c\left(I,q\right)\le W\left(I\right).$$ E13

In addition, for any density $\mu$ on $\left[0,\infty \right)$ such that ${\int }_0^{\infty }\mathit{s\mu }\left(s\right)<\infty ,$

$$\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}\le \mathit{\beta W}\left(I\right),\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }.$$ E14

The proof of Lemma 3.3 is given in Section 6.

We denote by ${\mathbb{B}}_W$ the normed linear space of all measurable functions $u:\mathbb{I}\to \Re$ with finite weighted-norm ( $W-$ norm) $\parallel \cdot {\parallel }_W$ defined as

$${‖u‖}_W≔\underset{I\in \mathbb{I}}{\text{sup}}\frac{\left|u\left(I\right)\right|}{W\left(I\right)}.$$ E15

Essentially, Lemma 3.3 proves that the inventory system (1) falls within of the weighted-norm approach used to study general Markov decision processes (see, e.g., [11]). Hence, we can formulate, on the space ${\mathbb{B}}_W$ , important results as existence of solutions of the DP-equation, convergence of the value iteration algorithm, as well as existence of optimal policies, in the context of the inventory system (1). Indeed, let

$$V^{\left(n\right)}\left(\gamma ,I\right)=E_I^{\gamma }\left[\sum _{t=0}^{n-1}{\alpha }^{t}c\left(I_t,q_t\right)\right]$$

be the $n$ -stage discounted cost under the policy $\gamma \in \mathrm{\Gamma }$ and the initial inventory level $I\in \mathbb{I},$ and

$$V^{\left(n\right)}\left(I\right)=\underset{\gamma \in \mathrm{\Gamma }}{\text{inf}}{V}^{\left(n\right)}\left(\gamma ,I\right);V^{\left(0\right)}\left(I\right)=0,I\in \mathbb{I}$$

the corresponding value function. Then, for all $n\ge 0$ and $I\in \mathbb{I},$ (see, e.g., [6, 10, 11]),

$$V^{\left(n\right)}\left(I\right)=\underset{q\in {\mathbb{Q}}^{\ast }}{\text{min}}\left\{c\left(I,q\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\left(n-1\right)}\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right\}$$ E16

Moreover, from [11, Theorem 8.3.6], by making the appropriate changes, we have the following result.

Theorem 3.4 (Dynamic programming) (a) The functions $V^{\left(n\right)}$ and $V^{\ast }$ belong to ${\mathbb{B}}_W.$ Moreover

$$V^{\left(n\right)}\left(I\right)\le \frac{W\left(I\right)}{1-\mathit{\alpha \beta }},V^{\ast }\left(I\right)\le \frac{W\left(I\right)}{1-\mathit{\alpha \beta }},I\in \mathbb{I}.$$ E17

1. (b) As $n\to \infty ,$ ${‖V^{\left(n\right)}-V^{\ast }‖}_W\to 0.$

2. (c) $V^{\ast }$ is convex.

3. (d) $V^{\ast }$ satisfies the dynamic programming equation:

$$\begin{array}{lll}{V}^{\ast }\left(I\right)& =& \underset{q\in {\mathbb{Q}}^{\ast }}{\text{min}}\left\{c\left(I,q\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right\}\\ & =& \underset{I\le y\le Q^{\ast }+I}{\text{min}}\left\{\mathit{cy}+L\left(y\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(y-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right\}-\mathit{cI},I\in \mathbb{I}.\end{array}$$ E18

1. (e) There exists a function $g^{\ast }:\mathbb{I}\to \mathbb{Q}$ such that $g^{\ast }\left(I\right)\in {\mathbb{Q}}^{\ast }$ and, for each $I\in \mathbb{I},$

$$V^{\ast }\left(I\right)=c\left(I,g^{\ast }\left(I\right)\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g^{\ast }\left(I\right)-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds},I\in \mathbb{I}.$$

Moreover, ${\gamma }^{\ast }=\left\{g^{\ast }\right\}$ is an optimal control policy.

4. Density estimation

As the density $\rho$ is unknown, the results in Theorem 3.4 are not applicable, and therefore they are not accessible to the IM. In this section we introduce a suitable density estimation method with which we can obtain an estimated DP-equation. This will allow us to define a scheme for the construction of optimal policies. To this end, let $D_0,D_1,\dots ,D_t,\dots$ be independent realizations of the demand whose density is $\rho .$

Theorem 4.1 There exists an estimator ${\rho }_t\left(s\right)≔{\rho }_t\left(s;D_0,D_1,\dots ,D_{t-1}\right),s\in 0,\infty )$ , of $\rho$ , such that (see (2) and (3)):

1. D.1. ${\rho }_t\in L_1\left[0,\infty \right)$ is a density.

2. D.2. ${\rho }_t\le \overline{\rho }\left(\cdot \right)$ a.e. with respect to the Lebesgue measure.

3. D.3. ${\int }_0^{\infty }s{\rho }_t\left(s\right)\mathit{ds}\le \overline{D}.$

4. D.4. $E{\int }_0^{\infty }\left|{\rho }_t-\rho \left(s\right)\right|\mathit{ds}\to 0,\mathit{as}t\to \infty .$

5. D.5. $E‖{\rho }_t-\rho ‖\to 0,\mathit{as}t\to \infty ,$ where

$$\Vert \mu \Vert ≔\underset{\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }}{\text{sup}}\frac{1}{W\left(I\right)}\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}$$ E19

for measurable functions $\mu$ on $\left[0,\infty \right).$

It is worth noting that for any density $\mu$ on $\left[0,\infty \right)$ satisfying (14), the norm $\Vert \mu \Vert$ is finite. The remainder of the section is devoted to prove Theorem 4.1.

We define the set $\mathcal{D}\subset L_1\left(\right[0,\infty \left)\right)$ as:

$$\mathcal{D}≔\left\{\mu :\mu \text{is}\mathrm{a}\text{density},\underset{0}{\overset{\infty }{\int }}\mathit{s\mu }\left(s\right)\mathit{ds}\le \overline{D},\mu \left(s\right)\le \overline{\rho }\left(s\right)\mathrm{a}.\mathrm{s}.\right\}.$$

Observe that $\rho \in \mathcal{D}$ .

Lemma 4.2 The set $\mathcal{D}$ is closed and convex in $L_1\left(\right[0,\infty \left)\right).$

Proof. The convexity of $\mathcal{D}$ follows directly. To prove that $\mathcal{D}$ is closed, let ${\mu }_t\in \mathcal{D}$ be a sequence in $\mathcal{D}$ such that ${\mu }_t\stackrel{L_1}{\to }\mu \in L_1\left(\right[0,\infty \left)\right).$ First, we prove

$$\mu \left(s\right)\le \overline{\rho }\left(s\right)a.e.$$ E20

We assume that there is $A\subset \text{[}0,\infty )$ with $m\left(A\right)>0$ such that $\mu \left(s\right)>\overline{\rho }\left(s\right),s\in A,$ $m$ being the Lebesgue measure on $\Re$ . Then, for some $\epsilon >0$ and $A^{\prime }\subset A$ with $m\left(A^{\prime }\right)>0,$

$$\mu \left(s\right)>\overline{\rho }\left(s\right)+\epsilon ,s\in A^{\prime }.$$ E21

Now, since ${\mu }_t\in \mathcal{D},t\ge 0,$ there exists $B_t\subset 0,\infty )$ with $m\left(B_t\right)=0,$ such that

$${\mu }_t\left(s\right)\le \overline{\rho }\left(s\right),s\in \text{[}0,\infty )\backslash B_t,t\ge 0.$$ E22

Combining (21) and (22) we have

$$\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|\ge \epsilon ,s\in A^{\prime }\cap \left(\right[0,\infty )\backslash B_t),t\ge 0.$$

Using the fact that $m(A^{\prime }\cap ([0,\infty )\backslash B_t)=m\left(A^{\prime }\right)>0$ , we obtain that ${\mu }_t$ does not converge to $\mu$ in measure, which is a contradiction to the convergence in $L_1.$ Therefore $\mu \left(s\right)\le \overline{\rho }\left(s\right)$ a.e.

On the other hand, applying Holder’s inequality and using the fact that $\overline{\rho }\in L_1\left[0,\infty \right)$ , from (20),

$$\begin{array}{lll}\left|1-\underset{0}{\overset{\infty }{\int }}\mu \left(s\right)\mathit{ds}\right|& =& \left|\underset{0}{\overset{\infty }{\int }}{\mu }_t\left(s\right)\mathit{ds}-\underset{0}{\overset{\infty }{\int }}\mu \left(s\right)\mathit{ds}\right|=\underset{0}{\overset{\infty }{\int }}{\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|}^{\frac{1}{2}}{\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|}^{\frac{1}{2}}\mathit{ds}\\ & \le & {\left(\underset{0}{\overset{\infty }{\int }}2\overline{\rho }\left(s\right)\right)}^{1/2}{\left(\underset{0}{\overset{\infty }{\int }}\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|\right)}^{1/2}\to 0\mathrm{as}t\to \infty ,\end{array}$$ E23

which implies ${\int }_0^{\infty }\mu \left(s\right)\mathit{ds}=1.$ Now, as $\mu \ge 0a.e.,$ we have that $\mu$ is a density. Similarly, from (4),

$$\begin{array}{lll}\underset{0}{\overset{\infty }{\int }}s\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|\mathit{ds}& =& \underset{0}{\overset{\infty }{\int }}s{\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|}^{\frac{1}{2}}{\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|}^{\frac{1}{2}}\mathit{ds}\\ & \le & {\left(\underset{0}{\overset{\infty }{\int }}{s}^{2}2\overline{\rho }\left(s\right)\mathit{ds}\right)}^{1/2}{\left(\underset{0}{\overset{\infty }{\int }}\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|\mathit{ds}\right)}^{1/2}\\ & \le & 2^{\frac{1}{2}}{M}^{\prime }{\left(\underset{0}{\overset{\infty }{\int }}\left|{\mu }_t\left(s\right)-\mu \left(s\right)\right|\mathit{ds}\right)}^{1/2},\end{array}$$ E24

for some constant $M^{\prime }<\infty .$ Letting $t\to \infty$ we obtain

$$\underset{0}{\overset{\infty }{\int }}s{\mu }_t\left(s\right)\mathit{ds}\to \underset{0}{\overset{\infty }{\int }}\mathit{s\mu }\left(s\right)\mathit{ds}$$

which, in turn, implies that

$$\underset{0}{\overset{\infty }{\int }}\mathit{s\mu }\left(s\right)\mathit{ds}\le \overline{D}.$$

This proves that $\mathcal{D}$ is closed.∎

Let ${\widehat{\rho }}_t\left(s\right)≔{\widehat{\rho }}_t\left(s;D_0,D_1,\dots ,D_t\right),s\in \text{[}0,\infty ),$ be an arbitrary estimator of $\rho$ such that

$$E{‖\rho -{\widehat{\rho }}_t‖}_{L_1}=E\underset{0}{\overset{\infty }{\int }}\left|\rho \left(s\right)-{\widehat{\rho }}_t\left(s\right)\right|\to 0\mathrm{as}t\to \infty .$$ E25

Lemma 4.2 ensures the existence of the estimator ${\rho }_t$ which is defined by the projection of ${\widehat{\rho }}_t$ on the set of densities $\mathcal{D}.$ That is, the density ${\rho }_t\in \mathcal{D}$ , expressed as

$${\rho }_t≔\text{arg}\underset{\sigma \in \mathcal{D}}{\text{min}}{‖\sigma -{\widehat{\rho }}_t‖}_{L_1},$$

is the “best approximation” of the estimator ${\widehat{\rho }}_t$ on the set $\mathcal{D},$ that is,

$${‖{\rho }_t-{\widehat{\rho }}_t‖}_{L_1}=\underset{\mu \in \mathcal{D}}{\text{inf}}{‖\mu -{\widehat{\rho }}_t‖}_{L_1}.$$ E26

Now observe that ${\rho }_t$ satisfies the properties D.1, D.2, and D.3. Hence, Theorem 4.1 will be proved if we show that ${\rho }_t$ satisfies D.4 and D.5. To this end, since $\rho \in \mathcal{D}$ , from (26) observe that

$${‖{\rho }_t-\rho ‖}_{L_1}\le {‖{\rho }_t-{\widehat{\rho }}_t‖}_{L_1}+{‖{\widehat{\rho }}_t-\rho ‖}_{L_1}\le 2{‖{\widehat{\rho }}_t-\rho ‖}_{L_1},t\ge 0,$$

which implies that, from (25),

$$E\underset{0}{\overset{\infty }{\int }}\left|\rho \left(s\right)-{\rho }_t\left(s\right)\right|\mathit{ds}\le 2E{‖{\widehat{\rho }}_t-\rho ‖}_{L_1}\to 0,\mathrm{as}t\to \infty .$$ E27

That is, ${\rho }_t$ satisfies Property D.4. In fact, since $\underset{0}{\overset{\infty }{\int }}\left|\rho \left(s\right)-{\rho }_t\left(s\right)\right|\mathit{ds}\le 2$ a.s., from (27) it is easy to see that

$$E{\left(\underset{0}{\overset{\infty }{\int }}\left|\rho \left(s\right)-{\rho }_t\left(s\right)\right|\mathit{ds}\right)}^q\to 0,\mathrm{as}t\to \infty ,\text{for}\mathrm{any}q>0.$$ E28

Now, to obtain property D.5, observe that from (12)

$$\Vert {\rho }_t-\rho \Vert =\underset{\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }}{\text{sup}}\frac{1}{W\left(I\right)}\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\left|\rho \left(s\right)-{\rho }_t\left(s\right)\right|\mathit{ds}=\phi \underset{0}{\overset{\infty }{\int }}\left|\rho \left(s\right)-{\rho }_t\left(s\right)\right|\mathit{ds}.$$ E29

Therefore, property D.4 yields

$$E\Vert {\rho }_t-\rho \Vert \to 0,\mathrm{as}t\to \infty ,$$ E30

which proves the property D.5.

5. Estimation and control

Having defined the estimator ${\rho }_t,$ we will now introduce an estimate dynamic programming procedure with which we can construct optimal policies for the inventory systems.

Observe that for each $t\ge 0,$ from (14),

$$\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right){\rho }_t\left(s\right)\mathit{ds}\le \mathit{\beta W}\left(I\right),\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }.$$ E31

Now, we define the estimate one-stage cost function:

$$\begin{array}{lll}{c}_t\left(I,q\right)& =& \mathit{cq}+h\underset{0}{\overset{I+q}{\int }}{\left(I+q-s\right)}^{+}{\rho }_t\left(s\right)\mathit{ds}+b\underset{I+q}{\overset{\infty }{\int }}{\left(s-I-q\right)}^{+}{\rho }_t\left(s\right)\mathit{ds}\\ & =& \mathit{cq}+L_t\left(I+q\right),\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },\end{array}$$ E32

where (see Remark 3.2) for $y=I+q,$

$$L_t\left(y\right)≔h\underset{0}{\overset{y}{\int }}{\left(y-s\right)}^{+}{\rho }_t\left(s\right)\mathit{ds}+b\underset{y}{\overset{\infty }{\int }}{\left(s-y\right)}^{+}{\rho }_t\left(s\right)\mathit{ds}.$$

In addition, observe that for each $t\ge 0,$ $L_t\left(y\right)$ is convex and

$$\underset{y\to \infty }{\text{lim}}{L}_t\left(y\right)=\infty .$$ E33

We define the sequence of functions $\left\{V_t\right\}$ as $V_0\equiv 0,$ and for $t\ge 1$

$$\begin{array}{lll}{V}_t\left(I\right)& =& \underset{q\in {\mathbb{Q}}^{\ast }}{\text{min}}\left\{c_t\left(I,q\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}_{t-1}\left({\left(I+q-s\right)}^{+}\right){\rho }_t\left(s\right)\mathit{ds}\right\}\\ & =& \underset{I\le y\le Q^{\ast }+I}{\text{min}}\left\{\mathit{cy}+L_t\left(y\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}_{t-1}\left({\left(y-s\right)}^{+}\right){\rho }_t\left(s\right)\mathit{ds}\right\}-\mathit{cI},I\in \mathbb{I}.\end{array}$$ E34

We can state our main results as follows:

Theorem 5.1 (a) For $t\ge 0$ and $I\in \mathbb{I},$

$$V_t\left(I\right)\le \frac{W\left(I\right)}{1-\mathit{\alpha \beta }}.$$ E35

Therefore, $V_t\in {\mathbb{B}}_W.$

1. (b) As $t\to \infty ,$ $E\left[\underset{\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }}{\text{sup}}\frac{\left|c_t\left(I,q\right)-c(I,q)\right|}{W\left(I\right)}\right]\to 0.$

2. (c) As $t\to \infty ,$ $E{‖V_t-V^{\ast }‖}_W\to 0.$

3. (d) For each $t\ge 0,$ there exists $K_t\ge 0$ such that the selector $g_t:\mathbb{I}\to \mathbb{Q}$ defined as

$$q_t^{\ast }=g_t\left(I\right)≔\left\{\begin{array}{lll}{K}_t-I& \mathit{\text{if}}& 0\le I\le K_t\\ 0& \mathit{\text{if}}& I>K_t\end{array}\right.$$

attains the minimum in (34).

Remark 5.2 From [10, Proposition D.7], for each $I\in \mathbb{I}$ , there is an accumulation point $g_{\infty }\left(I\right)\in {\mathbb{Q}}^{\ast }$ of the sequence $\left\{g_t\left(I\right)\right\}$ . Hence, there exists a constant $K^{\ast }$ such that

$$g_{\infty }\left(I\right)≔\left\{\begin{array}{lll}{K}^{\ast }-I& \mathit{\text{if}}& 0\le I\le K^{\ast }\\ 0& \mathit{\text{if}}& I>K^{\ast }\end{array}\right.$$ E36

Theorem 5.3 Let $g_{\infty }$ be the selector defined in (36). Then the stationary policy ${\gamma }^{\ast }≔\left\{g_{\infty }\right\}$ is an optimal base stock policy for the inventory problem.

6. Proofs

6.1 Proof of Lemma 3.3

Note that, for each $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },$

$$\begin{array}{lll}c\left(I,q\right)& \le & {\mathit{cQ}}^{\ast }+h\left(I+Q^{\ast }\right)+b\overline{D}\\ & \le & \left(c+h\right)Q^{\ast }+\mathit{hI}+b\overline{D}\le \mathit{MG}\left(I\right),\end{array}$$ E37

where $M≔\text{max}\left\{\left(c+h\right)Q^{\ast }+b\overline{D},h\right\}$ and $G\left(I\right)=I+1.$ Moreover, for every density function $\mu$ on $\left[0,\infty \right)$ and $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },$

$$\underset{0}{\overset{\infty }{\int }}G\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}\le G\left(I\right)+Q^{\ast }.$$ E38

On the other hand, we define the sequence of functions $\left\{w_t\right\},$ $w_t:\mathbb{I}\to \Re$ , as

$$w_0\left(I\right)≔1+\mathit{MG}\left(I\right)$$ E39

and for $t\ge 1$ and any density function $\mu$ on $\left[0,\infty \right)$

$$w_t\left(I\right)≔\underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\underset{0}{\overset{\infty }{\int }}{w}_{t-1}\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}.$$

Observe that, for each $I\in \mathbb{I}$ ,

$$\begin{array}{lll}{w}_1\left(I\right)& =& \underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\underset{0}{\overset{\infty }{\int }}\left[1+\mathit{MG}\left({\left(I+q-s\right)}^{+}\right)\right]\mu \left(s\right)\mathit{ds}\\ & \le & 1+\mathit{MG}\left(I\right)+{\mathit{MQ}}^{\ast }.\end{array}$$

Thus,

$$\begin{array}{lll}{w}_2\left(I\right)& =& \underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\underset{0}{\overset{\infty }{\int }}\left[1+\mathit{MG}\left({\left(I+q-s\right)}^{+}\right)+{\mathit{MQ}}^{\ast }\right]\mu \left(s\right)\mathit{ds}\\ & \le & 1+\mathit{MG}\left(I\right)+{\mathit{MQ}}^{\ast }+{\mathit{MQ}}^{\ast },I\in \mathbb{I}.\end{array}$$

In general, it is easy to see that for each $I\in \mathbb{I}$ ,

$$w_t\left(I\right)\le \mathit{MG}\left(I\right)+1+\sum _{j=0}^{t-1}{\mathit{MQ}}^{\ast }=\mathit{MG}\left(I\right)+1+{\mathit{MQ}}^{\ast }t.$$ E40

Let ${\alpha }_0\in \left(\alpha ,1\right)$ be arbitrary, and define

$$W\left(I\right)≔\sum _{t=0}^{\infty }{\alpha }_0^t{w}_t\left(I\right).$$ E41

Then, from (40),

$$\begin{array}{lll}W\left(I\right)& \le & \sum _{t=0}^{\infty }{\alpha }_0^t\left[\mathit{MG}\left(I\right)+1+{\mathit{MQ}}^{\ast }t\right]\\ & =& \sum _{t=0}^{\infty }{\alpha }_0^t\left(\mathit{MG}\left(I\right)+1\right)+{\mathit{MQ}}^{\ast }\sum _{t=0}^{\infty }t{\alpha }_0^t\le \frac{\mathit{MG}\left(I\right)+1}{1-{\alpha }_0}+\frac{{\mathit{MQ}}^{\ast }{\alpha }_0}{{\left(1-{\alpha }_0\right)}^2}.\end{array}$$ E42

Therefore, $W\left(I\right)<\infty$ for each $I\in \mathbb{I}$ , and since $w_0>1,$ from (41),

$$W\left(I\right)>1.$$ E43

Furthermore, using (42) and the fact that $W\left(\cdot \right)\ge w_0\left(\cdot \right),$ a straightforward calculation shows that

$$\phi ≔\underset{\left(I,q,s\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }\times 0,\infty )}{\text{sup}}\frac{W\left({\left(I+q-s\right)}^{+}\right)}{W\left(I\right)}<\infty .$$ E44

Now, from (37) and (39), $c\left(I,q\right)\le w_0\left(I\right),$ which yields, for all $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },$

$$c\left(I,q\right)\le W\left(I\right).$$ E45

In addition, for every density function $\mu$ on $\left[0,\infty \right)$ and $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },$

$$\begin{array}{lll}\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}& =& \underset{0}{\overset{\infty }{\int }}\sum _{t=0}^{\infty }{\alpha }_0^t{w}_t\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}\\ & =& \sum _{t=0}^{\infty }{\alpha }_0^t\underset{0}{\overset{\infty }{\int }}{w}_t\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}\\ & \le & \sum _{t=0}^{\infty }{\alpha }_0^t{w}_{t+1}\left(I\right)={\alpha }_0^{-1}\left[\sum _{t=0}^{\infty }{\alpha }_0^t{w}_t\left(I\right)-w_0\left(I\right)\right]\\ & =& {\alpha }_0^{-1}\left[W\left(I\right)-w_0\left(I\right)\right]\le {\alpha }_0^{-1}W\left(I\right).\end{array}$$

Therefore, defining $\beta ≔{\alpha }_0^{-1},$ we have that $0<\mathit{\alpha \beta }<1,$ and

$$\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\mu \left(s\right)\mathit{ds}\le \mathit{\beta W}\left(I\right),\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast },$$

which, together with (43), (44), and (45), proves Lemma 3.3.∎

6.2 Proof of Theorem 5.1

1. (a) Since ${\int }_0^{\infty }s{\rho }_t\left(s\right)\mathit{ds}\le \overline{D}$ , from (32) (see (37)) $c_t\left(I,q\right)\le \mathit{MG}\left(I\right)$ for each $t\ge 0,\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }.$ Hence, it is easy to see that $c_t\left(I,q\right)\le W\left(I\right)$ for each $\left(I,q\right)\in \mathbb{I}\times {\mathbb{Q}}^{\ast }$ (see (45)). Then we have $V_1\left(I\right)\le W\left(I\right),$ and from (31), and by applying induction arguments, we get

$$V_t\left(I\right)\le \frac{W\left(I\right)}{1-\mathit{\alpha \beta }},t\ge 0,I\in \mathbb{I}.$$ E46

1. (b) Observe that from (39), for each $I\in \mathbb{I},$

$$W\left(I\right)\ge w_0\left(I\right)=1+\mathit{MG}\left(I\right),$$

1. which implies that (see (43))

$$\frac{\mathit{MG}\left(I\right)}{W\left(I\right)}\le 1-\frac{1}{W\left(I\right)}<\infty .$$ E47

1. In addition, from (37),

$$h\left(I+Q^{\ast }\right)\le \mathit{MG}\left(I\right).$$ E48

1. On the other hand, similarly as (24), from (4), it is easy to see that

$$\underset{0}{\overset{\infty }{\int }}s\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\le 2^{\frac{1}{2}}{M}^{\prime }{\left(\underset{0}{\overset{\infty }{\int }}\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\right)}^{1/2},$$ E49

1. for some constant $M^{\prime }<\infty .$ Hence, combining (47)–(49), from the definition of $c_t\left(I,q\right)$ and $c\left(I,q\right)$ , we have

$$\begin{array}{lll}\frac{\left|c_t\left(I,q\right)-c(I,q)\right|}{W\left(I\right)}& \le & \frac{h}{W\left(I\right)}\underset{0}{\overset{\infty }{\int }}\left(I+Q^{\ast }\right)\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}+\frac{b}{W\left(I\right)}\underset{0}{\overset{\infty }{\int }}s\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\\ & \le & \frac{\mathit{MG}\left(I\right)}{W\left(I\right)}\underset{0}{\overset{\infty }{\int }}\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}+b{2}^{\frac{1}{2}}{M}^{\text{'}}{\left(\underset{0}{\overset{\infty }{\int }}\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\right)}^{1/2}.\end{array}$$

1. Finally, taking expectation, (28) and Property D.4 prove the result.

2. (c) For each $I\in \mathbb{I}$ and $t\ge 0,$ by adding and subtracting the term $\alpha \underset{0}{\overset{\infty }{\int }}{V}_{t-1}\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}$ , we have

$$\begin{array}{l}\left|V_t\left(I\right)-V^{\ast }\left(I\right)\right|\le \underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\left|c_t\left(I,q\right)-(I,q)\right|+\underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\alpha \underset{0}{\overset{\infty }{\int }}{V}_{t-1}\left({\left(I+q-s\right)}^{+}\right)\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\\ +\alpha \underset{0}{\overset{\infty }{\int }}\left|V_{t-1}\left({\left(I+q-s\right)}^{+}\right)-V^{\ast }\left({\left(I+q-s\right)}^{+}\right)\right|\rho \left(s\right)\mathit{ds}\\ \le \underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\left|c_t\left(I,q\right)-(I,q)\right|+\frac{\alpha }{1-\mathit{\alpha \beta }}\underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\underset{0}{\overset{\infty }{\int }}W\left({\left(I+q-s\right)}^{+}\right)\left|{\rho }_t\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\\ +\mathit{\alpha \beta }\parallel V_{t-1}-V^{\ast }{\parallel }_{W}W\left(I\right),\end{array}$$

where the last inequality is due to (35), (17), (14), and (15). Therefore, from (15) and (19) and by taking expectation,

$$E\parallel V_t-V^{\ast }{\parallel }_W\le E\underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\left|c_t\left(I,q\right)-c(I,q)\right|+\frac{\alpha }{1-\mathit{\alpha \beta }}E‖{\rho }_t-\rho ‖+\mathit{\alpha \beta E}\parallel V_{t-1}-V^{\ast }{\parallel }_W.$$ E50

Finally, from (17) and (35), $\eta ≔\text{lim}{\text{sup}}_{t\to \infty }E{‖V^{\ast }-V_t‖}_W<\infty$ . Hence, taking limsup in both sides of (50), from part (a) and property D.5 in Theorem 4.1, we get $\eta \le \mathit{\alpha \beta \eta }$ , which yields $\eta =0$ (since $0<\mathit{\alpha \beta }<1$ ). This proves (c).

1. (d) For each $t\ge 0,$ let $H_t:\mathbb{I}\to \Re$ be the function defined as

$$H_t\left(y\right)≔\mathit{cy}+L_t\left(y\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}_{t-1}\left({\left(y-s\right)}^{+}\right){\rho }_t\left(s\right)\mathit{ds}.$$

Hence, (34) is equivalent to

$$V_t\left(I\right)=\underset{q\in {\mathbb{Q}}^{\ast }}{\text{min}}{H}_t\left(I+q\right)-\mathit{cI},I\in \mathbb{I}.$$ E51

Moreover (see (33)), observe that $H_t$ is convex and ${\text{lim}}_{y\to \infty }{H}_t\left(y\right)=\infty .$ Thus, there exist a constant $K_t\ge 0$ such that

$$H_t\left(K_t\right)=\underset{I\le y\le Q^{\ast }+I}{\text{min}}{H}_t\left(y\right),$$

and

$$g_t\left(I\right)=\left\{\begin{array}{lll}{K}_t-I& \text{if}& 0\le I\le K_t\\ 0& \text{if}& I>K_t\end{array}\right.$$

attains the minimum in (51).∎

6.3 Proof of Theorem 5.3

We fix an arbitrary $I\in \mathbb{I}$ . Since $g_{\infty }\left(I\right)$ is an accumulation point of $\left\{g_t\left(I\right)\right\}$ (see Remark 5.2), there exists a subsequence $\left\{t_m\left(I\right)\right\}$ of $\left\{t\right\}$ ( $t_m=t_m\left(I\right))$ such that

$$g_{t_m}\left(I\right)\to g_{\infty }\left(I\right)\mathrm{as}m\to \infty .$$

Moreover, from (34) and Theorem 5.1(d), letting $t_m=m,$ we have

$$V_m\left(I\right)=c_m\left(I,g_m\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+g_m-s\right)}^{+}\right){\rho }_m\left(\mathit{ds}\right).$$ E52

On the other hand, following similar arguments as the proof of Theorem 5.1(c), for each $m\ge 0$ and $\left(I,q\right)\in \mathbb{I}\times \mathbb{Q},$ we have

$$\begin{array}{l}\left|\alpha \underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+q-s\right)}^{+}\right){\rho }_m\left(s\right)\mathit{ds}-\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right|\\ \le \alpha \underset{0}{\overset{\infty }{\int }}\left|V_{m-1}\left({\left(I+q-s\right)}^{+}\right)-V^{\ast }\left({\left(I+q-s\right)}^{+}\right)\right|{\rho }_m\left(s\right)\mathit{ds}+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+q-s\right)}^{+}\right)\left|{\rho }_m\left(s\right)-\rho \left(s\right)\right|\mathit{ds}\\ \le \mathit{\alpha \beta }\parallel V_{m-1}-V^{\ast }{\parallel }_{W}W\left(I\right)+\frac{\alpha }{1-\mathit{\alpha \beta }}‖{\rho }_m-\rho ‖.\end{array}$$

Then, for each $I\in \mathbb{I},$

$$E\underset{q\in {\mathbb{Q}}^{\ast }}{\text{sup}}\left|\alpha \underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+q-s\right)}^{+}\right){\rho }_m\left(s\right)\mathit{ds}-\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+q-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right|\to 0,\mathrm{as}m\to \infty .$$ E53

Now,

$$\begin{array}{l}\alpha \underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+g_m-s\right)}^{+}\right){\rho }_m\left(\mathit{ds}\right)\\ =\left[\alpha \underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+g_m-s\right)}^{+}\right){\rho }_m\left(\mathit{ds}\right)-\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g_m-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\right]\\ +\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g_m-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}.\end{array}$$ E54

Taking expectation and liminf as $m\to \infty$ on both sides of (54), from (53) we obtain

$$\begin{array}{lll}\underset{m\to \infty }{\text{lim}\text{inf}}\mathit{E\alpha }\underset{0}{\overset{\infty }{\int }}{V}_{m-1}\left({\left(I+g_m-s\right)}^{+}\right){\rho }_m\left(\mathit{ds}\right)& =& \underset{m\to \infty }{\text{lim}\text{inf}}\mathit{E\alpha }\underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g_m-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\\ & \ge & \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g_{\infty }-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds},\end{array}$$

where the last inequality follows by applying Fatou’s Lemma and because the function $q\to {\left(I+q-s\right)}^{+}$ is continuous. Hence, taking expectation and liminf in (52), we obtain

$$c\left(I,g_{\infty }\right)+\alpha \underset{0}{\overset{\infty }{\int }}{V}^{\ast }\left({\left(I+g_{\infty }-s\right)}^{+}\right)\rho \left(s\right)\mathit{ds}\le V^{\ast }\left(I\right),I\in \mathbb{I}.$$ E55

As $I$ was arbitrary, by (18), the equality holds in (55) for all $I\in \mathbb{I}$ . To conclude, standard arguments on stochastic control literature (see, e.g., [10]) show that the policy ${\gamma }^{\ast }=\left\{g_{\infty }\right\}$ is optimal.∎

7. Concluding remarks

In this chapter we have introduced an estimation and control procedure in inventory systems when the density of the demand is unknown by the inventory manager. Specifically we have proposed a density estimation method defined by the projection to a suitable set of densities, which, combined with control schemes relative to the inventory systems, defines a procedure to construct optimal ordering policies.

A point to highlight is that our results include the most general scenarios of an inventory system, e.g., state and control spaces either countable or uncountable, possibly unbounded costs, finite or infinite inventory capacity. This generality entailed the need to develop new estimation and control techniques, accompanied by a suitable mathematical analysis. For example, the simple fact of considering possibly unbounded costs led us to formulate a density estimation method that was related to the weight function W, which, in turn, defines the normed linear space ${\mathbb{B}}_W$ (see (15)), all this through the projection estimator. Observe that if the cost function c is bounded, we can take $W\equiv 1$ and we have $‖\cdot ‖={‖\cdot ‖}_{L_1}$ (see (19) and (25)). Thus, any $L_1-$ consistent density estimator ${\rho }_t$ can be used for the construction of optimal ordering policies.

Finally, the theory presented in this chapter lays the foundations to develop estimation and control algorithms in inventory systems considering other optimality criteria, for instance, the average cost or discounted criteria with random state-action-dependent discount factors (see [14, 15] and references therein).