Fluid Inventory Models under Markovian Environment

Abstract

Today’s products are subject to fast changes due to market conditions, short life cycles, and technological advances. Thus, an important problem in inventory planning is how to effectively manage the inventory control in a dynamic and stochastic environment. The traditional Economic Order Quantity (EOQ) and Economic Production Quantity (EPQ) both are widely and successfully used models of inventory management. However, both models assume constant and fixed parameters over time. Unfortunately, most of these assumptions are unrealistic. In this study, we generalize the EOQ and EPQ models and study production-inventory fluid models operating in a stochastic environment. The inventory level increases or decreases according to a fluid-flow rate modulated by an n-state continuous time Markov chain (CTMC). Our main objective is to minimize the expected discounted total cost which includes ordering, purchasing, production, set up, holding, and shortage costs. Applying regenerative theory, optional sampling theorem (OST) to the multi-dimensional martingale and fluid flow techniques, we develop methods to obtain explicit formulas for these cost functionals. As such, we provide managers with a useful framework and an efficient and easy-to-implement tool to coop with different demand–supply patterns.

1. Introduction

An important problem in inventory planning is how to effectively manage the inventory control in a dynamic and stochastic environment. Particularly, today’s products are subject to fast changes due to market conditions, short life cycles, and technological advances. The uncertainties in the supply-chain hierarchy make inventory control very challenging and the structure of the optimal policy is still unknown; thus, both researchers and practitioners are focusing on relatively simple control policies. This focus is especially true in practice where there are uncertainties in the production rate, demand rate, and lead time. The specification of the production rates, order quantities, the storage capacity and the backlog possibilities has to take into account the costs for ordering, production, the holding of inventory, backlogging, and lost sales. For example, for a factory Manufacturer, fixing the production rate at some high level or ordering a large quantity avoids backlogs but may cause high production/order cost and inventory costs; however, fixing the production rate at some low level, or ordering many but small quantities may lead to severe backlogging costs and the loss of sale opportunities. An inventory control policy governs inventory replenishment decisions by specifying when and how many items should be ordered or produced, thus, the planning of such a supply chain subject to market uncertainty is challenging in terms of dealing with uncertainties and variations.

The traditional Economic Order Quantity (EOQ) and Economic Production Quantity (EPQ) both are widely and successfully used models of inventory management (see Nahmias [1]). In the classical EOQ model the demand occurs at a constant rate, and each time the inventory level hits level 0 an immediate and fixed order of size $Q$ is placed. In the classic EPQ model, every production cycle is composed of ON and OFF deterministic periods. There exists a predetermined level, say $Q,$ such that the system is ON and the inventory level increases from level 0 up to level $Q$. When level $Q$ is reached, the production is stopped and the inventory decreases down to 0. The time it takes from $Q$ to 0 is the OFF period. Notice that the EOQ model, in which the items are obtained from an outside suppliers, is just a special case of the EPQ model, by letting the production rate to be almost infinite.

The EOQ model derives the optimum order size that should be placed with a vendor to minimize the holding and ordering costs. On the other hand, the EPQ model determines the optimum production size that is to be manufactured to avoid unnecessary blockage of funds and excess storage costs. Both models consider the timing of reordering or production, the cost incurred due to order/production, and the holding costs to store items; holding cost can further be in the form of rentals for the storage area, salaries of personnel looking after the inventory, electricity bills, repairs, maintenance, etc. Thus, both models describe the trade-off between fixed ordering/producing costs and variable holding costs. Furthermore, both models assume that the demand and production rates are constant over time. The traditional EOQ model assumes that the order arrives with no time, and its replenishment will happen as soon as it reaches the minimum threshold level (usually level 0). Similarly, the traditional EPQ model assumes that the production can starts immediately as the stock goes down below a minimum level (usually level 0). The price is fixed and constant while making a purchase under EOQ or producing under EPQ. The key difference between the two is that the EOQ model is applied when the items are ordered from a third party, and the EPQ model comes into use when the company is the producer itself of the products.

Unfortunately, in real-world, most of the above assumptions are unrealistic; holding and ordering costs may vary due to change in rentals, salaries of personnel, and other overhead expenses. The demand rate, as well as the price of a product, can hardly be constant. They fluctuate a lot in the real world. Consumer income, tastes, and preferences, prices of inputs and raw materials, seasonal variation in demand, etc. are key factors that will affect demand as well as price. Similarly, under the EPQ model, the production process also does not remain constant because of factors like an interruption in power supply, breakages, and repairs in plant and machinery, overheating, change in the quality of inputs and raw materials, etc.

Moreover, as a consequence of home shopping, changes in customer preferences, technological advancements, and competition, modern sales and production companies often offer a take-back guarantee. Companies soften customers’ risk by offering a trial period for their products, thus, policies such as the right to return goods have become a part of daily routine. As a result, companies realize that a better understanding of returning items can provide a competitive advantage (Beltran and Krass [2], Fleischmann et al. [3], Pinçe et al. [4], Shaharudin et al. [5], and Barron [6, 7]).

The real need for guidance on how best to handle these uncertainties in demands, returns, productions, and costs motivates this study. We consider a continuous stochastic fluid inventory model for a single-item infinite horizon. Our main focus is to provide contributions to the study of inventory systems modulated by a Markovian environment. In the literature, dynamic control of stochastic inventory systems have been classified as periodic review models and continuous review models. In the case of continuous model, the inventory level (i.e., the number of on-hand items) can be viewed as a fluid process in which the production and demand rates undergo recurring changes in a stochastic fashion, and may be modeled as Markovian.

Markov-modulated fluid flows models have been an active area of research in recent years; one of their main applications is to the modeling the traffic evolution in communication channels. A standard example of a fluid flow is given by an infinite capacity buffer with inflow and outflow rates controlled by a Markov chain. The buffer level increases or decreases linearly at the current rate; when the buffer becomes empty, several strategies can be applied; it can remain empty until the inventory content level reaches a certain barrier (see, e.g., Boxma et al. [8], Kulkarni and Yan [9], Bean and O’reilly [10], and Barron and Hermel [11], Baek et al. [12], and Baek et al. [13]) or it can have positive jumps at the boundary (see e.g. Kulkarni and Yan [9, 14], and Barron [6, 15]). Fluid flow models are appropriate in situations where the arrival is comprised of a discrete unit, but the inter-arrival time between successive arrivals is negligible. Therefore, the arrival can be approximated by a continuous flow of fluid as individual units have less impact on the performance of the system. Such fluid queues are used as modeling tools of high-speed communication networks, transportation systems, congestion control systems, risk processes, and production-inventory systems.

In this study, the on-hand inventory level $\mathrm{ℐ}=\left\{I\left(t\right):t\ge 0\right\}$ increases or decreases according to a fluid-flow rate modulated by an $n$-state Continuous-Time Markov Chain (CTMC). The fluid process is the inventory position or inventory level under continuous review where the environment process represents the varying background state. A jump in the fluid level represents an external order arrival, and the transition at the background state can be the result of repairs or production facilities, etc. The cost structure includes an ordering cost for each order, a variable cost that is proportional to the actually replenished amount (both for EOQ), a set up cost for production line initialization, a production cost per item (both for EPQ), a holding cost per unit of inventory during time unit, and a penalty cost in case of shortage.

Due to the complexity of the optimal policy, these inventory/production fluid systems are challenging to optimize, and great effort in the past focused on constructing various heuristic policies (Mohebbi [16], Kouki et al. [17], Barron [18], and Barron and Dreyfuss [19]). Fluid versions of the EOQ model are studied in Kulkarni and Yan [9], Yan and Kulkarni [20], Kulkarni [21], Berman et al. [22], Berman et al. [23], Berman and Perry et al. [24], and Barron [15] and the references therein. We also mention another related model, so-called clearing system (see Kella et al. [25], Berman et al. [26], and Barron [27]), which can be regarded as a dual EOQ stochastic model. In a clearing system, the fluid process jumps back to zero when it reaches a certain positive level. For background on stochastic EPQ models, we cite Vickson [28], Kella and Whitt [29], Boxma et al. [8, 30], and Barron [7] among others.

In this chapter, our main objective is to minimize the expected discounted total cost using a discount factor $\beta >0$. For that, we develop techniques enabling us to determine all the costs in such vendor-managing-inventory models in a closed-form. Our analysis is based on a combination of a certain martingale technique and an application of fluid flow theory to semi-regenerative processes. The martingale approach was introduced by Asmussen and Kella [31] and was frequently used in the study of inventory models (see, e.g., Boxma et al. [8], Kella et al. [25], and Barron et al. [27, 32]). The matrix-analytic approach and the theory of Markov-modulated fluid flows was initiated by Ramaswami [33] and Ahn et al. [34], who developed a unified methodology for studying a large class of insurance risk models via fluid flows by making use of the connection between the surplus process of an insurance and a particular fluid flow.

As we will show, the exit-time results are used to efficiently derive LST (Laplace–Stieltjes transform) functionals associated with the discrete-type measures, while the combination with the martingales yields simple expressions for the continuous-type measures. These explicit expressions can then be used for an analysis of the dependence of the cost functionals on the system parameters or for optimization purpose when some of these parameters (e.g. the order amount, the threshold levels, or the costs) are taken as decision variables. As such, we provide managers with a useful framework and an efficient and easy-to-implement tool to derive the best parameters and to compare the results of different demand–supply patterns.

We start by introducing the main tools of our analysis to be used.

2. Main tools

In the following, we will briefly introduce our main tools: (a) the matrix-analytic approach and the theory of Markov-modulated fluid flows, initiated by a series of papers by Ahn and Ramaswami [34, 35, 36, 37] and Ramaswami [33] and (b) an application of the optional sampling theorem (OST) to the multi-dimensional martingale of Asmussen and Kella [31].

Here and in the following, we use bold symbols to denote vectors and blackboard to denote matrices. To be consistent, for any matrix $\mathbb{B}$, we shall denote its elements by ${\left(\mathbb{B}\right)}_{\mathit{ij}}$ or by ${\left[\mathbb{B}\right]}_{\mathit{ij}}$ and reserve the notation ${\mathbb{B}}_{\mathit{ij}}$ for the sub-matrix of $\mathbb{B}$ with row indices in the set ${\Im }_i$ and column indices in the set ${\Im }_j$. Moreover, for $n$-vector $\mathbf{x}$, we use ${\mathrm{\Delta }}_{\mathbf{X}}$ for a diagonal matrix ${\mathrm{\Delta }}_{\mathbf{x}}=\mathit{diag}\left(x_1,x_2,\dots ,x_n\right).$ We further use $E$ and $E_i$ to represent expectation and conditional expectation operators, respectively. $\mathbb{E}$ ($\mathbf{E}$) represents a matrix (a vector) of expectations. We denote by $\mathbf{e}$ a column vector with ones, i.e. $\mathbf{e}={\left(1,1,\dots ,1\right)}^T$, by ${\mathbf{e}}_i$ a row vector with the i_th component equal to 1 and all the other components 0, by $\mathbb{I}$ the identity matrix, by $\mathbf{0}$ the zero matrix (all with the appropriate dimensions); finally, by ${\mathbf{1}}_{\left\{A\right\}}$ the indicator of an event A.

2.1 Markov modulated fluid flow (MMFF)

Let $\Im$ be a state space that can partitioned into two sets: $\Im ={\Im }_1\cup {\Im }_2,$ with $\left|{\Im }_1\right|=n_1,$$\left|{\Im }_2\right|=n_2,$ and $\left|\Im \right|=n_1+n_2=n.$ Now, we introduce a modulating continuous time Markov chain (CTMC) $\{\mathcal{J}\left(t\right);$$t>0\}$ with that state $\Im$, and a fluid process $\left\{\mathcal{F}\left(t\right);t\ge 0\right\}$ that is modulated as follows: whenever the Markov chain is $\mathcal{J}\left(t\right)=i\in {\Im }_1,$ the fluid flow increases linearly at rate $c_i>0$ and whenever it is in $\mathcal{J}\left(t\right)=j\in {\Im }_2,$ the fluid flow decreases linearly at rate $\left|c_j\right|>0$$\left(c_j<0\right).$ The two-dimensional stochastic process $\left\{\mathcal{F}\left(t\right),\mathcal{J}\left(t\right),t\ge 0\right\}$ is called a MMFF (Markov Modulated Fluid Flow) process. We denote by $\mathbb{Q}$ the infinitesimal generator matrix of $\mathcal{J}\left(t\right);$$\mathbb{Q}$ is given in a block form according to transitions between the sets ${\Im }_i$ ($i=1,2$). Let ${\mathbb{C}}_1,{\mathbb{C}}_2$ and $\mathbb{C}$ be diagonal matrices as follows:

$$\begin{array}{c}{\mathbb{C}}_j=\mathit{diag}\left\{\left|c_i\right|,i\in {\Im }_j\right\},j=1,2.\\ \mathbb{C}=\mathit{diag}\left({\mathbb{C}}_1,{\mathbb{C}}_2\right).\end{array}$$

Let $\tau \left(x\right)=inf\left(t>0,\mathcal{F}\left(t\right)=x\right)$ be the first passage time to level $x.$ Let $\mathrm{\Psi }\left(\beta \right)$ be an $\left(n_1\times n_2\right)$ matrix whose ij_th component is

$${\left[\mathrm{\Psi }\left(\beta \right)\right]}_{\mathit{ij}}=E\left(e^{-\mathit{\beta \tau }\left(0\right)},\mathcal{J}\left(\tau \left(0\right)\right)=j|\mathcal{F}\left(0\right)=0,\mathcal{J}\left(0\right)=i\right)i\in {\Im }_{1,}j\in {\Im }_2,$$

which is the LST of $\tau \left(0\right)$ restricted to the event that the fluid process hits level 0 in-phase $j\in {\Im }_2$ and given that $\mathcal{F}\left(0\right)=0,\mathcal{J}\left(0\right)=i\in {\Im }_1.$ In the literature, a few algorithms, including some quadratically convergent ones, were established for computing $\mathrm{\Psi }\left(\beta \right)$ (and all other LSTs); see, e.g., Ahn and Ramaswami [36]. Let $\tau \left(x,y\right)$ be the first passage time of $\mathcal{F}$ from level $x$ to level $y$, and ${}_a^b\tau \left(x,y\right)$ be the first passage time of $\mathcal{F}$ from level $x$ to level $y$ avoiding a visit below $a$ or above $b$ (for simplicity, we use ${}^b\tau \left(x,y\right){\equiv }_0^b\tau \left(x,y\right)$ and ${}_a\tau \left(x,y\right){\equiv }_a^{\infty }\tau \left(x,y\right)$). Let $\widehat{f}\left(x,y,\beta \right)$ and ${}_a^b\widehat{f}\left(x,y,\beta \right)$ denote, respectively, the LST matrices of the joint distribution of the first passage times $\tau \left(x,y\right),$${}_a^b\tau \left(x,y\right)$ and the state of the phase process at each first passage time.

An important variant of the fluid flow $\mathcal{F},$ a Reflected Fluid Flow, is particularly useful in the analysis of our inventory level process. The reflected fluid flow ${\mathcal{F}}^r$ is obtained by reversing the roles of the up and down environment states. Analogous, ${\mathrm{\Psi }}^r\left(\beta \right)$ is the matrix (of order $\left(n_2\times n_1\right)$) whose $\left(i,j\right)$ component is the LST of the time to reach the level 0 for the process ${\mathcal{F}}^r$ restricted to ${\mathcal{J}}^r\left(\tau \left(0\right)\right)=j\in {\Im }_1$, given that ${\mathcal{F}}^r\left(0\right)=0$ and ${\mathcal{J}}^r\left(0\right)=i\in {\Im }_2,$ where ${\mathcal{J}}^r\left(t\right)$ is the modulated state process for ${\mathcal{F}}^r$ (we use notations ${\widehat{f}}^r\left(x,y,\beta \right)$ and ${}_a^b{\widehat{f}}^r\left(x,y,\beta \right)$ to denote quantities similar to those above defined for ${\mathcal{F}}^r$).

All these matrices, for hitting times, that we will use are straightforward to evaluate once we have computed $\mathrm{\Psi }\left(\beta \right).$Table 1 displays the basic elements (matrices) for the derivation of these LSTs; the first three matrices are associated with to flow $\mathcal{F},$ while the next three matrices are associated to the rate-reverse flow ${\mathcal{F}}^r$ by interchanging the indices 1 and 2. The LST matrices and their sizes are given in Table 2; all matrices have nice probabilistic interpretations. For more details see Ramaswami [33], Ahn et al. [34], and Bean et al. [38].

Quantity	Matrix size	The process
$\mathbb{K}\left(\beta \right)={\mathbb{C}}_1^{-1}\left({\mathbb{Q}}_{11}-\beta \mathbb{I}\right)+\mathrm{\Psi }\left(\beta \right){\mathbb{C}}_2^{-1}{\mathbb{Q}}_{12}$	$n_1\times n_1$	$\mathcal{F}$
$\mathbb{H}\left(\beta \right)={\mathbb{C}}_2^{-1}\left({\mathbb{Q}}_{22}-\beta \mathbb{I}+{\mathbb{Q}}_{21}\mathrm{\Psi }\left(\beta \right)\right)$	$n_2\times n_2$	$\mathcal{F}$
$\mathrm{\Xi }\left(\beta ,x\right)=\mathrm{\Psi }\left(\beta \right){\int }_{\left(0,x\right)}{e}^{\mathbb{H}\left(\beta \right)y}\left({\mathbb{C}}_2^{-1}{\mathbb{Q}}_{21}\right)e^{\mathbb{K}\left(\beta \right)y}\mathit{dy}$	$n_1\times n_1$	$\mathcal{F}$
${\mathbb{K}}^r\left(\beta \right)={\mathbb{C}}_2^{-1}{\mathbb{Q}}_{22}+{\mathrm{\Psi }}^r\left(\beta \right){\mathbb{C}}_1^{-1}{\mathbb{Q}}_{12}$	$n_2\times n_2$	${\mathcal{F}}^r$
${\mathbb{H}}^r\left(\beta \right)={\mathbb{C}}_1^{-1}\left({\mathbb{Q}}_{11}-\beta \mathbb{I}+{\mathbb{Q}}_{12}{\mathrm{\Psi }}^r\left(\beta \right)\right)$	$n_1\times n_1$	${\mathcal{F}}^r$
${\mathrm{\Xi }}^r\left(\beta ,x\right)={\mathrm{\Psi }}^r\left(\beta \right){\int }_{\left(0,x\right)}{e}^{{\mathbb{H}}^r\left(\beta \right)y}\left({\mathbb{C}}_1^{-1}{\mathbb{Q}}_{12}\right)e^{{\mathbb{K}}^r\left(\beta \right)y}\mathit{dy}$	$n_2\times n_2$	${\mathcal{F}}^r$

Table 1.

Transform matrices.

LST	The LST of the First Passage Time	Matrix size
${}_0{\widehat{f}}_{12}\left(x,0,\beta \right)=\mathrm{\Psi }\left(\beta \right)e^{\mathbb{H}\left(\beta \right)x}$	$\mathit{\text{from}}\left(x,{\Im }_1\right)\mathit{\text{to}}\left(0,{\Im }_2\right)\mathit{\text{avoiding}}0\mathit{\text{in}}F$	$n_1\times n_2$
${}_0{\widehat{f}}_{22}\left(x,0,\beta \right)=e^{\mathbb{H}\left(\beta \right)x}$	$\mathit{\text{from}}\left(x,{\Im }_2\right)\mathit{\text{to}}\left(0,{\Im }_2\right)\mathit{\text{avoiding}}0\mathit{\text{in}}F$	$n_2\times n_2$
${}_0{\widehat{f}}_{11}\left(0,x,\beta \right)=e^{\mathbb{K}\left(\beta \right)x}{\left(\mathbb{I}+\mathrm{\Xi }\left(\beta ,x\right)\right)}^{-1}$	$\mathit{\text{from}}\left(0,{\Im }_1\right)\mathit{\text{to}}\left(x,{\Im }_1\right)\mathit{\text{avoiding}}0\mathit{\text{in}}F$	$n_1\times n_1$
${}^x{\mathrm{\Psi }}^r\left(\beta \right)={\mathrm{\Psi }}^r\left(\beta \right){-}_0{\widehat{f}}_{22}^r\left(0,x,\beta \right){\mathrm{\Psi }}^r\left(\beta \right)e^{{\mathbb{H}}^r\left(\beta \right)x}$	$\mathit{\text{from}}\left(0,{\Im }_2\right)\mathit{\text{to}}\left(0,{\Im }_1\right)\mathit{\text{avoiding}}x\mathit{\text{in}}{F}^r$	$n_2\times n_1$
${}_0{\widehat{f}}_{22}^r\left(0,x,\beta \right)=e^{{\mathbb{K}}^r\left(\beta \right)x}{\left(\mathbb{I}+{\mathrm{\Xi }}^r\left(\beta ,x\right)\right)}^{-1}$	$\mathit{\text{from}}\left(0,{\Im }_2\right)\mathit{\text{to}}\left(x,{\Im }_2\right)\mathit{\text{avoiding}}0\mathit{\text{in}}{F}^r$	$n_2\times n_2$
$\begin{array}{ll}{\widehat{f}}_{11}\left(0,x,\beta \right)& ={\left[\mathbb{I}-{}^x\mathrm{\Psi }\left(\beta \right){\left(\beta \mathbb{I}-{\mathbb{Q}}_{22}\right)}^{-1}{\mathbb{Q}}_{21}\right]}^{-1}\\ & {\times }_0{\widehat{f}}_{11}\left(0,x,\beta \right)\end{array}$	$\mathit{\text{from}}\left(0,{\Im }_1\right)\mathit{\text{to}}\left(x,{\Im }_1\right)\mathit{\text{in}}F$	$n_1\times n_1$
${\widehat{f}}_{21}\left(0,x,\beta \right)={\left(\beta \mathbb{I}-{\mathbb{Q}}_{22}\right)}^{-1}{\mathbb{Q}}_{21}{\widehat{f}}_{11}\left(0,x,\beta \right).$	$\mathit{\text{from}}\left(0,{\Im }_2\right)\mathit{\text{to}}\left(x,{\Im }_1\right)\mathit{\text{in}}F$	$n_2\times n_1$

Table 2.

LST of first passage times.

2.2 The multi-dimensional martingale

Let $\left\{\mathcal{X}\left(t\right),t\ge 0\right\}$ be a right continuous Markov modulated Lévy process with modulating process $\left\{\mathcal{J}\left(t\right),t\ge 0\right\}$ which is a right continuous irreducible finite state space continuous time Markov chain. Let $\left\{Y\left(t\right),t\ge 0\right\}$ be an adapted continuous process with a finite expected variation on finite intervals and let $\mathcal{Z}\left(t\right)=\mathcal{X}\left(t\right)+Y\left(t\right).$ Asmussen and Kella [31] have shown, that for such a process, the matrix with elements $E_i\left[e^{\alpha \mathcal{X}\left(t\right)};\mathcal{J}\left(t\right)=j\right]$ has the form of $e^{t\mathbb{K}\left(\alpha \right)}$ for some matrix $\mathbb{K}\left(\alpha \right).$ Theorem 2.1 in Asmussen and Kella [31] yields that under certain conditions on $\left\{\mathcal{Z}\left(t\right),t\ge 0\right\},$ the multi-dimensional process

$$M\left(\alpha ,t\right)={\int }_{v=0}^t{e}^{\alpha \mathcal{Z}\left(v\right)}{1}_{\mathcal{J}\left(v\right)}\mathit{dv}\mathbb{K}\left(\alpha \right)+e^{\alpha \mathcal{Z}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}-e^{\alpha \mathcal{Z}\left(t\right)}{\mathbf{1}}_{\mathcal{J}\left(t\right)}+\alpha {\int }_{v=0}^t{e}^{a\mathcal{Z}\left(v\right)}{1}_{\mathcal{J}\left(v\right)}\mathit{dY}\left(v\right)$$E1

is a (row) vector-valued zero mean martingale. Some of the relevant functionals in this paper will be obtained by applying the OST (or Doob’s optional sampling theorem, see Doob [39]) to appropriate special cases of For our models, the inventory level $\mathrm{ℐ}$ is a special case of $\mathcal{X}\left(t\right)$ and has piecewise linear sample paths, with slope $c_j$ on intervals where $\mathcal{J}\left(t\right)$$=j.$

The outline of the chapter is as follows. We start with the ordering model and introduce an extension to the EOQ model, a fluid EOQ-type inventory model. Then, using the fluid EOQ-type model as a key, our study is generalized to include production facility, the fluid EPQ-type model.

3. A fluid EOQ-type inventory model

The simple Economic Ordering Quantity (EOQ) model assumes an inventory model of infinite horizon for a single item. The demand occurs at a constant rate and each time the inventory level hits a level 0, a fixed and immediate order of size $Q$ is places. EOQ considers the timing of reordering, the cost incurred to place an order, and the costs to store merchandise. Here, we generalize the traditional EOQ model and consider demand and return rates that depend on the background environment, and varying state-dependent holding and ordering costs. We start with the mathematical description of the model.

Let $\mathrm{ℐ}=\left\{I\left(t\right):t\ge 0\right\}$ be the on-hand inventory level at time $t$. The rate of change of the level is modulated by a continuous time Markov chain (CTMC) $\left\{\mathcal{J}\left(t\right):t\ge 0\right\}$ on a finite state space $\Im =\left\{1,2,\dots ,n\right\}$ with a generator matrix $\mathbb{Q}=\left[Q_{\mathit{ij}}\right]$. As long as $\mathcal{J}\left(t\right)$ is in state $i,$ there is a demand at rate $d_i$ and a return at rate $r_i.$ The net rate is thus $c_i=r_i-d_i.$ Note that $c_i$ may be either negative or positive. Accordingly, we have two disjoint sets $\left({\Im }_1,{\Im }_2\right)$, $\Im ={\Im }_1\cup {\Im }_2,$ where ${\Im }_1$ is a non-empty set of increasing rates ${\Im }_1=\left\{i\in \Im :c_i>0\right\}$ and ${\Im }_2$ is a non-empty set of the decreasing rates ${\Im }_2=\left\{i\in \Im :c_i<0\right\}.$ Let $\left|{\Im }_1\right|=n_1$ and $\left|{\Im }_2\right|=n_2$; thus $n_1+n_2=n$. Let $\mathit{\pi }=\left[{\pi }_1,\dots ,{\pi }_n\right]$ be the limiting distribution of the $\mathcal{J}\left(t\right)$ process, i.e. $\mathit{\pi }$ is the unique solution to

$$\mathit{\pi }\mathbb{Q}=0,\mathit{\pi }\mathbf{e}=1.$$

The system is stable if and only if the expected input rate is negative, i.e.,

$$\sum _{i=1}^n{\pi }_i{c}_i<0.$$

When $I\left(t\right)$ down-crosses level 0, an order of size $Q>0$ from an external supplier is placed which arrives instantaneously. Thus, immediately after the down-crossing to emptiness by the inventory level in state $i$, the process restarts at level $Q.$ It should be noted, that the EOQ model is a special case of the base–stock polices, in particular, the $\left(s,S\right)$-type, where the reorder level $s=0,$ the replenishment up-to-level $S=Q$ and zero lead time.

Let $T_k$ be the time of the $k$_th jump $\left(T_0=0\right)$ and ${\mathcal{J}}_k=\mathcal{J}\left(T_k\right)$ be the environmental state at $T_k$ (just after the jump). Note that since an order is placed when the inventory drops to level 0, the state ${\mathcal{J}}_k$ has to be a descending one, i.e., ${\mathcal{J}}_k\in {\Im }_2.$ We assume that over $\left[T_{k-1},T_k\right)$$k=1,2..,$ the process $\left\{\mathcal{J}\left(t\right),t\in T_{k-1},T_k\right)\}$ is an irreducible CTMC (continuous time Markov chain) on $\Im$. We call the points where the process jumps up (the replenishment times) order points. They form a semi-renewal process where $T_k^{\text{'}}s$ are the semi regenerative points of the process (see Ross [40]). Define the k_th cycle as the time elapsed between $T_{k-1}$ and $T_k,k=1,2,..$. Denote by $C_k=T_k-T_{k-1},$$k=1,2\dots \left(C_0=0\right)$ the inter-replenishment times; let $C=C_1$. Figure 1 illustrates a sample path of the process $I\left(t\right),t\ge 0$; for simplicity, we assume that $I\left(0\right)=Q,\mathcal{J}\left(0\right)\in {\Im }_n$ and let $\gamma =\left[{\gamma }_1,{\gamma }_2,\dots ,{\gamma }_{n_2}\right]$ be the initial probability vector of $\mathcal{J}\left(0\right)$.

Figure 1.

A typical path of the fluid EOQ-type inventory process.

3.1 The cost structure

Our cost structure composed of two components: (a) an order cost, including a fixed cost whenever an order is placed and a purchasing cost; (b) a holding cost for the stock. We further assume that the cost rates are determined by the environment at the order points; in real-world, order rates usually depend on the state of the economy, the season, and change accordingly. Specifically, if the state at an order point is ${\mathcal{J}}_k=$$i\in {\Im }_2,$ the fixed ordering cost is $K_i$ for an order (typically cost of ordering and shipping and handling), the cost to purchase one item from an external supplier is $q_i,$ and the cost to hold one item in inventory during a time interval of length $\mathit{dt}$ is $h_i\mathit{dt}$ during all the cycle $k$ in which ${\mathcal{J}}_k=i\in {\Im }_2$ (meaning that the holding cost for one unit is constant between two consecutive order replacements, i.e. $h\left(t\right)=h_i$ for $t\in T_k,T_{k+1})$ and ${\mathcal{J}}_k=i\in {\Im }_2$). We next introduce the functionals indicating the expected discounted costs using a discount factor $\beta >0.$

1. Order cost. If ${\mathcal{J}}_k=i\in {\Im }_2,$ an order of size $Q$ is immediately replenished up to level $I\left(T_k\right)=Q$. The order cost is $K_i+q_{i}Q.$ Let $\mathit{OC}\left(\beta \right)$ be the expected discounted order cost and let $\widehat{\mathbf{O}}\left(\beta \right)$ be an $\left(n_2\times 1\right)$ vector whose i_th component ${\widehat{O}}_i\left(\beta \right)$ is given by

$${\widehat{O}}_i\left(\beta \right)=E_i\left[\sum _{k=0}^{\infty }{e}^{-\beta T_k}\left(K_i+q_{i}Q\right)\right],$$E2

i.e., the expected discounted order cost, given $\mathcal{J}\left(0\right)=i\in {\Im }_2,I\left(0\right)=Q.$ Then we have $\mathit{OC}\left(\beta \right)=\gamma \cdot \widehat{\mathbf{O}}\left(\beta \right).$ Applying Table 2, let ${}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)$ be an $\left(n_2\times n_2\right)$ matrix whose $\mathit{ij}$th components is

$${\left({}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)}_{\mathit{ij}}=e^{\mathbb{H}\left(\beta \right)Q},i\in {\Im }_2,j\in {\Im }_2.$$E3

The component${\left({}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)}_{\mathit{ij}}$ represents the LST of the time until the process hits level 0 in state $j\in {\Im }_2$, given $\mathcal{J}\left(0\right)=i\in {\Im }_2,I\left(0\right)=Q$ (for a proof, see Ramaswami [33], Theorem 5).

Lemma 3.1The total expected discounted order cost vector$\widehat{\mathbf{O}}\left(\beta \right)$of order$\left(n_2\times 1\right)$satisfies the following equation:

$$\widehat{\mathbf{O}}\left(\beta \right)={\left(\mathbb{I}-{}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)}^{-1}{\mathrm{\Delta }}_{\mathbf{K}+\mathbf{q}Q}\mathbf{e}.$$E4

where ${\mathrm{\Delta }}_{\mathbf{K}+\mathbf{q}Q}=\mathit{diag}\left(K_i+q_{i}Q\right),i{\in }_{{\Im }_2}.$

Proof. It is easy to verify that $\widehat{\mathbf{O}}\left(\beta \right)$ can be written as (recall that C is the time of the next order):

$$\widehat{\mathbf{O}}\left(\beta \right)={\mathrm{\Delta }}_{\mathbf{K}+\mathbf{q}Q}\mathbf{e}+\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\widehat{\mathbf{O}}\left(\beta \right),$$E5

where the $\left(n_2\times n_2\right)$ matrix $\mathbb{E}\left(e^{-\mathit{\beta C}}\right)$ is the LST of the cycle length; its ij_th component is given by

$$\mathbb{E}{\left(e^{-\mathit{\beta C}}\right)}_{\mathit{ij}}=E\left(e^{-\mathit{\beta C}}{\mathbf{1}}_{\left\{\text{level}0\mathrm{hit}\mathrm{at}\text{time}C\text{in phase}j\right\}}|\mathcal{J}\left(0\right)=i,I\left(0\right)=Q\right).$$E6

Applying the fluid model $\mathbb{E}\left(e^{-\mathit{\beta C}}\right)=$${}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)$ and solving (5) for $\widehat{\mathbf{O}}\left(\beta \right)$ we obtain (4)_■

1. Holding cost. The expected discounted holding cost can be expressed as

$$\mathit{HC}\left(\beta \right)=E\left(\underset{0}{\overset{\infty }{\int }}h\left(t\right)e^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right).$$E7

Let $\widehat{\mathbf{H}}\left(\beta \right)$ be an $\left(n_2\times 1\right)$ vector whose i_th component is given by

$${\widehat{H}}_i\left(\beta \right)=E_i\left(\underset{0}{\overset{\infty }{\int }}h\left(t\right)e^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)$$E8

i.e., the expected discounted holding cost, given $\mathcal{J}\left(0\right)=i\in {\Im }_2.$ Thus, we have $\mathit{HC}\left(\beta \right)=\gamma \cdot \widehat{\mathbf{H}}\left(\beta \right).$

Lemma 3.2The vector$\widehat{\mathbf{H}}\left(\beta \right)$of order$\left(n_2\times 1\right)$satisfies the following equation:

$$\widehat{\mathbf{H}}\left(\beta \right)={\left(\mathbb{I}-{}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)}^{-1}{\mathrm{\Delta }}_{\mathbf{h}}\mathbf{E}\left({\int }_{t=0}^C{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right).$$

Proof. Applying the regenerative theory, and similar to Lemma 3.1, the vector $\widehat{\mathbf{H}}\left(\beta \right)$ can be written as

$$\widehat{\mathbf{H}}\left(\beta \right)={\mathrm{\Delta }}_{\mathbf{h}}\mathbf{E}\left({\int }_{t=0}^C{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)+\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\widehat{\mathbf{H}}\left(\beta \right)$$

(The first vector $\mathbf{E}\left({\int }_{t=0}^C{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)$ is the expected discounted inventory level of the first cycle). Now, we use the OST to the multi-dimensional martingale to find the $\left(n_2\times 1\right)$ vector $\mathbf{E}\left({\int }_{t=0}^C{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right).$ For $\mathcal{J}\left(0\right)=i\in {\Im }_2,I\left(0\right)=Q,$ consider a Lévy process $\left\{{\mathcal{X}}_i\left(t\right)\right\}$ as follows:

$${\mathcal{X}}_i\left(t\right)={\mathcal{X}}_i\left(0\right)-\underset{v=0}{\overset{t}{\int }}{c}_{\mathcal{J}\left(v\right)}\mathit{dv}0\le t<C,{\mathcal{X}}_i\left(0\right)=-Q.$$E9

It is not difficult to see that the latter process up to time $C$, i.e., ${\left({\mathcal{X}}_i\left(t\right)\right)}_{0\le t<C},$ has the same distribution as ${\left(-I\left(t\right)\right)}_{0\le t<C}.$ Chapter XI, p. 311 of Asmussen [41] yields that

$$E_i\left[e^{\alpha {\mathcal{X}}_i\left(t\right)};\mathcal{J}\left(t\right)=j\right]={\left(e^{t\mathbb{K}\left(\alpha \right)}\right)}_{\mathit{ij}}$$

where

$$\mathbb{K}\left(\alpha \right)=\mathbb{Q}-\alpha \mathbb{C}.$$E10

Let $Y\left(t\right)=-\left(\beta /\alpha \right)t$ (for an arbitrary $\alpha >0)$ and let ${\mathcal{Z}}_i\left(t\right)={\mathcal{X}}_i\left(t\right)+Y\left(t\right).$ Since $Y\left(t\right)$ is adapted and has paths of a finite expected variation, the process

$$\begin{array}{ll}{M}_i\left(\alpha ,t\right)& =\underset{v=0}{\overset{t}{\int }}{e}^{\alpha {\mathcal{Z}}_i\left(v\right)}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dv}\mathbb{K}\left(\alpha \right)+e^{\alpha {\mathcal{Z}}_i\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}-e^{\alpha {\mathcal{Z}}_i\left(t\right)}{\mathbf{1}}_{\mathcal{J}\left(t\right)}+\alpha \underset{v=0}{\overset{t}{\int }}{e}^{\alpha {\mathcal{Z}}_i\left(v\right)}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dY}\left(v\right)\\ & =\underset{v=0}{\overset{t}{\int }}{e}^{\alpha {\mathcal{X}}_i\left(v\right)-\mathit{\beta v}}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dv}\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)+e^{\alpha {\mathcal{X}}_i\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}-e^{\alpha {\mathcal{X}}_i\left(t\right)-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\end{array}$$E11

is an $n_2$-dimensional row vector-valued zero mean martingale. The OST yields ${\mathit{EM}}_i\left(\alpha ,0\right)={\mathit{EM}}_i\left(\alpha ,C\right)=0$, i.e.,

$$E_i\left(\underset{t=0}{\overset{C}{\int }}{e}^{\alpha X_i\left(t\right)-\mathit{\beta t}}\mathit{dt}\right)=\left[{\mathbb{E}}_i\left(e^{\alpha {\mathcal{X}}_i\left(C\right)-\mathit{\beta C}}{\mathbf{1}}_{\mathcal{J}\left(C\right)}\right)-{\mathbb{E}}_i\left(e^{\alpha {\mathcal{X}}_i\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)\right]{\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\mathbf{e}.$$E12

Obviously, we have $E_i\left(e^{\alpha {\mathcal{X}}_i\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)=e^{-\mathit{\alpha Q}}$for$i\in {\Im }_2,$ or in an $\left(n_2\times n\right)$ matrix form

$$\mathbb{E}\left(e^{\alpha \mathcal{X}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)=\left(\mathbf{0}{e}^{-\mathit{\alpha Q}}\mathbb{I}\right).$$E13

(Here, the $\left(n_2\times n_1\right)$ zero matrix arising due to the fact that $\mathcal{J}\left(0\right)\in {\Im }_2$). Next, we have to derive ${\mathbf{E}}_i\left(e^{\alpha X_i\left(C\right)-\mathit{\beta C}}{\mathbf{1}}_{\mathcal{J}\left(C\right)}\right).$ Since $X_i\left(C\right)=0,$ the fluid method yields the $\left(n_2\times n\right)$ matrix form

$$\mathbb{E}\left(e^{-\mathit{\beta C}}{\mathbf{1}}_{\mathcal{J}\left(C\right)}\right)=\left({\mathbf{0}}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right).$$E14

Substituting (13) and (14) in (12) we obtain the $\left(n_2\times 1\right)$ vector

$$\mathbf{E}\left(\underset{t=0}{\overset{C}{\int }}{e}^{\alpha \mathcal{X}\left(t\right)-\mathit{\beta t}}\mathit{dt}\right)=\left(\left({\mathbf{0}}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)-\left(\mathbf{0}{e}^{-\mathit{\alpha Q}}\mathbb{I}\right)\right){\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\mathbf{e}.$$E15

Finally, take the derivative of both sides of (15) with respect to $\alpha ,$ let $\alpha =0$ and note that

$$\mathbf{E}\left(\underset{t=0}{\overset{C}{\int }}{e}^{-\mathit{\beta t}}\mathcal{X}\left(t\right)\mathit{dt}\right)=-\mathbf{E}\left(\underset{t=0}{\overset{C}{\int }}{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)$$

leads to

$$\mathbf{E}\left(\underset{t=0}{\overset{C}{\int }}{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)=-\frac{d}{\mathit{d\alpha }}{\left.\left[\left({\mathbf{0}}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)-\left(\mathbf{0}{e}^{-\mathit{\alpha Q}}\mathbb{I}\right){\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\right]\mathbf{e}\right|}_{\alpha =0}.$$E16

Accordingly, the expected discounted total cost is

$$\mathit{TC}\left(\beta \right)=\mathit{OC}\left(\beta \right)+\mathit{HC}\left(\beta \right).$$E17

■

Remark 1It is easy to extend this model to include an order size determined by the environment at the order point, i.e., if$\mathcal{J}\left(T_k\right)=i\in {\Im }_2$, an order of$Q_i$items is placed (and, accordingly,$I\left(0\right)=Q_i$with probability${\gamma }_i,i\in {\Im }_2$). For that we define the vector$\mathbf{Q}=\left(Q_1,\dots ,Q_{n_2}\right).$Now, the entries${\left({}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\right)}_{\mathit{ij}}$are obtained by replacing$Q$with$Q_i$in(3). The order cost is given by(4)with${\mathrm{\Delta }}_{\mathbf{K}+\mathbf{qQ}}$replacing${\mathrm{\Delta }}_{\mathbf{K}+\mathbf{q}Q},$and the holding cost is given by substituting${\mathrm{\Delta }}_{e^{-\alpha \mathbf{Q}}}$instead$e^{-\mathit{\alpha Q}}\mathbb{I}$in(13)(here,${\mathcal{X}}_i\left(0\right)=-Q_i$).

Using the approach for the fluid EOQ-type model as a key, we next include a production facility and introduce the fluid EPQ-type model.

4. A fluid EPQ-type inventory model

Similar as above, let $I\left(t\right)$ be the inventory level at a time $t.$ The process $I\left(t\right)$ can be partitioned into two parts, $I^{+}\left(t\right)$ and $I^{-}\left(t\right).$ The first part of the cycle, $I^{+}\left(t\right),$ is the ON period and this period ends whenever $I^{+}\left(t\right)$ reaches a predetermined level $Q$. The second part of the cycle, $I^{-}\left(t\right),$ is the OFF period and this is the time until $I^{-}\left(t\right)$ drops to level 0. The ON period is characterized by stochastic inputs, production and returns, and outputs, demand. However, no production occurs during the OFF period, and thus, the OFF period is characterized by stochastic returns and demand. Here, the analysis is more challenging due to the periods ON and OFF. Denote by $\left\{p_1,\dots ,p_n\right\}\subset \left(0,\infty \right)$ the production rates, by $\left\{d_1,\dots ,d_n\right\}\subset \left(0,\infty \right)$ the demand rates, and by $\left\{r_1,\dots ,r_n\right\}\subset \left(0,\infty \right)$ the returns rates. The rate at which the inventory is filled at time t is determined by the current environmental state $\mathcal{J}\left(t\right)$. During the ON period (OFF period) and as long as $\mathcal{J}\left(t\right)=i,$ the growth rate is the difference of the production and return rates (only the return rate) and the demand rate, $c_i^{+}=r_i+p_i-d_i$ ($c_i^{-}=r_i-d_i$); similarly as before, $c_i,i=1,..,n$ may be either negative or positive. We do not allow backlog; thus when $I^{+}\left(t\right)$ drops to level 0 (due to high demands), it stays there as long as the environmental growth rate is negative and until the environmental state changes to some positive growth rate. Note that the behavior of the process during each period is different. During the OFF period, we enforce that ${\sum }_{i=1}^n{\pi }_i{c}_i^{-}<0$ (a necessary and sufficient condition for the stability of the OFF process). During the ON period, we assume that ${\sum }_{i=1}^n{\pi }_i{c}_i^{+}>0$ (although with the absence of this, $I^{+}\left(t\right)$ is stable due to the reflection at level 0, however, it is a realistic assumption in order to avoid high lost demand).

A typical sample path of the inventory process is given in Figure 2. As we see, the inventory process is a semi-regenerative process which alternates between ON periods and OFF periods. We further note that if the ON periods are deleted from the sample path and the OFF periods are glued together we obtain a fluid EOQ-type model with refilling every time level 0 is reached. This holds with one exception; while each cycle under the fluid EOQ-type process starts with a descending state, it’s not necessarily holds in the fluid EPQ-type model (a detailed explanation is given below).

Figure 2.

A typical path of the fluid EPQ-type inventory process.

Define the following stopping times:

$$\begin{array}{l}{T}_0=0\\ T_k=inf\left\{t>L_k:I\left(t\right)=0\right\}k=1,2\dots \\ L_k=inf\left\{t\ge T_{k-1}:I\left(t\right)=Q\right\}k=1,2,\dots \end{array}$$

$T_k,k=1,2,\dots$ are the times of switchings from OFF to ON, and $L_n,n=1,2,\dots$are the time instants of switchings from ON to OFF. Thus, $I\left(t\right)$ is a semi-regenerative process with $T_n^{\text{'}}s$ are semi-regenerative points. Define the k_th cycle as the time elapsed between $T_{k-1}$ and $T_k,k=1,2,..$ and let $C_k=T_k-T_{k-1},k=1,2\dots$ be the kth cycle length. We use the generic form $L=L_1,T=T_1,T^{\prime }=T-L$ and $C=C_1$ (so $C=L+T^{\prime }$). Note that conditioning on the state at time $L$ and the common background environmental process, the two process $I^{+}{\left(t\right)}_{0\le t\le L}$ (the ON period) and $I^{-}{\left(t\right)}_{L\le t\le T}$ (the OFF period) are independent.

We construct diagonal matrices ${\mathbb{C}}_j^{+}=\mathit{diag}\{\left|c_i^{+}\right|,$$i\in {\Im }_j\}$, ${\mathbb{C}}_j^{-}=\mathit{diag}\{\left|c_i^{-}\right|,$$i\in {\Im }_j\},j=1,2$ and ${\mathbb{C}}^{+}=\mathit{diag}\left({\mathbb{C}}_1^{+},{\mathbb{C}}_2^{+}\right)$ and ${\mathbb{C}}^{-}=\mathit{diag}\left({\mathbb{C}}_1^{-},{\mathbb{C}}_2^{-}\right)$ from these rates. Regarding the fluid EPQ-type model, each ON/OFF period has one type of rates $({\mathbb{C}}^{+}$ or ${\mathbb{C}}^{-})$; hence, given the state at switching epoch and the common environment, the ON/OFF periods are independent. Now, we can analyze the inventory level within each period independently using MMFF process. Specifically, we consider $I^{+},$${\mathbb{C}}^{+}$ and ${\mathrm{\Psi }}^{+}\left(\beta \right)$ ($I^{-},$${\mathbb{C}}^{-},$${\mathrm{\Psi }}^{-}\left(\beta \right)$) corresponding to the ON (OFF) period. Note that, for this model, each LST matrix should be derived for ON (marked as +) and OFF (marked as −) processes. However, we do not insert the marks + or − corresponding to ON/OFF processes; it should be clear from the context which of the marks applies.

4.1 The cost functionals

We consider four costs: (a) the setup cost, (b) the holding cost of the inventory, (c) the production cost, and (d) the unsatisfied demand cost. For the derivation of the costs, we first need to derive the expected discounted cycle length $\left(n\times n\right)$ matrix $\mathbb{E}\left(e^{-\mathit{\beta C}}\right)$.

4.2 The matrix $\mathbb{E}\left(e^{-\mathit{\beta C}}\right)$

Given the state $\mathcal{J}\left(L\right),$ the ON period and the OFF period are independent. Clearly, the ON period ends at state in ${\Im }_1$ (at time $L$). However, at that point of time, the production stops, and thus, the state at which the OFF period starts can be either in ${\Im }_1$ or in ${\Im }_2.$ Similarly, the OFF period ends at state in ${\Im }_2$ (at time $T$). However, since the production starts at that point, the ON period can start either in state in ${\Im }_1$ or in ${\Im }_2$. This needs to be considered, particularly at switching times. Hence, at these switching times, the phases are labeled by arranging the entries according to the new states. Given the state $\mathcal{J}\left(L\right),$ and due to that independency, the matrix $\mathbb{E}\left(e^{-\mathit{\beta C}}\right)$ is given by

$$\mathbb{E}\left(e^{-\mathit{\beta C}}\right)=\mathbb{E}\left(e^{-\mathit{\beta L}}\right)\mathbb{E}\left(e^{-\beta T^{\text{'}}}\right).$$E18

We first introduce two similar LST matrices which differ only with their initial environment: (i) an $\left(n_1\times n_1\right)$ matrix ${\widehat{f}}_{11}\left(0,Q,\beta \right)$ whose $\mathit{ij}$th component ${\left({\widehat{f}}_{11}\left(0,Q,\beta \right)\right)}_{\mathit{ij}}$ represents the LST of the time until hitting $Q$ in state $j\in {\Im }_1$, given $I\left(0\right)=0,\mathcal{J}\left(0\right)=i\in {\Im }_1$ and (ii) an $\left(n_2\times n_1\right)$ matrix ${\widehat{f}}_{21}\left(0,Q,\beta \right)$ whose $\mathit{ij}$th component ${\left({\widehat{f}}_{21}\left(0,Q,\beta \right)\right)}_{\mathit{ij}}$ represents the LST of the time until hitting $Q$ in state $j\in {\Im }_1$, given $I\left(0\right)=0,\mathcal{J}\left(0\right)=i\in {\Im }_2$. Now, define the $\left(n\times n_1\right)$ matrix:

$${\widehat{f}}_{\cdot 1}\left(0,Q,\beta \right)=\left(\begin{array}{c}{\widehat{f}}_{11}\left(0,Q,\beta \right)\\ {\widehat{f}}_{21}\left(0,Q,\beta \right)\end{array}\right).$$

Since the ON period ends with environment in ${\Im }_1$ (at time $L$), the $\left(n\times n\right)$ matrix $\mathbb{E}\left(e^{-\mathit{\beta L}}\right)$ has the form:

$$\mathbb{E}\left(e^{-\mathit{\beta L}}\right)=\left({\widehat{f}}_{\cdot 1}\left(0,Q,\beta \right)\mathbf{0}\right).$$E19

Similarly, define the $\left(n\times n_2\right)$ matrix:

$${}_0{\widehat{f}}_{\cdot 2}\left(Q,0,\beta \right)=\left(\begin{array}{c}{}_0{\widehat{f}}_{12}\left(Q,0,\beta \right)\\ {}_0{\widehat{f}}_{22}\left(Q,0,\beta \right)\end{array}\right).$$

(See also (3)). The OFF period ends with environment in ${\Im }_2$ (at time T), and thus, the $\left(n\times n\right)$ matrix $\mathbb{E}\left(e^{-\beta T^{\prime }}\right)$ has the form of

$$\mathbb{E}\left(e^{-\beta T^{\prime }}\right)=\left({\mathbf{0}}_0{\widehat{f}}_{\cdot 2}\left(Q,0,\beta \right)\right).$$E20

(Recall that all matrices are given in Table 1). Next, we derive the expected discounted costs.

1. Set up cost. Let $K_1$ be the setup cost to switch from OFF to ON (at time T) and $K_2$ be the setup cost to switch from ON to OFF (at time $L$). Let $\mathit{SC}\left(\beta \right)$ be the expected discounted set up cost and let $\widehat{\mathbf{S}}\left(\beta \right)$ be an $\left(n\times 1\right)$ vector whose i_th component is the expected discounted set up cost given $\mathcal{J}\left(0\right)=i\in \Im ,I\left(0\right)=0,$

   $$\widehat{\mathbf{S}}\left(\beta \right)=E_i\sum _{n=1}^{\infty }\left(K_{2}exp\left(-\beta L_n\right)+K_{1}exp\left(-\beta T_n\right)\right).$$E21

   Similar technique as before arrives at

   $$\begin{array}{l}\mathit{SC}\left(\beta \right)=\gamma \widehat{\mathbf{S}}\left(\beta \right)\\ \widehat{\mathbf{S}}\left(\beta \right)={\left(\mathbb{I}-\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\right)}^{-1}\left(K_1\mathbb{E}\left(e^{-\mathit{\beta C}}\right)+K_2\mathbb{E}\left(e^{-\mathit{\beta L}}\right)\right)\mathbf{e}.\end{array}$$E22

   Substituting (18) and (19) finalizes the derivation.

2. Holding cost. The total expected discounted holding cost $\mathit{HC}\left(\beta \right)=\gamma \widehat{\mathbf{H}}\left(\beta \right),$ where the vector $\widehat{\mathbf{H}}\left(\beta \right)$ is of order $\left(n\times 1\right)$; its i_th components is the discounted expected holding cost given $\mathcal{J}\left(0\right)=i\in \Im ,I\left(0\right)=0.$ Revoking the ergodic theorem for regenerative process, we can write $\widehat{\mathbf{H}}\left(\beta \right)$ in terms of the first cycle and have

$$\widehat{\mathbf{H}}\left(\beta \right)={\left(\mathbb{I}-\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\right)}^{-1}{\mathrm{\Delta }}_{\mathbf{h}}\left[\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{I}^{+}\left(t\right)\mathit{dt}\right)+\mathbb{E}\left(e^{-\mathit{\beta L}}\right)\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{-\mathit{\beta t}}{I}^{-}\left(t+L\right)\mathit{dt}\right)\right].$$E23

The basic tool we use to derive $\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{I}^{+}\left(t\right)\mathit{dt}\right)$ and $\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{-\mathit{\beta t}}{I}^{-}\left(t+L\right)\mathit{dt}\right)$ is OST to the Asmussen–Kella multi-dimensional martingale, as introduced in Lemma 3.2. Here, we consider the process

$$\mathcal{X}\left(t\right)={\int }_{v=0}^t{c}_{\mathcal{J}\left(v\right)}\mathit{dv}0\le t\le L,\mathcal{X}\left(0\right)=0.$$E24

Chapter XI, p. 311 of Asmussen [41] yields that $E_i\left[e^{\alpha \mathcal{X}\left(t\right)};\mathcal{J}\left(t\right)=j\right]={\left(e^{t\mathbb{K}\left(\alpha \right)}\right)}_{\mathit{ij}}$ where $\mathbb{K}\left(\alpha \right)=\mathbb{Q}+\alpha \mathbb{C}$ (specifically, ${\mathbb{K}}^{+}\left(\alpha \right)=\mathbb{Q}+\alpha {\mathbb{C}}^{+}$). let $\mathcal{L}\left(t\right)=-{min}_{0\le v\le t}\mathcal{X}\left(v\right).$ The process $\mathcal{L}\left(t\right),$ known as the local time, is a non-decreasing process that increases only whenever $I^{+}\left(t\right)=0$ (for more details on local time and its properties, we refer the interested reader to [41]). Next, we let $\mathcal{Z}\left(t\right)=\mathcal{X}\left(t\right)+\mathcal{L}\left(t\right).$ It is not difficult to see that the latter process up to time $L,$ i.e., ${\left(\mathcal{Z}\left(t\right)\right)}_{0\le t<L},$ has the same distribution as ${\left(I\left(t\right)\right)}_{0\le t<L}.$ Finally, define $Y\left(t\right)=\mathcal{L}\left(t\right)-\left(\beta /\alpha \right)t,$ for arbitrary $\beta \ge 0$ and $\alpha <0,$ and $W\left(t\right)=\mathcal{X}\left(t\right)+Y\left(t\right)=\mathcal{Z}\left(t\right)-\left(\beta /\alpha \right)t.$ Since $Y$ is adapted and has paths of finite expected total variation on bounded intervals, Theorem 2.1 of Asmussen and Kella [31] leads to the next claim.

Claim 4.1The$\left(n\times 1\right)$vector$\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{I}^{+}\left(t\right)\mathit{dt}\right)$is given by:

$$\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{I}^{+}\left(t\right)\mathit{dt}\right)=\frac{d}{\mathit{d\alpha }}{\left.\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{\alpha \mathcal{Z}\left(t\right)-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\mathit{dt}\right)\right|}_{\alpha =0},$$E25

where

$$\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{\alpha \mathcal{Z}\left(t\right)-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\mathit{dt}\right)=\left[e^{\mathit{\alpha Q}}\mathbb{E}\left(e^{-\mathit{\beta L}}\right)-\mathbb{I}-\alpha \left(\mathbf{0},\widehat{\mathbf{L}}\left(\beta \right)\right)\right]{\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\mathbf{e}.$$E26

Proof. The proof and the derivation of $\widehat{\mathbf{L}}\left(\beta \right)$ are given in Appendix A. ■

In order to finish the holding cost we have to find the $\left(n\times 1\right)$ vector $\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{-\mathit{\beta t}}{I}^{-}\left(t\right)\mathit{dt}\right).$ Recall that since we are now dealing with the OFF period, for all the matrices in Table 1, we have to add the index (−). The OFF period stars at time $L$. We shift the time origin to $L$ so that the OFF period starts at time 0. Consider $\tilde{\mathcal{X}}\left(t\right)$ similar to (9) but with ${\mathbb{C}}^{-}$ and $\tilde{\mathcal{X}}\left(0\right)=Q,\mathcal{J}\left(0\right)\in \Im .$ Clearly, the latter process up to time $T^{\prime }$, i.e., ${\left(\tilde{\mathcal{X}}\left(t\right)\right)}_{0\le t<T^{\prime }}$, has the same distribution as ${\left(I\left(t\right)\right)}_{0\le t<T^{\text{'}}}.$ Similar arguments to (15) leads to:

$$\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{-\mathit{\beta t}}I\left(t\right)\mathit{dt}\right)=\frac{d}{\mathit{d\alpha }}{\left.\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{\alpha \tilde{\mathcal{X}}\left(t\right)-\mathit{\beta t}}\mathit{dt}\right)\right|}_{\alpha =0},$$E27

where

$$\begin{array}{cc}\mathbf{E}\left(\underset{0}{\overset{T^{\text{'}}}{\int }}{e}^{\alpha \tilde{\mathcal{X}}\left(t\right)-\mathit{\beta t}}\mathit{dt}\right)& =\left[\mathbb{E}\left(e^{\alpha \tilde{\mathcal{X}}\left(T^{\text{'}}\right)-\beta T^{\text{'}}}{\mathbf{1}}_{\mathcal{J}\left(T^{\text{'}}\right)}\right)-\mathbb{E}\left(e^{\alpha \tilde{\mathcal{X}}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)\right]{\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\mathbf{e}\\ & =\left(\mathbb{E}\left(e^{-\beta T^{\text{'}}}\right)-e^{-\mathit{\alpha Q}}\mathbb{I}\right){\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}\mathbf{e}.\end{array}$$E28

1. Production cost. Let $q_i$ be the production cost for one unit. We have:

$$\begin{array}{l}\mathit{PC}\left(\beta \right)=\gamma \widehat{\mathbf{P}}\left(\beta \right),\\ \widehat{\mathbf{P}}\left(\beta \right)={\left(\mathbb{I}-\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\right)}^{-1}{\mathrm{\Delta }}_{\mathbf{q}}\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{p}_{\mathcal{J}\left(t\right)}{e}^{-\mathit{\beta t}}\mathit{dt}\right).\end{array}$$E29

Note that the OFF period is characterized by no production. For the derivation of $\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{p}_{\mathcal{J}\left(t\right)}{e}^{-\mathit{\beta t}}\mathit{dt}\right)$ (ON period), let ${\mathbb{P}}_j^{+}=\mathit{diag}\left\{p_i,i\in {\Im }_j\right\},j=1,2$ and ${\mathbb{P}}^{+}=\mathit{diag}\left({\mathbb{P}}_1^{+},{\mathbb{P}}_2^{+}\right).$ Thus,

$$\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{p}_{\mathcal{J}\left(t\right)}{e}^{-\mathit{\beta t}}\mathit{dt}\right)=\mathbb{E}\left({\int }_0^L{e}^{-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\mathit{dt}\right){\mathbb{P}}^{+}\mathbf{e}\mathbf{.}$$E30

Applying (26) with $\alpha =0$ yields

$$\mathbb{E}\left({\int }_0^L{e}^{-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\mathit{dt}\right)=\left(\mathbb{E}\left(e^{-\mathit{\beta L}}\right)-\mathbb{I}\right){\left(\mathbb{K}\left(0\right)-\beta \mathbb{I}\right)}^{-1},$$E31

which completes the derivation of the production cost.

1. Lost demand cost. In our model, backlog is not allowed; any demand which cannot be satisfied immediately is lost. Clearly, there is no unsatisfied demand during the OFF period. During ON period, once level 0 is reached the process stays there until the environment changes to state with a positive growth rate. Assume the process hits 0 at state $i$ (for some $c_i^{+}<0),$ the demand is lost with rate $\left(-c_i^{+}\right)$ until the environmental state changes. Let $\mathit{wdt}$ be the cost for a lost unit during a time interval of length $\mathit{dt}$ ($w>0$). As a measure for the expected discounted lost demand cost one can use the functional

$$\begin{array}{l}\mathit{UC}\left(\beta \right)=\gamma \widehat{\mathbf{U}}\left(\beta \right)\\ \widehat{\mathbf{U}}\left(\beta \right)=-\mathit{w\gamma }{\left(\mathbb{I}-\mathbb{E}\left(e^{-\mathit{\beta C}}\right)\right)}^{-1}\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{c}_{\mathcal{J}\left(t\right)}^{+}{\mathbf{1}}_{\left\{I^{+}\left(t\right)=0\right\}}\mathit{dt}\right)\end{array}$$E32

The right term is the expected discounted production loss during the first cycle; it can be written in terms of the local time process.

Corollary 4.1It is easy to verify that

$$\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{c}_{\mathcal{J}\left(t\right)}^{+}{\mathbf{1}}_{\left\{I^{+}\left(t\right)=0\right\}}\mathit{dt}\right)=\mathbf{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}d\mathcal{L}\left(t\right)\right)=\widehat{\mathbf{L}}\left(\beta \right)\mathbf{e},$$E33

where$\widehat{\mathbf{L}}\left(\beta \right)$is given in(40).

A simple cost function for the entire system would be the sum, $\mathit{TC}\left(\beta \right)$, of these four expected discounted costs:

$$\mathit{TC}\left(\beta \right)=\mathit{SC}\left(\beta \right)+\mathit{PC}\left(\beta \right)+\mathit{HC}\left(\beta \right)+\mathit{UC}\left(\beta \right).$$E34

5. Summary

During the past few decades, the problem of control of inventory systems has been widely investigated. Many stochastic factors inherent in inventory systems can make it more difficult for managers to plan and control the inventory. Dealing with the randomness of demand, production, and returns, this study considers a continuous-review inventory system where the inventory level is characterized as a fluid process modelled by Markovian environment. The cost structure includes an order cost, a purchase cost, a set up cost, a production cost, an inventory cost, and a lost cost due to unsatisfied demands. By taking a simple probability approach and by applying stopping time theory to fluid processes and martingales, the explicit components of the resulting costs are derived. These cost components can be used for optimization purposes. Moreover, the closed-form expression of the components allows us to obtain efficiently and numerically the optimal parameters and enables us to investigate the behavior of the system and to study its properties. From a managerial perspective, our framework can be applied to many industries, in situations where the system is subject to uncertain environment. Our approach appears to be a powerful way to address related inventory problems. The models considered here assume only discrete demand and return sizes. For instance, it seems to be possible to adapt a similar approach in the case of continuous demand size distributions, such as exponential, uniform, or gamma. Another avenue of research would be to extend this framework to include random lead times; in that case, two policies can be applied while shortage, backordering or lost sales. All of these nontrivial extensions are tractable and worthy of study.

Proof of claim 4.1. Theorem 2.1 of Asmussen and Kella [31] yields that the process

$$\begin{array}{l}M\left(\alpha ,t\right)=\underset{v=0}{\overset{t}{\int }}{e}^{\mathit{\alpha W}\left(v\right)}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dv}\mathbb{K}\left(\alpha \right)+e^{\mathit{\alpha W}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}-e^{\mathit{\alpha W}\left(t\right)}{\mathbf{1}}_{\mathcal{J}\left(t\right)}+\alpha \left(\underset{v=0}{\overset{t}{\int }}{e}^{\mathit{\alpha W}\left(v\right)}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dY}\left(v\right)\right)\\ =\underset{v=0}{\overset{t}{\int }}{e}^{\alpha \mathcal{Z}\left(v\right)-\mathit{\beta v}}{\mathbf{1}}_{\mathcal{J}\left(v\right)}\mathit{dv}\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)+e^{\alpha \mathcal{Z}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}-e^{\alpha \mathcal{Z}\left(t\right)-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\\ +\alpha \left(\underset{v=0}{\overset{t}{\int }}{e}^{-\mathit{\beta v}}{\mathbf{1}}_{\mathcal{J}\left(v\right)}d\mathcal{L}\left(v\right)\right)\end{array}$$E35

is an n-row vector-valued zero mean martingale. The OST yields $\mathbb{E}\left(M\left(\alpha ,L\right)\right)=\mathbb{E}\left(M\left(\alpha ,0\right)\right)=0$,

$$\begin{array}{l}\mathbb{E}\left(\underset{0}{\overset{L}{\int }}{e}^{\alpha \mathcal{Z}\left(t\right)-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}\mathit{dt}\right)=[\mathbb{E}\left(e^{\alpha \mathcal{Z}\left(L\right)-\mathit{\beta L}}{\mathbf{1}}_{\mathcal{J}\left(L\right)}\right)-\mathbb{E}\left(e^{\alpha \mathcal{Z}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)\\ -\alpha \mathbb{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}d\mathcal{L}\left(t\right)\right)]{\left(\mathbb{K}\left(\alpha \right)-\beta \mathbb{I}\right)}^{-1}.\end{array}$$

Since $\mathcal{Z}\left(0\right)=0$ and $\mathcal{J}\left(0\right)\in \Im ,$ we obtain the $\left(n\times n\right)$ matrix $\mathbb{E}\left(e^{\alpha \mathcal{Z}\left(0\right)}{\mathbf{1}}_{\mathcal{J}\left(0\right)}\right)=\mathbb{I}$. Applying $\mathcal{Z}\left(L\right)=Q$ leads to

$$\mathbb{E}\left(e^{\alpha \mathcal{Z}\left(L\right)-\mathit{\beta L}}{\mathbf{1}}_{\mathcal{J}\left(L\right)}\right)=e^{\mathit{\alpha Q}}\mathbb{E}\left(e^{-\mathit{\beta L}}\right).$$E36

To finish, we have to derive the $\left(n\times n\right)$ matrix $\mathbb{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}d\mathcal{L}\left(t\right)\right),$ which is the expected discounted lost demand until time L. Notice that a loss occurs only during states in ${\Im }_2.$ For that, we introduce the $\left(n_2\times n_2\right)$ matrix $\mathtt{\Upsilon }\left(\beta \right)$ whose ij_th component, ${\mathrm{\Upsilon }}_{\mathit{ij}}\left(\beta \right),$ is the expected discounted loss in state $j\in {\Im }_2$ until exiting level 0, given that the process starts at level 0 with state $i\in {\Im }_2.$

Lemma A.1${\mathrm{\Upsilon }}_{\mathit{ij}}\left(\beta \right)$ satisfies the following system of linear equations:

$$\begin{array}{l}{\mathrm{\Upsilon }}_{\mathit{ii}}\left(\beta \right)=\frac{-c_i}{\beta }\left(1+\frac{{\mathbb{Q}}_{22}\left(i,i\right)}{\beta -{\mathbb{Q}}_{22}\left(i,i\right)}\right)+\sum _{j\ne i}\frac{{\mathbb{Q}}_{22}\left(i,j\right)}{\beta -{\mathbb{Q}}_{22}\left(i,i\right)}{\mathrm{\Upsilon }}_{\mathit{ji}}\left(\beta \right),\\ {\mathrm{\Upsilon }}_{\mathit{ij}}\left(\beta \right)=\sum _{k\ne i}\frac{{\mathbb{Q}}_{22}\left(i,k\right)}{\beta -{\mathbb{Q}}_{22}\left(i,i\right)}{\mathrm{\Upsilon }}_{\mathit{kj}}\left(\beta \right).\end{array}$$E37

Proof. Once the inventory process hits 0 in state $i\in {\Im }_2,$ it stays there for an exponential random time ${\xi }_i$ with parameter $-{\mathbb{Q}}_{22}\left(i,i\right).$ With probability $\frac{{\mathbb{Q}}_{22}\left(i,j\right)}{-{\mathbb{Q}}_{22}\left(i,i\right)}$ the state changes to $j\in \Im$, and the expected discounted loss from state $j$ is ${\mathrm{\Upsilon }}_j\left(i\right).$ By conditioning on the first state visited after $i$, we readily obtain that

$$\begin{array}{l}{\mathrm{\Upsilon }}_{\mathit{ii}}\left(\beta \right)=-E\left(\underset{t=0}{\overset{{\xi }_i}{\int }}{c}_i{e}^{-\mathit{\beta t}}\mathit{dt}\right)+E_i\left(e^{-{\mathit{\beta \xi }}_i}\right)\sum _{\begin{array}{c}i\ne j\\ j\in {\Im }_2\end{array}}\frac{{\mathbb{Q}}_{22}\left(i,j\right)}{-{\mathbb{Q}}_{22}\left(i,i\right)}{\mathrm{\Upsilon }}_{\mathit{ji}}\left(\beta \right),\\ {\mathrm{\Upsilon }}_{\mathit{ij}}\left(\beta \right)=E_i\left(e^{-{\mathit{\beta \xi }}_i}\right)\sum _{\begin{array}{c}k\ne i\\ k\in {\Im }_2\end{array}}\frac{{\mathbb{Q}}_{22}\left(i,k\right)}{-{\mathbb{Q}}_{22}\left(i,i\right)}{\mathrm{\Upsilon }}_{\mathit{kj}}\left(\beta \right).\end{array}$$E38

Solving (38) with respect to ${\mathrm{\Upsilon }}_{\mathit{ii}}\left(\beta \right)$ and ${\mathrm{\Upsilon }}_{\mathit{ij}}\left(\beta \right)$ returns (37) (note that $E\left(\underset{t=0}{\overset{{\xi }_i}{\int }}{c}_i{e}^{-\mathit{\beta t}}\mathit{dt}\right)=\frac{c_i}{\beta }\left(1+\frac{{\mathbb{Q}}_{22}\left(i,i\right)}{\beta -{\mathbb{Q}}_{22}\left(i,i\right)}\right)$ and $E_i\left(e^{-{\mathit{\beta \xi }}_i}\right)=\frac{-{\mathbb{Q}}_{22}\left(i,i\right)}{\beta -{\mathbb{Q}}_{22}\left(i,i\right)})$). ■.

Now, we apply $\mathtt{\Upsilon }\left(\beta \right)$ to the derivation of the loss until time $L.$ Since the lost demand occurs only for states in ${\Im }_2,$ the matrix $\mathbb{E}\left(\underset{v=0}{\overset{L}{\int }}{e}^{-\mathit{\beta v}}{\mathbf{1}}_{\mathcal{J}\left(v\right)}d\mathcal{L}\left(v\right)\right)$ has the form

$$\mathbb{E}\left(\underset{0}{\overset{L}{\int }}{e}^{-\mathit{\beta t}}{\mathbf{1}}_{\mathcal{J}\left(t\right)}d\mathcal{L}\left(t\right)\right)=\left(\mathbf{0},\widehat{\mathbf{L}}\left(\beta \right)\right)$$E39

where $\widehat{\mathbf{L}}\left(\beta \right)$ is an $\left(n\times n_2\right)$ matrix whose ij_th component is the expected discounted loss in state $j\in {\Im }_2$ until $L,$ given $\mathcal{J}\left(0\right)=i\in \Im$. Let ${\widehat{\mathbf{L}}}_i\left(\beta \right),i=1,2$ be an $\left(n_i\times n_2\right)$ sub-matrix of $\widehat{\mathbf{L}}\left(\beta \right)$ includes all rows corresponding to states in ${\Im }_i,i=1,2,$ such that

$$\widehat{\mathbf{L}}\left(\beta \right)=\left(\begin{array}{c}{\widehat{\mathbf{L}}}_1\left(\beta \right)\\ {\widehat{\mathbf{L}}}_2\left(\beta \right)\end{array}\right).$$E40

It is easy to verify that

$$\begin{array}{l}{\widehat{\mathbf{L}}}_1\left(\beta \right){=}^Q\mathrm{\Psi }\left(\beta \right)\left(\mathtt{\Upsilon }\left(\beta \right)+{\left(\beta \mathbb{I}-{\mathbb{Q}}_{22}\right)}^{-1}{\mathbb{Q}}_{21}{\widehat{\mathbf{L}}}_1\left(\beta \right)\right),\\ {\widehat{\mathbf{L}}}_2\left(\beta \right)=\mathtt{\Upsilon }\left(\beta \right)+{\left(\beta \mathbb{I}-{\mathbb{Q}}_{22}\right)}^{-1}{\mathbb{Q}}_{21}{\widehat{\mathbf{L}}}_1\left(\beta \right).\end{array}$$E41

Lost demand occurs when the process drops to level 0 avoiding $Q,$ with LST ${}^Q\mathrm{\Psi }\left(\beta \right).$ From that point, $\mathtt{\Upsilon }\left(\beta \right)$ is the discounted lost demand. The term ${\left(\beta \mathbb{I}-{\mathbb{Q}}_{22}\right)}^{-1}{\mathbb{Q}}_{21}$ is the discounted time until exiting level 0 with environmental state in ${\Im }_1$ and starting again with LST ${\widehat{\mathbf{L}}}_1\left(\beta \right).$ The matrix ${\widehat{\mathbf{L}}}_2\left(\beta \right)$ is derived similarly.