Open access peer-reviewed chapter

Fluid Inventory Models under Markovian Environment

Written By

Yonit Barron

Reviewed: 02 March 2022 Published: 25 June 2022

DOI: 10.5772/intechopen.104183

From the Edited Volume

Logistics Engineering

Edited by Samson Jerold Samuel Chelladurai, Suresh Mayilswamy, S. Gnanasekaran and Ramakrishnan Thirumalaisamy

Chapter metrics overview

69 Chapter Downloads

View Full Metrics


Today’s products are subject to fast changes due to market conditions, short life cycles, and technological advances. Thus, an important problem in inventory planning is how to effectively manage the inventory control in a dynamic and stochastic environment. The traditional Economic Order Quantity (EOQ) and Economic Production Quantity (EPQ) both are widely and successfully used models of inventory management. However, both models assume constant and fixed parameters over time. Unfortunately, most of these assumptions are unrealistic. In this study, we generalize the EOQ and EPQ models and study production-inventory fluid models operating in a stochastic environment. The inventory level increases or decreases according to a fluid-flow rate modulated by an n-state continuous time Markov chain (CTMC). Our main objective is to minimize the expected discounted total cost which includes ordering, purchasing, production, set up, holding, and shortage costs. Applying regenerative theory, optional sampling theorem (OST) to the multi-dimensional martingale and fluid flow techniques, we develop methods to obtain explicit formulas for these cost functionals. As such, we provide managers with a useful framework and an efficient and easy-to-implement tool to coop with different demand–supply patterns.


  • Inventory/production
  • Markov chain
  • Fluid flow
  • Renewal theory
  • Martingales
  • EOQ
  • EPQ

1. Introduction

An important problem in inventory planning is how to effectively manage the inventory control in a dynamic and stochastic environment. Particularly, today’s products are subject to fast changes due to market conditions, short life cycles, and technological advances. The uncertainties in the supply-chain hierarchy make inventory control very challenging and the structure of the optimal policy is still unknown; thus, both researchers and practitioners are focusing on relatively simple control policies. This focus is especially true in practice where there are uncertainties in the production rate, demand rate, and lead time. The specification of the production rates, order quantities, the storage capacity and the backlog possibilities has to take into account the costs for ordering, production, the holding of inventory, backlogging, and lost sales. For example, for a factory Manufacturer, fixing the production rate at some high level or ordering a large quantity avoids backlogs but may cause high production/order cost and inventory costs; however, fixing the production rate at some low level, or ordering many but small quantities may lead to severe backlogging costs and the loss of sale opportunities. An inventory control policy governs inventory replenishment decisions by specifying when and how many items should be ordered or produced, thus, the planning of such a supply chain subject to market uncertainty is challenging in terms of dealing with uncertainties and variations.

The traditional Economic Order Quantity (EOQ) and Economic Production Quantity (EPQ) both are widely and successfully used models of inventory management (see Nahmias [1]). In the classical EOQ model the demand occurs at a constant rate, and each time the inventory level hits level 0 an immediate and fixed order of size Q is placed. In the classic EPQ model, every production cycle is composed of ON and OFF deterministic periods. There exists a predetermined level, say Q, such that the system is ON and the inventory level increases from level 0 up to level Q. When level Q is reached, the production is stopped and the inventory decreases down to 0. The time it takes from Q to 0 is the OFF period. Notice that the EOQ model, in which the items are obtained from an outside suppliers, is just a special case of the EPQ model, by letting the production rate to be almost infinite.

The EOQ model derives the optimum order size that should be placed with a vendor to minimize the holding and ordering costs. On the other hand, the EPQ model determines the optimum production size that is to be manufactured to avoid unnecessary blockage of funds and excess storage costs. Both models consider the timing of reordering or production, the cost incurred due to order/production, and the holding costs to store items; holding cost can further be in the form of rentals for the storage area, salaries of personnel looking after the inventory, electricity bills, repairs, maintenance, etc. Thus, both models describe the trade-off between fixed ordering/producing costs and variable holding costs. Furthermore, both models assume that the demand and production rates are constant over time. The traditional EOQ model assumes that the order arrives with no time, and its replenishment will happen as soon as it reaches the minimum threshold level (usually level 0). Similarly, the traditional EPQ model assumes that the production can starts immediately as the stock goes down below a minimum level (usually level 0). The price is fixed and constant while making a purchase under EOQ or producing under EPQ. The key difference between the two is that the EOQ model is applied when the items are ordered from a third party, and the EPQ model comes into use when the company is the producer itself of the products.

Unfortunately, in real-world, most of the above assumptions are unrealistic; holding and ordering costs may vary due to change in rentals, salaries of personnel, and other overhead expenses. The demand rate, as well as the price of a product, can hardly be constant. They fluctuate a lot in the real world. Consumer income, tastes, and preferences, prices of inputs and raw materials, seasonal variation in demand, etc. are key factors that will affect demand as well as price. Similarly, under the EPQ model, the production process also does not remain constant because of factors like an interruption in power supply, breakages, and repairs in plant and machinery, overheating, change in the quality of inputs and raw materials, etc.

Moreover, as a consequence of home shopping, changes in customer preferences, technological advancements, and competition, modern sales and production companies often offer a take-back guarantee. Companies soften customers’ risk by offering a trial period for their products, thus, policies such as the right to return goods have become a part of daily routine. As a result, companies realize that a better understanding of returning items can provide a competitive advantage (Beltran and Krass [2], Fleischmann et al. [3], Pinçe et al. [4], Shaharudin et al. [5], and Barron [6, 7]).

The real need for guidance on how best to handle these uncertainties in demands, returns, productions, and costs motivates this study. We consider a continuous stochastic fluid inventory model for a single-item infinite horizon. Our main focus is to provide contributions to the study of inventory systems modulated by a Markovian environment. In the literature, dynamic control of stochastic inventory systems have been classified as periodic review models and continuous review models. In the case of continuous model, the inventory level (i.e., the number of on-hand items) can be viewed as a fluid process in which the production and demand rates undergo recurring changes in a stochastic fashion, and may be modeled as Markovian.

Markov-modulated fluid flows models have been an active area of research in recent years; one of their main applications is to the modeling the traffic evolution in communication channels. A standard example of a fluid flow is given by an infinite capacity buffer with inflow and outflow rates controlled by a Markov chain. The buffer level increases or decreases linearly at the current rate; when the buffer becomes empty, several strategies can be applied; it can remain empty until the inventory content level reaches a certain barrier (see, e.g., Boxma et al. [8], Kulkarni and Yan [9], Bean and O’reilly [10], and Barron and Hermel [11], Baek et al. [12], and Baek et al. [13]) or it can have positive jumps at the boundary (see e.g. Kulkarni and Yan [9, 14], and Barron [6, 15]). Fluid flow models are appropriate in situations where the arrival is comprised of a discrete unit, but the inter-arrival time between successive arrivals is negligible. Therefore, the arrival can be approximated by a continuous flow of fluid as individual units have less impact on the performance of the system. Such fluid queues are used as modeling tools of high-speed communication networks, transportation systems, congestion control systems, risk processes, and production-inventory systems.

In this study, the on-hand inventory level =It:t0 increases or decreases according to a fluid-flow rate modulated by an n-state Continuous-Time Markov Chain (CTMC). The fluid process is the inventory position or inventory level under continuous review where the environment process represents the varying background state. A jump in the fluid level represents an external order arrival, and the transition at the background state can be the result of repairs or production facilities, etc. The cost structure includes an ordering cost for each order, a variable cost that is proportional to the actually replenished amount (both for EOQ), a set up cost for production line initialization, a production cost per item (both for EPQ), a holding cost per unit of inventory during time unit, and a penalty cost in case of shortage.

Due to the complexity of the optimal policy, these inventory/production fluid systems are challenging to optimize, and great effort in the past focused on constructing various heuristic policies (Mohebbi [16], Kouki et al. [17], Barron [18], and Barron and Dreyfuss [19]). Fluid versions of the EOQ model are studied in Kulkarni and Yan [9], Yan and Kulkarni [20], Kulkarni [21], Berman et al. [22], Berman et al. [23], Berman and Perry et al. [24], and Barron [15] and the references therein. We also mention another related model, so-called clearing system (see Kella et al. [25], Berman et al. [26], and Barron [27]), which can be regarded as a dual EOQ stochastic model. In a clearing system, the fluid process jumps back to zero when it reaches a certain positive level. For background on stochastic EPQ models, we cite Vickson [28], Kella and Whitt [29], Boxma et al. [8, 30], and Barron [7] among others.

In this chapter, our main objective is to minimize the expected discounted total cost using a discount factor β>0. For that, we develop techniques enabling us to determine all the costs in such vendor-managing-inventory models in a closed-form. Our analysis is based on a combination of a certain martingale technique and an application of fluid flow theory to semi-regenerative processes. The martingale approach was introduced by Asmussen and Kella [31] and was frequently used in the study of inventory models (see, e.g., Boxma et al. [8], Kella et al. [25], and Barron et al. [27, 32]). The matrix-analytic approach and the theory of Markov-modulated fluid flows was initiated by Ramaswami [33] and Ahn et al. [34], who developed a unified methodology for studying a large class of insurance risk models via fluid flows by making use of the connection between the surplus process of an insurance and a particular fluid flow.

As we will show, the exit-time results are used to efficiently derive LST (Laplace–Stieltjes transform) functionals associated with the discrete-type measures, while the combination with the martingales yields simple expressions for the continuous-type measures. These explicit expressions can then be used for an analysis of the dependence of the cost functionals on the system parameters or for optimization purpose when some of these parameters (e.g. the order amount, the threshold levels, or the costs) are taken as decision variables. As such, we provide managers with a useful framework and an efficient and easy-to-implement tool to derive the best parameters and to compare the results of different demand–supply patterns.

We start by introducing the main tools of our analysis to be used.


2. Main tools

In the following, we will briefly introduce our main tools: (a) the matrix-analytic approach and the theory of Markov-modulated fluid flows, initiated by a series of papers by Ahn and Ramaswami [34, 35, 36, 37] and Ramaswami [33] and (b) an application of the optional sampling theorem (OST) to the multi-dimensional martingale of Asmussen and Kella [31].

Here and in the following, we use bold symbols to denote vectors and blackboard to denote matrices. To be consistent, for any matrix B, we shall denote its elements by Bij or by Bij and reserve the notation Bij for the sub-matrix of B with row indices in the set i and column indices in the set j. Moreover, for n-vector x, we use ΔX for a diagonal matrix Δx=diagx1x2xn. We further use E and Ei to represent expectation and conditional expectation operators, respectively. E (E) represents a matrix (a vector) of expectations. We denote by e a column vector with ones, i.e. e=111T, by ei a row vector with the ith component equal to 1 and all the other components 0, by I the identity matrix, by 0 the zero matrix (all with the appropriate dimensions); finally, by 1A the indicator of an event A.

2.1 Markov modulated fluid flow (MMFF)

Let be a state space that can partitioned into two sets: =12, with 1=n1,2=n2, and =n1+n2=n. Now, we introduce a modulating continuous time Markov chain (CTMC) {Jt;t>0} with that state , and a fluid process Ftt0 that is modulated as follows: whenever the Markov chain is Jt=i1, the fluid flow increases linearly at rate ci>0 and whenever it is in Jt=j2, the fluid flow decreases linearly at rate cj>0cj<0. The two-dimensional stochastic process FtJtt0 is called a MMFF (Markov Modulated Fluid Flow) process. We denote by Q the infinitesimal generator matrix of Jt;Q is given in a block form according to transitions between the sets i (i=1,2). Let C1,C2 and C be diagonal matrices as follows:


Let τx=inft>0Ft=x be the first passage time to level x. Let Ψβ be an n1×n2 matrix whose ijth component is


which is the LST of τ0 restricted to the event that the fluid process hits level 0 in-phase j2 and given that F0=0,J0=i1. In the literature, a few algorithms, including some quadratically convergent ones, were established for computing Ψβ (and all other LSTs); see, e.g., Ahn and Ramaswami [36]. Let τxy be the first passage time of F from level x to level y, and abτxy be the first passage time of F from level x to level y avoiding a visit below a or above b (for simplicity, we use bτxy0bτxy and aτxyaτxy). Let f̂xyβ and abf̂xyβ denote, respectively, the LST matrices of the joint distribution of the first passage times τxy,abτxy and the state of the phase process at each first passage time.

An important variant of the fluid flow F, a Reflected Fluid Flow, is particularly useful in the analysis of our inventory level process. The reflected fluid flow Fr is obtained by reversing the roles of the up and down environment states. Analogous, Ψrβ is the matrix (of order n2×n1) whose ij component is the LST of the time to reach the level 0 for the process Fr restricted to Jrτ0=j1, given that Fr0=0 and Jr0=i2, where Jrt is the modulated state process for Fr (we use notations f̂rxyβ and abf̂rxyβ to denote quantities similar to those above defined for Fr).

All these matrices, for hitting times, that we will use are straightforward to evaluate once we have computed Ψβ.Table 1 displays the basic elements (matrices) for the derivation of these LSTs; the first three matrices are associated with to flow F, while the next three matrices are associated to the rate-reverse flow Fr by interchanging the indices 1 and 2. The LST matrices and their sizes are given in Table 2; all matrices have nice probabilistic interpretations. For more details see Ramaswami [33], Ahn et al. [34], and Bean et al. [38].

QuantityMatrix sizeThe process

Table 1.

Transform matrices.

LSTThe LST of the First Passage TimeMatrix size

Table 2.

LST of first passage times.

2.2 The multi-dimensional martingale

Let Xtt0 be a right continuous Markov modulated Lévy process with modulating process Jtt0 which is a right continuous irreducible finite state space continuous time Markov chain. Let Ytt0 be an adapted continuous process with a finite expected variation on finite intervals and let Zt=Xt+Yt. Asmussen and Kella [31] have shown, that for such a process, the matrix with elements EieαXtJt=j has the form of etKα for some matrix Kα. Theorem 2.1 in Asmussen and Kella [31] yields that under certain conditions on Ztt0, the multi-dimensional process


is a (row) vector-valued zero mean martingale. Some of the relevant functionals in this paper will be obtained by applying the OST (or Doob’s optional sampling theorem, see Doob [39]) to appropriate special cases of For our models, the inventory level is a special case of Xt and has piecewise linear sample paths, with slope cj on intervals where Jt=j.

The outline of the chapter is as follows. We start with the ordering model and introduce an extension to the EOQ model, a fluid EOQ-type inventory model. Then, using the fluid EOQ-type model as a key, our study is generalized to include production facility, the fluid EPQ-type model.


3. A fluid EOQ-type inventory model

The simple Economic Ordering Quantity (EOQ) model assumes an inventory model of infinite horizon for a single item. The demand occurs at a constant rate and each time the inventory level hits a level 0, a fixed and immediate order of size Q is places. EOQ considers the timing of reordering, the cost incurred to place an order, and the costs to store merchandise. Here, we generalize the traditional EOQ model and consider demand and return rates that depend on the background environment, and varying state-dependent holding and ordering costs. We start with the mathematical description of the model.

Let =It:t0 be the on-hand inventory level at time t. The rate of change of the level is modulated by a continuous time Markov chain (CTMC) Jt:t0 on a finite state space =12n with a generator matrix Q=Qij. As long as Jt is in state i, there is a demand at rate di and a return at rate ri. The net rate is thus ci=ridi. Note that ci may be either negative or positive. Accordingly, we have two disjoint sets 12, =12, where 1 is a non-empty set of increasing rates 1=i:ci>0 and 2 is a non-empty set of the decreasing rates 2=i:ci<0. Let 1=n1 and 2=n2; thus n1+n2=n. Let π=π1πn be the limiting distribution of the Jt process, i.e. π is the unique solution to


The system is stable if and only if the expected input rate is negative, i.e.,


When It down-crosses level 0, an order of size Q>0 from an external supplier is placed which arrives instantaneously. Thus, immediately after the down-crossing to emptiness by the inventory level in state i, the process restarts at level Q. It should be noted, that the EOQ model is a special case of the base–stock polices, in particular, the sS-type, where the reorder level s=0, the replenishment up-to-level S=Q and zero lead time.

Let Tk be the time of the kth jump T0=0 and Jk=JTk be the environmental state at Tk (just after the jump). Note that since an order is placed when the inventory drops to level 0, the state Jk has to be a descending one, i.e., Jk2. We assume that over Tk1Tkk=1,2.., the process JttTk1Tk} is an irreducible CTMC (continuous time Markov chain) on . We call the points where the process jumps up (the replenishment times) order points. They form a semi-renewal process where Tk's are the semi regenerative points of the process (see Ross [40]). Define the kth cycle as the time elapsed between Tk1 and Tk,k=1,2,... Denote by Ck=TkTk1,k=1,2C0=0 the inter-replenishment times; let C=C1. Figure 1 illustrates a sample path of the process It,t0; for simplicity, we assume that I0=Q,J0n and let γ=γ1γ2γn2 be the initial probability vector of J0.

Figure 1.

A typical path of the fluid EOQ-type inventory process.

3.1 The cost structure

Our cost structure composed of two components: (a) an order cost, including a fixed cost whenever an order is placed and a purchasing cost; (b) a holding cost for the stock. We further assume that the cost rates are determined by the environment at the order points; in real-world, order rates usually depend on the state of the economy, the season, and change accordingly. Specifically, if the state at an order point is Jk=i2, the fixed ordering cost is Ki for an order (typically cost of ordering and shipping and handling), the cost to purchase one item from an external supplier is qi, and the cost to hold one item in inventory during a time interval of length dt is hidt during all the cycle k in which Jk=i2 (meaning that the holding cost for one unit is constant between two consecutive order replacements, i.e. ht=hi for tTk,Tk+1) and Jk=i2). We next introduce the functionals indicating the expected discounted costs using a discount factor β>0.

  1. Order cost. If Jk=i2, an order of size Q is immediately replenished up to level ITk=Q. The order cost is Ki+qiQ. Let OCβ be the expected discounted order cost and let Ôβ be an n2×1 vector whose ith component Ôiβ is given by


i.e., the expected discounted order cost, given J0=i2,I0=Q. Then we have OCβ=γÔβ. Applying Table 2, let 0f̂22Q0β be an n2×n2 matrix whose ijth components is


The component0f̂22Q0βij represents the LST of the time until the process hits level 0 in state j2, given J0=i2,I0=Q (for a proof, see Ramaswami [33], Theorem 5).

Lemma 3.1The total expected discounted order cost vectorÔβof ordern2×1satisfies the following equation:


where ΔK+qQ=diagKi+qiQ,i2.

Proof. It is easy to verify that Ôβ can be written as (recall that C is the time of the next order):


where the n2×n2 matrix EeβC is the LST of the cycle length; its ijth component is given by

EeβCij=EeβC1level0hitattimeCin phasejJ0=iI0=Q.E6

Applying the fluid model EeβC=0f̂22Q0β and solving (5) for Ôβ we obtain (4)

  1. Holding cost. The expected discounted holding cost can be expressed as


Let Ĥβ be an n2×1 vector whose ith component is given by


i.e., the expected discounted holding cost, given J0=i2. Thus, we have HCβ=γĤβ.

Lemma 3.2The vectorĤβof ordern2×1satisfies the following equation:


Proof. Applying the regenerative theory, and similar to Lemma 3.1, the vector Ĥβ can be written as


(The first vector Et=0CeβtItdt is the expected discounted inventory level of the first cycle). Now, we use the OST to the multi-dimensional martingale to find the n2×1 vector Et=0CeβtItdt. For J0=i2,I0=Q, consider a Lévy process Xit as follows:


It is not difficult to see that the latter process up to time C, i.e., Xit0t<C, has the same distribution as It0t<C. Chapter XI, p. 311 of Asmussen [41] yields that




Let Yt=β/αt (for an arbitrary α>0) and let Zit=Xit+Yt. Since Yt is adapted and has paths of a finite expected variation, the process


is an n2-dimensional row vector-valued zero mean martingale. The OST yields EMiα0=EMiαC=0, i.e.,


Obviously, we have EieαXi01J0=eαQfori2, or in an n2×n matrix form


(Here, the n2×n1 zero matrix arising due to the fact that J02). Next, we have to derive EieαXiCβC1JC. Since XiC=0, the fluid method yields the n2×n matrix form


Substituting (13) and (14) in (12) we obtain the n2×1 vector


Finally, take the derivative of both sides of (15) with respect to α, let α=0 and note that


leads to


Accordingly, the expected discounted total cost is


Remark 1It is easy to extend this model to include an order size determined by the environment at the order point, i.e., ifJTk=i2, an order ofQiitems is placed (and, accordingly,I0=Qiwith probabilityγi,i2). For that we define the vectorQ=Q1Qn2.Now, the entries0f̂22Q0βijare obtained by replacingQwithQiin(3). The order cost is given by(4)withΔK+qQreplacingΔK+qQ,and the holding cost is given by substitutingΔeαQinsteadeαQIin(13)(here,Xi0=Qi).

Using the approach for the fluid EOQ-type model as a key, we next include a production facility and introduce the fluid EPQ-type model.


4. A fluid EPQ-type inventory model

Similar as above, let It be the inventory level at a time t. The process It can be partitioned into two parts, I+t and It. The first part of the cycle, I+t, is the ON period and this period ends whenever I+t reaches a predetermined level Q. The second part of the cycle, It, is the OFF period and this is the time until It drops to level 0. The ON period is characterized by stochastic inputs, production and returns, and outputs, demand. However, no production occurs during the OFF period, and thus, the OFF period is characterized by stochastic returns and demand. Here, the analysis is more challenging due to the periods ON and OFF. Denote by p1pn0 the production rates, by d1dn0 the demand rates, and by r1rn0 the returns rates. The rate at which the inventory is filled at time t is determined by the current environmental state Jt. During the ON period (OFF period) and as long as Jt=i, the growth rate is the difference of the production and return rates (only the return rate) and the demand rate, ci+=ri+pidi (ci=ridi); similarly as before, ci,i=1,..,n may be either negative or positive. We do not allow backlog; thus when I+t drops to level 0 (due to high demands), it stays there as long as the environmental growth rate is negative and until the environmental state changes to some positive growth rate. Note that the behavior of the process during each period is different. During the OFF period, we enforce that i=1nπici<0 (a necessary and sufficient condition for the stability of the OFF process). During the ON period, we assume that i=1nπici+>0 (although with the absence of this, I+t is stable due to the reflection at level 0, however, it is a realistic assumption in order to avoid high lost demand).

A typical sample path of the inventory process is given in Figure 2. As we see, the inventory process is a semi-regenerative process which alternates between ON periods and OFF periods. We further note that if the ON periods are deleted from the sample path and the OFF periods are glued together we obtain a fluid EOQ-type model with refilling every time level 0 is reached. This holds with one exception; while each cycle under the fluid EOQ-type process starts with a descending state, it’s not necessarily holds in the fluid EPQ-type model (a detailed explanation is given below).

Figure 2.

A typical path of the fluid EPQ-type inventory process.

Define the following stopping times:


Tk,k=1,2, are the times of switchings from OFF to ON, and Ln,n=1,2,are the time instants of switchings from ON to OFF. Thus, It is a semi-regenerative process with Tn's are semi-regenerative points. Define the kth cycle as the time elapsed between Tk1 and Tk,k=1,2,.. and let Ck=TkTk1,k=1,2 be the kth cycle length. We use the generic form L=L1,T=T1,T=TL and C=C1 (so C=L+T). Note that conditioning on the state at time L and the common background environmental process, the two process I+t0tL (the ON period) and ItLtT (the OFF period) are independent.

We construct diagonal matrices Cj+=diag{ci+,ij}, Cj=diag{ci,ij},j=1,2 and C+=diagC1+C2+ and C=diagC1C2 from these rates. Regarding the fluid EPQ-type model, each ON/OFF period has one type of rates (C+ or C); hence, given the state at switching epoch and the common environment, the ON/OFF periods are independent. Now, we can analyze the inventory level within each period independently using MMFF process. Specifically, we consider I+,C+ and Ψ+β (I,C,Ψβ) corresponding to the ON (OFF) period. Note that, for this model, each LST matrix should be derived for ON (marked as +) and OFF (marked as −) processes. However, we do not insert the marks + or − corresponding to ON/OFF processes; it should be clear from the context which of the marks applies.

4.1 The cost functionals

We consider four costs: (a) the setup cost, (b) the holding cost of the inventory, (c) the production cost, and (d) the unsatisfied demand cost. For the derivation of the costs, we first need to derive the expected discounted cycle length n×n matrix EeβC.

4.2 The matrix EeβC

Given the state JL, the ON period and the OFF period are independent. Clearly, the ON period ends at state in 1 (at time L). However, at that point of time, the production stops, and thus, the state at which the OFF period starts can be either in 1 or in 2. Similarly, the OFF period ends at state in 2 (at time T). However, since the production starts at that point, the ON period can start either in state in 1 or in 2. This needs to be considered, particularly at switching times. Hence, at these switching times, the phases are labeled by arranging the entries according to the new states. Given the state JL, and due to that independency, the matrix EeβC is given by


We first introduce two similar LST matrices which differ only with their initial environment: (i) an n1×n1 matrix f̂110Qβ whose ijth component f̂110Qβij represents the LST of the time until hitting Q in state j1, given I0=0,J0=i1 and (ii) an n2×n1 matrix f̂210Qβ whose ijth component f̂210Qβij represents the LST of the time until hitting Q in state j1, given I0=0,J0=i2. Now, define the n×n1 matrix:


Since the ON period ends with environment in 1 (at time L), the n×n matrix EeβL has the form:


Similarly, define the n×n2 matrix:


(See also (3)). The OFF period ends with environment in 2 (at time T), and thus, the n×n matrix EeβT has the form of


(Recall that all matrices are given in Table 1). Next, we derive the expected discounted costs.

  1. Set up cost. Let K1 be the setup cost to switch from OFF to ON (at time T) and K2 be the setup cost to switch from ON to OFF (at time L). Let SCβ be the expected discounted set up cost and let Ŝβ be an n×1 vector whose ith component is the expected discounted set up cost given J0=i,I0=0,


    Similar technique as before arrives at


    Substituting (18) and (19) finalizes the derivation.

  2. Holding cost. The total expected discounted holding cost HCβ=γĤβ, where the vector Ĥβ is of order n×1; its ith components is the discounted expected holding cost given J0=i,I0=0. Revoking the ergodic theorem for regenerative process, we can write Ĥβ in terms of the first cycle and have


The basic tool we use to derive E0LeβtI+tdt and E0T'eβtIt+Ldt is OST to the Asmussen–Kella multi-dimensional martingale, as introduced in Lemma 3.2. Here, we consider the process


Chapter XI, p. 311 of Asmussen [41] yields that EieαXtJt=j=etKαij where Kα=Q+αC (specifically, K+α=Q+αC+). let Lt=min0vtXv. The process Lt, known as the local time, is a non-decreasing process that increases only whenever I+t=0 (for more details on local time and its properties, we refer the interested reader to [41]). Next, we let Zt=Xt+Lt. It is not difficult to see that the latter process up to time L, i.e., Zt0t<L, has the same distribution as It0t<L. Finally, define Yt=Ltβ/αt, for arbitrary β0 and α<0, and Wt=Xt+Yt=Ztβ/αt. Since Y is adapted and has paths of finite expected total variation on bounded intervals, Theorem 2.1 of Asmussen and Kella [31] leads to the next claim.

Claim 4.1Then×1vectorE0LeβtI+tdtis given by:




Proof. The proof and the derivation of L̂β are given in Appendix A. ■

In order to finish the holding cost we have to find the n×1 vector E0T'eβtItdt. Recall that since we are now dealing with the OFF period, for all the matrices in Table 1, we have to add the index (−). The OFF period stars at time L. We shift the time origin to L so that the OFF period starts at time 0. Consider X˜t similar to (9) but with C and X˜0=Q,J0. Clearly, the latter process up to time T, i.e., X˜t0t<T, has the same distribution as It0t<T'. Similar arguments to (15) leads to:




  1. Production cost. Let qi be the production cost for one unit. We have:


Note that the OFF period is characterized by no production. For the derivation of E0LpJteβtdt (ON period), let Pj+=diagpiij,j=1,2 and P+=diagP1+P2+. Thus,


Applying (26) with α=0 yields


which completes the derivation of the production cost.

  1. Lost demand cost. In our model, backlog is not allowed; any demand which cannot be satisfied immediately is lost. Clearly, there is no unsatisfied demand during the OFF period. During ON period, once level 0 is reached the process stays there until the environment changes to state with a positive growth rate. Assume the process hits 0 at state i (for some ci+<0), the demand is lost with rate ci+ until the environmental state changes. Let wdt be the cost for a lost unit during a time interval of length dt (w>0). As a measure for the expected discounted lost demand cost one can use the functional


The right term is the expected discounted production loss during the first cycle; it can be written in terms of the local time process.

Corollary 4.1It is easy to verify that


whereL̂βis given in(40).

A simple cost function for the entire system would be the sum, TCβ, of these four expected discounted costs:


5. Summary

During the past few decades, the problem of control of inventory systems has been widely investigated. Many stochastic factors inherent in inventory systems can make it more difficult for managers to plan and control the inventory. Dealing with the randomness of demand, production, and returns, this study considers a continuous-review inventory system where the inventory level is characterized as a fluid process modelled by Markovian environment. The cost structure includes an order cost, a purchase cost, a set up cost, a production cost, an inventory cost, and a lost cost due to unsatisfied demands. By taking a simple probability approach and by applying stopping time theory to fluid processes and martingales, the explicit components of the resulting costs are derived. These cost components can be used for optimization purposes. Moreover, the closed-form expression of the components allows us to obtain efficiently and numerically the optimal parameters and enables us to investigate the behavior of the system and to study its properties. From a managerial perspective, our framework can be applied to many industries, in situations where the system is subject to uncertain environment. Our approach appears to be a powerful way to address related inventory problems. The models considered here assume only discrete demand and return sizes. For instance, it seems to be possible to adapt a similar approach in the case of continuous demand size distributions, such as exponential, uniform, or gamma. Another avenue of research would be to extend this framework to include random lead times; in that case, two policies can be applied while shortage, backordering or lost sales. All of these nontrivial extensions are tractable and worthy of study.


Proof of claim 4.1. Theorem 2.1 of Asmussen and Kella [31] yields that the process


is an n-row vector-valued zero mean martingale. The OST yields EMαL=EMα0=0,


Since Z0=0 and J0, we obtain the n×n matrix EeαZ01J0=I. Applying ZL=Q leads to


To finish, we have to derive the n×n matrix E0Leβt1JtdLt, which is the expected discounted lost demand until time L. Notice that a loss occurs only during states in 2. For that, we introduce the n2×n2 matrix ϒβ whose ijth component, ϒijβ, is the expected discounted loss in state j2 until exiting level 0, given that the process starts at level 0 with state i2.

Lemma A.1ϒijβ satisfies the following system of linear equations:


Proof. Once the inventory process hits 0 in state i2, it stays there for an exponential random time ξi with parameter Q22ii. With probability Q22ijQ22ii the state changes to j, and the expected discounted loss from state j is ϒji. By conditioning on the first state visited after i, we readily obtain that


Solving (38) with respect to ϒiiβ and ϒijβ returns (37) (note that Et=0ξicieβtdt=ciβ1+Q22iiβQ22ii and Eieβξi=Q22iiβQ22ii)). ■.

Now, we apply ϒβ to the derivation of the loss until time L. Since the lost demand occurs only for states in 2, the matrix Ev=0Leβv1JvdLv has the form


where L̂β is an n×n2 matrix whose ijth component is the expected discounted loss in state j2 until L, given J0=i. Let L̂iβ,i=1,2 be an ni×n2 sub-matrix of L̂β includes all rows corresponding to states in i,i=1,2, such that


It is easy to verify that


Lost demand occurs when the process drops to level 0 avoiding Q, with LST QΨβ. From that point, ϒβ is the discounted lost demand. The term βIQ221Q21 is the discounted time until exiting level 0 with environmental state in 1 and starting again with LST L̂1β. The matrix L̂2β is derived similarly.


  1. 1. Nahmias S. Production and Operations Analysis. 3rd edition. New York: Irwin/McGraw-Hill; 1997
  2. 2. Beltran JL, Krass D. Dynamic lot sizing with returning items and disposals. IIE Transactions. 2002;34:437-448
  3. 3. Fleischmann M, Kuik R, Dekker R. Controlling inventories with stochastic item returns: A basic model. European Journal of Operational Research. 2002;138(1):63-75
  4. 4. Pinçe Ç, Gürler Ü, Berk E. A continuous review replenishment-disposal policy for an inventory system with autonomous supply and fixed disposal costs. European Journal of Operational Research. 2008;190:421-442
  5. 5. Shaharudin MR, Zailani S, Tan KC. Barriers to product returns and recovery management in a developing country: investigation using multiple methods. Journal of Cleaner Production. 2013;96:220-232
  6. 6. Barron Y. An order-revenue inventory model with returns and sudden obsolescence. Operations Research Letters. 2018;46(1):88-92
  7. 7. Barron Y. Performance analysis of a reflected fluid production/inventory model. Mathematical Methods of Operations Research. 2016;83(2016):1-31
  8. 8. Boxma OJ, Kella O, Perry D. An intermittent fluid system with exponential on-times and semi-Markov input rates. Probability in the Engineering and Information Sciences. 2001;15:189-198
  9. 9. Kulkarni VG, Yan K. A fluid model with upward jumps at the boundary. Queueing Systems. 2007;56:103-117
  10. 10. Bean N, O’reilly M. Performance measure of a multi-layer Markovian fluid model. Annals of Operations Research. 2008;160:99-120
  11. 11. Barron Y, Hermel D. Shortage decision policies for a fluid production model with MAP arrivals. International Journal of Production Research. 2017;55(14):3946-3969
  12. 12. Baek JW, Lee HW, Lee SW, Ahn S. A Markov modulated fluid flow queueing model under D-policy. Numerical Linear Algebra with Applications. 2011;18:993-1010
  13. 13. Baek JW, Ahn S. Analyses of the Markov modulated fluid flow with one-sided ph-type jumps using coupled queues and the completed graphs. Journal of the Korean Statistical Society. 2014;43(2014):415-424
  14. 14. Kulkarni VG, Yan K. Production-inventory systems in stochastic environment and stochastic lead times. Queueing Systems. 2012;70:207-231
  15. 15. Barron Y. A fluid EOQ model with Markovian environment. Journal of Applied Probability. 2015;52(2):473-489
  16. 16. Mohebbi E. A production-inventory model with randomly changing environmental conditions. European Journal of Operational Research. 2006;174:539-552
  17. 17. Kouki C, Legros B, Babai MZ, Jouini O. Analysis of base-stock perishable inventory systems with general lifetime and lead-time. European Journal of Operational Research. 2020;287(3):901-915
  18. 18. Barron Y. Critical level policy for a production-inventory model with lost sales. International Journal of Production Research. 2019A;57(6):1685-1705
  19. 19. Barron Y, Dreyfuss M. A triple Ss-thresholds base-stock policy subject to uncertainty environment, returns and order cancellations. Computers & Operations Research. 2021;134:105320
  20. 20. Yan K, Kulkarni VG. Optimal inventory policies under stochastic production and demand rates. Stochastic Models. 2008;24:173-190
  21. 21. Kulkarni VG. Modeling and analysis of stochastic systems. 2nd edition. Boca Raton: CRC Press; 2009
  22. 22. Berman O, Perry D, Stadje W. A fluid EOQ model with a two-state Random environment. Probability in the Engineering and Informating Sciences. 2006;20:329-349
  23. 23. Berman O, Perry D, Stadje W. Optimal replenishment in a Brownian motion EOQ model with hysteretic parameter changes. International Journal of Inventory Research. 2008;1:1-18
  24. 24. Berman O, Perry D. An EOQ model with state dependent demand rate. European Journal of Operational Research. 2006;176:255-272
  25. 25. Kella O, Perry D, Stadje W. A stochastic clearing model with Brownian and a compound Poisson component. Probability in the Engineering and Informational Sciences. 2003;17:1-22
  26. 26. Berman O, Parlar M, Perry D, Posner MJM. Production/Clearing models under continuous and sporadic review. Methodology and Computing in Applied Probability. 2005;7:203-224
  27. 27. Barron Y. Clearing control policies for MAP inventory process with lost sales. European Journal of Operational Research. 2016;251(2):495-508
  28. 28. Vickson RG. A single product cycling problem under Brownian motion demand. Management Science. 1986;32:1336-1345
  29. 29. Kella O, Whitt W. A Storage model with a two-stage random environment. Operations Research. 1992;40:s257-s262
  30. 30. Boxma O, Kaspi H, Kella O, Perry D. On/off storage systems with state-dependent input, output, and switching rates. Probability in the Engineering and Informational Sciences. 2005;19:1-14
  31. 31. Asmussen S, Kella O. A multi-dimensional martingale for Markov additive process and its applications. Advances in Applied Probability. 2000;32:376-393
  32. 32. Barron Y. An skS fluid inventory model with exponential leadtimes and order cancellations. Stochastic Models. 2016;32:301-332
  33. 33. Ramaswami V. Passage times in fluid models with application to risk processes. Methodology and Computing in Applied Probability. 2006;8:497-515
  34. 34. Ahn S, Badescu AL, Ramaswami V. Time dependent analysis of finite buffer fluid flows and risk models with a dividend barrier. Queueing Systems. 2007;55:207-222
  35. 35. Ahn S, Ramaswami V. Fluid flow models and queues- a connection by stochastic coupling. Stochastic Models. 2003;19:325-348
  36. 36. Ahn S, Ramaswami V. Efficient algorithms for transient analysis of stochastic fluid flow models. Journal of Applied Probability. 2005;42:531-549
  37. 37. Ahn S, Ramaswami V. Transient analysis of fluid models via elementary level-crossing arguments. Stochastic Models. 2006;22:129-147
  38. 38. Bean N, O’reilly M, Taylor P. Algorithms for the Laplace-Stieltjes Transforms of First Return Times for Stochastic Fluid Flows. Methodology and Computing in Applied Probability. 2008;10:381-408
  39. 39. Doob J. Stochastic Processes. New York: Wiley; 1953
  40. 40. Ross M. Stochastic Processes. 2nd edition. New York: Wiley; 1996
  41. 41. Asmussen S. Applied Probability and Queues. Second edition. New York, Inc: Springer-Verlag; 2003

Written By

Yonit Barron

Reviewed: 02 March 2022 Published: 25 June 2022