Journal of Applied Mathematics and Stochastic Analysis, 10:4 (1997), 423-430. A HIDDEN MARKOV MODEL FOR AN INVENTORY SYSTEM WITH PERISHABLE ITEMS

This paper deals with a parametric multi-period integer-valued inventory model for perishable items. Each item in the stock perishes in a given period of time with some probability. Demands are assumed to be random and the probability that an item perishes is not known with certainty. Expressions for various parameter estimates of the model are established and the problem of finding an optimal replenishment schedule is formulated as an optimal stochastic control problem.


Introduction
This paper deals with a parametric multi-period integer valued inventory model for perishable items.Each item in the stock is assumed to perish in period n with probability (1-an) where 0 < a n < 1.The sequence {an} is not known with certainty but it is known to belong to one of a finite set of states of a Markov chain to be specified below.The above model is inspired from a special type of time series models called First Order Integer-Valued Autoregressive process (INAR(1)).These models were introduced independently by hl-Osh and il-Zaid [1] and McKenzie [6]   for modeling counting processes consisting of dependent variables.
Let X be a nonnegative integer-valued random variable.Then for any a E (0, 1), define the operator o by: x a o X E Yi, (1.1) i=1 where Y is a sequence of i.i.d, random variables, independent of X, such that: P(Yi 1) 1-P(Yi O) a.
(1.2) Then the INAR(1) process {X,:n 0, + 1, + 2,...} is given by: X n a o X n 1 at-Vn, where V n is a sequence of uncorrelated nonnegative integer-valued random variables with mean #n and variance r 2 One possible application of model (1.3) is in its monitoring the number of patients in a hospital.Here the number X n of patients at epoch n is composed of patients left from the previous epoch (each patient stays in the hospital with probabi- lity a or leaves with probability (1-a)), and a new set V n of arriving patients.Now let X n represent the number of items at the end of period n in our inven- tory and consider the following extension of (1.3): x=oX_-v+u, (1.4) x with X 0 constant (integer).Here an o Xn_ 1 E Y as defined in (1.1) for non- i--1 negative integer random variables X n_ 1 and we allow X n_ 1 to take on negative values in which case a n o X n 1 =-0.Note that a negative X n is interpreted as short- age.V n is a sequence of 7/----valued independent random variables with probability distribution en and independent of Y.The variable U n is a 7/+-valued sequence representing the replenishment quantity at time n (which later on will be considered as a control variable).Further we assume that the sequence {an} is a homogeneous Markov chain with finite state space S. For instance, {an} expresses changes in a caused by changes in the environment (temperature, humidity, air pressure, etc.) at different epochs n.
The objective of this paper is to: (i) find the conditional probability distribution of {an} given the information accumulated about the level of inventory {Xn} up to time n, (ii) estimate the transition probabilities of the Markov chain {an} (iii) formulate the optimal stochastic control problem related to (1.4).
The proposed model is an example of a hidden Markov model.Our approach shall use measure change techniques (see Elliot, Aggoun, and Moore [5]).A reference probability measure is introduced under which processes of interest are independent of each other.This greatly simplifies computations and allows derivations of interesting results.
Note that in analogy with engineering applications of hidden Markov models, we may think of {an} as a signal hidden in some noisy environment and can only be ob- served through noisy observations or measurements given by {Xn}.
Most of existing inventory models for perishable items in the literature are con- cerned with determining suitable ordering policies that optimize some of their per- formance measures.According to Nahmias [7], all perishable models fall in two main categories: (a) fixed life perishability models, (b) random lifetime models.
Our model is closely related to (b).However, due to the complexity inherent to the analysis of models where units have a random lifetime, very little progress has been made on that front, apart from suggesting equivalent deterministic or using queueing models for their analysis.Our paper proposes a new model for perishable items which we believe is useful.For related deterministic models see Benkherouf and Mahmoud [2] and Benkherouf [3].
The paper is organized as follows" In Section 2 recursive unnormalized condition- al distributions of {an} given the information accumulated about the inventory level- surviving items processes are derived.Section 3 contains estimates of the probability transitions of the Markov chain {an}.Finally, in Section 4, an optimal control form- ulation for finding the replenishment schedule is proposed.The last section contains a summary and some open problems.
2. Conditional Probability Distribution of {an} Let I1,...,I M be a partition of the interval (0,1).Also, let 81 be any point in I1, s 2 be any point in I2,...,s M be any point in IM, and S-{81,...,8M}.Suppose that the sequence {an} is a homogeneous Markov chain with state space S describing the evolution of the parameter c n of the inventory model.
i-1 lecall that X n represents the inventory level at time n.Now let G, cr{Xk, Y,ck,l <_ n,i >_ 1}, (2.3) be the complete filtrations generated by the parameter process {an} the inventory level-surviving items processes {Xn, Y]}, and the inventory level-surviving items and the parameter processes respectively.
Note that A n and n C G n for n-1, Following the techniques of Elliot, Aggoun, and Moore [5], we introduce a new probability measure P under which {c} is a sequence of i.i.d, random variables uniformly distributed on the set S-{Sl,...,SM} and {Xn} is a sequence of independent -valued random variables with probability distributions Cn independent of {a}.For this, define the factors" 0 1, An (Ma) I<n Si) n(Xn n o X n --1 Un) (2.4) .(x.) where I( si)is the indicator function of the event (a si) and M a.
j=l Remark 1: Note that.{a} is An_ 1 adapted and E[I( that I( n si)-a n" A is an A-martingale increment.

Now define
Then {An} is a Gn-martingale such that E[An]-1.Here E denotes the expectation under P. We can define a new probability measure P on (f2, Gn) by setting dP an The existence of on (a h=l Gn) is due to Kolmogorov's extension theorem (see, e.g., Chung [4]).Then under P, {Xn} is a sequence of independent random variables with probability distributions Cn and {c%} is a sequence of i.i.d, random variables uniformly distributed over S.
(2.9)Here E denotes the expectation under P and A n is given by (2.6).

Estimation of the Transition Probabilities Pij
In Section 2, the probabilities Pij were assumed to be known.However, in practice we seldom know Pij exactly and consequently, estimates of the transition probabilities would be reasonable requirements.These estimates are calculated each time new information regarding the behavior of the system becomes available.In this model, the state space S-{Sl,...,SM} is fixed a priori.This is done by partitioning the open unit interval into M subintervals I1,...,I M according to some prior information about the behavior of a. When, at time n, the new information is made available through Yn-measurable estimates {ij(n)}, a new partition of I can replace the initial one updating the state space of {an}" To replace, at time n, the parameter {pij(n)} by {ij(n)} in the Markov chain {an}, define Again, the existence of P is due to Kolmogorov's extension theorem.
It is easy to see that under P, the Markov chain {n} has transition probabilities given by {ij(n)}.
Theorem 2: The new estimates {ij(n)} at lime n of lhe probability transitions given lhe observalion of the inventory level from lime 0 to lime n are given by" 7.(N#) and Nn counts the number of transitions from slate j to state up to time n.
Proof: Note that from (3.1), n M M LgXn-E E E I(Ck e Ii)I(Crk_ 1 e Ij){Logij(n In this section we formulate an optimal control problem for model (1.4).However, {an} is not fully observed.Using results of Section 2 we transform the problem into a fully observed one.Dynamic programming results and minimum principle are ob- tained in terms of separated controls (i.e., controls which depend on the Markov chain only through qn)" Recall from (1.4) that our inventory model has the form: X n c n o X n-1 Vn + Un" (4.1) Assume that at each time n, 0 <_ n <_ T, the control U n takes on values in some finite subset of 7/+ and that U n is 5n-measurable where 5 n is given by (2.2).Also assume the existence of bounded measurable functions cn, dn, and fn.k=l where qn(" are given by Theorem 1. So, the expected cost is expressed in terms of the conditional measures qn(" )" The value function for this control problem is as follows for 0 _< t _< T" Vt(qIc, inf 7 {cn(Xn) -t-dn(Un)} qn(l) + fn(k)qn(k) qt q UT) n 1 k 1 inf Vt(q, U).