Why Least Squares and Maximum Entropy? An Axiomatic Approach to... Inverse Problems Author(s): Imre Csiszar
Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems Author(s): Imre Csiszar Source: The Annals of Statistics, Vol. 19, No. 4 (Dec., 1991), pp. 2032-2066 Published by: Institute of Mathematical Statistics Stable URL: http://www.jstor.org/stable/2241918 . Accessed: 03/07/2013 17:53 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp . JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected]. . Institute of Mathematical Statistics is collaborating with JSTOR to digitize, preserve and extend access to The Annals of Statistics. http://www.jstor.org This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions The Annals ofStatistics 1991,Vol. 19, No. 4, 2032-2066 WHY LEAST SQUARES AND MAXIMUM ENTROPY? AN AXIOMATIC APPROACH TO INFERENCE FOR LINEAR INVERSE PROBLEMS' BY IMRECSISZAR Institute Mathematical oftheHungarianAcademyofSciences An attemptis made to determinethe logicallyconsistentrules for when selectinga vectorfromanyfeasibleset definedbylinearconstraints, or thosewithpositivecomponents or the probability eitherall n-vectors ifand onlyifthe Somebasicpostulatesare satisfied vectorsare permissible. selectionruleis to minimizea certainfunction which,ifa "priorguess" is available,is a measureof distancefromthe priorguess. Two further naturalpostulatesrestrictthe permissibledistancesto the author's fAs corollaries,axand Bregman'sdivergences, divergences respectively. iomaticcharacterizations of the methodsof least squares and minimum thelatterare also information are arrivedat. Alternatively, discrimination As a specialcase,a characterized bya postulateofcomposition consistency. froma smallset ofnatural derivation ofthe methodofmaximumentropy axiomsis obtained. 1. Introduction. A frequently occurring problemin statisticsand applied frominsufficient is thata function has to be inferred information mathematics that specifiesonlya feasibleset of functions. Problemsofthiskindare often ofa signalor calledinverseproblems.Typicalexamplesare thereconstruction and the assignmentof a image fromthe resultsof certainmeasurements, that is, probability densityor mass functionsubjectto momentconstraints, thatspecify theexpectations ofcertainfunctions oftheunderlying constraints randomvariable.Oftenthe practicalsolutionto such problemsis to selectan elementofthefeasiblesetbya moreor less ad hocrule,usuallybyminimizing somefunctional such as the L2-normor negativeentropy.If somefunction is specifiedas a "priorguess," it is naturalto minimizea measureof distance fromthelatter,mostoftenthe L2-distance, densityor mass or,forprobability Kullback'sI-divergence fordiscrimination functions, (also calledinformation or cross-entropy). In thispaperwe addressthe questionofwhatselectionrulesare "good" in this framework and, in particular,whetherthe mentionedstandardones are indeedthe "best." Unfortunately, meaningto it is hardto givea mathematical thisquestion.It does not seempossibleto definea generalcriterion by which December Received 1990. May1989;revised ofTokyo,supported by 'This researchwas startedduringtheauthor'svisitat theUniversity forthePromotion ofScience. theJapanSociety Its completion wassupported bytheHungarian National ScienceFoundation, Scientific Research Grant1806. AMS 1980subject classifications. Primary 62A99;secondary 68T01,94A17,92C55. andphrases. linearconstraints, logically consistent inference, Keywords Imagereconstruction, minimum discrimination nonlinear distance, selection rule. information, projection, nonsymmetric 2032 This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2033 the goodness of selectionrules could be compared.Still, whereas people apparently feellittleneedforanyspecialjustification ofleastsquares(L2-norm variousreasonshave been put forwardto justifyI-divergence minimization), intostatisticsby Kullback(1959) as the methodof minimization [introduced minimumdiscrimination and entropymaximization. The recent information] widespreadapplicationsof "maximumentropy"have been pioneeredto a greatextentbyJaynes[forhis viewscf.Jaynes(1982)].The presentauthorhas arguedelsewhere[Csisza6r (1985)] that the conditionallimittheoremsofVan Campenhoutand Cover(1981) and Csiszar (1984) suggestthe interpretation thatthe minimumI-divergence distribution "updating"ofa priorprobability is a limitingformofBayesianupdating. to meetmomentconstraints Here we adoptan axiomaticapproachand considerthoseselectionrulesas in the sense of "good" thatlead to a logicallyconsistentmethodofinference, not natural The term inference is meant in a some satisfying postulates. even statisticalsense. Indeed, our considerationswill be nonprobabilistic, distributions. The relation thoughtheobjectsto be inferred maybe probability ofthisworkto previousaxiomaticapproacheswillbe discussedat the end of thissection. As a typicalexample,we briefly sketcha modelofthe imagereconstruction and in variousotherfields problemthat occursin computerized tomography [cf.,e.g.,Hermanand Lent (1976) or Censor(1983)].An imageis represented functionf definedon some domain.The availableinforby a positive-valued mation consistsin the measuredvalues of some linear functionalsR if, i = 1,.. ., k. In X-raytomography, f is the unknownX-rayattenuationfunctionand Ri f is itsintegralalongthepathofthe i's ray.Now,thedomainof f is partitioned intoa finitenumberofpictureelements,calledpixels,numbered in some way from1 to n. Assumingthat f is nearlyconstantwithineach pixel,we can write n (1.1) f=Ev j=1 fi, where fj is the indicatorfunctionof the jth pixel.Then,settingaij and bi = Ri f, we have (1.2) = Ri fj n , aijvj j=1 = bi, i = 1,...,k. Thus the unknownfunctionf is represented by the vectorv = (v1,.. ., vn)T and the feasibleset is identified withthe set of thosevectorsv withpositive thatsatisfythe linearconstraints components (1.2). The reconstruction problemis to selecta "suitable"elementofthisfeasibleset (possiblydependingon a "priorguess" of f represented by a vectoru). In thispaper,we concentrate on linearinverseproblemsofform(1.2). The functions extensionofour resultsto the continuouscase, thatis, to inferring definedon some domainand not representable by finite-dimensional vectors, should not be hard but will not be entered.The above example will be This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2034 repeatedlyused as an illustration,but it should be emphasizedthat our axiomaticapproachis by no means limitedto imagereconstruction. On the other hand, since a generalapproachinevitablyinvolvesidealizations,the solutionsit leads to are not necessarily"best" forspecificpracticalproblems, includingimagereconstruction. Our goal is to determinethe "logicallyconsistent"rules forselectingan elementfromanypossiblefeasibleset.We adopttheidealizedassumptionthat all conceivablelinearconstraints mayoccur;thusthepossiblefeasiblesetsare all thosesubsetsofa basic set S ofpermissible vectorsthatcan be definedby constraintsof form(1.2). Threecases willbe consideredin a parallelmanner namelywhen S consistsofall n-vectors, or ofthosewithpositivecomponents (as in imagereconstruction) or ofthe probability vectorswithpositivecomponents.The choiceofvectorswithpositiveratherthannonnegative components has been preferred in orderto reducetechnicaldifficulties; also, thisensures that a nonnegativequantityis never inferredto be 0 when the available information permitsit to be positive,whichis generallyconsidereddesirable. Our postulateswill be stated and intuitively justifiedin Section2. The resultswillbe statedin Section3 and provedin Section5, usingthelemmasin Section4. The keyresultis Theorem1, namelythat the basic postulatesof "regularity"and "locality"of a selectionrule are satisfiedif and onlyifthe selectionis by minimizing a certainfunction. If a priorguess is available,this functionis a measureof distance(nonsymmetric in general)fromthe prior guess. The subsequenttheoremsshow how certainadditionalpostulatesrestricttheclass offunctions thatmaybe used. Perhapsthemoststriking result in Theorem5. It providesa parallelcharacterization of the methodsof least squares and minimumI-divergence as the onlyones satisfying, in additionto and locality,a postulateof"composition regularity The intuitive consistency." meaningofthispostulateis thatifthe objectofinference is composedoftwo and the availableinformation components, says nothingabout theirinteraction,we shouldinferthatno interaction is present. It shouldbe mentionedthatin practice(1.2) is oftenrelaxedto (1.3) n E aij + ei =bi, j=1 i = ly,...,)k, wheree = (el, ..., ek)' is an errorvector.For the reconstruction problemof positronemissiontomography, Vardi,Sheppand Kaufman(1985) describeda modelequivalentto (1.3) withdata bi that were Poisson randomvariables (countsofdetectedemissionsin "tubes" determined by pairsofdetectors). A vectore as in (1.3) did notexplicitly entertheirmodel,but,definingei as the ofthe actualand expectedcountsforthe ith tube,(1.3) wouldhold. difference The suggestedreconstruction was the maximumlikelihoodestimateofv, and forits computationthe EM algorithmof Dempster,Laird and Rubin(1977) was used. This reconstruction techniqueis sometimesconsideredas relatedto "maximumentropy"[cf.,e.g.,Millerand Snyder(1987)],fortheformalrather thanconceptualreasonthatthe EM algorithm is equivalentto an alternating This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2035 I-divergence minimization; Vardi,Sheppand Kaufman(1985) haveshownthat the convergence of theiralgorithmis an instanceof a resultof Csiszar and minimization. Tusnady(1984) on alternatingI-divergence Maximumlikelihoodestimationis not a generallyapplicablemethodfor "solving"inverseproblemsof form(1.3) because (i) the "errors"ei maybe nonrandomor else theirjointdistribution maybe unknownand (ii) evenifthe the maximumlikelihoodesti"errors"are randomwithknowndistribution, mate is typicallynonuniqueif n > k. Vardi, Shepp and Kaufman(1985) theobjectintofewerpixelsthanthe avoidedthelatterdifficulty bypartitioning numberof "tubes" thatwas sufficiently large.If boundson the magnitudeof the errorsare known,it maybe convenientto interpret (1.3) as a systemof inequalitiesofform (1.4) bi n j=l1 aijvj < btzi=l,i , k, or as an inequalityofform k (1.5) k i=l ( n j=l 2 aiivi - b) < c. Our axiomaticapproachcould be extendedto the problemof selectingan elementfromanyset definedbyinequalityconstraints suchas (1.4) or (1.5) or, moregenerally, fromanyconvexsubsetofthebasic set S. Alternatively, (1.3) could be interpreted as determining a "feasible set" of pairs (v,e), and a selectionfromthis set could be made by any methodthat has been deemed "good" for the problem(1.2). Indeed, this is oftendone in practice,for somequadraticfunction ofthe pair(v,e) [cf.Herman example,by minimizing and Lent (1976) or Censor(1983)]. It remainsto be seen whether"solutions" of this kind to the problem(1.3) can be coveredby an extensionof our axiomaticapproach;one of the difficulties is that the possiblelinear constraintson pairs(v, e) are ofthe veryspecialform(1.3). The approachin thispaperwas strongly motivatedby Shore and Johnson (1980), where-forthe problemofassigninga probability distribution subject to momentconstraints-an intuitively appealingaxiomaticderivationof the methodsof maximumentropyand minimumI-divergencewas provided. forinferring Skilling(1988) gave a similarderivation arbitrary positive-valued because of the greaterliberty, this case turnedout to be considerfunctions; ablysimpler.BothShoreand Johnson(1980) and Skilling(1988) startedfrom the assumptionthat inferencewas based on minimizingsome functionor, equivalently,on some transitiverankingthat could be describedby real numbers.Afterhavingsubmitted thispaper,theauthorlearnedthatParis and ofmaximumentropy"from Vencovska(1990) had arrivedat "the inevitability axiomsthat-like ours-did not a prioriassume the minimization of some fromtheirs, function;in otherrespects,our approachappearsquite different in the axiomsdo exist.The authoris indebtedto Profesalthoughsimilarities sor Paris forsendinghimmanuscripts ofthisand relatedworks. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2036 I. CSISZAR Our resultsalso provideaxiomaticcharacterizations ofmeasuresofdistance whoseminimization leads to "good" methodsofinference. Whereasthereis an of entropy,I-divergence extensiveliteratureon axiomaticcharacterizations and theirgeneralizations [cf.,e.g.,Aczeland Dar6czy(1975)],ourcharacterizadiffer fromthe usual ones: Our postulatesinvolvenot the tionssubstantially but ratherthe inferencemethodit leads to. In measureto be characterized Theorem2(ii), a class of distancesintroduced by Csisza6r (1963) [and independentlybyAli and Silvey(1966)] is characterized; relatedresultsalso appearin Shoreand Johnson(1980) and Skilling(1988). Theorem3 characterizes a class ofdistancesintroduced by Bregman(1967); sincethispaperhad been submitted,a different axiomaticcharacterization ofthisclass (in thecontinuouscase) was givenbyJonesand Byrne(1990). Finally,we commenton theone-parameterfamilyofdistancescharacterized in Theorem4(ii). In theoriginalversionof this paper,the previouslynot consideredmembersof that familyhad been mentionedas candidatesforbecomingpractically useful.Recently, Jonesand Trutzer(1989) reportedapplicationsof (continuousversionsof) these distancesthatseemto confirm thatprediction. 2. Preliminaries, postulates. The reallineand thepositivehalf-line are denotedbyR and R+, respectively. We emphasizethat R+ does notcontain0. The vectorsin Rn whosecomponents are all 0 or all 1 are denotedby0 or 1. All vectorsare columnvectors. The set of n-dimensional vectorswith positivecomponentsof sum 1 is denotedby An nthat is, (2.1) A = (v: vE R, Tv= 1}. The familyofaffinesubspacesof Rn, thatis, the familyofnonvoidsubsets of Rn definedby linearconstraints, will be denotedby .Zn. In otherwords # LE iff L 0 and n L = {v: vE Rn,Av = b} forsomeA E Mk xn (k x n matrix)and b E Rk. We denoteby Yn' thefamily of all nonvoidsubsetsof R+ of form(2.2), withv E Rn replacedby v E R+. The familyof those L E Yn+ whichare subsets of An will be denotedby Y4+(1). For any fixeddimensiond < n, the set of d-dimensional elementsof 4n (or 4+k) is endowedwith the natural topology,that is, the quotient topologyderivedfromthe Euclideantopology oftheset ofthosepairs(A,b) E elementof .4n (or Yn+). M(n-d)xn x Rn-d thatdefinea d-dimensional Throughoutthis paper,our basic set S willbe eitherof Rn, Rn and An The set ofcomponents ofvectorsin S willbe denotedby V, thatis, V stands for R, R + or the open interval(0, 1), accordingto the choiceof S. Unless statedotherwise,u, v and w willalwaysdenoteelementsof V, whereasu, v and w denotevectorsin Vn. Further, we denote by ./ the familyof nonvoid subsetsof S determined by linearconstraints.Thus, accordingto the three cases, ? equals .n, .+ or .n'(1). For convenience,we will call the elementsof Y subspacesof S also if S $ Rn. (2.2) This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2037 Noticethat S itselfis an elementof ..f; amongthe propersubsetsof S the maximalsubspacesare thoseofdimensionn - 1 (if S = Rn or R+) or n - 2 (if S = An). The subfamily of Y consisting ofthesemaximalsubspaceswillbe denotedby X4/. Thus X/ consistsofthe (nonvoid)sets (2.3) _ ({v:a av a =b 0, if S = R~ or R+, {v: aTv = b, ITv = 1}, a A1, if S = Azn As mentionedpreviously, the conditionv E Vn is implicitin the notationin (2.3). Having in mindinferenceproblemsas in Section 1, we are interestedin rulesthatspecify foreach L E Y (and "priorguess" u) an elementof L, to be selectedif L is in the feasibleset (and the priorguess was u). The vector selectedfromL whena priorguessu was availableis regardedas a nonlinear projectionon L ofu, denotedby fl(Llu). (2 L DEFINITION 1. A selectionrule (withbasic set S) is a mapping[l: Yt-*S suchthat 11(L) E L foreveryL E .. A projectionrule is a familyofselection rules fIl( Iu), u e S, such thatu E L impliesfl(Liu) = u. A selectionrule is generatedbya functionF(v), v E S, if foreveryL E Y, fl(L) is the unique elementof L whereF(v) is minimizedsubjectto v E L. A projectionrule is generatedbya function F(vlu), u E S, v E S, if its componentselectionrules fl( Iu) are generatedbythe functionsF( Iu). REMARK.A necessaryconditionforF(viu) to generatea projectionrule is thatforanyfixedu E S theuniqueglobalminimumof F on S be attainedat v = u. Attention maybe restricted to functionsF forwhichthisminimum is 0 [because a projectionrule generatedby some F(vlu) is also generatedby F(vIu) = F(vIu) - F(uIu)I. A functionF(vIu), u E S, v E S, withtheproperty F(vlu) 2 0, withequalityiffv = u, willbe calleda measureofdistanceon S. EXAMPLE1. In the case S = Rn, the Euclidean distance F(vlu) = ilu - vil generatesa projectionrule that gives rise to the ordinaryprojectionin Euclideangeometry. It willbe calledthe least squaresprojectionrule and its componentselectionrules will be called least squares selectionrules. In particular,the selectionrule generatedby F(v) = livilis the standardleast squares selectionrule. A projectionrule generatedby a weightedL2-distance (E n 1ai(vi - ui)2)1/2 willbe calleda weightedleastsquaresprojectionrule.Of course,"least squares" is a standardmethodof a verylong history.Notice, however, thatit is notsuitablein thecases S = R or An; thenF(v) = liv- uil does not generatea selectionrule forany u E S (neitherdoes any weighted L2-distance)becauseit does notattaina minimumon somesubspacesL E .Y. EXAMPLE2. definedby (2.4) Let S = R or An. The I-divergence ofv E S fromu E S is I(viiu) = f (vilog - vi+ ui. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2038 I. CSISZAR This generalization ofKullback'sformula I(v||u) n v vilog,U u E- Anx V E- A\n retainsthe propertyI(vllu) ? 0, withequalityif and onlyif u = v. Clearly, F(vlu) = I(vllu) generatesa projectionrule (forany fixedu E S, it attainsa unique minimumon any L E /, and if u E L, this minimumis attainedat v = u); this will be called the I-divergence projectionrule,and eitherof its componentselectionrulesis an I-divergence selectionrule.The selectionrule generatedby the negativeentropyF(v) = E n 1vilogVi will be called the maximumentropy selectionrule.This is a specialcase of I-divergence selection rulescorresponding to u = (1/n)1 if S = A\ and u = (1/e)1 if S = Rn. As a comparisonof the naturalityof these choicesof u indicates,the maximum role mainlyin the case S = An, entropyselectionrule plays a distinguished thatis, forinferring probability distributions. We emphasizethat we do not a priorirestrictattentionto selectionrules 2 and 3 generatedby some function.However,the postulatesin Definitions belowwillsuffice to provesuch generatedness. Givena selectionrule II withbasic set S, we willdesignateHI(S) byv?(H). Then the componentselectionrules H( Iu) of a projectionrule satisfy,by Definition1, (2.5) vo(H(. Iu)) = u. DEFINITION2. A selectionrule H:Y-+ S is regular if it satisfiesthe axioms: following if L' c L and H(L) e L', then E(L')=l(L); (i) (consistency) (ii) (distinctness) if L # L' are bothin X#,then H(L) M H(L') unlessboth L and L' containv?(HI); (iii) (continuity) the restriction of Hl to the subspacesof any fixeddimension is continuous. A projectionrule is regularifits componentselectionrulesare such. The consistencyaxiom formalizesthe intuitiveidea that if v = H(L) selected on the basis of constraintsspecifyingL also satisfiesthe stronger constraintsspecifying L', thenthe additionalconstraints provideno reasonto changethe selectionof v. The case forthis axiomappears quite strong,for one wouldhardlywantto use a selectionrule example,in imagereconstruction not satisfying it. Nevertheless, forcertain this axiom may be inappropriate problems.Whenthe selectedelementoughtto resemblethe otherelementsof thefeasibleset L as muchas possible,it is reasonableto selectthev E L that minimizes-forsome given measure of distance d-either sup,, L d(v,w) of d(v,w) ["barycenter expectation method,"Perez (1984)] or the conditional on the conditionw E L (Bayesianrule,requiresa priordistribution on S). This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2039 These selectionrulesdo not satisfythe consistency axiomand are outsidethe scopeofthispaper. in a The distinctness axiomsaysthatdifferent bothconsisting information, singlelinear constraint,must lead to different conclusions,unless both are This consistent withtheselectionthatwouldbe madewithoutanyconstraints. is a technicalpostulateand it wouldbe desirableifit couldbe dispensedwith. Notice that the distinctnessaxiom is certainlysatisfiedfor selectionrules functionsF. generatedby differentiable is an obviousregularity Continuity hypothesis.Notice,however,that (for projectionrules) we did not postulatethe continuousdependenceof H(LIu) on u. For any set of indices J = Ul, ... , jk, 1 < il < < Jk < n, and any vectora E Rn, we denoteby aj the vectorin Rk definedby (2.6) aj = (ajlX ... , aik) For a selectionrule [I, we willdenote(II(L))j briefly by rlj(L). S is local ifforeveryJ c {1, ... , n} DEFINITION3. A selectionrule II: Y ofarbitrary size k, and any L' and L" in Y ofform (2.7) L'= {v: vj e L, vjcEL'}, L" = {v: vjE L, vjcEL"), L" E4k whereLo E..4, L' E Yk, (if S = Rn) or Lo E E4yn+-k (if S = Rn orAn), we have L" -k XL' E/A-k, (2.8) H( L') = 11JW) . A projectionruleis local ifits componentselectionrules IL( Iu) are local and, in addition,forL' and L' as abovewe have (2.9) HIj(L'u') = llJ(L"'Iu") ifu', = U'. = +,then anyL' and L" of REMARK.If S=Rn, )Y=n orS=R+, form(2.7) necessarily belongto ..?. On theotherhand,if S = An, ..= 4+ (1), the sets L' and L" in (2.7) belongto Y iffthe sum of componentsof each vectorin Lo is equal to the same 0 < c < 1, and the sum of componentsof each vectorin L' and L" is equal to 1 - c. Localitymeans,in otherwords,thatforL' E Y as in (2.7), [Ij(L') depends onlyon LO) and [IH(L'Iu') dependsonlyon Lo and u'j. the localityaxiomsays thatwheneverthe availableinformation Intuitively, consistsof two piecesthat involvecomplementary sets of componentsof the vectorto be inferred, each component ofthevectorselectedon thebasis ofthis whichinvolvesthe willbe determined information bythatpieceofinformation componentin question.In the reconstruction problemof X-raytomography, this means that if two sets of ray paths coveringdisjointsets of pixelswere wouldbe determined used,thenforeach pixelthereconstruction bythoserays whose paths are in the set coveringthat pixel. This axiom appears very This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2040 I. CSISZAIR natural,but strictadherenceto it mayperhapsbe criticized in thetomography exampleon thebasis ofsmoothnessproperties oftheunknownX-rayattenuationfunction.For the case ofinferring probability mass functions, Shoreand Johnson(1980) used a similarbut strongerpostulatecalled "subset independence." In this paper, regularityand localitywill be the basic postulates.The selectionand projection rulessatisfying themwillbe characterized in Theorem 1. In the restofthissectionwe formulate someotherdesirableproperties that suggestthemselvesas additionalpostulates,ifwe wantto arriveat methodsof practicalinterest. An important rolewillbe playedbythe specialsubspaces ({v: vj + Vj=t}, (2.10) Lij(t) | v: + vi = t, E if S =R v= 1-t}, orR, if S = 'A lo,j Given a selectionrule fl, we will writev ij v' to designatethat v and v' equal the ith and jth componentsof il(Lij(t)) forsome t; of course,then necessarilyt = v + v'. Similarly,givena local projectionrule,we will write (vlu) -ij (v'u') to designate that v and v' equal the ith and jth components of [l(Lij(t)Iu) ifthe ith and jth component ofu are u and u' (and t = v + v'). Clearly,v <ij v' means the same as v' *ji v. Further,we will show (corollaryof Lemma 3) that if v -ij V' and v' jk V" for a regular,local selectionrule II, then also v *>ik V", providedin the case S = An that v + v' + v" < 1. Ofcourse,similarstatements holdfortherelations(v u) i*- (v'Iu') associatedwitha regular,local projectionrule. DEFINITION4. (i) A local projectionrule is semisymmetric if for every Lij(t) as in (2.10), the ith and jth componentsof Hl(Lij(t)u) are equal wheneverui = uj, that is, (vyu) <-ij (v'lu) iffv = v' (providing, in the case S = An, that v < 1/2, u < 1/2). (ii) A projectionrule withbasic set R+ or An is statisticalifit is regular, local,and the relation(ylu) *-ii (v'lu') is equivalentto v/u = v'/u',withthe additionalconditionsv + v' < 1, u + u' < 1 if S = An. The term"semisymmetric" refersto the factthatthisweak and plausible postulateoftenimpliesthe apparentlymuchstrongerpropertyof symmetry (permutation invariance);see Theorem2(i). It wouldnot be unreasonableto use symmetry as a postulate,as did Shore and Johnson(1980), but by not doingso we will obtain strongermathematicalresultswithlittleadditional effort. Also the strongerpostulatein (ii), applicablewhen S = R n or An, appears in thelattercase. Namely,ifa priorguess natural;it is particularly compelling about a probabilitymass functionhas to be updated subject to a single ofa givenset,it is standardto assign constraintthatspecifiesthe probability This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2041 to the prior ones. probabilitiesto the elementsof this set proportionally 4(u) Definition requires this for two-elementsets only. The proportional the probabilitiesof several updating,in general for constraintsspecifying pairwisedisjointsets ["Jeffrey's rule,"cf.Diaconisand Zabell(1982)] appears uncontroversial. It was also partof the axiomsof Shore and Johnson(1980). 5. (i) A projectionrule withbasic set S = R' or R+ is scaleDEFINITION invariantifforeveryA > 0, L E Y and u E S we have [l(ALJAu) Afl(LIu). (ii) A projectionrule withbasic set S = Rn is translation-invariant if for everyL E Y, u E S and ,uE R, fl(L + ,1Iu + 1) = fl(LIu) + 41. Here AL and L + ,i1 denotethe set ofvectorsAvand v + At1,respectively, such thatvE L. The intuitivemeaningof these invariancepropertiesis obvious,and they are clearlydesirable.The characterization ofall (regularand local) projection rules satisfying eitheror both of thesepostulatesis not difficult but will be omittedbecause of its limitedpracticalinterest.Rather,thesepostulateswill be used in connection withthe nextone only. DEFINITION 6. (i) A projectionrule is subspace-transitive if forarbitrary L' c L in Y and anyu E S we have fH(L'Ju)= fl(L'jfl(L1u)). (ii) A projectionrule is parallel-transitive if (2.11) holds forsubspaces L and L' that can be representedin the form{v: Av = b} [cf.(2.2)] withthe same matrixA. (2.11) Subspacetransitivity meansthatifupdatinga "priorguess" u based upon information thefeasibleset L resultsin v = Hl(LJu),and additional specifying information leads to L' as the actual feasibleset,thenupdatingthe "present guess" v on the basis of all availableinformation leads to the same resultas wouldthe directupdatingof the "priorguess" u. For example,if an image reconstruction has beenobtainedfrommeasurements Ri f, i = 1,. . ., k, using some priorguess,and thenfurther measurementsRi f, i = k + 1,. . ., k + 1, are made,thereconstruction fromall themeasurements Ri f,i = 1,..., k + 1, willbe the same no matterwhetherthe originalpriorguess or the previous reconstruction is used as " priorguess." Parallel transitivity has a similar intuitivemeaningforthecase whenafterhavingobtainedthefirstreconstruction,the same measurements are repeatedwithresultsdifferent fromthose before;the second reconstruction is now based on the new results,the previouscontradicting ones beingdiscarded. Subspace transitivity is a highlydesirableproperty also because it implies (and is actuallyequivalentto) the commutativity of two-stageupdatings. Indeed,letus be giventwosetsoflinearconstraints determining subspacesL1 and L2 withL1 n L2 = 0. Then (2.11) impliesthatupdatinga priorguess u based upon the firstset of constraints and thenupdatingthe resultfl(L1Iu) This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2042 I. CSISZAR based uponbothsets ofconstraints, theresultwillbe H(L1 n L21u); thesame resultwouldbe obtainedifthe firstupdatingwerebased on the secondset of constraints. Ofcourse,thiscommutativity holdsfor"proper"two-stage updatingsonly,thatis, whentheconstraints used in thefirststageare notdiscarded in the secondstage. For regularprojectionrules,paralleltransitivity impliessubspacetransitivity,and more generally,also that (2.11) holds wheneverL = {v: Av = b}, L {v: Av = b'}, wherethe matrixA containsall rowsof A. In fact,then L = {v: A'v = A'v*} c L, wherev* = fl(Llu), and thus fl(Llu) = v* by the axiom.Then paralleltransitivity consistency impliesthat = I(L'Il(LIu)), [(L'ju) and (2.11) holdsas claimed. We will show that for regular,local projectionrules the two kinds of definedabove are actuallyequivalent.The familyof all regular, transitivity local and transitive in Theorem3. projectionruleswillbe characterized Combiningresultscharacterizing projectionruleswithpropertiesstatedin Definitions4-6 will lead us to the least squares and I-divergence projection rules as the onlyones that simultaneously satisfysome intuitively appealing postulates.On theotherhand,a singlepostulate(in additionto regularity and locality)willalso suffice to uniquelycharacterize theseprojection rules,as well as the least squares and I-divergenceselectionrules. To formulatethis postulate(Definition7 below),it is necessaryto considervectorsindexedby pairsofintegers(i, j), 1 ? i < m, 1 < j < n, ratherthanbyintegers1 < i < n. This is the natural representation for vectorsdescribingtwo-dimensional objects,such as images. The marginalsof v = fvij: 1 < i < m, 1 <j < n} E Rmn are the vectors E Rm, v = (v,.. =(i1,.. . ., v (2.12) v= n j=1 .., vn vij, E R, where m V E i=l vij. As before,we considerthreechoicesof our basic set S, namely,S Rmn or Amn = Rmnor For any v = {vij} E S, we denoteby L, the subspaceof S consistingof thosevectorsthathave the same marginalsas v, thatis, (2.13) We say thatv (2.14) Lv = = (w: w = v; w = v}. {vij} is of sum or productformif vij = si + tj or vij = s t, respectively. DEFINITION 7. A selection rule H with basic set S as above is composition-consistent, more precisely,sum- or product-consistent, if This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2043 A projecHI(L,) = v wheneverv E S is of sum or productform,respectively. if its compotion rule is composition-consistent (sum-or product-consistent) wheneveru itselfis of sum or nentselectionrules fl( Iu) have thisproperty productform,respectively. the vectorsv = {vij, 1 < i < m, 1 < j < n} are interpreted Intuitively, as oftwointeracting compositions the individualcomponents components, being represented by the marginalsv and v. Then Lv beingthe feasibleset means thattheavailableinformation but nothing specifiesthe individualcomponents else. The postulateofcomposition formalizes theintuitiverequireconsistency mentthaton the basis ofsuch information we shouldinfer"no interaction," unless a prior guess is available that impliesinteraction.Implicitin this interpretation is that "no interaction"is represented by the sum or product formof v. In particular,for inferring probabilitymass functions,product is a hardlyavoidablepostulate.It could be argued,thoughwith consistency less weight,that productconsistency is an appropriateaxiomalso forimage that is, that if the marginalsof an image were knownand reconstruction, nothingelse (an unlikelysituationin practice),the "best" reconstruction wouldbe ofproductform.In inverseproblemswithouta positivity constraint, the sum formofv appearsto be the naturaldescription of "no interaction," suggestingthat in this case the postulateof sum consistencyshould be adopted. REMARKS. (i) Product-consistentselection rules cannot exist for S = Rmn because in that case different elementsof S, each of productform,can have thesame marginals.On theotherhand,sum-consistent selectionrules,satisfyingthecontinuity axiomin Definition 2, cannotexistforS = R + n or S = Amn To see this,considera sequencev(k) of elementsof S of sum formv(*)-*t wees1t= s(k) >O> suhtaS(k) S (k) +t (k) > jh0,, such t5(k)O, -_> that SI -*+( SSi' t(k) i tj, wheres, = t, = 0 I( *j Ij)Si* and si > 0 fori > 1, tj > 0 forj > 1. Then v(k) - v and L(k) - Lv, where selectionrulewe shouldhave vij = si + tj. Thus fora sum-consistent fl(Lv) = limfl(Lv(k)) k- = lim v(k) = v k- a contradiction becausev i S. (ii) In the case S = Rmn,everyLv as in (2.13) containsan elementofsum form;hence fora sum-consistent selectionrule II, fl(Lv) is alwaysof sum form.It follows,subjectto regularity, thatv? = v0(01)is ofsumform,because = v? II(Lvo) by the consistency axiom. Similarly,in the cases S = Rmn or if a is HI product-consistent (regular)selectionrule,then v0(HI) is of Amn productform. (iii) For thecase S = Amn, thepostulateofproductconsistency is similarto but weakerthan Shore and Johnson's(1980) "systemindependence"postulate. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2044 I. CSISZAR 3. Statement of results. Recallthatourbasic set S is eitherRn, R n or An,the familyofaffinesubspacesof S is denotedby /, and V denotesR, R + or theinterval(0, 1), accordingto thechoiceof S. Throughout thissection,we assumethat n 2 3 (if S = Rn or R+) or n ? 5 (if S = An). The following willfacilitatethe statementofour results. terminology DEFINITION 8. A n-tupleoffunctionsf1,. . . ,fn definedon V willbe called a standardn-tuplewith0 at vo = (vI, .. ., v? E S if: (i) fi is continuously differentiable and vanishestogetherwithits derivativeat v ,i = 1, ..., n, (ii) in the cases S = Rn or A ,lim oif(V) = -0i =L. n (iii) the function n (3.1) F(v)= E fi(vi) i=l is nonnegativeand strictlyquasiconvexon S, that is, forany v and v' in S and any 0 < a < 1 we have (3.2) F(av + (1 - a)v') < max(F(v), F(v')). REMARK. The functionsfi in a standardn-tuplemustbe convexif S = Rn or R n but not necessarilyif S = A\. The proofofthisis omittedin orderto save space. (i) For anyregular,local selectionrule [: s-- 5, thereexists a standardn-tuplef1,.. ., fn with0 at v0 = v0(l) such thatr1 is generated bythefunctionF in (3.1). Conversely, any F as in (3.1) generatesa regular, local selectionrule 1l withv?(JII)= v?. (ii) For any regular,local projectionrule, thereexistfunctionsfi(vlu), u e V, v E V, such thatforeveryfixedU = (U1,..., Un)T E S, thefunctions fi(vIu) forma standardn-tuplewith0 at u and thegivenprojectionruleis generatedby THEOREM 1. (3.3) n F(vlu) = E fi(vilui). i=1 Conversely, anysuch F generatesa regular,localprojectionrule. (iii) Two functions F(v)= E= if(vi), F(v)= En1fi(vi) or F(vlu)= E Z=lfi(vIui) F(VIU) = E1 fi(viIui) as in (i) or (ii) generatethesame selection or projectionrule, respectively, if and onlyif fi= cfi,i = 1,... , n, for someconstantc > 0. REMARK. In part (iii),the assumptionthat F and F be ofthe statedform is essential.Otherwise,F(v) and F = 4(F(v)) generatethesame selectionrule for any strictlyincreasingfunctionFD,and F(vlu) and F = F(F(v1u),u) generatethe same projectionrule whenever(?(* Iu) is strictlyincreasingfor everyfixedu E S. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2045 THEOREM 2. (i) A projectionrule withbasic set S = Rn or Rn is regular, local and semisymmetric (Definition4) ifand onlyifit is generatedbyF(vlu) as in Theorem1(i) withfi notdependingon i. (ii) A projectionrule withbasic set S = R or An is statistical(Definition 4) ifand onlyif it is generatedby (3.4) F(vlu) = E ui fI-'-I) for some continuously strictlyconvexfunctionf on R+, with differentiable, = = = f(l) f'(1) 0 and limt 0 f'(t) oo. Functionsofprobability ofform(3.4), as measuresofdistance distributions motivated byinformation theory,wereintroduced by Csiszar(1963) underthe name f-divergences [cf. also Csiszar (1967)] and independently by Ali and Silvey(1966). For theirapplicationsin statistics,see, forexample,Liese and Vajda (1987). The conditionson thederivative of f are notpartoftheoriginal definition of f-divergences. Notice that the conditionf'(1) = 0 is essential onlyin the case S = R+ because if S = AnX thenany f(t) and f(t) = f(t) + c(t - 1) definethe same F in (3.4). The conditionlimt 0 f'(t)= -o is necessaryforF in (3.4) to generatea projection rule,thatis, to ensurethatits minimumin v be attainedon everyL E Y. THEOREM 3. (i) For any regular,local and subspace-transitive projection rule (Definition6), thereexists a standard n-tuple(P1, , fpnsuch that ?(V)= E=p1,i(vi) is strictlyconvexon S and the givenprojectionrule is generatedby F(vlu) (3.5) = ?(v) n E i=l - ?(u) ((v) - - (grad 1(u))T i(Ui) - (v - u) 0(Up(iu)(V-ui)). (ii) AnyF as in part (i) generatesa regular,local and parallel-transitive projectionrule. On accountofthistheorem,in the sequel we neednotdistinguish between the twokindsoftransitivity. COROLLARY.The only transitivestatisticalprojectionrule (with S An) is theI-divergence projectionrule (cf. Example2). = Rn or The class of measuresof distanceassociatedas in Theorem3 withstrictly convex functions'F (not necessarilyof a sum form)was introducedby 'F under Bregman(1967). He developedan iterativealgorithm forminimizing linear(and, moregenerally, underlinearinequality)constraints, whosesteps involvedprojectionsin the sense of the distancecorresponding to 'P [cf.also Censorand Lent (1981)]. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2046 Bregman'sdivergencesinclude squared Euclidean distanceand I-divergence,and sharewiththesethe property (3.6) F(vlu) + F(wlv) = F(w1u) if [l(Llu) = v, w E L. Notice that for squared Euclidean distance,(3.6) is just the Pythagorean also has thisPythagorean property playsa theorem.The factthat I-divergence keyrolein its applicationsin statistics[cf.Kullback(1959)]. A resultrelatedto but notdirectly comparablewithTheorem3 was recently thepreviousresultsofJones obtainedbyJonesand Byrne(1990),generalizing (1989). They showedthat amongthe continuousanalogs of the distancesof form(3.3), onlythe analogsofthosein Theorem3 satisfieda postulatecalled projectivity. That postulateis stronglymotivatedfromthe pointof view of inference, but it also involvesthe distanceitself,as opposedto transitivity which is a propertyof the projectionrule alone. Jones and Byrne(1990) pointedout that theirdistancessatisfied(3.6), and, in fact,that (3.6) was equivalentto the projectivity postulate. THEOREM4. (i) A projectionrule withbasic set S = Rn is regular,local, transitive and bothlocationand scale-invariant ifand onlyifit is a weighted least squares projectionrule (cf. Example 1). The above propertiesand semisymmetry uniquelycharacterize theleastsquaresprojectionrule. (ii) A projectionrulewithbasic setS = Rn+is regular,local, transitive and scale-invariantif and only if it is generatedby F(vlu)= ELz alha(vilhu4 wherea1, ... , an are positive constants,a < 1, and v v log-u (3.7) ha(VIU) v + u, if = 1, u v logg-+---1, 1 _(ua a v if a =0, u - va) + Ua l(v - u), if 0 < a < 1 or a < 0. A projectionrulewiththeaboveproperties ifand onlyif is also semisymmetric it is generatedby (3.8) Fa(vlu) = n E i=1 ha(vilui), a < 1. REMARK. The one-parameter family(3.8) containstwo well-knowndistances: I-divergence (a = 1) and Itakura and Saito (1968) distance (a = 0). RecentresultsofJonesand Trutzer(1989) indicatethat(continuousversions of) the membersof this familywith a = 1/m (m some integer)may be a tradipreferable to the Itakura-Saitodistancein spectrumreconstruction, tionalfieldofapplicationofthe latter. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2047 ofthe least squaresand I-diverOur last resultprovidesa characterization genceselectionand projectionrulesthat,insteadofinvarianceor transitivity, relyon composition consistency (Definition 7). We emphasizethatthischaracterizationalso appliesto individualselectionrules,ratherthan to projection rules only.On the otherhand,we have to assume thatthe basic set S is as the inferenceproblemin question describedbeforeDefinition7. Intuitively, shouldrelateto objectswith(at least) twocomponents. THEOREM5. Let S be as in Definition7, S = Rmnor R+n or Ain' with m > 2, n > 2, and if S = Amn, m + n ? 5. (i) In the case S = Rmnn,the regular, local, sum-consistentselection and projectionrulesare exactlythoseleastsquaresselectionrulesforwhichv0 is of sum form,and theleast squaresprojectionrule,respectively (cf. Example 1). (ii) In the cases S = R mnor A the regular, local and product-consistent selectionrulesfor selectionand projectionrulesare exactlythoseI-divergence whichv? is ofproductform,and theI-divergence projectionrule,respectively (cf. Example 2). (i) For S = Rmnn,thestandardleast squares selectionrule is theuniqueregular,local, sum-consistent selectionruleforwhichv0(H) = 0. (ii) For S = Almn'themaximumentropy selectionrule is theunique regular, local, product-consistent selectionruleforwhichv0(Lt) is the "uniform distribution" (1/mn)l. The same holdsalso forS = R+ n ifthelast condition is replacedby vo(H) = (1/e)1. COROLLARY. 4. Basic lemmas. LEMMA1. For a regularselectionrule H: S , for everyL' E Y of dimension less than n - 1 (if S = Rn or Rn) or less than n - 2 (if S = A thereexists L E X/ [cf. (2.3)] such that L' c L and H(L') = [1(L). - COROLLARY . For a regularH and L' E ., [1(L) impliesL' c L unless H(L') = v0(H). L E X#,theequality[I(L') = PROOF. Clearly, it sufficesto prove that if dim L' = d with 0 < d < n - 1 (or 0 < d < n - 2), then thereexistsan L e .. such that L D L', dimL = d + 1, H(L') = H(L). Further,insteadofthe last equality,it suffices to show that [(L) E L' becausethis,bytheconsistency axiomin Definition 2, already implies (L') = [(L). L1 D L' and supposethat [1(Ll) t L'. Now, pick any (d + 1)-dimensional Then we will "rotate" L1 to obtaina family{Lt: 0 < t < 2} and show that 1(Ld) E L' forsome t. To this end, pick any vo 0 L1 in S, set v, = [M(L,) and let v2 be such that some interiorpointof the segment[vO,v2] is in L'. Set vt = (1 - t)vo + tv1 if 0 < t < 1 and vt = (2 - t)v1 + (t - 1)v2 if 1 < t < 2, This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2048 I. CSISZAR and let Lt denotethe subspaceof S spannedby L' and vt.Then Lo = L2 and Ltn Lt2= L' if0< t1<t2 <2. By the continuity axiom,{I1(Lt): 0 < t < 21 is a continuousclosedcurvein the subspaceL spannedby L1 and vo (t = 0 and t = 2 representing the same point).For ? > 0 sufficiently small,fl(L1) and fl(Ll+E)-which are arbitrarily close to 11(L1)= vl-are separatedby L1 withinL; hence thereexists some t with It- 11> E forwhich fl(Lt) E L1. Then fl(Lt) E Lt n L, = L' and theproofofLemma 1 is complete. The corollaryis immediate,because for L E X containingL' such that axiomimpliesL = L. El ME(L')= 11(L),the distinctness LEMMA2. The restriction ofa regularselectionrule Vt:.-* S to X'\ Y0 is a homeomorphism ontoS \ {v?}, wherevo = vo(H) and Y0 - {L: vo E LI. PROOF. ApplyingLemma 1 to the zero-dimensional subspace L' = {v}, it followsthatforeach v E S thereexistsL EX-#suchthat11(L) = v. Ifv 0 v?, thenL 4 ?, bytheconsistency axiom.Thus 11maps X#\f ? ontoS \ {v0}. This mappingis one-to-one and continuousbythe distinctness and continuity axioms.It remainsto provethe continuity of the inverse.In otherwords,we have to showthat U(Lk) --+H(L) #v? impliesLk-- L. We do this by showingthat everysubsequenceof {Lk* containsa subseto L. Write quenceconverging L k J{v: akv =bk {v: aTv = b; 1Tv1if= ifS =Rn orS =R+, S =A [cf.(2.3)]. Here we may supposethat Ilakll= 1 and in the case S = An also thatak ? 1. Now, any subsequenceof {L*} containsa subsequence{Lkj such that by asa, say. Writefl(Ld) = Vk, fl(L) = v*, aTv* = b. As Vkv* aki = ak.Vk that b. This a means that if a aTv= sumption, implies aki bki the set -* LJ[{v:aTv-b}, ifS=R orRn, (v: &Tv= b, 1Tv = 1}, if S = An, is in we have Lk. L. But L EX,# holds because (i) L ? 0 (namely, = 1 and,if S = An, also a ? 1. We have provedthat everysubsequenceof {L*) containsa convergent subsequenceLk. -+ L. Bythecontinuity axiom,here1l(L) = limi lH(Lk.)= 11(L) # v?, and hence necessarilyL = L by the distinctnessaxiom. This completestheproofofLemma2. oJ c@, -* v*~E L bythe definition of b) and (ii) IlaI We will need the following notation,fork < n: The vectorsin Rk whose are all 0 or all 1, willbe denotedby Ok and lk, respectively components (thus This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE o= on, 1 = In). Two vectorsin a a, iffforsome A # 0, 2049 will be called equivalent,denotedby Rk if S =Rn orS =RRn+ if S = An- {Aa+ XAa+ ,k, AE R, Observethatin the representation L (4.1) ) -ftv:av = t{v: aTv blTV ifS=R orR+, fS = A = 1} ofa subspaceL E X1,thevectora E Rn is determined up to equivalence,and it is notequivalentto 0. LEMMA3. Let H: ..t-* S be a regular, local selectionrule, and let therelationintroducedin thepassage containing(2.10). *-*ij be (i) Let L fe, [(L) = v* * v? = vo(H). Then vi* ij vj* if and only if (4.1) of L; further,in the cases S = Rn or ai - aj in the representation S =R+n, v = v ifand onlyifa =0. (ii) Let L and L be both in Xt' with [(L) # v?, [(L) 0 vo, and let = Hj(L) impliesaj J c{1, ..., n}. Then[j(L) iaj fora and -a representL ing and L as in (4.1). In otherwords,aj is determined by jII(L) up to equivalence. COROLLARY.If vi +>ij also vi +ik Vj and vj *->jk Vk for some {i, j, k) c (1, .. ., n}, then = An thatvi + Vj + Vk < 1. Vk, providedin thecase S PROOF. We will show that for L as in (i), arbitraryJ = {j1, {1, . . ., n}, a E Rk, and for *, iv= a -avTv (4.2) L a={ aTvj* i J l lTvVjc), iv ifS =Rn ,ik} C orR+, = if S = 'An) the following holds:aj _a is a sufficient conditionfor (4.3) [j(L) = [rj(L') and thisconditionis also necessaryunlessaj To provethis,set (4.4) L" = L n {V: vjc ?k. = VJc. Then,by locality,Hj(L') = r1i(L"). Since,of course,[ljc(L") = V5c, it follows that(4.3) holdsifand onlyif [(L") = v*. Now, if aj a, then L" c L. By the consistency axiom,the latterimplies H(L") = v*. Thus aj a is sufficient for(4.3), as claimed. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2050 if(4.3) and therefore Conversely, [(L") = v* hold,the corollaryofLemma 1 yieldsL" c L. Comparing(4.1), (4.3) and (4.4), L" c L meansthat aTvi = = aivj (subjectto the additionalconstraintE jeJVj = aTvj impliesaiv v if S = An). Clearly,this implicationholds iffthe two conditions E iO representthe same constraint, exceptforthe case a1 0?k. This provesthat for indeed unlessajk.? a" is (4.3) necessary aj, To proveassertion(i) of the lemma,supposefirstthat S = R' or S = R+ and applythe resultjust provedto J = {i}, a = 0. Then L' in (4.2) equals S and (4.3) meansthat vi = vo.Thus we obtainthat ai = 0 impliesvi = v,0and if ai # 0, then vi*= vO cannot hold. Next, apply our resultto conversely, J = {i, j}, a = (1, 1)T. Then L' in (4.2) equals the specialsubspaceLij(t) (with t = V* + VJ) thatdefines therelation+-*ij [cf.(2.10)].Hence(4.3) is equivaa = (1,1)T, except when S = R' lent to v* v Now if ai = aj, then a VJ. or S = R+ and ai = a =O. Thus we have that ai = aj impliesvO*-*jV, because the mentionedexceptionalcase has alreadybeen covered. a; hence v* VJ if ai # aj, thenneitheraj - 02 nor ai Conversely, i cannothold. To proveassertion(ii), supposethat IIj(L) = lJ4(L)= vj and considerL' as in (4.2) with a = &j. Then ljl(L) = Hj(L) = HII(L'), by assumption and the sufficient conditionfor(4.3). This implies,bythenecessarycondition, that when L L Since the roles of and except perhaps 0k' are aj ~aj, aj we similarlyhave aj when is symmetric, not equivalent to ?k' aj aj whereasthe same trivially holdswhenbothaj ?k and aJOk. To provethe corollary, pick any v* E S with vi*= vi, Vj*= V, vk = Vk. If # v* v?, the hypothesesvi ij vj and vj **jk Vk implyby Lemmas2 and 3(i) that v* = 11(L) for some L as in (4.1) with ai = aj = ak. This, in turn, implies[again,byLemma3(i)] that Vi jik Vk whereasthelatteris obvious(by the consistency axiom)ifv* = v?. LEMMA4. (i) Let Fij(u, v), i s j, {i, j) c {1, . . ., n), n > 3, be real-valued functionsdefinedforu E Ai, v c Aj, whereA, ..., An are arbitrary sets. If theseFij satisfythefunctionalequations (4.5) Fi(u, v)Fjk(V, w) = Fik(u, w), Fij(u, v)Fji(v, u) = 1, thenthereexistfunctions gi definedon Ai, i = 1,... , n, such thatforeveryi andj, (4.6) Fij1u,v) = gi(u) (ii) Let n 2 4 and letBij c Ai x Aj, Cijkc Ai x Ai x Ak be setssuch that (1) (u, v)E Bij if and onlyif (v,u) E Bji; (2) (u, v,w) C Cijk impliesthat (u, v) E Bij, (u, w) E Bik, (v, w) E B3k; (3) forany distincti, j, k,I and any uEAi, veAj, wEAk thereexists s e Al with (u, s) e Bil, (v, s) e Bjl, (w, s) E Bkl; and (4) for any distincti, j, k,I and any u, v,s, s' with (u,s),(u,s') in BiI and (v,s),(v,s') in Bjl, there exists w EAk with This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2051 (u, w,s), (u, w,s') in Cikl and (s, w, v),(s', w,v) in Clki. Thenifthefunctions Fij are definedon thesetsB i and theequations(4.5) holdfor(u, v,w) E Cijk theconclusionofpart (i) stillholds. PROOF. (i) Fix some k and w~E Ak, and writegi(u) = FW(u, W) fori # k. Then gi(u) # 0 bythesecondpartof(4.5), and thefirstpartof(4.5) gives(4.6) if i, j are bothdifferent fromk. In addition,(4.5) impliesthat gi (u) g,(v) (V Fjk , W ) Fi k (U, W ) foreveryi * k, j * k, u E A,, vE Ai. Denotingthe commonvalue of these quotientsby gk(W),we obtain(4.6) also forj = k. Finally,the validityof(4.6) forthe remainingcase i = k followsfromthe secondequationin (4.5). (ii) On account of (i) it sufficesto provethat the functionsFij can be extendedfromBij to Ai x Ai so thattheequations(4.5) remainvalid.To this end,givenu E Ai, v E Aj, pickarbitrarily 1 = k (bothdifferent fromi, j) and s and s' in Al such as in hypothesis(4); choose w E Ak accordingto that hypothesis. Then,applyingthe firstand thenthe secondpartof(4.5), we get Fil(U, s)F1j(s, v) = Fik(U, W)Fkl(W, S)Flk(S, = Fik(U, W)Fkj(w, v) V) W)Fkj(W, and similarly Fil(U, s ( S%v) )Flj = Fik(U, v) W)Fkj(W, This meansthat Fij( u, v) = Fil( u, s) F,j(s, v) (4.7) is welldefinedbecausetheright-hand sidedoes notdependon I and s [subject to (u, s) E Bii, (v, s) E Bjl, thelatterbeingequivalentto (s, v) E B,j]. Clearly, (4.7) definesan extensionof Fij to Ai x Ai. To see thattheseextensionsFj satisfythe functional equations(4.5) foreveryu E Ai, v E Aj, wE Ak, let l be different fromi, j, k and pick s E Al accordingto hypothesis (3). Then by (4.7) and the secondpartof(4.5) Fij(u, V)Fjk(V, w) = = Fij(u, v) Fji(u Fil(U, s)F,j(s, v)Fjl(v, FEA(u, S)Flk(s, W) s) Flj(s, v) Fjl(v, I v) = Fil(u, = Fil(u, s)F,i(s, u) s)Flk(s, = Fk(U, = w) W), s) Fli(s, u) 1. 5. Proof of the main results. PROOFOF THEOREM 1. (i) Let H: S be a regular,local selectionrule, and let v? = (v9,... , v2)T= v'(H1). By Lemma 2, foreveryv / v? there exists This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2052 L(v) dependscontinua unique L = L(v) E X suchthatfl(L) = v; moreover, on v. Write L = L(v) as ously (5.1) (5.1) ~L w aT {w: S= R'or R', =aTV,if aTw = aTv; lTW = 1}, if S = A". up to equivalence(cf.thepassagebefore Here thevectora E R' is determined Lemma3). Our firstclaimis that thereexistcontinuousfunctionsgi(v), v e V, with (5.1) of L = L(v) (for gi(v) = 0, i = 1,...,n, such thatin the representation arbitrary v #v?) a - (91(V J (5.2) gn(Vn)) For laterreference, observethat(5.2) impliesby Lemma3(i) that Vi ij vj ifand onlyif gi(vi) =gj(vj) (5.3) in the case S = An that vi + vj < 1. providing To proveour firstclaim,we startwiththe simplercases S = Rn or Rn. By up to Lemma3(i), appliedto J = {i, j}, it followsthat(a1, aj )T is determined equivalenceby vi and vj. Thus (5.4) Fij(vi,vj) = of vi and vj wheneverv; 'vj9[which,by is a well-defined continuousfunction (5.4) Lemma3(i), is necessaryand sufficient foraj # 0]. Clearly,thefunctions equations satisfythe functional Fij(Vi Vj)Fjk(Vj, Vk) = Fik(Vi, Uk), Fij(vi,vj)Fji(v, vi) = 1 forvi E V\ {v9), i = 1,. . ., n (recallthatV = R or R+ accordingas S = Rn or Rn). It followsby Lemma4 that (5.5) Fij(vi,vj) = gi(vi) if vi #v10,vj #vjo,for suitable functionsdefinedand not equal to 0 on V \ {v(0.Lettinggi(vo)= 0, (5.5) also holdsforvi = vo?wheneverFij(vi,vj) is thatis, vj 0 vu?. defined, Comparing(5.4) and (5.5), we obtain(5.2). The functionsgi are continuous becausethe Fij are such. case S = An, applyLemma 3(i) with J= Turningto the more difficult {i, j, l} to obtainfora E Rn in (5.1) that(ai, aj, ai)T is uniquelydetermined, up to equivalence,by vi,vj,vI. Hence a (5.6) Fijl(vi, vj,v1) = a, -- a, a, is a well-defined continuousfunctionof (vi,vj,v,) subjectto vi + vj+ v, < 1 This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2053 foraj = a,, by Lemma exceptforvj *j v1[whichis necessaryand sufficient 3(i)]. Clearly,the functions (5.6) satisfythe functional equations (5.7) if VI + V (5.8) (5.9) (5.10) Fijl(Vi + Vk + Vl <1, Vk) VI) = Fikl(Vi) Vk, Vl) vI vjlV)Fjkl(vjy as wellas Fijl(vi, Vj,vl)Fjil(vj, vi, vl) Fijl(vi v, vj)Fjli( , vjv, vvi)Fij( vl,Vij) + Fij(vi,vl,vj) Fij(vi,vj,vl) = 1, = -1, = 1 if vi + vj + v1< 1, assumingin each case thatall functions are defined. These functional equationscan be solvedapplyingLemma4(i) threetimes. First,we use (5.7) and (5.8) fixingI and v1and restricting the domainofthe Fij1's-as functionsof vi and vj-by excludingvi *il vl, that is, Fil = 0. Thenthehypotheses ofLemma4(i) are easilychecked,takingintoaccountfor hypotheses (3) and (4) that(forfixedvl) the relationv >il vl neverholdsif v is sufficiently small;the latterfollowsfromthe continuity axiomin Definition 2. Lemma4 gives (5.11) G-1(vi, vl) Fijl(vi vj,vl) = G1('v) for suitable functionsGil defined(and nonzero) for ui + v, < 1, unless Vi *4il V1 Substituting (5.11) into(5.9), we obtain,afterrearranging, Gji(vj, vi) Glj(vl, vj) Gij(vi, vj) Gjl(vj,vl) Gli(vl vi) 'Gil(vi, vl) NowweapplyLemma4(i) to thefunctions -Gl1(vl, sG V(v)-vl Hil( vg VI) Hi1(vi, vi) (definedfor vi + v, < 1, unless vi il vl), yielding h vi) (( .Hi1(vi,v1) = h1v1 forsuitablefunctionshi defined(and nonzero)in V= (0, 1). This meansthat the functions G 1(v ,v1) Gil(v) vl) = Gjl(vi, v/) h (v1) satisfy (5.12) Gil(vi, vl) = Gli(vl vi) This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2054 I. CSISZAR and,by(5.11), (5.13) Fijl(vi,vj,vj) = G"(vi ) At this point we removethe temporaryexclusionof vi ji, v1 fromthe ofthe domainof Fijl; defining definition Gil(vi,vl) = 0 if vi 'a v1,(5.13) will alwaysholdwheneverFijl in (5.6) is defined. Finally,substituting (5.13) into (5.10) gives,afterrearranging, using also (5.12), (5.14) vj) = Gij(vi,vj), G6i(vi, vi) + G1j(v1, whenevervj + v + v1< 1. Moreexactly,the givenderivation of(5.14) is valid unless Vj -jl v1and a similarderivation from(5.10), withtherolesof i and j interchanged, is validunlessvi*->il vl; ifbothvi*->il v1 and v3 >l vl,then also vi ij vj and (5.14) holdstrivially. (5.14) and (5.12) mean that Lemma 4(i) is applicableto the functions Fij = expGij. It followsthat G7ij(vi,vi) = gi(vi) - gi(vi), forsuitablefunctionsgi definedon V = (0, 1). SinceGij(vi,vj) = 0 if vi +->ijvj andthus,in particular, if vi = v0,vj = here gi(v ) is independentof i, and we may assume that gj(v0) = 0, i = 1,... I n. Substituting (5.15) into(5.13), we obtain (5.15) (5.16) Fijl(vi, vjvl) = g(vi) -- g1(v1) g1(v3) g1(v1) whenevertheleft-hand sideis defined. As thefunctionsFijl are continuous, so are the gi's, too. Comparing(5.16) with(5.6), we obtain(5.2). Havingestablishedour firstclaim,we define v (5.17) fi(v) = f g(t) dt, v? n F(v) = E i=l fi(vi). We willshowthat f1,..., fn forma standardn-tuple[noticethatproperty (i) in Definition 8 obviouslyholds]and that F(v) generatesII. With(5.17), (5.2) becomesa - gradF(v). As the vectora in (5.1) is determinedup to equivalenceonly,it followsthat(forarbitrary v 0 vo)L(v) is the set ofall w E S satisfying (5.18) (gradF(v) )T(W - V) = 0. Observenextthat forany L E -t we have by the consistency axiomand Lemma1 thatfl(L) = v ifand onlyifv e L c L(v), providing v # v?, whereas ifv? E L, thenalways[1(L) = v?. Thus we mayassertwithoutanyrestriction This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2055 on v and forarbitraryL E t, that [1(L) = v if and onlyifv E L and (5.18) fulfilled ifv = vo). holdsforeveryw E L (the latterbeingautomatically For any distinctv and w in S satisfying (5.18), the result in the last paragraphappliedto the line L' throughv and w givesthat (grad F(w))T(w - v) # 0; herewe used the factthatthe difference ofanytwoelementsof L' is a scalar multipleofw - v. By continuity, this nonzeroinnerproductmust be of constantsign for w 0 v satisfying (5.18),whenv E S is fixed.Further,againbycontinuity, this sign cannotactuallydependon v, either.Withoutrestricting generality, we may assume that this constantsign is positive.Indeed,the functionsgi in (5.17) maybe multiplied by(-1) ifnecessary,withoutchangingtheirproperties assertedin our firstclaim. We have obtainedthat(5.18) withw 0 v alwaysimplies (grad F(w))T(w - v) > 0. (5.19) It followsthat forany line L' in S, F(w) is strictlyincreasingas w moves away fromv = 11(L') in eitherdirection.This immediately givesthat F has property (iii) in Definition 8. Further,foranyLEYE, thefactthat F(w) is strictly increasingas w moves awayfromv = 11(L) on anyline L' c L provesthat H(L) is the uniquepoint whereF is minimizedoverL. Thus fl is generatedby F. To provethat (ii) in Definition8 also holds,noticefirstthat as v = v? trivially satisfies(5.18) foreveryw e S, (5.19) gives (grad F(w)j)T(W- v?) > 0 forallw # v' in S. (5.20) IfS = R' or A, let v = ( vl,... , Qn)" be a boundarypointof S such that = s 0 if j 0 i, further,fj'(Vj) f/(01) v forsome j and 1 different vi 0 and VQ fromi. (5.20) impliesthat liminf(grad F(w)"( v w -r )>0; V hence f'(v) is bounded from above as v -- 0. Thus, if the assertion = -mowere false, fi' would have a finitelimiton a suitable limv 0/f7'(v) sequence of positivenumbersapproaching0. Then therewould exist a sequence vk -e v (withvk e S) such that gradF(vk) -- a, say, wherea is not equivalentto 0. This providesthe desiredcontradiction becausethe sequence ofsubspaces Lk = (w: (gradF(vk)j)T(w-vk) = 0, wES} Es X convergesto L-={w: and the sequence[(Lk) aT(w = -_ ) = 0, w e S} E, g to 11(L), forit goesto v Vk does notconverge This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions S. I. CSISZAR 2056 This completesthe proofof the assertionsmade in the passage containing (5.17). Finally,let fl,..., fnbe any standardn-tuplewith0 at v0 = (v?,.. ., and set F(v) = E 1fi(vi). Notice firstthat in the cases S = Rn or Rn property (iii) in Definition8 impliesthat fi(v) is strictly increasing(decreasthelatter, ing)forv > v? (v < v) and also that fi(v) Xc as v -- oo.To verify that fi(v) - c < ooas v xc, choosevi> vf', > v) with supposeindirectly fi(vi)+ fj(vj) = c, and apply(3.2) to v = (v,..., vn)T withv, = vI for1 # i, j and v' = (vi, ... V,v)T withv' = v' for1 # i. It followsthat (5.21) fi(avi + (1 - a)v') + fj(avj + (1 - a)vJ) < f-(vi) + fj(vj), foreveryvi > v? and 0 < a < 1. Lettingfirstvi -> and then a -* 0, (5.21) resultsin thecontradiction c < fi(vi).One verifies in the samewaythatin the case S = R , fi(v) > ooalso as v or An, F(v) has a limit . If S = (finiteor + cm)as v convergesto a boundarypointof S, becauseproperty (ii) in Definition8 impliesthe monotonicity of each fi near 0. It followsthat F, extendedby continuityto the closure of S if S = R+ or A attains its minimumon any L e /, moreexactly,if S = R n or An,on the closureof L. (ii) rules out the minimumbeingattainedon the boundary.By But property property (iii),the pointwhereF attainsits minimumoverL mustbe unique; hence F does generatea selectionrule [I. Property(i) impliesthatthis HI is regularand it is obviouslylocal. (ii) Let us be givena regular,local projectionrule withcomponentprojection rules fl( Iu), u E S. For arbitrary v ? u, let L(vlu) denotethe unique L E X#withrl(LIu) = v. We claimthatthe following modified versionofour firstclaimin the proofofpart(i) is valid: There existfunctionsgi(vIu), u, v E V, continuousin v and vanishingfor v = u, i = 1,.. , n, such that in the representation(5.1) of L = L(vlu) we have T U a - (g(V1 UI 1), (5.22) gn(Vn I n)) In the proofofpart(i), the functionF generating the selectionrule LI was constructed fromfunctionsgi withthe property (5.2). If (5.22) is established, the functionsgi(vlui) therecan be takenas functionsgi(v) corresponding to fl = H(I Iu). Then [cf.(5.17)] Hf(Iu) willbe generatedby (5.23) n F(vlu) = E fi(vilui), i=1 fi(vilui)= f v. ui gi(vlui)dv as a functionofv. This means,by definition, thatthe givenprojectionrule is generatedby F(vlu). Thus it suffices to provethe claimabout(5.22). This can be donealongthe linesofpart(i); hencewe onlysketchthe proof,forthe hardcase S = An . We needan obviousmodification ofLemma3, namelythatforL as in (4.1) with rl(Llu) = v # u, (a) (vilui) ij (vjluj) iffai = aj and (b) the vectoraj is determined by uj and vj up to equivalence. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2057 Applying thisto J = {i, j, 1),it followsthatthe functions (5.24) Fijl(vi,vj,vljui,uj, ul) = a, - a1 are well definedand continuousin vi,vj,v1 subjectto ui + u; + u < 1, vi + vj + v, < 1, exceptwhen(vjIuj) -jl (vilu,). The functions(5.24) satisfyfunctionalequations similarto (5.7)-(5.10), each variablevi in thelatterbeingreplacedbya pairofvariablesvi,u i, where witheach constrainton sums of variablesvi a similarconstrainton sums of variables u is imposed.This systemof functionalequationscan be solved similarlyto (5.7)-(5.10), again applyingLemma4(u)threetimes.It is convein Lemma4 werenot requiredto be nientthatthe variablesofthe functions reals; presentlywe have to let them stand forpairs of real numbersv,u. of the functions(5.24) analogousto Finally,we arriveat a representation (5.16), namely, (5.25) gi(vilui) -- g1(v1ul) riJ~U~UJ,U1tt~U1,It1 Fiji (vi,ij, VIlgi,X j,Xal) = g1(v1iu1) =gj(vjtu1) part Comparing (5.24) and (5.25) provesthedesiredrelation(5.22) and thereby (ii) ofTheorem1. (iii) Suppose that F(v) = E ,7 1fi(vi) and F(v) =E f(vi) generatethe same selectionrule,where(f1,. , fn) and ( f,..., fn) are standardn-tuples. to prove fi = cfi forthecase when( fl,. . ., fn) is arbitrary, Clearly,it suffices with0 at vo = (v?,.. ., v')T, say,and ( to the (regufn) is constructed lar,local) selectionrule 11generatedby F as in theproofofpart(i) [cf.(5.17)]. Now,since F generatesII, its minimumon L = L(v) is achievedat the point v. Hence foreveryv + v' we have (gradF(v))T(w - v) = 0 forall w E L(v). Since L(v) is the set ofall w E S satisfying (5.18), it followsthatforv + v', (5.26) (5.26) gradF(v) =A(v)gradF(v) gTad A(v)gTadF(v) ifS ifS = Rn orRn+, F(v) = gradF(v) = A(v)gradF(v) + i(v)1 if S = An; the same holdstrivially also forv = v?, withj,(v?) = 0 in the case S = An. As thecomponents ofthegradientvectorsdependonlyon thecorresponding componentsof v, the scalar functionsA and ,u in (5.26) and (5.27) mustbe constantand, in particular,,u in (5.27) is identically 0. Thus we actuallyhave (5.27) gradF(v) = c gradF(v). Since f1,..., fn and f1,..., f, have the property (i) in Definition8, (5.28) rules impliesthat fi = cfi,as claimed.The proofofassertion(iii) forprojection is similar.El (5.28) PROOF OF THEOREM 2. By theproofofTheorem1(i), anyregularand local projectionrule is generatedby a functionas in (5.23), where gi(vlu) is a continuousfunctionof v thatvanishesat v = u, i = 1,. . . , n; moreover, for anyu # v thesubspaceL definedby(5.1) withai = gi(viIu1)has theproperty This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2058 of Lemma 3(i) used in that proof,it that Il(Llu) = v. By the modification followsthat (vilui) ij (vjluj) ifandonlyif gi(vilui) =gj(vjluj) (5.29) providedin the case S = An that ui + uj < 1, vi + vj < 1. ruleis semisymmetric, Now,if S = Rn or Ri+ and thegivenprojection that is, (vlu) jj (vyu) foreveryu and v in V and everyi, j E {1, ... , n}, (5.29) impliesthatthe functionsgi do not dependon i; hence,by (5.23), neitherdo the functionsfi. Further,if S = R n or Ai and thegivenprojection ruleis statistical, thatis, (vyu) * i1(v'lu')ifand onlyif v/u = v'/u',(5.29) impliesthat (5.30) (5 30) g~vJ~ = g(~u)if~~gi(vilui) gi(vilui) i V. Vi providedin the case S = An that ui + uj < 1, vi + vj < 1. Actually,the last constraintcan be dispensedwith, because for any given u1,1U jv , vj [in V = (O,1)] thereexist Uk, Vk such that ui + Uk < 1, vi + Vk < 1, uj + Uk < 1, Vj + Vk < 1 and vj/ui = Vk/U. (5.30) means that gi(vlu) is a one-to-one functionof v/u, notdependingon i, thatis, gj(vlu) = g (5.31) ). The continuityof gi as a functionof v and the one-to-onepropertyof g implies that g(t) is a continuous,strictlymonotonicfunctionof t, and g(l) = gi(u lu) = 0. Substituting (5.31) into(5.26) gives (5.32) = Uif fi(vijui) = |dv ) f (t) = fg(s) ds. This completesthe proofofTheorem2. o PROOFOF THEOREM3. (i) First,we showthatifa regularselectionruleis thenfordistinctelementsu, v,w of S, subspace-transitive, (5.33) L(vlu) n L(wlv) c L(wlu) ifwEE L(vju). Here,as in theproofofTheorem1(i), L(vlu) denotestheunique L E X. with H(LIu)= v. Indeed, let L' = L(vlu) n L(wiv). Then [I(L'iv) = w by the consistency axiom,and the transitivity postulateappliedto L' c L = L(vlu) gives fl(L'u) = [l(L'Iv) = w. But Hl(L'lu)= w impliesthat L' c L(wlu), by the corollaryof Lemma 1, proving(5.33). This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2059 Now let us be givena regular,local projectionrule,generatedby F(vlu) as in (5.23). In particular, (5.34) gi(vlu) = -fi(vju) dv is a continuousfunctionof v, vanishingat v = u. Then L(vlu) consistsof thosew E S thatsatisfy n (5.35) E i-l gi(vilui)(wi - vi) = 0. Hence,ifthe givenprojection ruleis subspace-transitive, it followsfrom(5.33) thatforany distinctu, v,w in S satisfying there existscalars a, ,l, y, (5.35), possiblydependingon u, v,w, withy = 0 unless S = An, suchthat (5.36) i - 1,..., n. agi(uilui) + fgi(wi(vi) + y = gi(wilui), We claimthatactually (5 37) gj(vlu) + gj(wlv)= gj(wlu), foreveryu, v,w in V and i 1,.. . , n. Clearly,it sufficesto provethis for = i = 1. Considerfirstthe simplercases S = Rn or R+ . Then forany u, v,w in V, thereexistu,v,win S satisfying(5.35) suchthat u1 = u, v1 = v, w1 = w and, in addition, = (5.38) U2 = V2 + W29 U3 ? V3 W3. Withthese u,v,w, (5.38) impliesthat in (5.36), wherenow y- 0, we have a = f = 1 [usingthat,by Lemma3, gi(vlu) # 0 if v + u]. This proves(5.37) fori = 1. If S = An then v # u does not necessarilyimplygi(vlu) / 0. It follows, however,from(iii) in Definition 8, thatforat mostone index i can u < 1/2 and an intervalI c (0, 1/2) be foundsuch that fi(vlu)is constantforv E I. This implies(assuming,withoutany loss ofgenerality, thatthe exceptionali, if any, is different from2 and 3) that for any 8 < 1/2, the numbersE satisfying (5.39) g2(Ej8) + 0, g3(cJ) # 0 are densein theinterval(0, 1/2). Usingthis,it is easyto see thatforanyfixed v and w sufficiently close to v, thereexist u, v, w in S = An satisfying (5.35), such that u1 = u, v1= v, w1 = w and, withsome E and 8 satisfying (5.39), ui, (5.40) U2 = V2 = U3 W2 = V3 = W3 =? and (5.41) U4 = V4 = W4. Withtheseu, v,w, (5.41) impliesthatin (5.36) we have y = 0, then(5.40) and This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2060 I. CSISZAR (5.39) implythat a = f3= 1. This proves(5.37) (fori = 1) if w is sufficiently closeto v. Observenext that givenany u and v in V = (0,1), the numbersw E V satisfying (5.37) fori = 1 forman openset. Indeed,bythe last paragraph,for closeto w we have w' sufficiently gl(w'lw) + gl(wlv) = g1(w'v), g1(w'Jw)+ gl(wlu) = gl(w'Iu); it followsthatif w satisfies(5.37) fori = 1, thenso does w'. of w, the(nonvoid)set Since gl(wlu) and gl(wlv) are continuousfunctions of those w E V that satisfy(5.37) fori = 1 can be open onlyif it equals the wholeV = (0, 1). This completesthe proofof(5.37) in the case S = An The functionalequations(5.37) implythat the functionsgi can be representedas (5.42) gi(VIU) = iV(V) - i(u), whereit may be assumed that Ofi(s)= 0, i = 1,... , n, forsome s E V [set i(v) = gi(slv), say]. Since gi(vlu) is a continuousfunctionof v, i mustbe continuous,too.Writing (5.43) (Pi(v)= v i(t) dt, the functions'Pi .. ., (Pn satisfy(i) in Definition 8, withvo = s 1, and (in the cases S = R n or An) the validityof (ii) in Definition8 forthe functions'pi followsfromthat forgif(lu), by (5.42). Property(iii) for (Pl, Pn will,of of C4(v)= E ' 1= that we course,followfromthe strictconvexity are going li(vd) to verify immediately. From(5.34), (5.42) and (5.43) we get (5.44) fi(vIu) =] v gi(tlu) dt = fi(v) - fi(u) - M(u)(v - u). This provesthat F(vlu) = E= fi(v(Iu ) has the claimedform F(vlu) = ?F(v) - ?F(u) - (grad?(u) ) T (V - U). Since F(vlu) 2 0, withequalityifand onlyifv = u, thisresultalso provesthe strictconvexity of CFon S. (ii) Let (P1 'Pn be a standardn-tuplesuch that C(v) = E=>pi(vi) is convexon S, and let fi(vlu)be definedby(5.43) where 0i((u)= f(u). strictly Then foranyfixedu E S, the functionsfif(Iun) forma standardn-tuplewith 0 at u; hencebyTheorem1(ii),F(vlu) = E. lfi(vilui) generates a regular, local projectionrule. To provethat the latteris parallel-transitive, suppose that v = H(LIu), w = H[('Iv), where L and L' are "parallel" subspaces. Then gradF(vlu) and gradF(wlv) are both orthogonalto these subspaces (wherethe gradientrefersto the firstvariable)and henceso is gradF(wlu) = gradF(vlu) + gradF(wlv). This meansthat l(L'Iu) = w, as claimed. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2061 To provethe corollary, recallthatbyTheorem2, everystatisticalprojection ruleis generatedby an f-divergence F(vlu) z f Ui where f(t) is a continuouslydifferentiable functionwith f(1) = f'(1) = 0. Then the functionsgi(vlu) in (5.34) do not dependon i and are equal to g(v/u), whereg(t) = f'(t). If thisprojectionrule is transitive, we musthave g()q = 1(v) -q(u) [cf.(5.42)] forsome continuousfunctionq. This impliesthat g satisfiesthe functional equation g(ts) =g(t) +g(s), t,s eR+, whose onlycontinuoussolutionsare g(t) = c logt [cf.Aczel (1966), Section 2.1.2]. Hence f(t) = Jtg(t)dt = c(tlogt - t) and this means that F(vlu) = cI(vIIu). o (5.45) PROOFOF THEOREM4. (i) First we show that a (regular, local) projection rulewithbasic set S = R', generatedby (5.46) n F(vlu) = E i=l fi(vilui) as in Theorem1(i), is translation-invariant ifand onlyif (5.47) fi(v + Alu + A) = c(u) fi(vlu), foreveryu, v and ,Ain R, wherec(,u) is a suitablepositive-valued function. Indeed,sincev* = fl(L + ullu + Al) minimizesF(vlu + Al) subjectto v E L + ,ul, v* - ,ul minimizesF,(vlu) = F(v + ,Illu + Al) subject to v E L. Hence the givenprojectionrule is translation-invariant, that is, Il(Llu) = v* - 4l, if and onlyif this projectionrule is also generatedby Fu(vIu).The latteris, by Theorem1(iii),equivalentto (5.47), as claimed. By Theorem3, if(5.46) generatesa tranlsitive projection rule,then (5.48) fi(vIu) = gPi(v)-si(u) - PJ(u)(v - u), where(,P.. ., Spn) is a standardn-tuple.By the paragraphcontaining(5.43), we mayassumewithoutanyloss ofgenerality thatthisstandardn-tuplehas 0 at v? = 0. Our nextgoal is to determine whatfunctionsfi(vlu) ofform(5.48) satisfy(5.47). Observethat (5.47) implies c(GL1+ tL2) = c(t1)c(A2) and that (5.47) and (5.48) implythe continuity of c(,u).It followsthat c(,u) = el [cf.Aczel(1966), Section2.1.2]. (5.49) forsomef3e R This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2062 From(5.47) (with,u = -u) and (5.49), we obtain (5.50) = ePufi(v fi(vIu) ulO); - thisand (5.48)-where, by assumption,(pi(O)= fp(O)= 0- resultin (5.51) fi(vlu) = e- a). - i(V by v, it followsthat Comparing(5.48) and (5.51) and differentiating (p'(v) - (p'(u) = epuSp(v - u). (5.52) This means, substitutingv = u + t, that ifr= < satisfiesthe functionalequa- tion (5.53) f(u + t) = qf(u) + ePUif,(t). This functional equationis solvedeasily.First, (5.54) +lr(t)= at if,B= 0 [cf.Aczel (1966), Section 2.1.2]. For f3 0 0, observethat (5.53) impliesby symmetry +fi(u) + eiSulk(t) = +(t) + ept`O(u); thus ept - 1 +f(t) q_(u) =eBU - 1 This gives 1) if,B=kO. Thus (p' mustbe eitherofform(5.54) or ofform(5.55), wheretheconstant factora (but not ,B)maydependon i. Clearly,the positivity or negativity of these factors,accordingas p 2 0 or p < 0, is necessaryand sufficient for gettinga standardn-tuple(Pi,. I n fortranslationinvariFinally,just as (5.47) was necessaryand sufficient if ance,one sees thatthe projectionrulegeneratedby(5.46) is scale-invariant and onlyif (t (5.55) a(ept - (5.56) fi(AvlAu) = c(A) fi(vIu), foreveryA > 0. Now, fi(vlIu)definedby(5.48) with'p' ofform(5.55) does not satisfy(5.56), whereaswith'p ofform(5.54) it does. In the lattercase (5.46) becomes n (5.57) F(vju) = E i=l ai(vi - ui)2, ai > O,i = 1,...,n, thatis, onlythe weightedleast squaresprojectionrulesare transitiveas well as location-and scale-invariant. impliesthatall By Theorem2(i), the additionalpostulateofsemisymmetry in coefficients (5.57) must be equal. ai This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2063 (ii) Supposenowthat S = R+ and a projection rulegeneratedbya function and scale-invariant. Then(5.48) and (5.56) hold;in the as in (5.46) is transitive former we nowsupposethatthestandardn-tuple(pl, .. ., 9n)has 0 at vo = 1. As (5.56) impliesthat c(AlA2)= c(A1)c(A2)and (5.48) and (5.56) implythe of c(A), we have c(A) = Aa forsome a e R [Acz6l(1966), Section continuity 2.1.2]. Hence from(5.56) (with A = l/u) and (5.48) [wherenow 'i(l)= po(1) = 0], we obtain (5.58) fi(Vju) = Oaf - 1i) = U'9i Comparing(5.48) and (5.58) and differentiating by v, it followsthat = u q'i(U) -V) fi u v = tu, or,substituting (5.59) p'i(tu) = 0'i(u) + U-'p,(t). Since (5.59) means that p(t) = '9(et) satisfiesthe functionalequation(5.53) (with ( = a - 1), we obtain from(5.54) and (5.55) (5.60) 'p4(t) failog t, a,=(ta- if a =1, - 1), if a 0 1. Clearly,(5.60) with (pi(l) = 0 definesa standardn-tuplefor S = Rn iff a < 1 and, in addition,a > 0 in the case a = 1 or ai < 0 in the case a < 1. Withthis choiceof pi,(5.48) becomes fi(vlu) = lailha(Vlu),with ha defined by (3.7). On the otherhand,(5.46) withthese functionsfi does generatea transitiveand scale-invariant projectionrule. If this projectionrule is also it followsby Theorem2(i) that it is generatedby (3.8). The semisymmetric, proofis complete.0 PROOFOF THEOREM5. Supposethat S Rmn,Rn or Ain the elements of S beingrepresentedas v= {vij), i = 1,...,m, j = 1, . . ., n. Let H be a regular,local selectionrulewithbasic set S. By Theorem1, HIis generatedby a function (5.61) F(v) = m n E E fii(Vii) i=lj=1 wherethe functionsfij forma standardmn-tuplewith 0 at vo {v0} = v?(11).In particular, the functionsgij(t) = (d/dt) fj(t) are continuousand - (5.62) fij(V&?) = gij (v9j) = 0. Considerthe subspaces Lv definedby (2.13), that is, Lv= {w: w =v; = v}. Then if HI(Lv) = v forsomev E S, thatis, ifthe minimum of(5.61) on Lv is attainedat v, we have This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions I. CSISZAR 2064 Ai), . It followsfrom(5.63) that forsuitable"Lagrangemultipliers" (5.64) + gkl(Vkl) gij(Vij) = + gkj(vkj), gil(vil) forevery i,j,k,1. then Nowwe proceedto provepart(i) ofTheorem5. If fl is sum-consistent, by Definition7, rl(L,) = v always holds ifv is of sum form,vij = si + tj. Thus (5.64) givesthe systemoffunctional equations (5.65) gij(si + tj) + gkl(sk + tl) = gil(si + tl) + gkj(Sk + tj). of the (continuous) Observefirstthat (5.65) impliesthe differentiability bothsidesof(5.65) can for this be functions seen, example, by integrating gij; withrespectto Sk* bothsides of(5.64) by tj on the one hand and by Sk on the Differentiating other,we obtain i'j(si + tj) = gkj(Sk + tj) = gjl(Sk + tl). This meansthat g' equals the same constantc foreveryi, j; hence gij(v) = cv + dij. (5.66) Recalling(5.62), it followsthat (5.67) gjj(v) = c(v - V?)g = fij(V) _ (u V)2. it mustbe a least squares selection This provesthat if 17is sum-consistent, rule. Recall that,as remarkedafterDefinition7, vo mustbe of sum form. it is easy to see thatthe least squares selectionruleswithvo of Conversely, sum formare sum-consistent. The resultjust provedeasilyimpliesthatthe only(regular,local) sum-consistentprojectionrule is the least squares projectionrule. In fact,let this projectionrulebe generatedby m F(vlu) n = i=l j=1 f fij(VijUij), wherethe functionsfij( luij) forma standardn-tuplewith0 at u forevery fixedu = (uijs E S. Then (5.67) givesthatif u is of sum form,the termsof thisstandardmn-tuple mustbe constantmultiplesof(v - u ij)2. Sinceforany fixed u there exists u = {uij} of sum formwith uij = u, it followsthat fij(vlu) alwaysequals a constanttimes(v - u)2. The proofof part (ii) is similar.If fl generatedby (5.61) is product-consistent,thenby Definition 8, [l(L,) = v alwaysholdsifv is ofproductform, = Thus (5.64) gives vij sitj. + gkl(Sktl) + 9kj(Skt;)(5.68) =gil(Sitl) 9ij(Sitj) Noticethatwhereasin thecase S = Amn theconditionE vij = 1 represents on thepermissiblesi and t, in (5.68), the*equationmustcertainly a constraint bvai-oanfiei, j kg 1*i si+S 1 t+ < 1. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions AXIOMATICAPPROACHTO INFERENCE 2065 to (5.65). The systemof functionalequations(5.68) can be transformed Namely,if the functionsgij(v) satisfy(5.68), then gij(v) = gij(ev) satisfy (5.65). Hence,from(5.66), gij(v) = gij(log v) (5.69) c log v + dij. Recalling(5.62), it followsthat gij(v) = clog V[oivlo O, fj( v) = cv log- - v + v9j. (5.70) This provesthata product-consistent selectionrulemustbe an I-divergence selectionrule(cf.Example2) withv? ofproductform.Conversely, it is easyto see that these I-divergence selectionrules are product-consistent. The assertionthattheonlyproduct-consistent projection ruleis the I-divergence projectionrule followsin the same wayas did its analogin part(i). The corollaryis immediate.o REFERENCES J.(1966). Lectureson FunctionalEquationsand TheirApplications. ACZfL, NewYork. Academic, ACZEL,J. and DAR6czY,Z. (1975). On Measuresof Information and Their Characterizations. NewYork. Academic, ALI,S. M. and SILVEY, S. D. (1966). A generalclass ofcoefficients ofdivergence ofone distribution fromanother.J. Roy.Statist.Soc. Ser. B 28 131-142. BREGMAN,L. M. (1967). The relaxation methodoffinding thecommonpointofconvexsetsand its applicationto the solutionof problemsin convexprogramming. U.S.S.R. Comput. Math. and Math. Phys.7 200-217. VANCAMPENHOUT,J. M. and COVER, T. M. (1981). Maximumentropy and conditional probability. IEEE Trans. Inform.TheoryIT-27 483-489. CENSOR, Y. (1983). Finiteseries-expansion reconstruction methods.Proceedings oftheIEEE 71 409-419. CENSOR, Y. and LENT, A. (1981). An iterative row-action methodforintervalconvexprogramming. J. Optim.TheoryAppl.34 321-353. CsiszAR,I. (1963). Eine Informationstheoretische Ungleichungund ihre Anwendungauf den Beweis der Ergodizitiit von Markoffschen Ketten.Publicationsof theMathematical Institute oftheHungarianAcademyofSciences8 85-108. CsisZAiR, I. (1967). Information-type measuresofdifference ofprobability distributions and indirectobservations. Studia Sci. Math. Hungar.2 299-318. I. (1984). Sanovproperty, CsIsZAR, generalizedI-projection and a conditional limittheorem.Ann. Probab.12 768-793. CsIsZAR, I. (1985). An extendedmaximumentropyprincipleand a Bayesianjustification (with discussion).In BayesianStatistics2 (J.M. Bernardo, M. H. DeGroot,D. V. Lindleyand A. F. M. Smith,eds.) 83-89. North-Holland, Amsterdam. CSISZAR, I. and TUSNADY, G. (1984). Information and alternating geometry minimization procedures.Statist.DecisionsSuppl. 1 205-237. DEMPSTER, A. D., LAIRD, N. M. and RUBIN, D. B. (1977). Maximum likelihood fromincomplete data via theEM algorithm. J. Roy.Statist.Soc. Ser. B 39 1-37. DIACONIS, P. and ZABELL, S. L. (1982). Updatingsubjective probability. J. Amer.Statist.Assoc. 77 831-834. HERMAN, G. T. and LENT, A. (1976). Iterativereconstruction algorithms. in Biology Computers and Medicine6 273-294. This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions 2066 I. CSISZAR telephony basedon themaximumlikelihood F. and SAITO,S. (1968). Analysissynthesis ITAKURA, Congresson Acoustics(Y. Kohasi,ed.) method.In ReportsoftheSixthInternational 17-20. Tokyo,Japan. oftheIEEE 70 entropy methods.Proceedings JAYNES,E. T. (1982). On therationaleofmaximum 939-952. entropyprinciplesfor theoreticderivationof logarithmic JONES, L. K. (1989). Approximation methodto incorporate ofthemaximum entropy inverseproblemsanduniqueextension SIAM J. Appl. Math.49 650-661. priorknowledge. C. L. (1990). Generalentropycriteriafor inverseproblems,with JONES, L. K. and BYRNE, and clusteranalysis.IEEE patternclassification applicationsto data compression, Trans.Inform.TheoryIT-36 23-30. minimum-distance feasiblehigh-resolution JONES, L. and TRUTZER, V. (1989). Computationally method.InverseProblems5 749-766. whichextendthemaximum-entropy procedures Theoryand Statistics.Wiley,NewYork. KULLBACK, S. (1959). Information Distances.Teubner,Leipzig. LIESE, F. and VAJDA,I. (1987). ConvexStatistical and entropyin incomplete-data MILLER, M. I. and SNYDER, D. L. (1987). The role of likelihood and Toeplitzconstrained intensities point-process to estimating Applications problems: oftheIEEE 75 892-907. covariances.Proceedings Internaofmaximumentropy. A. (1990). A noteon theinevitability J. B. and VENcOvsKA, PARIS, tionalJournalofInexactReasoning4 183-223. in statistical measuresand its application ofa set ofprobability PEREZ, A. (1984). "Barycenter" decision.Compstat Lectures154-159.Physica,Heidelberg. derivationof the principleof maximum SHORE, J. E. and JOHNSON, R. W. (1980). Axiomatic entropy.IEEE Trans. Inform.Theory entropyand the principleof minimum-cross IT-29 (1983) 942-943.] IT-26 26-37. [Correction SKILLING, J. (1988). The axioms of maximumentropy.In MaximumEntropyand Bayesian Methodsin Scienceand Engineering1 173-187.Kluwer,Amsterdam. VARDI, Y., SHEPP, L. A. and KAUFMAN, L. (1985). A statisticalmodel for positronemission tomography (withdiscussion).J. Amer.Statist.Assoc.80 8-35. MATHEMATICALINSTITUTE OF THE HUNGARIANACADEMYOF SCIENCES BUDAPEST, P.O.B. 127 H-1364 HUNGARY This content downloaded from 152.2.104.17 on Wed, 3 Jul 2013 17:53:26 PM All use subject to JSTOR Terms and Conditions
© Copyright 2024