i Contents Table of Contents

Some decision problems can be formulated as sorting models which consist in assigning alternatives evaluated on several criteria to ordered categories. The implementation of a multiple criteria sorting model requires to set the values of the preference parameters used in the model. Rather than ﬁxing directly the values of these parameters, an usual approach is to infer these values from assignment examples provided by the decision maker (DM), i.e. , alternatives for which he/she speciﬁes a required category or interval of acceptable categories. However, the judgments expressed by DMs through assignment examples can be inconsistent, i.e. , may not match the sorting model. In such situations, it is necessary to support the DMs in the resolution of this inconsistency. In this paper, we propose algorithms that calculate diﬀerent ways to modify the set of assignment examples so that the information can be represented in the sorting model. The modiﬁcations considered are either the removal of examples or the relaxation of existing assignments. These algorithms incorporate information about the conﬁdence attached by the DMs to each assignment example. These algorithms aim at ﬁnding and ranking the solutions to solve inconsistency that the DMs are most likely to accept.


Introduction
Many real-world decision problems can be represented by a model stating explicitly the multiple points of view from which alternatives under consideration should be evaluated, through the definition of n crit criterion functions g 1 , g 2 , . . ., g j , . . ., g n crit .Given a set A = {a 1 , a 2 , . . ., a i , . . ., a n alt } of potential alternatives evaluated on the criteria, the analyst conducting the decision aiding study may formulate the problem in different terms.B. Roy [14] distinguishes among three problem statements, i.e., problem formulations (choosing, sorting and ranking) that may guide the analyst in structuring the decision problem (see also [1]).Among these problem statements, a major distinction concerns relative vs absolute judgments of alternatives.This distinction refers to the way alternatives are considered and to the type of result expected from the analysis.
In the first case, alternatives are directly compared one to each other and the results are expressed using the comparative notion of "better " vs. "worse".Choosing (selecting a subset of the best alternatives) or ranking (defining a preference order on A) are typical examples of comparative judgments.The presence (or absence) of an alternative a i in the set of the best alternatives results from the comparison of a i to the other alternatives.Similarly, the position of an alternative in the preference order depends on its comparison to the others.
In the second case, each alternative is considered independently from the others in order to determine its intrinsic value by means of comparisons to norms or references; it consists of assigning each alternative to one of the pre-defined categories C 1 , C 2 , ..., C k , ..., C ncat .The assignment of an alternative a i results from its intrinsic evaluation on all criteria with respect to the norm defining the categories.Several methods have been proposed to handle multiple criteria sorting problems (MCSP), e.g., Trichotomic Segmentation [9], N-TOMIC [8], ORCLASS [6], ELECTRE TRI [15], PROAFTN [2], UTADIS [16] and a general class of filtering methods [12].
One of the main difficulties that an analyst must face when interacting with a decision maker (DM) in order to build a sorting model is the elicitation of various preference parameters used by the method.Even when these parameters can be interpreted, it is difficult to fix directly their values and to have a clear understanding of the implications of these values in terms of the output of the model.In order to avoid direct elicitation of the parameters, several authors have designed disaggregation procedures which allow to infer parameter's values from holistic judgments (such a disaggregation approach was first introduced in the UTA method [4]).Such procedures have been defined for MCSP e.g., [16] for UTADIS and [11] for ELECTRE TRI.
The holistic judgments required to infer sorting models are called assignment examples and correspond to alternatives (actual or fictitious) for which the DM can express a desired assignment, e.g., "a i should be assigned to C 3 " (a i → C 3 ), or "a i should be assigned to i.e., imprecise assignment examples can be considered).In some sorting methods (namely UTADIS and ELECTRE TRI when only the weights of criteria are inferred) such assignment examples define linear constraints on the model parameters.
In order to minimize the differences between the assignments made by the method and the assignments made by the DM, a mathematical program infers the values for these parameters that best restore the DM's judgments.Such a methodology requires from the DM much less cognitive effort than a direct elicitation of parameters (the elicitation of parameters is done indirectly using holistic information given by the DM) and provides a factual justification for the values assigned to the parameters.
Inference procedures are usually not designed as a problem to be solved only once, but rather several times in an interactive learning process, where the DM continuously revises the information they provide as they learn from the results of the inference programs (see [3]).At each iteration, the DM has the opportunity to revise assignment examples.This interactive process stops when the DM is satisfied with the values of the parameters and when the results of the model (i.e., assignment of alternatives to categories) match their view of the decision problem.
During this interactive process, the DM might provide inconsistent judgments, i.e., a set of assignment examples that cannot be satisfied simultaneously by the sorting model.Such inconsistencies can arise for several reasons (cognitive limitations, evolution of preferences during the process, ...).In such a situation, it is not always easy for the DM to identify the reasons for inconsistencies.Moreover, there usually exists more than one way to restore consistency.Hence, the DM need support in inconsistency analysis.
Consider a problem in which a DM has interactively specified assignment examples inducing linear inequalities on the preference parameters.This is namely the case with UTADIS [16] and ELECTRE TRI [3] when only the weights of criteria are inferred.Let x 1 , x 2 , . . ., x j , . . ., x n denote the n parameters of the considered sorting model.The assignment examples define a polyhedron of possible values for the parameters, T = {x ∈ R n : n j=1 α ij x j ≥ β i , i = 1, . . ., m}; when an inconsistent set of assignment examples is provided by the DM, this polyhedron is empty.There exist various ways by which the set of assignment examples can be modified so that the polyhedron T becomes non empty.
The problem is then to identify all the "minimal" subsets (minimal in the sense of the inclusion) that resolve inconsistency, i.e., subsets among which the DM must choose in order to make his/her information consistent.In [10] two algorithms are proposed to identify all the minimal subsets S q , q = 1, . . ., Q to be deleted (sorted by cardinality) that resolve inconsistency and whose cardinality is lower than (or equal to) maxcount (maxcount is an input to the algorithms that states the maximum number of solutions to be computed).
In this paper we propose alternative ways to resolve inconsistencies stemming from a set of assignment examples; namely, instead of deleting assignment examples, we consider relaxing them, i.e., enlarging the interval of the possible assignments for an alternative.Moreover, we consider that the DM may provide confidence levels associated with the assignment examples; such information can be exploited to find a way to solve inconsistency that best them.
The paper is organized as follows.Section 1 defines inconsistency relaxation and shows that the algorithms proposed by [10] still apply when considering constraints relaxation rather than constraints deletion.The section 2 considers the case where the DM is able to provide confidence levels associated to the assignment examples.We provide two ways to account for such information in order to rank the solutions according to the confidence levels provided by the DM.Section 3 provides an illustrative example within the context of the ELECTRE TRI.

Inconsistency resolution via constraints relaxation
Resolving the inconsistencies can be performed by deleting a subset of constraints.Let us denote I = {1, 2, ..., m} the set of indices of the constraints and T ∅ = {x ∈ R n : n j=1 α ij x j ≥ β i , ∀i ∈ I} the initial empty polyhedron, i.e., with all the constraints.Let S ⊆ I denote a subset of indices of constraints.We will say that S resolves the inconsistency if and only if the polyhedron In [10] two algorithms are proposed to compute alternative ways to restore consistency by constraints deletion.We consider here the case where consistency can be solved by relaxing constraints rather than deleting them.Considering an infeasible system of linear inequalities (that can correspond to assignment examples), relaxing constraints rather than deleting them (in order to restore feasibility) has already been studied in the general case by (e.g., [13] and [7]).The relaxations considered by these authors are continuous and deal with the right-hand-side of the constraints only.In our case, we will define the relaxations differently: • the relaxations will be performed by changing the technical coefficients of the constraints rather than the right-hand-side; • a discrete set of relaxations will be considered which have a meaning in the sorting model, namely increasing the interval of categories to which an alternative can be assigned.
Suppose the DM has specified a set of assignment examples, i.e., a subset of alternatives A * ⊆ A such that each a i ∈ A * is associated with max(a i ) (min(a i ), respectively) the index of the maximum (minimum, respectively) category to which a i should be assigned according to their holistic preferences (a i → [min(a i ), max(a i )], a i ∈ A * ).From the DM's perspective, max(a i ) represents the statement "a i should be assigned at most to category C max(a i ) " and, min(a i ) express that "a i should be assigned at least to category C min(a i ) ".For each a i ∈ A * , min(a i ) and max(a i ) induce (when considering UTADIS and ELECTRE TRI) two linear constraints.Note that trivial constraints such as min(a i ) = C min and/or max(a i ) = C max do not need to be taken into account.

Let us consider the assignment example a
, with at least one strict inequality.Let us consider the system of inequalities containing the constraints corresponding to all the possible relaxations of the assignment example a i → [min(a i ), max(a i )] (it also contains the constraints corresponding to the original assignment example).It should be noticed that all the constraints corresponding to a relaxation of one of the two initial constraints are redundant.Consider S the set of all indices of constraints induced from a set of assignment examples corresponding to a relaxation of the initial assignment examples.Therefore we can note that S contains many redundancies.
If we apply the algorithms proposed by [10] (i.e., inconsistency resolution via constraints deletion) to the set S, the solutions correspond to constraints relaxation and/or deletion.It follows from the preceding remark that it is possible to use the algorithm proposed by [10] to solve inconsistencies by relaxation (rather than deletion) of assignment examples.Hence, in the rest of the paper, we will talk in terms of constraints deletion, knowing that it embraces the case of constraints relaxation.

Attributing confidence levels to assignment examples
In the course of the interactive process that aims at inferring the parameters of a sorting model, the DM provides assignment examples.For each assignment example, DMs might be more or less confident in their statements.Let us suppose that they are able to express confidence judgments during the interactive process.Such confidence judgments can be taken into account when an inconsistency arises.More precisely, algorithms that identify alternative ways for solving inconsistencies may use such information.Intuitively, these algorithms should provide solutions in an order such that the least confident constraints are relaxed/deleted with a higher priority than solutions relaxing more confident statements.
From the DM's perspective, a i → max(a i ) represents the statements "a i should be assigned at most to category C max(a i ) " and, min(a i ) express that "a i should be assigned at least to category C min(a i ) ".These two statements induce two constraints.The DM can attach a confidence level to each of the above mentioned statements.This information will be interpreted as confidence levels attached to the corresponding constraints (for example, a 1 → C 2 implies "a 1 should be assigned at least to C 2 " and "a 1 should be assigned at most to C 2 " and the DM may have different confidence levels concerning these two statements, e.g., they may say that if a 1 is not assigned to C 2 , then it is more likely to be assigned to a higher category than to a lower one).For each relaxed constraint, the attached confidence level corresponds to the confidence level of the original constraint from which it was derived (unless the DM provides specific information).

A lexicographic ranking procedure
Let us consider an inconsistent set of assignment examples provided by the DM and the set of linear constraints associated with these examples.Any relaxation of these assignment examples (see §1) will be also considered here.Following the notation introduced previously, m denotes the total number of constraints and I = {1, 2, . . ., i, . . ., m} denotes the set of indices of these constraints.The resulting polyhedron is, Let I p denote the subset of constraints whose confidence level is equal to ψ p .Hence, I 0 , I 1 , ..., I p , ..., I τ define a partition of I. Furthermore, we will denote I ≤p = p l=0 I l the set of constraints whose confidence level is lower than or equal to ψ p .Now, consider S l ⊆ I ≤l a subset of indices of constraints whose confidence level is lower than or equal to ψ l .We will say that S l resolves the inconsistency at a confidence level ψ l if and only if A simple way to account for the confidence level attached to each constraint is to proceed as follows: 1. Identify (by increasing order of cardinality) all minimal subsets S 0 1 , S 0 2 , ..., S 0 q 0 that resolve the inconsistency at level ψ 0 (i.e., relaxations whose confidence level is equal to ψ 0 that make the original system of inequalities feasible).
3. Proceed in the same way until finding minimal subsets S τ 1 , S τ 2 , ..., S τ qτ that resolve the inconsistency at level ψ τ or finding a total number of subsets equal to maxcount.
The program P M 0 1 identifies the smallest set of constraints S 0 1 whose confidence level is equal to where, M is a positive large number; the variables y i , i ∈ I ≤0 , are the binary variables assigned to each constraint index in S whose confidence index is lower than or equal to ψ 0 .The indices of constraints for which y * i = 1 (at the optimum of P M 0 1 ) constitute the subset S 0 1 .
P M 0 2 is defined in order to compute S 0 2 .This new program is derived from P M 0 1 by adding the single constraint i∈S 0 1 This constraint makes it impossible to find S 0 1 (the optimal solution of P M 0 1 ) or any solution that includes this set.A third program P M 0 3 is then defined by adding the constraint i∈S 0 2 y i ≤ |S 0 2 | − 1, and so on until we reach an infeasible program, meaning that there are no more solutions in I ≤0 .When all the solutions in I ≤0 are found, the algorithm starts to search for solutions in I ≤1 .The first solution S 1 1 is found by solving P M 1 1 which is derived from the previous program by replacing the constraints The algorithm continues until finding maxcount solutions (or no more solution exists).

Begin
p ← 0 count ← 0 While (p ≤ τ ) and (count ≤ maxcount) q ← 1 moresol ← true While moresol Solve P M p q If (P M p q has no solution) or (count>maxcount) Then moresol ← false Else S p q ← {i ∈ I :

End while End
This algorithm requires to solve several 0-1 programs.Note that it is possible to design an algorithm for the same purpose using only linear programming (see the second algorithm presented in [10]).

Defining a penalty function
Another approach to our problem consists of defining a penalty function π(S) associated to each subset of constraints indices S ⊆ I, and to rank the subsets that resolve the inconsistency by decreasing penalty order: the larger π(S), the greater the insatisfaction of the DM in removing S from I.This approach generalizes the lexicographic ranking as it is possible to define the penalty function π in a way that the penalty ranking coincides with the lexicographic ranking.
Given a subset S ⊆ I, S ∩ I p denotes the subset of S corresponding to constraints indices whose confidence level is equal to ψ p , p = 0, ..., τ .Let |S ∩ I p | denote the cardinality of S whose confidence level is equal to ψ p .
In order to define the semantic of the penalty function π, we impose a few suitable conditions on π: These conditions express natural properties for a function π to define a consistent penalty function: • Condition 2.1 (non-negativity) states that π has a lower bound (arbitrarily set to 0).Although it is not necessary, it would be possible to impose also an upper bound on the penalties, for instance π(I) = 1.In such a case, the penalty function π could be understood as a "disutility" function related to a utility function u(.) = 1 − π(.).This would be of interest in that the questioning techniques used to elicit multi-attribute utility functions [5] from the DM can be used in this context.
• Condition 2.2 (anonymity) states that the penalty of a set S only depends on the number of constraints of each confidence level contained in S regardless of the constraints "label".
• Condition 2.3 (confidence monotonicity) states that considering a solution S, if the confidence level of one constraint in S decreases, then π(S) should also decrease.
• Condition 2.4 (cardinality monotonicity) states that if a solution S contains less (or equal) constraints than another solution S ′ , for each confidence level, the penalty should be lower (or equal) for S than for S ′ .
Among the possible penalty functions, one of the simplest can be defined considering that each constraint in S of a given confidence level ψ p contributes to increase π(S) by an amount ∆ p (the values ∆ p are to be defined by the DM): Considering condition 2.3, the amounts for ∆ 0 , ..., ∆ τ should be such that u < v ⇒ ∆ u < ∆ v .This simple model can be generalized by considering that the penalty does not increase linearly with respect to the number of constraints: where π p (n) is a function (to be defined by the DM) denoting the penalty of removing n constraints of confidence level ψ p (given the assumed conditions, π p must be an increasing function).
More sophisticated non-additive models may be envisaged, namely those taking into account preference dependencies among different confidence levels.

Ranking solutions according to the penalty function
Given a penalty function π, it is necessary to define an algorithm to rank by increasing penalty the subsets of I that, if removed, yield a consistent system.In order to design an algorithm to identify the maxcount subsets of constraints that solve inconsistency and rank them according to the penalty function π, we will adapt the algorithms presented by [10] which rank by increasing cardinality all minimal subsets that resolve inconsistency : S 1 , S 2 , . . ., S q , . . .
The algorithms presented in [10] provide the set of the solutions ordered by cardinality without any consideration about confidence levels.In our case, we want to provide the set of (at most) maxcount solutions ordered by increasing penalty.It should be noticed that the smallest cardinality solutions might not correspond to those of the smallest penalty.Therefore, we can proceed by computing the solutions by increasing cardinality and stop when we are sure that the solutions of a higher cardinality have a greater penalty than the ones we already obtained.
Let S x,p denote an arbitrary set of x constraints, all of confidence levels equal to ψ p .
, the penalty of any solution after the q-th is not lower than the penalty that would be awarded to the q-th solution if all the constraints indexed by S q were of the lowest confidence.
In the following algorithm, TOP-N denotes a list of solutions of at most maxcount elements, ordered by increasing penalty, and S tail denotes the last solution (i.e., the solution with the highest penalty) of list TOP-N.At the end of the algorithm, TOP-N contains the maxcount first solutions ordered by increasing penalty.

Begin
q ← 0 TOP-N ← empty list S tail ← first solution repeat S q ← q th solution provided by the algorithm in [10] If (q ≤ maxcount) or (π(S q ) < π(S tail )) Then S q enters TOP-N End If Until S q = ∅ or (q ≥ maxcount and π(S tail ) ≤ π(S |Sq|,0 )) End In this algorithm, when S q enters TOP-N it does so respecting the penalty ranking.If the list is full (when it contains maxcount elements), this implies removing the highest penalty element (S tail ), and the variable S tail will be updated.Proposition 2.1 allows us to define the stopping condition π(S tail ) ≤ π(S |Sq|,0 )).

Illustrative example
Let us consider a situation in which a set of 40 alternatives has to be assigned to 5 categories using the ELECTRE TRI pessimistic method.Each alternative is evaluated on the basis of a set of 7 criteria (see Appendix A).The limits of categories are known but the criteria importance coefficients are to be defined (see Appendix B).Suppose the DM provides assignment examples with associated level of confidence on a scale (absolutely confident ≻ quite confident ≻ not so confident), where means that alternative a i must be assigned to a category between C k and C k ′ , (k ≤ k ′ ): From these assignment examples, it is possible to define relaxations as defined in Section 1.In this example, we will suppose that the relaxations have the same confidence level than their corresponding assignment examples.For instance, the relaxations corresponding to the assignment example amounts at removing the assignment example a 1 → C 5 ).
These assignment examples and their relaxations generate a set of 41 constraints on the criteria weights w j , j = 1, ..., 7 and cutting level λ which are presented in Appendix C. The first 17 constraints correspond to the original assignment examples and the remaining ones correspond to the relaxations.The linear system associated with the assignment examples is infeasible, which means that the information provided by the DM is inconsistent, i.e. there is no way to represent this information in the ELECTRE TRI sorting model.
Considering this infeasible linear system, there exist 11 minimal subsets of constraints that resolve the inconsistency, where I = {1, 2, ..., 41}.These 11 subsets (ordered by cardinality) are listed below.Let us remark that due to the limited size of this example, we have computed all the solutions.Such a way of proceeding is time consuming when dealing with real world problems of large size .
When interacting with the DM to solve an inconsistency, it is not reasonable to propose him/her a large number of alternative solutions.It is convenient to propose a limited number of solutions that might be interesting for him/her.In our case, we wish to propose approximately 5 solutions to the DM (maxcount=5).

Conclusion
In this paper, we considered the problem of supporting the DM in the resolution of inconsistent judgments expressed in the form of assignment examples in multiple criteria sorting model.We have proposed the concept of relaxation of an assignment example, which is helpful in this context.To resolve the inconsistency, it is useful to obtain from the DM confidence statements associated with the assignment examples.We have proposed procedures that account for this information to assist the DM in finding the most relevant ways to restore consistency.
An illustrative example has been provided to show how the proposed procedures can be used within the context of the ELECTRE TRI sorting method.However, our procedures are general and apply to any sorting method for which assignment examples generate linear constraints on the preference-related parameters.
An interesting extension of this work consists in considering the possibility of associating different confidence levels to the original assignment examples constraints and their corresponding relaxation.This extension amounts at considering that the various relaxations of an assignment example are not judged as equivalent as regards their confidence levels.

Appendices
Appendix A: Evaluation matrix S original assignment relaxed assignment S original assignment relaxed assignment