Fundamental values and their sub criteria.
Participatory management establishes a dynamic of democratization of the public administration, since it associates planning and widespread participation, through political definitions and adjustments and changes. Its purpose is to discuss and define the population, in a democratic way, the distribution of investment resources. Major challenges put pressure on governments at the federal, state, and municipal levels, requiring them to be more creative and effective ways of achieving results, from the employment of increasingly scarce resources and an increasingly demanding and participatory population. Considering that the process of managing public resources is defined in the budget, it is necessary that it contains all the elements that facilitate and allow a correct application of these, among the universe of interests presented by the population through the participatory management process. Moreover, we propose a model for the management of optimization of the Public Budget, whose objective is to provide the administrator with the instruments necessary to optimize the application of available public resources. The methodology in the analysis of the feasibility, and the application of a multicriteria structured model and mathematical programming applied to the public budget, having as a case study a macro view of the budget for Municipal City Halls.
- public budgeting
- mathematical programming
- linear programming
- municipal city
Participatory management establishes a dynamic of democratization of the public administration, since it associates planning and popular participation, through political definitions and adjustments and adjustments in the process, when necessary. Its purpose is to discuss and define the population, in a democratic way, the distribution of investment resources. Exactly because it is the result of the discussion and the decisions of the society, the works or investments prioritized, in general, involve the most complex problems of the communities. They are works that require more detailed projects and careful executions. Significant challenges put pressure on governments at the federal, state, and municipal levels, requiring them to be more creative and effective ways of achieving results, from the employment of increasingly scarce resources and an increasingly demanding and participatory population.
Considering that the process of managing public resources is defined in the budget, it is necessary that it contains all the elements that facilitate and allow the correct application of these, among the universe of interests presented by the population through the participatory management process.
Within this context, we present a methodology for modeling the problem that seeks in science based on multicriteria and linear programming the optimal solution for its proposition for the Public Budget, whose objective is to provide the public administrator with the necessary tools to optimize the application of the available public resources, considering the environment in which it is inserted. To determine the optimal way of applying the funds authorized by the Legislative Power, in order to maintain the administrative machinery, enforce the laws and still meet the desires of the population with increasingly scarce financial resources. Our proposal is to apply these technologies to public finances, whose objective is to model the public budget in order to maximize the available resources, taking into account the government programs and the current legislation.
In the Section 2, presents Budget Guidelines Law. Moreover, Section 3, is given Hybrid Model for Optimization. More details of the Multicriteria Modeling provided is Section 4. On the other hand, in Section 5 provides more information on the applied methodologies. Furthermore, in Section 6 the computational results obtained using. Finally, Section 6 presents conclusions and future works.
2. The budget guidelines law
In the Budget Guidelines Law (BGL), expenditure is classified through several criteria, the most important of which are functions, programs, nature of expenditure and elements, which leads to a more complex and more difficult codification, different interpretations for some of its components. The Law requires that some principles be observed in the execution of the budget, especially flexibility, that is, adaptation to new or unforeseeable situations. This flexibility necessary for public managers is extremely limited due to the restrictions imposed by the current legislation. In this sense, public administrators have great difficulty in defining the application of financial resources, by the legal limitation and by the population’s demand for the services that they considered to be priorities.
Moreover, the system has laws that restrict, and on the other, increasingly demanding citizens who always demand more quality public services, lies the difficulty of determining the solution to find the best way to distribute resources. This article focuses on government programs that define how public administration seeks solutions to social problems. Each government program is linked to activities, to which are associated resources among which the financial necessary to carry out the actions that underlie the programs in question. For any expenditure that is desired, it is necessary to pre-exist the corresponding budget allocation. We seek to define budget allocations for each government program to optimize the available resource and meet the population’s wishes. Budget allocations are distributed among the various Government Programs. Each Government program has an importance for the Public Administration in office, and must be defined to distribute the budget allocations for the activities related to it, and then the expenses necessary for the implementation of the Program can be made.
3. Hybrid model for optimization
The proposed model is composed of two steps: initially to construct the objective function from the multicriteria analysis [1, 2, 3]. Next, the construction and solution of a linear programming model based on the objective function constructed in the first phase and the legal, minimum and maximum constraints defined for each variable to be calculated, called budget allocations.
3.1. Model in linear programming
The aim is to maximize the actions to be carried out through budget allocations, considering the degree of importance of these actions (Government Programs). Budget allocations are defined as variables xi.
The best distribution of the budget will be the one that better attends to the actions of the Government Programs, considering the importance defined by the widespread participation coordinated by the administration in question. The optimization criterion is then the sum of the budget allocations of each activity weighted by the degree of importance of the Government Program to which it is linked. We can stand for the objective function in Eq. (1),
xi = Budget allocation of the activity related to the Government Program i;
ki = Degree of importance of the Government Program i;
n = Number of government program activities in the budget.
The evaluation of the pertinent legislation Budget Guidelines Law (BGL), and Fiscal Responsibility Law (FRL) allowed to find some legal restrictions, which we present below. The definitions given are merely illustrative for the assembly of the model; In their application, should follow what is recommended in the specific literature.
I. Expenditures may only be made up to the limit of the budget credits and additional credits, and following the disbursement schedule of the Unit, duly approved in Eq. (2),
n = Number of government program activities in the budget;
RCTot = Total revenue previously defined by the finance team, according to the pertinent legislation;
II. According to Eq. (3) for the FRL, personnel expenses are the sum of the expenses of the Federation entity with the assets; The expenses for inactive and pensioners; Fixed and variable salaries and benefits; The subsidies, retirement benefits; Reforms and pensions; The added ones of any nature; The gratuities, overtime and personal advantages; Social charges; The contributions collected by to private pension entities. The FRL limits spending on public sector personnel to 60% of net revenue for states and municipalities. In the municipal sphere, the 60% limit will be distributed among the Legislative Branch: 6% (including the Municipal Court of Audit) and Executive Branch: 54%.
p1…pn = activities of Government Programs in the budget related to personnel;
RCL = Net revenue.
III and IV. According to Eq. (4) and Eq. (5) for the Federal Constitution, the municipality must distribute to education, not less than 25% of its tax collection and transfers. Of these 25%, 60% should be earmarked for primary school funding and the remaining 40% for other levels of education. In addition, 60% of the resources should be earmarked exclusively for the payment of teachers’ salaries, with a view to enhancing the teaching profession and improving the quality of teaching.
e1, …, en = activities of Government Programs in the education budget;
fund1,…fundn = activities of Government Programs included in the budget for primary education;
RCE = income for education, defined in the Budget Guidelines Law (BGL);
RCFund = revenue for calculation of fundamental education, defined in the BGL.
V. Expenditure on health: the linking percentage is 15% of its tax revenue and constitutional transfers in Eq. (6). This rate must be reached within 5 years from the entry into force of the Amendment. In their first year, percentages below 15% for health financing should be increased gradually by the 5th year after their approval, reducing that gap by at least 1/5 per year. For modeling purposes, we are considering only the percentage of binding of 15%, understanding that the Administration in question is with this percentage stabilized.
s1…sn = government Program activities in the health budget;
RCS = For calculation of Health, defined in the BGL.
VI. The budget distributed to the City Council should be at most equal to 5% of tax revenue in Eq. (7).
c1 … cn = number of activities of Government Programs gave in the budget relative to City Hall;
RCT = Tax revenue, as defined in the BGL.
VII. In Eq. (8) to current expenditure cannot be financed by capital expenditure; That is, the value of current expenses must be less than or equal to the value of current revenues. Current spending is the cost of supporting public services; Without any return to equity of equal value. Capital expenditures are expenses where there is the reward for the direct entrance of a kind, of corresponding value, in the patrimony; that is, investments, financial investments, and capital transfers.
dc1…dcn = current expenses supported in the BGL;
RC = Current revenues (revenues directly from the Public Administration that are directly or indirectly consumed in expenses with the maintenance of public services previously created - do not include capital funds).
VIII. Finally, in Eq. (9) all Government Programs to which the budget allocations (model variables) are linked have a certain political importance for Administration (index k of the model); and, so, they must be positive, different from zero, greater of a value u and smaller of a value d, both decided by the Administration; or in the range:
4. Multicriteria modeling
The degrees of importance of government programs should be defined based on the multicriteria methodology, defined as k in the model, will define the maximum value of the objective function and will be a decisive factor in the definition of controlled variables of the model. The information required for the model to be provided :
The revenues, included in our model, that restrict the value of the controlled variables;
The relation of the activities of the Government Programs that will define the size of the vector of controlled variables. Government Programs will be treated as constants in the model;
The classification of the activities of the Government Programs, through Participatory Management, duly reviewed by the Principal Manager. This classification will be the input to calculate the degree of importance of each Government Program.
In addition, each government program is divided into activities and each of these activities should be classified into two levels: initially classifies the activity by fundamental values such as Health, Education, Environment, etc. Moreover, specifies each fundamental value, called here sub-criteria, creating a two-level classification. The value tree will be built with as many fundamental values as believed necessary by management, including the participation of society (fundamental value A, B, C, D, etc. …). Each fundamental value will have as many sub criteria as are defined by management and society (sub criteria I, II, III, etc. …). The proposed modeling considers that the software Measuring Attractiveness by a Categorical Based Evaluation Technique (M-MACBETH): method elaborated by [5, 6]) is applied to define the degrees of importance, from the tree of Values built. The attractiveness indexes of these fundamental values among themselves and the sub-criteria should be giving by consensus within a participatory management program of the actors.
Furthermore, a matrix of fundamental values (A, B, C, …) is created and the attractiveness of them is compared to each other, generating the index f of each of these values by the M-MACBETH method. The index f is the result of comparability of the first level of the value tree. Each fundamental value will have sub-criteria linked to it, which should also be compared to each other. A new value matrix is constructed for each fundamental value, comparing the attractiveness of its sub criteria, generating the index s of each sub-criterion by the M-MACBETH method (second level index of comparability).
The degree of importance k, related to the government activity to which the budget allocation x is bound, will be the product of the index set up for its fundamental value (first level) by the index set up for its sub-criterion (second level): k = f × s.
4.1. Range of variation of fi
The actors decide which fundamental value has the greatest and the least importance, they then receive grades 100 and 10, respectively. These levels of attractiveness, because they are defined as major and minor, by the actors must receive the maximum and minimum values, and therefore cannot assume intermediate values; Consequently, there is no variation interval.
4.2. Range of variation of s
For each fundamental value the actors’ levels of attractiveness were defined by their sub criteria. As in the valuation of f, in each case there is an item of greater and another of lesser importance. These items receive degrees 100 and 10, respectively. These attractiveness levels, since they are defined as major and minor by the actors, must receive the maximum and minimum values, and therefore cannot assume intermediate values; Consequently, there is no range of variation.
Furthermore, each parcel f and s that composes index k can assume a range of values, without, however, changing the level of attractiveness that is calculated by M-MACBETH. We call this range of “Interval” values. The M-MACBETH presents the ranges of variation that the f and s indexes generated by it can assume, guaranteeing the valuation of the levels of attractiveness defined by the actors.
A range of variation for the index k, defined from the solution of the problem in linear programming, considered objective function and constraints. Let’s call this interval “Interval θ “.
Moreover, each step of our proposal generates a set of values in which the degrees of importance of government activities (k) may be inserted: the intervals β and θ. The intersection of these sets stands for the interval at which, in fact, our objective function indexes k may vary, the “Interval γ“.
4.3. Variable time
The construction phase of the model supports the analysis of the time horizons of the proposed modeling. The budget part is a public instrument defined for the fiscal year (fiscal year - 12 months). Your legal restrictions must be respected at the end of each exercise. This hybrid model proposes a method for the construction of the budget piece for a given fiscal year, which guarantees compliance with the legislation, and other restrictions imposed. It can also be used in budget supplementation, during the exercise, to verify compliance with legislation; Since, once the restrictions have been respected throughout the year, we will be guaranteeing compliance with these restrictions at the end of the exercise. But in this case, we must evaluate the results in time every round of the model.
4.4. Duality analysis
The primal of the proposed model has as goal function the sum of budget allocations that are weighted by the degree of importance of the Government Program to each of them associated, degrees these, defined from multicriteria analysis by popular participation. Its restrictions are the legal ones regarding the revenue and the minimum and maximum limits defined by the public administration for each activity. These values of revenue and maximum and minimum limits are constant values in our proposal, defined at the time of the elaboration of the primal model. They change the distribution of the budget: if, for example, Total Revenue is changed, the values of the budget allocations will change. If the minimum or maximum amount to be applied in the school lunch activity is changed, this will have a greater or lesser impact than a variation in Total Revenue.
z = number of restrictions on budget allocations;
w = number of legal restrictions;
rj = values defined as restrictive revenue from the applications of budget allocations;
pj = values determined as lower/upper limits of the budget allocations for the activities of the Government Programs;
yj = degree of effect about values defined by the public administration as lower/upper limits and revenues in the budget distribution.
On the other hand, restrictive revenues (rj) are those present in the constraints of the primal as legal impositions. Some constraints of the primal are formed by equations that impose to the budget allocations xi minimum limit, others impose maximum limit. These minimum or maximum values determined by administration at the time of constructing the primal are pj.
By “degree of impact” we mean the relative importance that these lower (upper) limits on the main problem have in defining budget allocations. That is, yj measures the relative impact of that arbitrariness of the lower value (or higher) and of the revenue in the budget distribution. We can present in Eq. (10) the duality of the proposed model in the form:
The values “pj” and “rj” are “arbitrated” in the primal, that is, they are constant values defined at the time of the elaboration of the primal model. Once altered they affect, in some way, the solution of the primal, that is, the distribution of the budget.
The purpose of the dual is to MINIMIZE the impacts of the “arbitrated” upper (upper) limits and legal restrictions on revenues in budget distribution.
The greater the degree of effect y, the greater the impact of the arbitrated limits, or the revenue, on the distribution. That is, a change to a greater or lesser extent in the coefficient going with the y in the objective function of the dual will have a greater or lesser impact, respectively, in the distribution of budget allocations.
Each constraint of the dual will be formed by an inequality where a k will be present in its second term, which will stand for them by in Eq. (11):
m = number of constraints of the budget distribution;
v = number of variables budget allocations;
q = is the coefficient of x in the nth budget constraint.
5. Case study: Fortaleza city hall
We used as a case study the budget data for the 2013 fiscal year of the City Hall of Fortaleza, approved by the Municipal Council and published in the official gazette.
The City Hall of Fortaleza 2013 Budget is composed of several tables. Among them, we show the Government Programs and the General Summary of Revenue as being of most relevance for the development of this work. We look for the government programs defined as goals to be achieved in 2013 by the City Hall of Fortaleza, and their respective activities; And, from there, we apply the definition proposed in the model. The targets will be achieved through the application of the budget allocations defined for each of the activities planned for the government programs.
5.1. Application of the model
Initially, we start from the Government Programs table of the 2013 City Hall of Fortaleza budget and classify activities as fundamental values. We call the areas of general interest in society: Education, Infrastructure, Environment, Administration, Health, Economic Development and Social Assistance. For each fundamental value, we define classification sub criteria as presented in Table 1.
|Fundamental value||Classification subcriteria|
|Social assistance||Habilitation||Social assistance programs||Laser||Citizenship||Policies social assistance|
|Economic development||Training||Economic development programs||Policies||Tourism||Culture|
|Infrastructure||Road||Collective transportation||Lighting||Traffic||Policies infra|
|Health||Hospital||Basic attention||Medication||Surveillance||Health policies|
The areas and their sub criteria were defined based on interviews with several specialists in the areas of planning, education, health, administration, infrastructure, etc. … The proposed method consists in classifying each budget activity as the fundamental value, that is, if that activity refers to Education, Infrastructure, Environment, Administration, Health, Economic Development or Social Assistance; And the sub-criterion presented in the table above. We apply M-MACBETH to compare the attractiveness of fundamental values and their sub criteria and then define the “degree of importance” of each activity group. We understand by activity group the set of budgetary activities that have as fundamental value and sub-criterion the same classification. The definition of the attractiveness indices predicted in the M-MACBETH method was carried out based on research with some actors from different social levels, selected at random. We defined as “consensus” between the actors the arithmetic mean of their answers in each comparison, calculated from the one-to-one correspondence of the attractiveness, on a scale of 2 to 6 proposed in the model (from 2—Much less important to 6—much more important).
We understand that the choice of actors and the definition of consensus among them should be the object of specific research within a participatory management project. Our aim in this case study was to stand for an interest of a randomly defined community, without scientific research on the representativeness of the community since this research transcends the object of our research. On the other hand, to apply the M-MACBETH software, inserting the result “of this consensus” in steps:
5.2. Definition of the attractively index among the fundamental values
We present in Table 2 - Summary of the M-MACBETH Indices, column “f - Index” the result of the comparison of the attractiveness of the fundamental values (Health, Education, Social Assistance, Environment, Economic Development, Infrastructure and Administration) -MACBETH. On the other hand, Health was identified as the most important fundamental value and Management was of the least importance. In the table of values of M-MACBETH we defined with 10 points the lowest value of attractiveness for calculating the individual indexes.
|—||Social assistance programs||0.4093||68.69||53.06||0.2586||0.441||0.2919|
|Assist. social policies||0.0632||10||10||0.0399||0.0642||0.055|
|Economic development programs||0.191||45.99||37.01||0.0879||0.2529||0.1695|
|Economic development policies||0.046||10||10||0.0212||0.055||0.0458|
5.3. Definition of the attractiveness index among the subcriteria of the fundamental values
We present in Table 2 - Summary of the M-MACBETH Indices, column “S - Index” the result of the comparison of the attractiveness of the subcriteria of each fundamental value, using the method M-MACBETH. According to the proposed model, the degree of importance k for the government activity whose budget allocation x is bound will be the product of the index established for its fundamental value by the index established for its sub-criterion: k = f × s. Table 2 presents in the column “k - Index” the product of the columns “f - Index” and “s - Index” for each established subcriteria.
The objective function applied to the case study can be seen in the https://www.dropbox.com/s/ga4flhp61291ro4/Restrictions%20and%20degrees%20of%20attractiveness.pdf?dl=0 through the horizontal reading of the table, by the sum of the product of the fifth column (relation of the k’s) by the corresponding x.
Each budget activity may be present in more than one constraint. This model applied to the City Hall of Fortaleza in 2013 can be seen in tabular form in the https://www.dropbox.com/s/ga4flhp61291ro4/Restrictions%20and%20degrees%20of%20attractiveness.pdf?dl=0. The columns 1 and 2 present the budget allocations of government activity x; Columns 3 to 9 show whether the activity is present in restrictions 1 to 7, respectively; And columns 10 and 11 refer to the specific constraints of each activity (minimum and maximum values, if any).
The table reading can be done horizontally, where you can see each model variable, that is, the budget allocation of each activity and the vertical form of columns 3 to 9, the sum of the rows in each column is the composition of A restriction. Columns 10 and 11 shows whether there is a specific constraint, that is, whether there is a restriction on the maximum or minimum value, respectively, for the budget allocation of that activity. The revenues that make up the second term of the restriction can be found in the “General Summary of Revenue” table, in the fiscal year 2013 of the City Hall of Fortaleza, approved by the City Council and published in the official Gazette.
The first lines of the model, that is, the objective function and the first constraints (those imposed by the specific legislation) are formed by many variables. The other constraints of the primal problem are specific to each government activity, and therefore are made up of only one variable.
We applied LINDO/PC, release 6.1 for analysis and solution of the model in linear programming applied to the City Hall of Fortaleza. The software generated the result of the application of the model for budget optimization/City Hall of Fortaleza - 2013, presenting an array of 578 rows and 285 columns.
6. Computational results analysis
The graphs presented in the following figures were constructed based on the computational results generated by the implementation of the proposed model applied to the City Hall of Fortaleza, exercise 2013. In the abscissa, we find the budget allocations x (where i ranges from 1 to 285, number of variables of our Primal problem). In the ordinate, we have the values the values of k. These figures graphically show the sensitivity of the computational results. Figures 1 and 2 shows, respectively, the ranges. The curves in pink present the maximum value and in yellow the minimum value that k can assume.
Following the model, we must construct the intersection curve of these two graphs. Figure 3 shows the interval, intersection of the curves shown in the earlier graphs. The curve in pink shows the maximum value and the yellow line the minimum value that k can assume.
We can see that the variation spectrum of k is small, since the curves almost coincide. There is a small gap in which each k can vary without changing the budget distribution.
If we have two methodologies in the proposed model that generate two intervals (e) for variation of k, we must analyze which of the two methodologies contributes to the definition of the maximum limit and the minimum limit of the solution. The pink color curves presented in the graphs (Figures 2 and 3) that present the maximum values for k in the intervals (intersection) and (M-MACBETH), respectively, were superposed and are shown in Figure 4:
The dotted line in green stands for the maximum values of the M-MACBETH curve and the continuous in pink represents the maximum values of the intersection curve. It is the comparison of the two, in which the curve of the intersection of the maximum value of k coincides with that of M-MACBETH. We can see, then, that the definition of k by the actors (multicriteria method) imposes an upper limit of the values of the budget distribution. Let us therefore evaluate the main contribution of the lower limit. Figures 5 and 6, below, show the comparison of the minimum boundary curves of the intervals.
Except for a few points, the curve of the intersection of the minimum value of k coincides with that of LINDO. We conclude that legal and specific constraints (linear programming) impose the lower limit of the values of the budget distribution.
In the case of the City Hall of Fortaleza, participatory management (definition of k) by the actors (community) limits the budget distribution by assigning the maximum values of the priority grades of the budget activities. It is the legal and specific restrictions of each activity that impose the minimum amounts of amounts of the budget activities. Both seek a balance of budget distribution, reconciling the will of the population and laws.
7. Conclusion and future works
The present proposal defines a process of modeling in linear programming for the Municipal Public Budget, whose aim is to offer the administrator with the tools necessary to optimize the application of the available public resources, considering the environment in which it is inserted. This environment brings subjectivity in the definition of priorities through Participatory Management (made by so-called actors in our model) and imposes constraints, sometimes by the administration’s own criteria, sometimes by force of laws.
The proposed model confronts the subjectivity of the problem using the method of analysis of multicriteria, in which the actors define the scale of priorities, by comparing the attractiveness of the Government activities. It treats the constraints imposed through a linear programming modeling, in which the limits of applicability of the budget are decided and the objective function meets the scale of priorities defined by the actors.
The case study showed the applicability of the model, since it defines the value of each endowment, thus distributing the budget resource [7, 8, 9]. The model presented was constructed from impositions to the municipal public budget, since we used, as a case study, the data of the City Hall of Fortaleza. But we understand that the proposed method can be applied to any of the spheres of Government, since the federal and state budgets must be correctly distributed by the different and respective Government Programs. The modeling should follow the same principle of use of the two scientific methodologies, through the definition of scale of priorities through the multicriteria analysis, and the definition of limits and legal restrictions based on the norms of the federal or state, respectively, or both, if applicable. We emphasize that the critical point of our proposal is precisely in the construction of the scale of priorities by actors who defend the value system they stand for, since they must, necessarily, reach a consensus so that the considered value is represented in the model.
The authors are thankful to National Counsel of Technological and Scientific Development (CNPq), Foundation for Support of Scientific and Technological Development Ceara State (FUNCAP) and Coordination for the Improvement of Higher Level or Education- Personnel (CAPES) for the support received on this project.
Figueira J, Greco S, Ehrgott M. Multiple Criteria Decision Analysis: State of the Art Surveys. Boston, Dordrecht, London: Springer Verlag; 2005
Pinheiro PR, Souza Gilberto GCde. A multicriteria model for production of a newspaper. In: The 17th International Conference on Multiple Criteria Decision Analysis. British Columbia: Simon Fraser University; 2004. Vol. 17. pp. 315-325
Pinho Teresa Cristina Neves. Um Modelo para Otimização do Orçamento Público Municipal: Um Estudo de Caso na Prefeitura Municipal de Fortaleza, Dissertação, Mestrado em Informática Aplicada (MIA), Universidade de Fortaleza (UNIFOR); 2004
Pinheiro PR, Pinho Teresa Cristina Neves de. Modelo para Otimização do Orçamento Público. In: XXXV Simpósio Brasileiro de Pesquisa Operacional; 2003; Natal. Anais do XXXV Simpósio Brasileiro de Pesquisa Operacional; Rio de Janeiro
Bana e Costa CA, Sanchez-Lopez R, Vansnick JC, De Corte JM. Introducción a MACBETH. In: Leyva López JC, editors. Análisis Multicriterio para la Toma de Decisiones: Métodos y Aplicaciones. Plaza y Valdés, México; 2011. pp. 233-241
Bana e Costa CA, De Corte JM, Vansnick JC. MACBETH. International Journal of Information Technology and Decision Making. 2012; 11(2):359-387
Pinheiro PR, Souza Gilberto GCde. A multicriteria model for production of a newspaper. In: The 17th International Conference on Multiple Criteria Decision Analysis. British Columbia: Simon Fraser University; 2004. Vol. 17. pp. 315-325
Pinheiro PR, Souza GGC, De e Castro AKA. Estruturação do problema multicritério para produção de jornal. Pesquisa Operacional. 2008; 28(2):203-216
Silva CFG, Nery A, Pinheiro PR. A multi-criteria model in information technology infrastructure problems. Procedia Computer Science. 2016; 91:642-651