The Improved Model of the Method, Rights, and Resources (MRR) for the Evaluation of the EIA System: Revising the Sustainability Indicators

Abstract

Measuring effectiveness of environmental impact assessment systems is central to the implementation of environmental impact assessment, considering the debate about relevance and usefulness of environmental impact assessment systems. Many models for evaluating environmental impact assessment system’s effectiveness have been developed. Difficulties in quantifying environmental impacts have restricted the effectiveness mostly to procedural effectiveness evaluation, though substantive effectiveness evaluation is better. The method, rights, and resources (MRR) model was initially developed to harness the indicator-based evaluation theory into the evaluation of environmental impact assessment system’s effectiveness. This chapter reviews the method, rights, and resources model and proffers some improvement. The method, rights, and resources model evaluates environmental impact assessment systems using indicators of compliance, participation, and capacity. The indicators incorporate both procedural and substantive approaches; hence, it attempts to present a more indicative measure of environmental impact assessment system’s effectiveness. The guiding idea in this chapter is that monitoring and evaluating environmental impact assessment systems should be embedded in the environmental impact assessment system itself as opposed to being concepts that are externally and subsequently applied on existing environmental impact assessment systems.

1. Introduction

The concept of environmental impact assessment (EIA) was developed in the USA in 1960. From there, the concept quickly spread across the world. European countries adopted EIA very early after that. Most developing countries adopted EIA after 1992. For example, although no African country had mandatory EIA procedures prior to 1992, over 40% had established EIA systems by 1997 [1]. After 1992, EIA was rapidly adopted as a national decision-making tool by many countries. This rapid adoption is in most countries was undoubtedly influenced by the Rio Summit.

The Rio Declaration on Environment and Development and the Local Agenda 21 placed EIA firmly as an important concept in environmental management processes.

The EIA system’s primary role is to incorporate environmental issues into decision-making to ensure that new developments include steps to protect the environment and social well-being. EIA is therefore a series of steps that enable environmentally responsible decisions to be made. A logical question that follows is whether EIA systems actually contribute to environmental protection and, if so, how effectively.

Although the finer details of how EIA is implemented vary from country to country, the basic idea is the same. In this chapter, it is assumed that the reader has reasonable knowledge about the EIA process; therefore, only a brief description is given. It starts with a detailed knowledge of the proposed project. This detailed knowledge informs the possible impacts that may arise from the implementation of the project.

Studies are done to determine the environmental baseline on which possible environmental impacts of the proposed project may be benchmarked. The baseline also enables experts to understand details about the potential impacts such as magnitude, type, severity, and so on. The next step is to plan for ways of reducing any negative environmental impacts that would have been identified and analyzed while enhancing any positive impacts, if any. An environmental management plan is put together and married to project implementation. One of the important steps of EIA is the consultation of affected and interested parties (stakeholder consultation) to gain and incorporate their views.

The effectiveness of the EIA process is a growing subject of scholarly research [2, 3]. Governments and private companies commit many resources to implement the EIA process. The major question that remains partly unanswered is whether EIA is achieving environmental protection as expected. From this question arises another question which is the focus of this chapter. How can the effectiveness of an EIA system be measured?

The MRR model is one such a means introduced to evaluate EIA system effectiveness [4]. The purpose of this chapter is to introduce an updated method, rights, and resources (MRR) model for evaluating EIA system effectiveness [4], an alternative conceptual and practical model based on the evaluation theory. The next section describes the research method of this chapter followed by presenting different approaches to evaluating EIA system effectiveness and then discussing the challenges associated with evaluating EIA system effectiveness. After that the theoretical framework of the MRR model is presented followed by the MRR model and its application and finally the conclusion and recommendations.

2. Research methodology

This chapter sets out to review and improve the MRR model for EIA system evaluation. To do so, the literature review of the current EIA evaluation models was conducted together with the UN conventions on which the MRR model is based. After adding more theoretical context, the description of the MRR model was reviewed to add rigor and flow diagrams.

Further improvement was made by discussing the pros and cons of the MRR model using a SWOT analysis. Further recommendations toward the application of the MRR model were added.

3. EIA system evaluation approaches

Firstly, it is important to set apart EIA review models whose primary purpose is to assess the compliance and content of EIA reports/statements. Examples of these are the Lee and Colley review package [5], the European Commission Guidelines on EIS Review, the Oxford-Brookes University EIS review package, and the Guide to Technical Analysis of Environmental Impact Studies.

This chapter is concerned about evaluating the effectiveness of the entire EIA system. Effectiveness of EIA systems has been researched since EIA systems were introduced. EIA effectiveness evaluation approaches were originally divided into two categories, vis-à-vis the procedural and the substantive effectiveness [6].

Adherence to the stipulated method of conducting EIA is the focus of procedural effectiveness. For example, in procedural effectiveness, the focus is on whether EIA studies were conducted thoroughly, whether the public had adequate opportunity to air their views, and whether the views were taken into consideration. Hence procedural effectiveness emphasizes on assessing how well information was gathered and used for decision-making and much less on whether tangible environmental stewardship itself was actually achieved. The actual environmental protection objectives are assumed to be achieved once a certain method is followed in conducting EIA studies and making decisions about it.

A typical procedural effectiveness viewpoint is the “democratization of governmental decision-making processes” as suggested by Macintosh [7]. In this case, simply ensuring that affected stakeholders have contributed to the decision-making process would be considered an achievement of its objectives.

Procedural EIA system evaluation is the easiest to perform since the focus is on whether specific procedural steps have been complied with. There are many procedural EIA evaluation models that have been developed. Two examples are given below.

The first one is the systemic and foundation measures model which was developed by Ahmad and Wood [8]. Features of EIA (system) “that are designed to deliver quality assurance in both practice and the administration are called systemic measures” [9]. Foundation measures are those “actions undertaken to improve the effectiveness of the EIA system and ensure successful application of the systemic measures” [9].

The other model is the EIA evaluation criteria developed by Wood which consists of 18 questions grouped into 3 categories, namely, institutional aspects of the EIA system, EIA process, and other requirements of the EIA system [10]. The EIA evaluation criteria have been widely used [11, 12].

There are a number of other criteria that have been developed and applied such as the 21 criteria [13], 5 criteria [14], 62 criteria [15], and 80 criteria [16]. All these are primarily procedural effectiveness models ( Table 1 ).

Approach/model	Effectiveness evaluation	Focus
Lee and Colley review package European Commission Guidelines on EIS Review Oxford-Brookes University EIS review package and the Guide to Technical Analysis of Environmental Impact Studies	Procedural	Quality of EIA reports
Systemic and foundation measures model	Procedural	Presence of the basic conditions and requirements for EIA system implementation
EIA evaluation criteria (18 questions) 5 criteria 21 criteria 62 criteria 80 criteria	Procedural	Diverse including EIA report quality, conditions, and requirements for EIA system implementation; Historical view of EIA system
Cost-effective analysis (CEA) Cost-benefit analysis (CBA)	Transactive	More empirical measure of the effectiveness of EIA systems

Table 1.

List of approaches to EIA system effectiveness evaluation.

Substantive effectiveness focuses on whether EIA systems are actually achieving the tangible outcomes, the ultimate tangible outcome being the protection of the environment and improving social well-being. Without achieving the actual protection of the environment, the EIA process is not adequately effective. Therefore, understanding if EIA systems protect the environment is very important.

For that reason, substantive effectiveness of EIA systems is a better measure of EIA system effectiveness than procedural effectiveness because it addresses the actual environmental protection outcomes of the EIA system. To give an example, we can ask the question: “Has the implementation of the EIA system actually resulted in less pollution, less land degradation, or enhancement of the natural and social environment?”

To answer this question requires the use of some metrics which quantify impacts such as pollution, land degradation, and enhancement of the natural and social environment. Only with quantitative information is it possible to objectively measure impact attributable to EIA systems and, as a result, to assert that substantive effectiveness is present. Quantitative metrics go as far as assigning monetary value to environmental goods, services, and impact, to measure whether the society has actually benefitted from implementing EIA systems.

However, there are challenges with quantitatively measuring environmental and social impacts and even more challenges with monetizing the same. For example, since any country has implemented its EIA system, how much flora and fauna has been saved, and how much is better off because of it?

At a project level, it may be possible to quantify some of the impacts more objectively. For example, a single project can make a case that the level of pollution in an adjacent river has not increased, measured in terms of concentration of pollutants and compared to the baseline before the project started. However, to cascade this measurement to the EIA system level and involving all possible impacts cannot be easily demonstrated.

Therefore, substantive models of EIA system evaluation are much more challenging for two reasons. Firstly, methods of measuring environmental goods and services in order to measure environmental damage avoided through the EIA system are contestable. Secondly, even if there was such a widely accepted method, the attribution gap could be impossible to objectively account for.

Besides the procedural and substantive models postulated by Cashmore, two more models of EIA effectiveness have been added vis-a-vis the transactive and the normative [15, 17].

Transactive EIA effectiveness is one which focuses on relating the cost of carrying out the EIA process to the relative outcomes/benefits of doing so. Hence transactive EIA effectiveness places emphasis on efficiency of resource use. As noted by Veronez and Montaño [18], not much attention has been paid to transactive EIA effectiveness. However, in this chapter cost-effectiveness analysis and cost-benefit analysis are identified as possible tools to use for transactive EIA effectiveness.

CEA can be used to compare two alternative courses of action based on the ration of the cost of the course of action to an indicator of the outcomes. The outcomes are not quantified, but a suitable indicator can be used to quantify the outcomes. Although the author could not locate examples where CEA has been applied in EIA effectiveness, it is conceivable how application can be done. For example, the cost of operating the entire EIA system can be compared to its outcomes such as the number of EIA studies conducted, reviewed, and licensed. Similarly, the efficiency of monitoring EIA implementation in the post-certification stage can also be evaluated.

CBA [19] is similar to CEA except that in CBA, the outcomes are expressed in monetary value. For example, the efficiency of the extra costs incurred by businesses as a result of implementing the environmental management plan can be evaluated against the zero option or other options.

It is important to note that the use of CBA in EIA is common but at the EIA study level in the pre-certification phase of EIA [20]. How to use CBA for EIA system effectiveness evaluation is not clear yet and still needs to be developed.

The last and fourth approach to EIA effectiveness (after procedural, substantive, and transactive) also postulated by Loomis and Dziedzic [17] and Veronez and Montaño [18] is the normative approach. In normative effectiveness, the focus is on the broader impacts of EIA on “sustainable development and participatory process” involved. Normative EIA effectiveness takes into account EIA best practices for continuous improvement.

As noted by Veronez and Montaño [18], there is a conceptual basis for bringing together the different EIA effectiveness models into an integrative EIA effectiveness model. The method, rights, and resources model, as described in this chapter, is one such attempt to produce a more integrative model of evaluating EIA effectiveness.

4. Challenges to EIA effectiveness models

This section discusses the challenges that affect evaluation or measurement of EIA effectiveness.

4.1 Effectiveness of EIA: an afterthought

The first is that, by design, at both policy and project levels, EIA systems lack inherent evaluation mechanisms. Concerns by US citizens over environmental degradation triggered the birth of EIA. As a result, emphasis has always been placed on the processes which support popular participation and decision-making in the pre-certification phase but less in the post-certification phase. Another characteristic of its political origins is that the EIA steps did not include a clear path to evaluate its success. The need to evaluate the effectiveness of EIA systems was almost an afterthought.

Therefore, EIA systems usually do not have built-in evaluation mechanisms comparable to inherent evaluation mechanisms of policies planned through the logical framework analysis. Indicators of progress to outputs, outcomes, and impact are used in LFA planning approaches to inculcate evaluation before policy implementation even starts.

4.2 Quantifying impact

As discussed before, the second challenge is about quantifying environmental impact. For EIA to be considered effective, the question of how much of the environment and social well-being an EIA system saves in pertinent quantification of environmental damage and social well-being levels is necessary to ensure objective answers to this question. Empirical baseline using environmental and social well-being indicators enables future measurement of chance. For example, before a mine is set up, ground water quality can be measured as a baseline. Future periodic measurements will then be measured against the baseline to determine if negative or positive impact has been made.

However, other stakeholders may emphasize on monetary value of the impacts so that they can offset against the cost of implementing the EIA. Unfortunately, there are many models of putting a monetary value to the environment, but they all suffer from the challenge of what is the real value of nature. Hence, this challenge will continue to exist.

4.3 Attribution gap

Attribution gap is another challenge. In all policy interventions such as EIA, the impact of policy is often subject to other policies or forces such that the resultant change, whether negative or positive, cannot be attributed to one intervention. Hence the attribution gap requires accounting for unplanned forces of change and unplanned change itself. Such other forces may include environmental awareness campaigns affecting the attitude of citizens toward environmental issues in general and other policies such as local environmental action plans and waste management plans.

5. Theoretical framework of the MRR model

The theoretical framework for the method, rights, and resources model is derived from the outcomes of the Rio Summit of 1992. This may be unusual that a theoretical framework is derived from policy documents, but as will be demonstrated, that makes EIA effectiveness evaluation very practical and relevant.

The MRR model starts by recognizing that the EIA system is enshrined in the outcomes of the Rio Summit and national policy and legal documents. Therefore, the MRR attempts to identify what these documents intended to be the elements of the EIA system and draw from them. Method, rights, and resources are the three elements of the MRR model for EIA system effectiveness evaluation.

In the Rio Declaration on Environment and Development [21], the world governments agreed on 27 principles to guide the world toward sustainable development. Principle 17 states that:

“Environmental impact assessment, as a national instrument, shall be undertaken for proposed activities that are likely to have a significant adverse impact on the environment and are subject to a decision of a competent national authority.”

Principle 15 advanced the precautionary principle stating that:

“In order to protect the environment, the precautionary model shall be widely applied by States according to their capabilities. Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation.”

After placing EIA as a tool for environmental decision-making, the necessary elements for implementing sustainable development initiatives were also identified.

Section 8.3 (d) of the LA 21 stipulates the need for procedures:

“To establish domestically determined procedures to integrate environment and development issues in decision-making.”

Section 22 of the LA 21 [22] states the right of affected people to participation in decision-making:

“One of the fundamental prerequisites for the achievement of sustainable development is broad public participation in decision-making. … the need of individuals, groups and organizations to participate in environmental impact assessment procedures and to know about and participate in decisions, particularly those which potentially affect the communities in which they live and work.”

Principle 10 of the Rio Declaration on environment and development also promulgates that “environmental issues are best handled with the participation of all concerned citizens …” and promotes “access to information” and “the opportunity to participate in decision-making processes.”

We can therefore say, by design, compliance (with the method/procedure) is identified as a requirement for implementing EIA systems. The other element is participation (stakeholder’s rights), and the third one is capacity (or resources). It is proposed that EIA systems can be used to evaluate based on these three to gauge whether intended objectives are being achieved. Based on this discussion, we have the three elements as method (M), rights (R), and resources (R), respectively ( Figure 1 ).

Figure 1.

MRR model’s theoretical framework.

The figure above shows a schematic representation of the theoretical framework of the MRR model.

6. The updated MRR model

This chapter proposes the method, rights, and resources model, which is more flexible and all-encompassing while involving less of subjective assessment of the researcher. This model proposes that the entire EIA system of any country can be broken down into three elements, namely, method, rights, and resources. From this point of view, the MRR model is an integrative model, seeking to bring together procedural, substantive, transactive, and normative issues into EIA effectiveness evaluation.

By nature, evaluation is a process of collecting and analyzing information, with the aim of measuring the difference between the situation and a set standard. Since information collection is involved, decisions must be made about what information to collect and how to analyze it.

In MRR model, the set standard against which effectiveness can be evaluated is derived from the founding principles of the EIA origins as discussed below. On the other hand, what information to collect is guided by indicators specifically chosen for EIA systems.

6.1 Method

Method has to do with the procedures or process putting the EIA system into practice. The procedures or process ensures that stakeholder views and environmental issues are taken account of in decision-making. This is the pre-certification phase. Procedures or process also ensures mitigation/enhancement measures stipulated for a project are implemented (post-certification phase). The pre-certification phase is often laid out in the form of step-by-step guidelines (the EIA process). Splitting the EIA process into two (pre- and post-certifications phases) makes it easier to understand.

The main focus of the pre-certification phase is to produce an EIA report which a regulatory authority will use to decide whether a project should be allowed to be implemented or not. The report is produced through a scientific study of environmental and social issues relating to a specific project.

The pre-certification phase process can have the following general steps: screening to eliminate activities which do not need detailed studies followed by scoping the extent of the EIA study, collecting baseline information, stakeholder participation platforms, impact identification analysis, impact analysis (e.g., magnitude, severity, etc.), formulating mitigation/enhancement measures, and formulating the environmental management plan (EMP) and impact monitoring plan (IMP).

The role of the post-certification phase is the implementation of the EMP and IMP within the project activities. Often, reports about the progress of implementation of the EMP and IMP are required to be submitted to the regulatory authority on a regular basis. In this way, the method becomes a reasonable proxy for substantive effectiveness as companies report the achievement of environmental and social protection at that level.

Given the importance of the process in determining the information available to regulatory authorities for decision-making, compliance with the process is a necessity. In fact, the degree of compliance with the process is a measure of the effectiveness of EIA systems including the extent of integration of environmental issues into both decision-making and, if granted permission, the implementation of projects.

An important point to note here is that measurement of environmental and social impacts can be embedded in the method in the post-certification phase, hence giving the opportunity to measure substantive effectiveness, first at project level and by extension at a higher EIA system level. For example, periodic reporting of the success and failure in the implementation of the environmental and impact monitoring plan provides information about the post-certification phase, indicating potential substantive effectiveness of the EIA system.

6.2 Rights

Citizens who are affected by economic and social development have the right to have a say in such activities. The EIA system provides a platform for affected citizens to contribute their views to EIA decision-making. It is therefore essential to measure the effective participation of all stakeholders as this demonstrates the effectiveness of an EIA system. To effectively contribute their views, adequate information and an opportunity to contribute both need to be provided.

Some stakeholders participate because of their mandates as organizations. For example, a tourism project that has potential impacts on the river system may require input from the ministry responsible for water even though the ministry responsible for the environment may be responsible for facilitating the EIA system. Environmental associations may also have a say on the mandate given to them by their membership. Hence a wide array of stakeholders may participate in an EIA process where their interests may be at stake.

The participation of all stakeholders can also be measured both in the pre-certification and post-certification phases. In the pre-certification phase, participation is about stakeholders having their views included in decision-making before a project can be implemented. In the post-certification phase, participation is about stakeholders being heard about the environmental and social impacts of the project. Hence a strong post-certification participation is a proxy measure for substantive effectiveness.

6.3 Resources

Policy implementation requires resources. In fulfilling their roles, the capacity of the EIA system as well as that of the individual stakeholders is dependent on the resources they have. This includes financial, human, institutional, legal, and material resources for the EIA system to function. The LA21 constantly emphasizes on the need for “means of implementation” which imply capacity required to implement the dictates of the declaration.

Just as discussed for the method and rights above, resources can also be measured in both pre- and post-certification phases. Similarly, the post-certification measure of the resource capability of each stakeholder is a proxy measure of the substantive effectiveness of the EIA.

Therefore, the MRR model is about measuring the level of compliance (with the method), participation (upholding stakeholder rights), and capacity (in the form of each stakeholder’s resources) in the two phases of the EIA system which, together, give the measure of the effectiveness of the EIA system.

7. Application of the MRR model

In this section, the summarized overview of the MRR model is presented. Development of indicators of effectiveness of the EIA system at all levels is the basis for applying the MRR model. Indicators can be developed based on the three elements presented before, i.e., compliance, participation, and capacity.

Further to that, the indicators must clearly measure the pre- and the post-certification phases separately. The importance of this is that by design, the post-certification phase indicators must be proxies for the actual protection and enhancement of the natural and social environment.

The set of indicators suggested below are categorized according to the three pillars of the EIA system, namely, compliance, participation, and capacity. The author cannot formulate an exhaustive list of indicators. In addition, indicators can be formulated at different levels depending on the focus of the evaluation (e.g., national level or sector level). Similarly, indicators can be developed for each phase of the EIA process, i.e., pre- and post-certification phases.

The indicators individually or collectively measure the effectiveness of the EIA system. The rational further emphasizes that if compliance, participation, and capacity are effective, the objectives of the EIA system are being achieved. The objectives may be procedural, substantive, transactive, or normative.

Next, the application of the model indicates the sources of data from where data can be collected to inform the analysis of each indicator. Data can be characterized in several ways including whether it pertains to the pre- or post-certification phase of the EIA system, whether it is secondary, primary, qualitative, quantitative, and so on. The last part of the application suggests methods of data analysis suitable to answer the question about whether a particular indicator is being achieved or not. Emphasis is placed on quantitative analysis.

It is important to note that these indicators are derived from an EIA system with detailed EIA process guidelines. The EIA process guidelines provide the standard against which some of the indicators can be evaluated as shown in the following tables (Tables 2–4).

7.1 Compliance indicators

Below are listed some suggested compliance indicators:

Data source	Indicator explanation
Sample of approved EIA report (enumeration)	Average % of steps of the EIA process guideline that are complied with by the EIA studies
Questionnaire survey of EIA practitioners	Level of compliance in general; Level of compliance of the pre-certification stage in general; Level of compliance of the post-certification stage in general; Length of EIA process, from referral/prospectus/scoping to certification; Length of approval/review process; Integration of the approved environmental management plan and the project’s/business’s environmental management system (if any)
Report of the responsible authority	% of EMP monitoring reports submitted to responsible authority compared to overall number of approved projects; % projects stalled before certification; % projects stalled due to non-compliance after certification; % projects approved within prescribed timeframe; Length of EIA process, from referral/prospectus/scoping to certification; Length of approval/review process

Table 2.

Compliance indicators.

7.2 Participation indicators

Below are some suggested participation indicators:

Data source	Indicator explanation
Sample of approved EIA reports (enumeration)	Number of stakeholders consulted during the EIA study process; Stakeholders’ recommendations that are integrated into the EIA report for decision-making; Consultation methods used during EIA studies; Availability of information/feedback to stakeholders before certification
Report of the responsible authority	Involvement of other government departments in the EIA review stage (collaboration); Involvement of other government departments in impact monitoring (collaboration); Availability of legal complaint systems
Questionnaire survey of EIA practitioners	Involvement of other government departments in the EIA review stage (collaboration); Involvement of other government departments in impact monitoring (collaboration); Availability of information/feedback before certification; Availability of information/feedback after certification
EMP monitoring reports	Availability of information/feedback after certification

Table 3.

Participation indicators.

7.3 Capacity indicators

Below are some suggested capacity indicators:

Data source	Indicator explanation
Report of the responsible authority	Capacity to monitor EMP implementation; Capacity to enforce EIA conditions; Capacity to review EIA reports; Stakeholder capacity to contribute to EIA process; Capacity to measure the environmental baseline; Capacity to identify and analyze impacts; Availability of baseline information; Availability of competent experts; Capacity to meet EIA study costs; Capacity to meet EMP implementation costs; Capacity to monitor impacts
Questionnaire survey of proponents and practitioners	Capacity to monitor EMP implementation; Capacity to enforce EIA conditions; Capacity to model impacts; Capacity to review EIA reports; Stakeholder capacity to contribute to EIA process; Capacity to measure the environmental baseline; Capacity to identify and analyze impacts; Availability of baseline information; Availability of competent experts; Capacity to meet EIA study costs; Capacity to meet EMP implementation costs; Capacity to monitor impacts; Capacity to produce EIA reports
Sample of EIA reports	Capacity to model impacts; Capacity to review EIA reports; Stakeholder capacity to contribute to EIA process; Capacity to measure the environmental baseline; Capacity to identify and analyze impacts; Availability of baseline information; Availability of competent experts; Capacity to meet EIA study costs; Capacity to produce EIA reports
EMP monitoring reports	Capacity to monitor EMP implementation; Capacity to enforce EIA conditions; Capacity to measure the environmental baseline; Capacity to identify and analyze impacts; Availability of baseline information; Availability of competent experts; Capacity to meet EMP implementation costs; Capacity to monitor impacts

Table 4.

Capacity indicators.

7.4 Data sources and collection methods

Data collection follows the scientific research method conventions. There are both primary and secondary sources of data from which to choose from for each indicator. Some of the indicators allow the collection of both secondary and primary data, but some may not. From the tables above, it can be observed that most of the data comes from the EIA reports and questionnaire survey of all stakeholders.

7.4.1 Secondary data sources

Secondary data sources for analyzing the indicators are documents found within the stakeholders such as the government offices, regulatory authorities, proponents, consultants, and any other institution. Reports from the regulatory authority, the EIA reports reviewed by the regulatory authority, and the EMP monitoring reports are the main sources of secondary data. If the regulatory authority produces an annual report, relevant data such as the total annual number of EIA reports approved and rejected can be found. This data is relevant for evaluating the effectiveness of the EIA systems overseen by the regulatory authority.

Data collection from regulatory authority documents is by reviewing and capturing both qualitative and quantitative data.

EIA reports, environmental audit reports, impact monitoring reports, environmental management systems, prospectuses, referrals, project proposals, terms of references, environmental certificates (and conditions), and environmental management plan reviews are other important sources of secondary data. These secondary data sources are mostly found within the regulatory authority but can also be obtained from the proponents implementing approved EIA plans.

From the EIA reports, data can be collected by reading and enumerating. For example, EIA reports do show how much of the EIA guidelines are complied with during their compilation. Hence the number of the steps of the EIA process that each EIA report complies with can be counted or enumerated. Similarly, one can calculate the number of the different types of consultation methods used which indicates how well the stakeholders participated in the EIA process.

7.4.2 Primary data sources

Stakeholders’ experiences are the main source of primary data. The main stakeholders of the EIA system include the government officials, proponents of projects that have undergone the EIA process, consultants who undertake the EIA studies, the public that is affected by the projects undergoing EIA study, and the regulatory authority responsible for deciding on the EIA acceptance.

Any suitable primary data collection method can be used. Questionnaire instruments can be designed to collect data on most if not all the indicators. Interviews, if necessary, can be used to provide in-depth information to explain the results of quantitative analysis.

From the questionnaires, data can be collected to measure any of the indicators that show compliance, participation, and capacity. Questionnaires can be used to capture stakeholder experiences about the achievement of the procedural, substantive, transactive, and normative objectives of the EIA system. Questionnaires collect information that fills in the gaps in secondary data from the regulatory authority and proponents.

7.5 Data analysis

The MRR model places emphasis on quantitative analysis of data since the primary focus of EIA effectiveness evaluation is to benchmark the achieved versus the planned. Below are some suggested statistical tests which can be used to analyze the quantitative data that arise from enumeration and scoring of EIA reports and regulatory authority reports as well as questionnaire data (Tables 5 and 6).

Statistical analyses	Example
Test of association between observed compliance and expected compliance	For example, if desired compliance is set at say 70% percent, a test of association can determine whether the compliance level is significantly below, above, or equal to 70%
Calculate means of compliance, participation, or capacity for each project type, and compare means between the project types Do the same for economic sectors, consultants, project size, year, location, or any other categorization of projects	For example, projects in the mining sector can be compared to projects in the tourism sector in terms of compliance or participation or capacity indicators
Calculate compliance, participation, or capacity for the pre-certification and the post-certification phases, and compare the two phases	For example, the pre-certification phase can be compared to the post-certification phase in terms of compliance or participation or capacity indicators
Set an expected benchmark of compliance, participation, or capacity, and compare with the observed/actual compliance, participation, or compliance	For example, if desired capacity is set at say 70% percent, a test of association can determine whether the capacity level is significantly below, above, or equal to 70%
Collate all compliance indicators into one overall compliance indicator, and compare against a set benchmark. Do the same for participation and capacity	For example, from the questionnaire data, calculate the mean compliance, and perform a test of association to determine if the mean compliance level is equal or less than a set benchmark such as 70%
Calculate and graph the trends (trend analysis) in overall compliance and participation of capacity over several years	For example, grouping all EIA reports into the corresponding years of compilation, calculate the compliance indicators for each year, and observe if compliance has been increasing or decreasing
Rank stakeholders by participation levels

Table 5.

Suggested data analysis techniques.

Advantages	Weaknesses
Simplicity	Costly extensive data collection
Holistic	Unavailability of information
Improved objectivity
Flexible analysis approach
Triangulation

Table 6.

Advantages and weaknesses.

8. Advantages and weakness

8.1 Advantages and weaknesses

8.1.1 Advantages

This section will outline the advantages of the MRR model. Firstly, the MRR model has the advantage of simplicity. Being based on the simple pillars of the implementation of any policy intervention (compliance, participation, and capacity), the MRR model is very easy to conceptualize. Further, the MRR model uses the evaluation theory, a commonly accepted and widely used approach in policy interventions and development planning.

Secondly, the MRR model provides a framework to encompass all issues related to the implementation of an EIA system. Any conceivable EIA issues including effectiveness, efficiency, relevance, and sustainability of the EIA system can be brought down to indicators within the three elements, i.e., compliance, participation, and capacity (or method, rights, and resources).

Thirdly, the MRR model is a possible reduction to the subjectivity that sometimes underlies the evaluation of effectiveness using other models. The MRR model ensures that the EIA system is evaluated according to its fundamental principles more than subjective concepts of researchers. For example, the idea of systemic and foundation issues [9] discussed before is very dependent on the authors’ conceptualization of issues. However, compliance, participation, and capacity are more objectively defined.

In addition, the MRR model emphasizes on quantitative data obtained as objective data collection process using standardized data collection tools.

In the MRR model, analysis can be adapted to suit the availability of data. The indicators can also be crafted to suit the expected scope of EIA system evaluation.

Multiple sources of data used toward a single indicator are a means of triangulating data collection. This allows a more robust analysis process and a richer and more representative analysis outcome.

Lastly, the MRR model can be easily adapted to compare EIA systems from different geographical locations and jurisdictions. The MRR model is particularly suitable for use by regulatory authorities who have easy access to all the data required for a rigorous evaluation of the EIA system.

It is suggested that further fine-tuning of the MRR model can enable a more objective and in-depth inter-country comparison of the performance of EIA systems.

8.1.2 Weaknesses

There are also challenges with the MRR model. The main challenge is that although quantitative methods can be used with the MRR model, the model still falls short of completely addressing the inherent challenge of quantifying environmental impact. Secondly, applying the model can be very costly depending on other factors. Thirdly, application will vary considerably depending on how the respective country’s EIA system is set up. The MRR model will apply best where the EIA system is elaborate including specific guidelines, monitoring, and report systems from which data can be made available. Where no such elaborate EIA system guidelines exist, applying the MRR model is likely to be more challenging. However, the MRR model can be narrowed down to suit the situation.

Lastly, data collection proposed in the MRR model is extensive. It may not always be easy to obtain all the data. The availability of data also depends on having a well-developed system of recording and EIA reports and other sources of secondary data. Without a significant pool of practitioners effectively involved in the EIA system, it is not possible to obtain adequate information through the questionnaire surveys.

9. Conclusion and recommendations

This chapter introduced the MRR model for EIA system effectiveness evaluation. The chapter shows that the MRR model can address, albeit partially, some of the objectivity and comparability issues of EIA system evaluation. The MRR model does not intend to be a magic bullet to solve all the inherent challenges of quantifying environmental impact. It only suggests presumably a better approach to evaluating the effectiveness of EIA systems based on the evaluation theory and indicator formulation.

The intention of the chapter was not to give a step-by-step guide but the basic thinking behind the MRR model [4], hence opening it up to trial by other evaluators of EIA systems.