Step 4 - Evaluation and Deliberation

This step relies on the framework and information developed in steps 1 to 3 above.

Method:

After structuring the social choice problems and selecting a catalogue of indicators, you can choose a specific type of evaluation or deliberation exercise among those that are possible. The type of assessment chosen will depend on the objective of the exercise, but also on the assumptions of the team on sustainability (weak or strong), for instance.

Sustainability assessment can use a uni-dimensional or a multi-criteria framework. Uni-dimensional frameworks include: monetary evaluation, physical unit evaluation (matter, as in a life-cycle assessment, or energy).

Assessment can be conducted by a team of experts of one or several disciplines (economists, ecologists, sociologists, political scientists), or as a variation through interviews of resource people (workers, inhabitants, engineers…), or can be done with the participation of all stakeholders, as a deliberation exercise.

 

Types of evaluation:

Uni-dimensional assessment, for instance

  • Cost-benefit analysis
  • Energy footprint
  • Carbon footprint
  • Other physical measure

This type of assessment consists of reducing all information type to one common dimension and measuring the equivalence in that dimension type, which can be:

  • money (dollars, or euros, or another currency)
  • energy (KJoules for example)
  • matter (specific elements: carbon, or a table of elements through lifecycle analysis)

Multi-actor multi-criteria evaluations

  • Expert multi-criteria evaluation
  • Multi-stakeholder deliberation

In this assessment type, each valuation criteria is assessed apart. Multi-stakeholder evaluation assumes each stakeholder has an equal legitimacy to produce judgements on a given situation.

In the INTEGRAAL framework, each scenario, option or policy is measured against each of the different performance issues identified in Step 2. Moreover, in a multi-stakeholder assessment (evaluation or deliberation), each stakeholder group, or each actor within a stakeholder group, produces his or her own evaluation of the performance of each scenario, option or policy, against each of the performance issue.

  •  To better understand the concepts and issues of multi-criteria and multi-stakeholder assessment, you can refer to REEDS’ evaluation module on the Forest of Brocéliande http://broceliande.kerbabel.net/evaluation

KerDST (Deliberation Support Tool) is an on-line tool offering users a multi-stakeholder multi-criteria deliberation framework that can be applied to any situation of social choice or evaluation. It has for variations, which are the combinations of the two following criteria:

  •  with or without multiple actors per stakeholder group
  •  with or without indicators

For deliberations made with indicator, the KerBabel Indicator Kiosk (KIK) provides a convenient way of cataloguing indicators, and mobilising them for tasks of deliberation.

  •  You will find an online presentation and a downloadable PDF guide of the KerDST on the Evaluation module of the Forest of Brocéliande.

Deliberation exercises are undertaken from a multi-stakeholder multi-criteria perspective at appropriate scales (e.g., from farm to region to nation…), corresponding to defined contexts of collective debate and action. There may, in principle, be many discrete evaluation exercises that can be loosely coupled by engaging common typologies of stakeholders and performance values, or by considering the same or analogous strategies.

Whether an evaluation or a deliberation framework, uni-dimensional or multicriteria, is chosen, this step produces outcomes in the formal sense.  Multi-stakeholder deliberation also provides insights and learning to participants via the discussions that take place and observation of the respective positions adopted and of how these evolve through the collective learning that occurs.

A multi-criteria and multi-stakeholder deliberation exercise is different from conventional case analysis. Though this step is related to part 3 of the Outline for the core case studies “Analysis of the case”, the exercises and outputs will be quite different according to the type of exercise and related assumptions chosen.

A multi-stakeholder deliberation may or may not produce “Policy recommendations”, mentioned in part 4 of the Outline document. This depends on the original framing of the social choice problem, and on the objective of the assessment.

For instance, specific options or scenarios identifies in Step 2 may include policies. The assessment in Step 4 (this step) will allow each stakeholder group or actor to evaluate the performance of each scenario or policy against specific issues.

 

Advice from REEDS:

REEDS can provide advice on selecting an appropriate type of evaluation, according to the assumptions and the objectives of the EJO.

It can then suggest to the EJO specific techniques or tools for evaluation, and guide the team through the use of the KerDSTTM deliberation support tool if this is the tool chosen, and optionally with its Indicator Kiosk,

 

Output:

The output will be a short position statement by the EJO on the method chosen, and a summary of the evaluation or deliberation exercise.