| |
|
home > FAQs
Frequently Asked Questions -
- compiled by Sarah
Mosedale, former EDIAIS Project Research Officer
to top
IA Basics
What is impact assessment?
Impact assessment is the process of identifying either the
anticipated or the actual impacts of a development intervention,
on those social, economic and environmental factors which the
intervention is designed to affect or may inadvertently affect.
Impact assessment can be applied at both the project level,
and at the strategic (policies, programmes, plans) level. Ex
ante impact assessment takes place before an intervention is
approved and forecasts the potential impacts as part of the
planning and design of the intervention. Ex post impact assessment
identifies actual impacts during and after implementation, to
enable corrective action to be taken if necessary and to improve
the design of future interventions. Indicators of impact can
also be integrated into existing management information systems
so that information about progress towards desired goals can
be made available to staff immediately. In this way the impact
assessment process feeds into internal monitoring and evaluation
for ongoing learning. The key findings of impact assessments
should be shared with the stakeholders involved to increase
programme-level learning. Impact assessment should be regarded
as a dynamic process and not as a series of static reports.
For more information see "Basic
Impact Assessment at Project Level" and, for information
about strategic impact assessment of policies, plans and programmes
see "Strategic Impact
Assessment and Enterprise Development".
to top
What does impact assessment aim to achieve?
Impact assessment has three different, but interrelated, objectives:
· Accountability and transparency: to provide evidence about
the achievements of interventions and their costs.
· Improving programme/project effectiveness: to produce recommendations
about how present and future performance could be improved
· Policy development: to generate guidance about how government
and donor policies could be reformed so as to facilitate more
successful interventions.
Each of these objectives is likely to shape the design of
an impact assessment in different directions. In the past, impact
assessment has tended to be donor-led and to focus on "proving"
impact. More recently there has been a move towards "improving"
impact and using impact assessment to set up or develop sustainable
learning systems. Impact assessment can only contribute to lesson
learning if the information is used as a basis for asking intelligent
questions about project implementation and how it can be enhanced.
When policy development is an important aim it is particularly
important that the results of impact assessments are promptly
and widely disseminated. For more information see Basic
Impact Assessment at Project Level.
to top
Isn't impact assessment very expensive?
Certainly carrying out a full-scale impact assessment which
includes interviewing a large number of people and analysing
detailed information gathered over a considerable period of
time can be expensive. As yet there are no clear guidelines
on how much should be spent on impact assessments but good practice
suggests that the more important the findings are expected to
be then the more resources should be made available for the
impact assessment. For example if a major expansion of a project
is being considered then the initial project should be assessed
in detail. On the other hand if there is clear evidence that
a project is failing and most of those involved agree why this
is happening then a low cost impact assessment to verify it
may be all that is needed. Often a major resource constraint
is skilled impact assessment personnel. Therefore it may be
necessary to book consultants well in advance and to build timetables
around their availability. Where possible less experienced consultants
should be used to assist established experts in order to broaden
the human resource base. Thought also needs to be given to the
timescale. The time needed to carry out an impact assessment
increases with the scale of the intervention, the complexity
of its impacts and the degree to which these are likely to be
challenged by different stakeholders. It is common for insufficient
time to be allowed for a satisfactory assessment. See Basic
Impact Assessment at Project Level (page 17) for guidance
on the components of impact assessments to be considered and
costed when planning the budget. See also Strategic
Impact Assessment and Project Development for a description
of the process of deciding on the level of detail required.
to top
What are the main types of impacts?
Impacts are usually considered to fall into three categories:
economic, social and environmental. Sometimes impact assessments
focus on a single category. For example Environmental Impact
Assessments are well established, and are carried out as part
of the approval process to assess the likely impacts of a range
of proposed activities that may have potentially harmful effects
on the environment. Sometimes impact assessments produce reports
for more specific purposes. For example impact assessments may
focus exclusively on health, poverty, regulation or gender impacts.
Integrated impact assessments (which include sustainability
impact assessments) are increasingly being undertaken. Such
assessments cover a range of economic, social and environmental
impacts. Impacts can be analysed at a number of levels. For
example you might be interested in the impact the activity has
on individuals, on households, on enterprises, on the community
or on institutions (or on a combination of these). See "Basic
Impact Assessment at Project Level".
to top
What is the counterfactual or baseline?
Impact assessment attempts to identify what difference a project,
programme or policy has made or will make. To do this, a judgment
has to be made about what would have happened if the intervention
had not happened (or what will happen if the intervention does
not happen). This is known as the counterfactual or baseline.
It is important to realise that this baseline is different from
that baseline which refers to the situation at the start of
the intervention. It is of course very important to understand
the exact situation that you are starting from and indeed the
lack of this sort of information often causes problems for impact
assessment. Depending on the size and complexity of the impact
assessment it may be advisable to get external help to collect
baseline data at the start of the intervention. However the
baseline which is used to assess impact is the best estimate
of what the situation would have been if the intervention had
not happened. The fact that it is impossible for any particular
person, household or enterprise both to experience an intervention
and at the same time not experience it leads to lots of problems
in methodology. One way of establishing a counterfactual is
to identify sufficiently similar groups of people or enterprises
which have not been involved in the project in order to compare
their situations with the situations of those who have participated
in the project. This is complicated and however carefully it
is done problems remain. See "Basic
Impact Assessment at Project Level".
to top
What is attribution?
Attribution is one of the key challenges involved in conducting
impact assessments. Dealing with attribution means answering
questions such as: supposing significant changes are found to
have occurred after a project has been implemented how can we
be sure these changes were caused by the project? How do we
know that these changes would not have happened anyway? Attribution
is about demonstrating that observed changes are a result of
the intervention rather than coincidental. The use of causal
(or impact) chain analysis can help in establishing attribution.
See "Basic Impact Assessment
at Project Level".
to top
What is an impact chain?
Behind all interventions is the assumption that they will change
behaviour and practice in ways that will lead to desired outcomes.
All changes are influenced in ways that are difficult to predict,
by such things as the specific characteristics of the people
involved and of their economic, physical, social and political
environment. For example in a microfinance project a package
of technical assistance and capital changes the behaviour (and
products) of a microfinance institution (MFI). The MFI subsequently
provides different products to a client. These services lead
to the client changing his/her microenterprise activities which
in turn leads to increased or decreased microenterprise income.
This causes changes in household income which in turn lead to
greater or lesser household economic security. The modified
level of household economic security leads to changes in the
morbidity and mortality of household members, in educational
and skill levels and in future economic and social opportunities.
Ultimately, perhaps, these changes lead to modifications in
social and political relations and structures. This sort of
series of events, where one impact leads to another is known
as an impact chain (or causal chain). See "Basic
Impact Assessment at Project Level".
to top
IA Methods
What is screening?
Screening is undertaken to determine whether or not an intervention
requires an impact assessment and, if so, how detailed the assessment
should be. Small scale projects or minor modifications to existing
programmes may not need to be assessed at all and interventions
which are considered unlikely to have major economic, social
or environmental impacts may only need a simplified assessment.
Only a minority of interventions, with the potential for major
or complex impacts may need a full impact assessment. Screening
should help to ensure that assessment resources are used cost-effectively.
For interventions below the 'threshold' for an impact assessment,
it is assumed that generally applicable policies or regulations
for the planning and management of development activities are
sufficient. See "Strategic
Impact Assessment and Enterprise Development".
to top
What is scoping?
The purpose of scoping is to determine the terms of reference
for an impact assessment. For a full impact assessment the scoping
study might be expected to cover:
· The problem which the intervention is addressing, the
goals and targets it is expected to meet, and the alternative
interventions which might be considered
· The types of impacts to be assessed, the causal determinants
(impact chain) to be investigated and the level of detail
expected
· The types of mitigating and enhancing measures that might
be investigated to deal with any remaining potentially significant
problems or to enhance potential beneficial effects
· The types of data that will be needed for the assessment,
the assessment methods to be used and the consultations which
will be undertaken during the assessment process Scoping aims
to focus the assessment on those impacts that are of particular
concern, and not waste time and resources on others. See "Strategic
Impact Assessment and Enterprise Development".
to
top
What are indicators?
Indicators are variables which we choose to investigate in
order to provide information about progress (or lack of it)
towards desirable goals. To be useful indicators must be able
to be defined precisely and must be measurable. Economic indicators
might include changes in income, levels and patterns of expenditure,
consumption and assets. Social indicators might include educational
status, access to health services, nutritional levels or contraceptive
use. Socio-political indicators, attempting to measure empowerment
and often focusing on changes in gender relations, can include
measurement of individual control over resources, involvement
in household or community decision-making, levels of participation
in community activities and social networks and electoral
participation. Indicators are used to measure both outcomes
and processes. When measuring outcomes for a gender empowerment
project, for example, indicators might include such things
as: percentage of girls enrolled at primary school, numbers
of women traders with enhanced access to credit, numbers of
women receiving agricultural extension services etc. Indicators
measuring process might include: legislation on gender opportunities,
availability of finance for women's businesses etc. Indicators
can be quantitative (objective, expressed as numbers) or qualitative
(subjective, expressed in words). Cash income and expenditure,
numbers of livestock owned, acreage of ground cultivated and
birth weights of children are all examples of quantitative
indicators. Health status, self-confidence and satisfaction
are qualitative indicators. Sometimes, to facilitate comparisons,
qualitative indicators are expressed as numbers on a sliding
scale. Provided respondents are all using the scale in exactly
the same way these can then be treated as quantitative indicators.
For most impact assessments it is important to use both quantitative
and qualitative indicators and to cross check results. For
more information see "What
do we want to know? Selecting indicators", "Qualitative
Methods", and "The
Quantification of Impacts".
to
top
How do I choose indicators?
There is no one set of "correct indicators" for assessing
a particular type of impact since indicators will need to
reflect the stated objectives of the intervention. However
any indicators chosen will inevitably be partial and selective
and will be guided by an underlying theoretical, and often
political, understanding of what types of impact are important.
Also different stakeholders will have different priorities
leading to inevitably tricky questions about how the relative
significance of indicators is to be weighed and how trade-offs
made. Decisions as to which indicators to use to assess progress
are best made at the planning stage with appropriate consultation
so that the necessary information can be collected at suitable
points. The key task in selecting indicators is not therefore
to try to provide a total picture of "reality" which is impossible,
but rather to choose indicators which will be more useful,
less arbitrary and more accountable. In general both economic
and social indicators will be needed, both qualitative and
quantitative. Questions as to why the indicators are needed,
what hypotheses of impact the indicators are intended to assess,
the degree of precision needed and the accuracy of information
available, how unexpected impacts will be measured and so
on must be addressed when selecting indicators. For more information
on choosing indicators see
"What do we want to know? Selecting Indicators".
to
top
What is stakeholder analysis?
A stakeholder is any person, group or institution that has
an interest in a development activity, project or programme.
This includes both intended beneficiaries and intermediaries,
winners and losers, and those involved in or excluded from
decision-making processes. Primary stakeholders are those
who are ultimately affected, i.e. who expect to benefit from
or be adversely affected by the intervention. Secondary stakeholders
are those with some intermediary role. In an enterprise project
secondary stakeholders might include e.g. DFID, trades unions,
banks, Ministry of Finance, local government, export promotion
agencies and business service providers. Key stakeholders
are those who can significantly influence the project, or
are most important if objectives are to be met. Stakeholder
analysis aims to identify and define the characteristics of
key stakeholders, assess how they might affect or be affected
by the activity, understand the relations between stakeholders
(including actual or potential conflicts of interest) and
assess the capacity of different stakeholders to participate.
For guidance on doing a stakeholder analysis, see "Impact
Assessment and Stakeholder Analysis".
to
top
What is stakeholder participation?
Stakeholder participation is the process whereby stakeholders
- those with rights, responsibilities and/or interests in
the project or programme - play an active role in decision
making and in the consequent activities which affect them.
It is essential that primary stakeholders (those who expect
to benefit from or be adversely affected by the development
activity) participate in this process. Participatory methods
should be an integral part of any impact assessment. Firstly,
grassroots participation leads to more widely accepted identification
of impact goals and measurable indicators. Secondly, different
stakeholders are affected by development in different ways
and participatory methods enable better identification of
who exactly is affected in which ways. Thirdly, such methods
enable complex interactions between contexts, grassroots aspirations
and strategies, institutional structures and enterprise interventions
to be better understood. Finally participatory methods help
to identify the practical implications of impact assessment
findings through negotiation between different stakeholders.
Women, those from the most remote communities and the poorest
and most disadvantaged people can easily be excluded by the
process because of the extra effort required to include them.
Careful sampling is therefore essential to any credible impact
assessment both to ensure representation of different stakeholders
in the process and to counter the inherent bias against the
poorest and most disadvantaged individuals and communities.
Conventional approaches to sampling are inadequate as they
focus on statistical sampling alone. For impact assessment
careful sampling is equally important when using qualitative
and participatory methods. For an overview of the issues and
an outline of the questions to consider when planning to use
participatory methods see "Participatory
Methods". For a comparison of the strengths and weaknesses
of different types of sampling design and summary guidelines
see "Who
do we talk to? Issues in sampling".
to
top
What are the main methods for collecting
information for an impact assessment?
Methods for collecting information for impact assessments
include literature reviews, surveys, questionnaires, interviews,
focus groups, case studies, observation and workshops. Tools
include many developed through Participatory Rural Appraisal
and Participatory Learning and Action which can be used to
enable people who may not be literate to communicate their
knowledge, ideas, opinions and priorities to researchers.
Usually impact assessments will involve the use of both quantitative
methods and qualitative methods. See "How
do we find out? Issues in collecting information".
to
top
What are quantitative methods?
Quantitative information is information in numerical form
and quantitative methods are methods used to collect this
type of information. To find out the weight of something,
for example, you weigh it and your answer is a specific number
of grams. Many such directly quantitative methods are used
in impact assessment to, for example, record air or water
quality, count the number of people who have attended training
and so on. Vast amounts of social data are also collected
in quantitative form using surveys. In such surveys respondents
typically tick boxes to choose from a range of pre-determined
answers. Quantitative statements can then be made which comment
on the numbers, or percentages, of respondents who have answered
in a particular way. See The
Quantification of impacts.
to
top
What are qualitative methods?
Qualitative methods are usually understood to include interviews
which can vary from semi-structured questionnaires to open
ended ad hoc conversations, direct observation and case studies.
The focus is on understanding different perceptions and how
these influence accounts of "facts" and events rather than
seeking one "true" version of reality. Qualitative methods
seek to understand complexity as a more accurate reflection
of reality. Because of this emphasis on complexity the scope
and focus of qualitative research are continually redefined
as understanding of different parts of the process increases
and new issues arise. Qualitative methods are an essential
complement to quantitative methods in any impact assessment.
For an overview and review of the main qualitative methods
see "Qualitative
Methods".
to
top
How do I choose which information
collecting methods to use?
A variety of methods is likely to be needed both to answer
the range of questions an impact assessment will want to ask
and for crosschecking. Quantitative methods are good at answering
questions about what impacts have occurred whereas qualitative
methods are also useful for investigating how and why those
impacts have occurred. Choice of methods will also relate
to the purpose of the assessment, the particular issues to
be addressed, the stakeholders involved, the ways the findings
are to be used and the size of the available budget. For an
overview of the methods and key issues in integrated impact
assessment see "How
Do We Find Out? Issues in Collecting Information".
For a summary of the comparative strengths and weaknesses
of the different methods and guidance as to when each method
is and is not appropriate see Section 8 of "Basic
Impact Assessment at Project Level".
to
top
Relationship of IA to the project
cycle
Does IA only apply to projects?
No. There is currently much more experience of impact assessment
at the project level that at any other but there is also a
great deal of interest in, and some experience of, assessing
impact at programme level, across sectors and nationally.
For more information see "Strategic
Impact Assessment and Enterprise Development".
to
top
How does IA relate to planning?
Impact assessment needs to be integrated into all stages
of the planning cycle for projects or policies. At the early
stages it is important to involve all the stakeholders (not
only donors and their partner organisations but also the people
who are intended to benefit from the project) in agreeing
the objectives, outputs and activities of the project, determining
and weighing the risks and identifying the indicators for
measuring progress. Time spent developing key indicators of
achievement with stakeholders is an important investment in
the success of the project, leading to greater shared commitment.
The resulting framework should be understood by all parties
to be a working document that can be adapted as necessary
as the planned intervention develops. Even with the most carefully
designed interventions, unexpected outcomes, both positive
and negative, can occur. Critically, at this early stage,
baseline data must be collected. Impact will ultimately be
assessed by comparing the position reached after the intervention
is over with an informed estimate of what the position would
have been had it not been undertaken. To do this it is essential
both to know what the starting position is and to develop
a good understanding of non-attributable factors and their
effects. A failure to collect adequate baseline data often
impedes this later analysis. It is also essential at this
stage to design the monitoring systems which will collect
the necessary information throughout the project's or policy's
life to enable progress (or lack of it) to be assessed. Responsibilities
for carrying out this information gathering, integrating it
within management information systems and ensuring it will
be fed back into the management process need to be agreed
and clearly understood by all parties. Appropriate funding
for this activity needs to be written in to the project budget.
For more information see "Impact
assessment and project management cycle (PMC)" and "Strategic
impact assessment and enterprise development".
to
top
How does IA relate to appraisal?
Two types of impact assessment are commonly recognised. Ex-ante
impact assessment is the process of predicting what the impact
of a proposed project is likely to be and is closely related
to the task of designing interventions in such a way as to
maximise positive impacts and minimise negatives. (Ex-post
impact assessment is the process of determining what the impact
of an intervention actually was.) Ex-ante impact assessment
therefore contributes to the appraisal stage of the project
cycle where decisions as to whether or not to fund the project
are taken on the basis of what it is believed that the project
is likely to achieve. See "Basic
impact assessment at project leve"l.
to
top
How does IA relate to monitoring?
Effective monitoring is an important component of the impact
assessment process as it is through monitoring that some of
the information essential for impact assessment will be collected.
As described above the monitoring system and responsibilities
should have been agreed at planning stage. Monitoring needs
to be handled sensitively as it can be viewed by stakeholders
as a policing exercise and cause suspicion and distrust. This
is less likely if implementing agency staff have been involved
from the start in its design. As well as the planned and routinised
collection of information as an integral part of the project,
monitoring should provide opportunities for face-to-face contact
between stakeholders and can help develop good partnership
relationships. Indeed participatory monitoring can be an important
part of increasing the intended beneficiaries' involvement.
If impact assessment has been an integral part of the planning
process so that monitoring has been carefully planned and
implemented then subsequent external assessment can in some
cases reduce to a simple quality assurance and information
dissemination exercise. The external assessment would check
that the monitoring had been conducted appropriately and carry
out any necessary independent checks, review the lessons learned
by the implementing team and conduct investigations of any
unexpected effects but otherwise the lessons disseminated
by the external assessment will be those learned within the
intervention itself. At the opposite extreme, where monitoring
has not been planned and implemented with impact assessment
in mind then external assessments will become very much more
onerous and expensive tasks. See
"Basic impact assessment at project level".
to
top
How does IA relate to evaluation?
Ex post impact assessment, where actual (as opposed to predicted)
impacts are assessed during or after the lifetime of the development
activity is normally designed to make specific contributions
to the evaluation stage of the traditional management cycle.
See "Basic impact assessment
at project level".
to
top
More questions:
Is the impact assessment process
the same regardless of what kind of project or programme I am
trying to assess?
The underlying philosophy of impact assessment does not change
according to the context. But the questions which need answering
by the assessment will vary. This means that once the principles
and main methods for impact assessment are understood they
have to be applied to the particular activity in question.
Good quality impact assessments achieve a "fit" with the objectives
of the development intervention and the resources and time
available. Enterprise development is a broad field - practitioners
may be involved in, for example, microfinance, business development
services, ICTs, fair trade, rural enterprise, tourism, regulatory
reform or privatisation. For guidance on applying impact assessment
principles and methods in particular types of enterprise development
see our "Applications
Guidance Web Page".
to
top
How reliable are the results of
impact assessments?
The reliability of the results of an impact assessment will
depend on how the assessment was carried out which in turn
will depend on its goals, and the resources available. Assessments
may be carried out at simple, moderate or complex levels.
A simple assessment (the most numerous type) will have only
moderate reliability and this will based mostly on triangulation
of results. Such assessments are carried out to test the existing
understanding of impacts and to contribute to improvements
in programme operation. A variety of methods will be used
usually involving a small scale client survey with a comparison
group, cross checked using rapid or participatory appraisal
methods. A moderate approach will be significantly more costly
and yield higher levels of reliability through statistical
inference rather than triangulation. Its focus will be both
on demonstrating impact and on improving programmes. Its audience
would include policymakers and senior programme managers and
its methods would centre on a significant survey where respondents
are compared with a carefully matched control group. Crosschecking
might involve rapid appraisal techniques, participant observation
and case studies. At the complex and most costly level there
is a focus on high levels of reliability. The focus is on
"proving" impact and attributing the causes of impacts. Repeated,
large-scale, carefully constructed household surveys will
probably be involved. The main audience is policymakers and
researchers. The key task for the impact assessment designer
is to select an approach that can meet the objectives of the
assessment at an acceptable level of rigour, given the resources
available.
to
top
I want to commission a consultant
to carry out an impact assessment - how do I draw up terms of
reference for this work?
Model terms of reference for impact assessment consultants
are included on pages 25 to 28 of "Basic
Impact Assessment at Project Level".
to
top
|
|
