Evaluation Framework (SIEF)
The SUM Standardized Impact Evaluation Framework (SIEF) extends CIVITAS CEF to verify SUM’s core target: at least a 25% shift from daily private car use to NSM and/or public transport in each Living Lab. It is the shared method every Living Lab uses to measure whether new shared mobility (NSM) moves people out of private cars. It also measures the impact on the environment, society and the local economy.
The SUM Standardized Impact Evaluation Framework
Policies
1. Level of completion of SUMP (city plan) measures
2. Community involvement
3. Balance of pull-push planned/implemented measures
4. Number of NSM modes integrated in the system
Transport system
7. Actual door-to-door travel time
8. Active mobility infrastructure
9. Multimodal integration in stations/stops
10. Travel cost ratio
5. Acceptance level of planned/implemented measures
6. Intention level to use NSMs
11. Perceived door-to-door travel time
12. Perceived safety difference
13. User satisfaction
14. Perceived affordability
Modal split (main project outcome)
15a. Modal split in trips
15b. Modal split in passenger kilometers
15c. Modal split in vehicle kilometers
Sustainability outcomes
Environment
16. Greenhouse emissions
17. Air pollution
18. Noise hindrance
19. Energy consumption ratio
Society
20. Accessibility ratio
21. Vertical social equity
22. Horizontal social equity
23. Social inclusion
Economy
24. Congestion and travel delays
25. Social welfare
26. Profitability of NSM operators
27. Profitability of MaaS platforms
28. Operation revenue of NSMs
29. Service utilization rate
How to read the framework
SIEF is built on four horizontal levels, crossed by vertical dimensions. The chart below follows exactly that structure.
Four levels (top to bottom)
- 1. Policies KPIs 1–6 Push and pull measures, NSM modes integrated, community involvement.
- 2. Transport system KPIs 7–14 Alternative modes versus private car on travel time, cost and safety.
- 3. Modal split KPIs 15a–15c The main project outcome: the share of each mode.
- 4. Sustainability outcomes KPIs 16–29 Indirect effects of any modal shift on environment, society and economy.
Two evaluation phases
- Process evaluation: levels 1–2: how measures were planned and implemented, and what helped or blocked them.
- Impact evaluation: levels 3–4: the effects of those measures, measured before and after implementation.
Vertical dimensions
- Actual vs perceived: levels 1–2: what a city really implemented, and how citizens experience it. Travellers only change habits if they feel the difference.
- Environment, society, economy: level 4: the three dimensions of the sustainability assessment.
Global and local indicators. Global indicators are measured in every Living Lab at city level and are used to compare them. Local indicators are chosen by each Living Lab to explain its own results, and are not used for comparison.
How impact is measured
SIEF tracks each Living Lab before, during and after its measures are implemented.
- 1
Before
The baseline: the WP1 SUM survey, run at the start of the project with at least 200 respondents per Living Lab, plus the city's own records. - 2
During
Intermediate measurements, on the actual indicators only: policy implementation and transport system performance. - 3
After
The survey is repeated and the indicators are measured again. Modal split is measured at least twice: before and after.
Available Resources
KPI SIEF Framework & Formulas
Comprehensive documentation explaining each KPI, calculation methodologies, data requirements, and scientific framework for NSM-PT integration metrics.
KPI Calculation Spreadsheet
Pre-built Excel template with formulas and data validation to help you organize raw data and calculate KPI values. Includes examples and instructions.
Survey Template (English)
Sample survey questionnaire designed to collect citizen feedback and usage data for relevant KPIs. Can be adapted and translated for your city's specific context.
Where SIEF comes from
The SUM Standardized Impact Evaluation Framework (SIEF) extends the core principles of the CIVITAS Evaluation Framework (CEF). The SIEF is built to examine what works, what does not, and why, aligned to SUM's own objective.
Its indicators come from two sources. A systematic literature review of sustainable urban mobility indicators (SCOPUS: 8,943 documents identified, 3,362 retained after the inclusion criteria) produced the candidate list. That list was then validated and circulated with the Living Labs, whose own ranked needs guide which local indicators each of them selects.