Evaluation Process
About the Evaluation Rubric
The Boundless San Francisco Evaluation Rubric, below, will be used to assess applications during Participatory Review and during Evaluation Panel Review. During Participatory Review, each valid application will be reviewed by at least four (4) peer Applicants who will assess each application on the four criteria that comprise the Evaluation Rubric. Once Participatory Review is complete, top Applicants will advance to the Evaluation Panel, where they will be reviewed by at least four (4) Evaluation Panel Reviewers who will use the same Evaluation Rubric to assess each application.
The rubric is built upon four distinct criteria, guiding reviewers to consider each trait in isolation when evaluating applications. In both stages of review, reviewers will assign a score for each of the four criteria and provide a comment to justify each decision. Those scores and comments will be compiled into one composite assessment of each application, and each Applicant will receive its “package” of scores/comments from the assigned reviewers (who will be anonymous to the Applicants).
This process is designed to be transparent, fair, and open while providing a value-added experience for Applicants. Even Applicants who do not receive an award will receive feedback from peers and many will receive feedback from Evaluation Panel Reviewers about their applications.
The Evaluation Rubric includes four criteria that are equally weighted across a five-point scale. Each Applicant can receive between 1 and 5 points on each scale, representing a total possible scoring range between 5-20 from each reviewer. When reviewing the rubric, there are different descriptors within each scale. They are:
- Trait: The name attributed to each review criterion (e.g., “Durability”).
- Prompt: One or more questions, intended to clarify areas of consideration when defining a Trait.
- Evaluation Range Definitions: Clarifying statements to define sub-scoring ranges (e.g., 2 vs. a 4).
With four (4) reviews per application, the total maximum score from the four assigned reviewers for each trait is 20, and the maximum aggregate total score is 80 points.
#1 TRANSFORMATIVE AMBITION
INCREMENTAL
1: Does not demonstrate potential to shift conditions for San Franciscans. It focuses exclusively on service delivery without engaging the systems, policies, norms, or opportunity structures that sustain the problem. The theory of change is absent, unclear, or not proportionate to the resources requested.
2: Shows limited potential to shift conditions for San Franciscans beyond direct service delivery. It references broader change but does not articulate a credible path toward it. The theory of change is underdeveloped or only loosely proportionate to the resources requested.
3: Demonstrates some potential to shift conditions for San Franciscans by moving beyond service delivery to engage systems, policies, norms, or opportunity structures that contribute to the problem. The theory of change is coherent and generally proportionate to the resources requested.
4: Presents strong potential to shift conditions for San Franciscans by clearly targeting systems, policies, norms, or opportunity structures that sustain the problem. The theory of change is bold, coherent, and well-proportionate to the resources requested. Where service delivery is present, it serves as a foundation for that broader structural ambition rather than an end in itself.
5: Offers exceptional potential to fundamentally shift conditions for San Franciscans by targeting systems, policies, norms, or opportunity structures that sustain the problem at scale. The theory of change is bold, rigorous, and compellingly proportionate to the resources requested. Any direct service work is explicitly connected to and in service of that structural ambition.
TRANSFORMATIVE
#2 VIABILITY
IMPLAUSIBLE
1: Lacks a credible plan for achieving results in San Francisco. It shows little to no understanding of the city's context, the communities to be served, or the partnerships, relationships, and strategies needed to execute. Where applicable, applicants without an existing San Francisco presence offer no realistic path to establishing one.
2: Shows limited evidence of a credible plan for achieving results in San Francisco. Understanding of the city's context and the communities to be served is surface-level. Partnerships, relationships, or strategies described are vague and insufficient for successful implementation. Where applicable, applicants without an existing San Francisco presence offer a weak or unconvincing path to establishing one.
3: Offers a credible plan for achieving results in San Francisco. Demonstrates a working understanding of the city's context and the communities to be served. Partnerships, relationships, or strategies are adequately identified and suggest the team has a reasonable foundation for successful implementation. Where applicable, applicants without an existing San Francisco presence offer a plausible path to establishing one.
4: Presents a strong and specific plan for achieving results in San Francisco. Reflects a clear understanding of the city's context and the communities to be served. Partnerships, relationships, or strategies are well-defined and clearly positioned to support successful implementation. Where applicable, applicants without an existing San Francisco presence offer a realistic and well-reasoned path to establishing one.
5: Offers an exceptional and detailed plan for achieving results in San Francisco. Demonstrates deep understanding of the city's context and the communities to be served. Partnerships, relationships, or strategies are concrete and constitute a significant asset for successful implementation. Where applicable, applicants without an existing San Francisco presence offer a compelling and thoroughly reasoned path to establishing one.
COMPELLING
#3 EVIDENCE OF RESULTS
UNSUBSTANTIATED
1: Provides no meaningful evidence (quantitative or qualitative, internal or external) that the approach works. For established applicants, there are no results to speak of. For early-stage applicants, the theory of change is poorly reasoned and draws on no relevant analogous evidence.
2: Provides weak evidence (quantitative or qualitative, internal or external) that the approach works. For established applicants, results are limited. For early-stage applicants, the theory of change is underdeveloped and loosely connected to relevant analogous evidence.
3: Provides adequate evidence (quantitative or qualitative, internal or external) that the approach works. For established applicants, results are present but not compelling. For early-stage applicants, the theory of change is sound and grounded in relevant analogous evidence, even if direct results data is limited.
4: Provides strong evidence (quantitative or qualitative, internal or external) that the approach works. For established applicants, results are clearly documented and speak to the effectiveness of the approach. For early-stage applicants, the theory of change is well-reasoned and supported by meaningful analogous evidence.
5: Provides compelling evidence (quantitative or qualitative, internal or external) that the approach works. For established applicants, results are robust, well-documented, and directly relevant. For early-stage applicants, the theory of change is rigorous and supported by strong analogous evidence that makes a persuasive case for the proposed approach.
WELL-EVIDENCED
#4 DURABILITY
FLEETING
1: Lacks a credible vision for sustaining results beyond the grant period. There is no meaningful plan for financial sustainability, policy change, institutional adoption, community ownership, or other sustaining mechanisms. There is no articulation of what lasting benefit looks like for the people or systems served. No realistic plan for the five-year horizon and beyond.
2: Offers limited evidence of a credible vision for sustaining results beyond the grant period. Plans for sustainability are vague or rely on assumptions that are not well-supported. The plan for the five-year horizon and beyond is underdeveloped.
3: Demonstrates a vision for sustaining results beyond the grant period. Identifies plausible mechanisms for sustainability, whether financial, policy-related, institutional, or community-based, though the plan for the five-year horizon and beyond could be more fully developed.
4: Presents a strong and credible vision for sustaining results beyond the grant period. It articulates clear mechanisms for sustainability, whether financial, policy-related, institutional, or community-based, and offers a realistic picture of the lasting benefits those mechanisms are designed to protect. Offers a realistic plan for the five-year horizon and beyond.
5: Outlines an exceptional and well-substantiated vision for sustaining results well beyond the grant period. Demonstrates a comprehensive approach to sustainability through multiple reinforcing mechanisms and offers a compelling articulation of the lasting benefits those mechanisms are designed to protect. The proposal offers a thorough, realistic plan for the five-year horizon and beyond.
LASTING
Lever for Change addresses possible biases in the evaluation process both on the front end, before the scoring begins, and on the back end, after the reviewers have submitted their scores.
On the front end, Lever for Change’s Participatory Review and Evaluation Panel Review webinars review the Competition eligibility criteria and the Evaluation Rubric and includes training on recognizing and addressing the biases that we all bring as we read and assess proposals.
Lever for Change also works to address unconscious bias inherent in the evaluation process.
During both Participatory Review and Evaluation Panel Review, each reviewer only reviews a very small subsample of the entire set of applications. Therefore, each reviewer has a very different reference point by which to adjudicate each application. For example, a reviewer may only see very good applications. Because these applications are of similarly good quality, the reviewer may decide to score these applications lower than they deserve. By contrast, a reviewer that only sees poor applications may give better scores on average. Furthermore, reviewers may have intrinsic cognitive biases. Even under ideal conditions, reviewers are prone to making non-objective judgements about the applications they review, and so Lever for Change normalizes the raw scores provided by our evaluators to reduce the risk that bias in the assessment process eliminates a strong application from consideration.
Normalization relies on finding the mean score given for all applications and the mean score given by each reviewer. Reviewers whose mean score is higher than the global average will have their scores devalued to account for whatever biases led to inflated scores. Similarly, reviewers whose scores are lower than the global average will have their scores boosted to compensate for their tougher-than-average assessments.
The evaluation and normalization process will be conducted at two points during the Competition: first, after the conclusion of Participatory Review and, later, after the conclusion of Evaluation Panel Review. A submission that ranks among the top applications, as determined by Participatory Review scores, will be eligible to advance to Evaluation Panel Review. A submission that ranks among the top applications, as determined by Evaluation Panel Review scores, may be considered for selection as a Finalist.
As a further risk mitigation step, Lever for Change staff review reviewers’ scores and comments for every submission.
If at any point in the process, it is determined that an applicant does not meet the eligibility criteria of the Competition as laid out in the Competition Rules, the applicant will be notified and will not be eligible to progress to further stages of the Competition.
