NSW Department of Communities and Justice / Evaluation of Justice Advocacy Service - DJ2018-207

The service will have six core functions:

1.  • Supports for suspects with cognitive impairment in police custody
2. • Supports for victims and witnesses with cognitive impairment when interacting with police
3. • Supports for people with cognitive impairment attending court
4. • Supports for people with cognitive impairment attending legal appointments
5. • Training for justice agency staff on working with people with cognitive impairment in contact with the criminal justice system
6. • Capacity building and peer mentoring for people with cognitive impairment in contact with the criminal justice system

 The following provides a draft overview of the intended outcomes of the Justice Advocacy Service. Further details will be provided to the successful supplier in the form of a program logic which has been developed by the Diversity Services Unit.

1.  • Suspects with cognitive impairment can exercise their rights in police custody.
2. • Victims and witnesses with cognitive impairment can effectively report crime to the police
3. • Victims, witnesses and defendants attending court are able to understand and participate in the process.
4. • Victims, witnesses and defendants with cognitive impairment are able to effectively instruct and communicate with legal representatives.
5. • Improved skills and understanding within and across justice agency staff about the issues that affect people with cognitive impairment in contact with the criminal justice system.
6. • Improved capacity and knowledge amongst people with cognitive impairment of the criminal justice system, including the types of support and services that are available to them.

 In order to assess the effectiveness and help inform decisions about the potential continuation of the service across NSW, an evaluation will need to be undertaken. Depending on the effectiveness of the service, a proposal may be developed to continue the service beyond 30 June 2021.

 In addition, the evaluation will also:

 • Assess whether the service is implemented and delivered as intended across the state of NSW

• Identify the barriers and facilitators to implementation and delivery

• Assess whether the service achieved its stated objectives and aims

• Inform the continuous improvement of the service’s implementation

• Assess supply and demand for the service across the state of NSW and at various sites (e.g., court locations, police cells and legal appointments), taking into account 24 hour a day support requirements for some sites

• Assess whether better outcomes are achieved at different locations and/or regions, including a comparison between rural, regional and remote areas and more centralised or populous areas

• Assess whether better outcomes are achieved at different sites, including a comparison between court locations, police cells and legal appointments

 Scope:

 The services to be supplied will be a process evaluation, outcomes evaluation and a cost-benefit analysis by a single organisation, researcher or consultant.

 All three components of the evaluation will be conducted by an independent evaluator. An option for sub-contracting arrangements will be made available in the Request for Tender on the basis that the results of the combined evaluation are provided to the Diversity Services Unit in a singular report.

 The Diversity Services Unit will coordinate the engagement and ongoing relationship with the evaluator. The evaluation will involve quantitative and qualitative data analysis. The evaluation methodology will be developed by the evaluator, in collaboration with the Diversity Services Unit. The evaluator will be required to work across NSW and at various sites including NSW Police Area Commands and Police Districts, around central court locations and if required, onsite with the Diversity Services Unit.

 Stakeholders involved in the evaluation may include, however are not limited to, individuals receiving support via the Justice Advocacy Service, staff delivering supports via the Justice Advocacy Service, Local Court Magistrates and Registrars, Courts and Tribunal Services, Legal Aid NSW, the Aboriginal Legal Service, NSW Health, NSW Police Force, NDIA, NDIS providers and Corrective Services NSW staff.

 The following sets out a draft evaluation framework which will be subject to further negotiation and development with the successful supplier. The development of the evaluation framework should be in accordance with the Justice Advocacy Service program logic. A copy of the program logic will be provided to the successful supplier once a contract is awarded.

 Process Evaluation:

 The process evaluation will focus on the implementation of the service and answer specific questions about whether the service achieved its stated objectives and aims. It will also identify and inform of effective strategies and practices that have developed over its course. The findings will act as a formative guide to improve service delivery and provide feedback about unintended consequences and the possible structural, cultural and administrative challenges to continued implementation.

 Another key focus of the process evaluation is to provide a comprehensive description of how the Justice Advocacy Service is designed, implemented and delivered across the state of NSW, incorporating data collected while the service is operational. The evaluation will also determine if the service is operating as intended and if implementation supports the goals of the service.

 

The process evaluation will also answer specific questions about the impact of the Justice Advocacy Service on agencies and the interactions of different elements of service and the service system, and identify effective strategies and practices.

 The process evaluation will consider service delivery during the period from 1 July 2019 to 30 June 2021.

 The terms of reference for the process evaluation may include:

1. • Determine the efficiency and effectiveness of the provision of supports for suspects with cognitive impairment in police custody
2. • Determine the efficiency and effectiveness of the provision of supports for victims and witnesses with cognitive impairment when interacting with police
3. • Determine the efficiency and effectiveness of the provision of supports for people with cognitive impairment attending court
4. • Determine the efficiency and effectiveness of the provision of supports for people with cognitive impairment attending legal appointments
5. • Identification and extent of any underservicing or over-servicing
6. • Determine the success of training as measured against its intended outcomes for justice agency staff on working with people with cognitive impairment in contact with the criminal justice system
7. • Measure justice agency staff’s levels of satisfaction with training on working with people with cognitive impairment in contact with the criminal justice system
8. • Determine the success of capacity building and peer mentoring for people with cognitive impairment in contact with the criminal justice system
9. • Measure stakeholder’s levels of satisfaction with the operation of the service and their recommendations, if any, for improvement.

Outcomes Evaluation:

The outcomes evaluation will focus on whether the Justice Advocacy Service is delivering expected outcomes that are in line with the intended objectives of the service. For example, the outcomes evaluation will measure short and medium term impacts and consequences of the Justice Advocacy Service on stakeholders and individuals receiving support in accordance with the program logic.

 Another key focus of the outcomes evaluations will be measuring any change in circumstances for individuals receiving support via the Justice Advocacy Service and assess whether the changes can be attributed to the support they received via the service. Similarly, changes in circumstances for stakeholders, such as changes in their understanding regarding the types of support that should be provided to individuals with cognitive impairment, and whether this change can be attributed to the supports provided via the Justice Advocacy Service will also be measured.

 The outcomes evaluation will also assess the success (or otherwise) of the service and its contribution to providing support to individuals with cognitive impairment in contact with the criminal justice system. The findings will act as a formative guide to consider whether the service should be provided beyond 30 June 2021.

 The outcomes evaluation may be separated into short and medium term outcomes.

 Some specific questions measuring the short term outcomes of the Justice Advocacy Service may include:

1.  • Do individuals receiving support via the Justice Advocacy Service have increased understanding of their rights in police custody?
2. • Do individuals receiving support via the Justice Advocacy Service have increased understanding of their rights when interacting with police?
3. • Do individuals receiving support via the Justice Advocacy Service have increased understanding when communicating with legal representatives?
4. • Does receiving support via the Justice Advocacy Service increase participation rates at court for people with cognitive impairment?
5. • How well does the Justice Advocacy Service meet client / stakeholder needs?

 Some specific questions measuring the medium term outcomes of the Justice Advocacy Service may include:

1.  • Do individuals receiving support via the Justice Advocacy Service have increased understanding of their bail conditions?
2. • Do individuals receiving support via the Justice Advocacy Service have increased compliance with their bail conditions?
3. • Do individuals receiving support via the Justice Advocacy Service have increased ability to provide instructions to legal representatives?
4. • Are defendants who receive support via the Justice Advocacy Service more likely to appear for their court appearance?
5. • Do individuals receiving support via the Justice Advocacy Service have an increased number of s32 diversion orders?
6. • Is there an increase in the number of convictions involving a victim or witness who has received support via the Justice Advocacy Service?
7. • Do stakeholders have a better understanding regarding the types of support that should be provided to individuals with cognitive impairment who are in contact with the criminal justice system?
8. • Whether the Justice Advocacy Service could be successfully applied to broader contexts and locations (e.g., the civil justice system or more broadly across the criminal justice system, including more broadly across children’s court locations).
9. • Whether the Justice Advocacy Service should be provided beyond 30 June 2021.

 Cost-Benefit Analysis:

 The Cost-Benefit Analysis will calculate the savings generated for every dollar spent on the Justice Advocacy Service.

 The Cost-Benefit Analysis of the Justice Advocacy Service will be undertaken to:

1. • Calculate the savings generated for every dollar spent on the service
2. • Evaluate the outcomes for participants against cost to government
3. • Help inform decisions about the potential continuation of the service across the state of NSW beyond 30 June 2021. 

 Comparison group:

 Although applying a randomised control trial (RCT) design is regarded as the gold standard for evaluating the effectiveness of interventions, it would be unethical to select individuals who would be eligible to receive support and randomly allocate them to a control group which would deny them access to a support person who can assist them to exercise their rights in custody, understand bail conditions, give effective evidence and receive support to attend court and legal appointments.

 Given these difficulties, Tenderers will be asked to propose how they would design a suitable comparison group as a means to evaluate the Justice Advocacy Service. The proposal is subject to further negotiation with the Diversity Services Unit.

 Essential Experience:

• Demonstrated experience in conducting large scale or state-wide process evaluation, outcomes evaluation and cost benefit analysis

• Demonstrated experience in conducting evaluation across social policy and human service related areas such as disability, mental health, ageing, health, homelessness, social justice, social impact and the criminal justice system

• Demonstrated experience in conducting evaluation with vulnerable or disadvantaged individuals, cohorts and/or communities such as individuals experiencing health issues, dual diagnosis of mental health disorder and cognitive impairment, lack of adequate family and/or community support, relationship/family breakdown, housing issues (homelessness, risk of homelessness, or living in inappropriate dwelling conditions), financial stress, history of disengagement with supports, behavioural issues, substance and/or alcohol abuse, domestic/family abuse, and involvement with multiple services/systems.

• Ability to provide a state-wide evaluation including rural, regional and remote communities, and at various sites including NSW Police Area Commands and Police Districts and around central court locations

 Location:

The evaluation will be conducted across the state of NSW, including regional, rural and remote communities and at various sites including NSW Police Area Commands and Police Districts and around central court locations. The evaluator will be required to work across these locations and if required, onsite with the Diversity Services Unit.

NSW Regions: Far North Coast, Mid North Coast, New England, Central Coast, Hunter, Cumberland/Prospect, Nepean, Northern Sydney, Inner West, South East Sydney, South West Sydney, Central West, Orana/Far West, Riverina/Murray, Illawarra, Southern Highlands

Open Tenders - An invitation to tender by public advertisement with no restriction placed on who may tender. Tenderers will normally be required to demonstrate in their tender that they have the necessary skills, resources, experience, financial capacity, and in some cases licences, accreditations, etc., to fulfil the tender requirements.

No