Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Judicial Modernization Project


  
1. Project Data:   
ICR Review Date Posted:
08/11/2011   
Country:
El Salvador
PROJ ID:
P064919
Appraisal
Actual
Project Name:
Judicial Modernization Project
Project Costs(US $M)
 24.05  21.71
L/C Number:
L7135
Loan/Credit (US $M)
 18.20  16.90
Sector Board:
Public Sector Governance
Cofinancing (US $M)
   
Cofinanciers:
Board Approval Date
  08/01/2002
 
 
Closing Date
12/31/2008 11/30/2010
Sector(s):
General public administration sector (40%), Law and justice (20%), Information technology (20%), Vocational training (10%), General information and communications sector (10%)
Theme(s):
Other public sector governance (33% - P) Administrative and civil service reform (33% - P) Legal institutions for a market economy (17% - S) Judicial and other dispute resolution mechanisms (17% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Gita Gopal
Rene I. Vandendries Ismail Arslan IEGPS2

2. Project Objectives and Components:

a. Objectives:


    The Project Appraisal Document stated the objectives of the Judicial Modernization Project as follows:
      • The proposed Project's development objective is to improve El Salvador's judicial system by promoting measures aimed at enhancing the effectiveness, accessibility and credibility of its Judicial Branch, through a participatory process involving judges, technical and administrative staff and users of the judicial system.
    The Legal Agreement stated the objectives similarly.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

There were four components that included a variety of activities to support specific results including strategic planning and a model Integrated Judicial Center, greater awareness of justice issues by relevant stake holders, enhanced efficiency of the court, improved citizen accessibility, enhanced transparency, and improved professional quality and competence of judicial officers and employees. A fifth component supported a Project Coordinating Unit (PCU), responsible for monitoring and evaluating progress. The components were as follows:

    Component 1: Institutional Management Capacity of the Judicial Branch (appraisal: US$4.28 million; actual US$3.16 million).
      • Design and implementation of an integrated planning system including development of internal procedures, training systems for judges, provision of equipment for administrative units of the Judicial Branch; and the development of ICT systems.
      • Development of an international grant resource management program through technical assistance to the Judicial Branch to develop a program to centralize and manage international organizations’ grant resources for broader public participation in institutional reform and to carry out user feedback surveys and diagnostic studies.
    Component 2 - Court System Modernization (appraisal: US$11.52 million; actual US$14.53 million).
      • Development of a court re-mapping plan to increase efficiency of the Judicial Branch’s resources, including improvements in incentive systems for judges, and greater flexibility for senior judges to create courts and allocate resources in line with demand and other factors.
      • Strengthening of the automated judicial and administrative case management program linked to the integrated planning system proposed in Component A.
      • Establishing model Integrated Judicial Centers (e.g. a one-stop-judicial outlet) in the Municipality of Soyapango in the Metropolitan San Salvador area.
      • Expansion of judicial infrastructure through the construction of the Integrated Judicial Center in Soyapango to enhance judicial services there and in surrounding municipalities.
    Component 3 - Knowledge Sharing to Foster Access to Justice and Transparency (appraisal: US$ 1.39 million; actual US$1.04 million).
      • Design and implementation of a system for inspection, vigilance and control of judicial services to assure greater transparency through: (a) the development of preventive and corrective measures against corruption in the Judicial Branch; (b) the promotion of a Judicial Branch’s code of ethics; and (c) the strengthening of the Judicial Branch’s professional oversight and judicial oversight units through the development of a program of timely resolution of complaints against judges and lawyers
      • Carrying out of legal outreach programs aimed at civil society groups to: (a) enhance understanding of the role of the Judicial Branch and the use of basic judicial procedures; (b) support the legal profession and its professional associations; (c) prevent family and youth violence; and (d)empower women and other minority groups.
      • Modernization of the judicial documentation centers and libraries through: (a) new software, hardware, internet applications, books and publications; and (b) linkages to legal and judicial databases and to local and 23 international information networks, and to ICT systems in Components A and B.

    Component 4 - Development of Professional Quality and Competence of Judicial Officers and Employees (appraisal: US$1.02 million; actual US$0.84 million).
      • Improvement of the Judicial Branch’s human resources policies through: (a) the design and implementation of staff manuals; (b) their dissemination through the Judicial Branch’s intranet; (c) provision of continuing legal education and ethics training to judges, administrative, and technical staff. This would also involve the revision of the Judicial Career Law and review of the draft Administrative Career Law for proposal of improvements to the plenary chamber of the Supreme Court of Justice (SCJ) and the Congress.
      • Promotion of the role of judges in El Salvador’s economic and social development through seminars, workshops and public information campaigns.

    Component 5 - Project Management, Monitoring and Evaluation (appraisal: US$1.01 million; actual US$0.76 million).
      • Strengthening of the capacity of the PCU.
      • Carrying out of Project studies, audits and reviews. Annual audits, progress reports, mid-term review, user surveys, focus groups, dissemination materials, and modernization replication plans and other reports and sector studies would be supported under this subcomponent; and
      • Design/implementation of a Project monitoring and evaluation (M&E) Program

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Costs and Financing: Overall project costs were appraised at US$ 24.05 million; actual was US$21.71 million. The Bank loan was approved for US$18.20 million; actual was US$16.9 million (although the figure in Annex 1 of the ICR is $17.12 million, which the TTL explains was the estimated final disbursement at time of writing the ICR). In July 2010, the Government requested a reallocation of funds to reflect actual expenses across categories, as well as increasing the percentages of local expenditures to be financed under the credit to 90 percent.

    Borrower Contribution: Estimated at US$5.85 million at appraisal; actual was US$4.60 million
    Dates: Although the project was approved in August 2002, it became effective only in March 2004. The closing date of the project was extended twice: (i) from December 31, 2008 to June 30, 2010; and (ii) then to November 30, 2010. The first extension was partly due to a delay of 19 months in the project becoming effective. Constitutional provisions in El Salvador require Congressional approval of public debt with sovereign guarantee; this process took much longer than planned. Project effectiveness was also delayed when the Congress gave priority to reconstruction projects after the 2001 earthquake and reached external debt obligation limits, thereby slowing down all Bank projects. The second extension was to complete some ongoing activities. Despite these extensions and the restructuring in July 2010 that increased the percentage of local expenditures to be financed by the credit to 90 percent, and US$1.3 million was canceled due to incomplete activities.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:

Relevance of project objectives was substantial:

    • The objectives were consistent with the objectives of the Bank’s Country Assistance Strategy (Report No. 22932-ES, dated November 2, 2001), which promoted pro-poor growth accompanied by reduced crime and violence, and increased participation and empowerment. A strong judiciary is necessary for reducing crime. Although the project did not support criminal justice, it did focus on increasing access to justice by the people. The most recent CPS (2009) notes that this project would contribute to strengthening the rule of law.
    • While the objectives were consistent with the strategic priorities of the judicial branch modernization strategy of the SCJ, prepared with donor support, and Government commitment was reflected in its request for the project, changing priorities and intervening events reduced such commitment (reflected in the significant delays for constitutional approval of the project).
    • The objectives complemented assistance provided by other key donors such as USAID and Spain (for human rights activities and training programs), the IDB (for juvenile justice and justice sector commission coordination assistance), and Canada (for youth gangs, crime and violence) in the judicial sector.

b. Relevance of Design:
Relevance of design was substantial:

    • The overall results framework was sound, although it could have been simplified, and the links between the outcomes, intermediate outcomes, and outputs better clarified.
    • Although the weak capacity in the country was recognized as a risk, the design of the project implementing arrangements were not adequate to mitigate the risk and according to the Borrower's ICR, the implementing arrangements proved to be very challenging. (See also Country Partnership Strategy 2009 for El Salvador, which confirms that the complex project design slowed implementation (pg. 41).


4. Achievement of Objectives (Efficacy) :

Achievement is assessed based on specific impact and outcome indicators noted in the PAD and were to contribute to four main outcomes which included :

Degree to which institutional management of the Justice Branch is improved (Substantial):
    • Employee satisfaction with strategic planning and the new organizational model:
      • According to the ICR, the project has established an Integrated Judicial Center at Soyapango, which now provides a model for a one-stop shop for justice services. The automated case-management system and judicial information framework to support the Center's operations were still works in progress at closing, but the Government has satisfactorily completed the activities. The TTL confirms that the integrated court is now fully functional, The ICR notes increased efficiency of the one stop justice center in that it employs only 41 people to serve 10000 inhabitants, while at the national level, the average equivalent figure is 138. This model has apparently been replicated to two other locations in San Salvador.
      • The ICR notes that the administration department of SCJ has now been ISO9000 certified as an indicator that it has adopted best practices in resource use and internal efficiency and internal stakeholder outreach. 100 percent of the administrative units have Institutional Strategic Plans than help with resource management.
      • A judicial map has been prepared for improving the efficiency of resource allocation, approval of the map will require new policies and legal changes
      • A beneficiary impact assessment conducted in 2010 for the project concluded that by and large the majority of the beneficiaries were satisfied with different aspects of the integrated model, although there is no baseline or benchmark. However, this indicator was in reference to justice of peace/first instance courts in general.
    • Greater awareness of judicial progress and problems by policy makers, donors and NGOs:
      • The ICR notes that there was no policy in place to reach out to NGOs at the initiation of the project. By the end of the project, the SCJ set up a communication unit in the judicial branch for outreach to citizens and civil society organizations, especially in the area of family and youth matters.
Degree to which the operation of the courts is modernized and brought closer to the citizens and resource use is rationalized (Substantial):
    • Reduction in time for the publication of SCJ decisions on-line;
      • At project initiation, according to the ICR, SCJ had limited ICT capabilities for publication of decisions. Decisions were published on paper after significant delays. A Centro de Documentacion Judicial was created under the project and is today responsible for producing electronic copies as well as for on-line use by judges and other stakeholders.
      • The ICR notes that the backlog at the Soyapango center has improved from 5,478 active processes at the end of the period in 2007 to 3,431 in 2010.
    • Lawyers' (and users') satisfaction with Court's on-line calendar system:
      • The automated case management system was operational on a trial basis at project closing. It has to be fully institutionalized once SCJ's ICT department takes over the operation from the contractors. It is functioning in Soyapango and 60 percent of those surveyed expressed satisfaction.
    • Users', lawyers', businesses', employees', parliamentarians', and policy makers' satisfaction with pilot Integrated Justice Center (one stop-judicial center):
      • The impact evaluation suggests satisfaction with several aspects of the one-stop judicial center across different institutional dimensions, but provides no baseline or benchmark to assess progress. A quantitative analysis was also conducted using opinion polls to understand the degree of satisfaction of users (internal and external) on the operation of the Soyapango Integrated Judicial Center, compared to the Judicial Center at Santa Tecla. It found that the average number of days that takes to process a case (from plaint reception to verdict) was lower in 3 out of the 5 court types in Soyapango. For family court, processing time was higher in Soyapango while for peace courts, the processing time was the same.
Improved citizen accessibility, and transparency: (Modest):
    • User satisfaction with the disciplinary and vigilance system of judges and lawyers:
      • Consultancy works were still in progress, when the project closed. A study “Establishment of the Transparency Office” has been completed – as a result, the Office has been established and has become functional after project closing. The second and the third studies were: “Implementation of a Corruption Prevention System” and “Strengthening of the Lawyer’s Registration Office”. These were also completed after project closing and an action plan has been formulated although the Bank is not involved anymore.
      • Users' satisfaction (especially the poor and disadvantaged) with legal education and outreach programs and campaigns on the role and responsibilities of the Judicial Branch and social role of the judge:
        • About 32.800 people (communities, schools and users) have been benefited by the Popular Campaign Legal Education in Soyapango. The program has been replicated in the municipalities of San Martin and Ilopango. There is no information on any outcomes.
      • Judges' and legal professionals' satisfaction with the Judicial Document Centers and their Judicial Portal and Increased engagement of civil society organizations in legal outreach activities:
        • Perception surveys were not conducted and no information is available.
Degree to which professional quality and competence of judicial officers and employees of the Judicial Branch is improved (Modest):
    • Progress towards meeting the professional and organizational quality management standards of resource management:
      • The Judicial and Administrative career laws were approved by Congress, but according to the ICR "dynamics in the Supreme Court prevented timely implementation of associated administrative measures recommended by the project i.e. Human Resource Management procedures".
      • A comprehensive study on the Judiciary’s HR system was completed and delivered to SCJ; little progress was made by project closing.
    • Satisfaction of judges and court staff, and others with improvements in quality, and content of judicial education and learning opportunities:
      • The ICR notes that there are concerns about sustainability given that no specific provision has been made for future training programs.
      • During the project more than 90 percent of staff received training either in judicial matters or administrative systems. 185 training events were conducted between 2004 and 2010, 33 in coordination with the Judicial Training School, representing 18% of all training conducted.
      • E-learning has not been established, but according to the ICR, the Judiciary is fully capable of initiating the implementation of Virtual Training System e-learning method because of two training courses for administrators on how to do this. There is no information on distance learning.
      • There is only some anecdotal information on the satisfaction of the trainees. For example, women interviewed for the Impact Evaluation Survey remarked that they are now aware of the importance of training activities for their empowerment, especially in dealing with domestic violence.

5. Efficiency:

Overall, the Review concludes that efficiency was modest for the following reasons:
    • The project took 19 months to become effective after appraisal and project closing had to be extended by two years.
    • Judicial efficiency has improved in some areas but key activities remain incomplete or have not been institutionalized and some of the most important activities were completed after project closing at additional Government cost.
    • Consultant services were more than double the estimate at appraisal, and goods were also about 20 percent more than planned. More importantly, a restructuring paper was prepared in July 2010 to ensure efficient use of all project funds. Changes were made to ensure that all funds would be disbursed. However, at project closing, US$ 1.3 million remained undisbursed and was cancelled.

      a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


Rate Available?
Point Value
Coverage/Scope*
Appraisal:
No
%
%
ICR estimate:
No
%
%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of project objectives and of project design were substantial. Efficacy was substantial for two of four objectives (institutional management and resource use), and modest for the other two especially given the lack of information on outcomes. Efficiency was modest. Overall project outcome is considered to be moderately satisfactory, particularly given the difficult political conditions under which the project and the Bank operated, and that some of the desired results were achieved shortly after closing. .

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

    Risks to outcomes are high:
      • A new law, passed in July 2011, requires Supreme Court of Justice to pass unanimous decisions. If implemented, this will significantly affect the integrity of the judiciary. (www.un.org/apps/news/story.asp?NewsID=38917&Cr=knaul&Cr1=).
      • Some project results are still outputs, or at best intermediate outcomes, and further work is necessary to ensure sustainable reform. The Government has not asked for a follow-up project and the risks remain significant that the results will not be strengthened or sustained.

        a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:
Quality at entry was moderately satisfactory:

  • The QAG rated the quality at entry as satisfactory. The strategic relevance of the project was commended. It noted that project preparation was based on significant research and wide consultation, and that at appraisal the right risks were identified.
  • The Bank team did not establish clear objectives and associated indicators.
  • The lack of an effective coordinating mechanism for the multiple agencies involved in project implementation presented several implementing challenges (Borrower's ICR).
  • Additionally, the assumption that weak capacity in the country would be addressed by strengthening the capacity in the PCU turned out to be unrealistic.
  • The Loan agreement required a PCU to be established prior to loan effectiveness. This did not happen and required an amendment to the loan agreement. Furthermore, given the lack of funds for the PCU meant that although the PCU was eventually established in March, 2004 (when the project became effective), funds were authorized for disbursement only in December, 2004.

Quality-at-Entry Rating: Moderately Satisfactory

b. Quality of supervision:

  • Although Management allocated adequate resources for implementation support, and supervision missions were conducted regularly, several issues appear to have been overlooked such as the need for improved monitoring indicators. This also led to insufficient basis for the ratings in ISRs.
  • The mid-term review (MTR) was conducted four years after project effectiveness and one year prior to the original closing date. The Bank did not review the adequacy of, or revise, the indicators at MTR, although by then they had been found to be insufficient to assess progress in achieving objectives.
  • The Bank was unable to build adequate capacity in the PCU, especially in the area of procurement. Weak PCU capacity resulted in poor coordination of various activities, and lack of knowledge of Bank procedures constrained project implementation. Actions to strengthen capacity were taken only in 12/2009.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Overall, Government performance was moderately satisfactory, taking into consideration factors outside its purview such as the political changes and the earthquake:

  • The Government and the SCJ worked closely with the Bank in preparing the project in a very consultative fashion. However, after project approval, while the SCJ was fully committed to the project objectives, such broad commitment was not always evident at the higher levels of the Government, particularly when it came to politically difficult exercises such as court remapping.
  • The Government took timely steps to ensure that it provided the agreed contribution, although the full amount was not necessary given the revisions that increased the percentage of local costs that could be financed under the credit.
  • The Government also followed through with some of the incomplete activities after project closing.
  • While the Government kept the PCU staffed at all times, lack of adequate incentives led to constant turnover of staff; this constrained implementation.

Government Performance Rating: Moderately Satisfactory

b. Implementing Agency Performance:

  • The ICR notes that the PCU delivered all contractually required reports by the World Bank and those who have been required to submit to government agencies (Ministry of Finance)
  • However, it was very weak in fulfilling one of its important obligations - M&E.
  • A candid and thoughtful ICR was also prepared and attached to the ICR.
  • Procurement capacity was weak and this resulted in several delays to project implementation.

  • Implementing Agency Performance Rating: Moderately Satisfactory

    Overall Borrower Performance Rating: Moderately Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:

      The project design planned for collection of timely information and statistics through an M&E system to be established under the auspices of a M&E facility in the PCU that would collect relevant data and conduct impact evaluations including capturing the gender dimensions. Development of baseline data for performance indicators was initiated very early on. The PAD noted that M&E would be adjusted as needed during the MTR Review. However, some of the indicators were descriptive, based on perception.

    b. M&E Implementation:

    Despite the fact that development of baseline data was initiated early on, the ICR notes that such data was a problem during implementation. The indicators determined at appraisal proved to be inadequate and the indicators were not revised at the time of the MTR as planned. An Impact evaluation was undertaken in 2010 to understand the results of the Soyapango Integrated Judicial Center.

    a. M&E Utilization:

    Given the weaknesses in design and implementation, the ICR notes that utilization of data was minimal.

    M&E Quality Rating: Modest

    11. Other Issues:

    a. Safeguards:

    No safeguard issues were trigged.

    b. Fiduciary Compliance:

    The ICR notes that although procurement was delayed and tedious, given the weak capacity, there were no issues and that annual audited project financial statements found no major issues.

    c. Unintended Impacts (positive or negative):

    The Soyapango Integrated Judicial Center has reportedly been implemented in a gender-aware manner with USAID assistance. To facilitate easier access of judicial services to battered women, day care facilities have been created in the Center to allow women to go to court.

    d. Other:

    Gender Issues were integrated into the project's monitoring system. However, as with other dimensions, this was not followed through.



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Moderately Satisfactory
    Moderately Satisfactory
     
    Risk to Development Outcome:
    Significant
    High
    The new law requiring Supreme Court of Justice to pass unanimous decisions is a risk that was not anticipated at the time of writing the ICR Review. 
    Bank Performance:
    Moderately Satisfactory
    Moderately Satisfactory
     
    Borrower Performance:
    Moderately Satisfactory
    Moderately Satisfactory
     
    Quality of ICR:
     
    Satisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
      The Review agrees with the ICR Lessons in general. It adds the following:
        • A clear and prioritized action plan that supports achievement of strategic objectives is helpful. A project can then be designed as part of this action plan with realistic and phased objectives, and with implementing arrangements that are consistent with existing capacity.
        • Pilot activities, such as establishing one or two integrated court models, which can then be used to demonstrate results of judicial reform can be effective in complex and low-capacity contexts.
        • The selection of clear, relevant, economic, adequate, and monitorable indicators is important to understand project effectiveness. When initially-selected indicators are found to be inadequate or uneconomic, they should be revised in a timely manner.

    14. Assessment Recommended?

    No

    15. Comments on Quality of ICR:

    Overall, the ICR is satisfactory.
      • It is detailed and provides a basis for its rating. The ICR focuses on gender issues, which is to be commended.
      • The ICR could have been strengthened with better organization of the results and a greater focus on outcomes. It should have also buttressed the evidence for the satisfactory performance of the Soyapango Justice Center by contrasting the results with that of the comparator at Santa Tecla.


        a. Quality of ICR Rating: Satisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions