Independent Evaluation - Home > Search

Implementation Completion Report (ICR) Review - Erwanda Project


  
1. Project Data:   
ICR Review Date Posted:
04/04/2013   
Country:
Rwanda
PROJ ID:
P098926
Appraisal
Actual
Project Name:
Erwanda Project
Project Costs(US $M)
 10.0  9.6
L/C Number:
CH253
Loan/Credit (US $M)
 10.0  9.6
Sector Board:
Global Information/Communications Technology
Cofinancing (US $M)
 0  0
Cofinanciers:
Board Approval Date
  09/07/2006
 
 
Closing Date
12/31/2010 12/31/2010
Sector(s):
Central government administration (68%), Sub-national government administration (24%), Postal services (8%)
Theme(s):
Administrative and civil service reform (33% - P) Health system performance (17% - S) Regulation and competition policy (17% - S) Infrastructure services for private sector development (17% - S) Rural services and infrastructure (16% - S)
         
Prepared by: Reviewed by: ICR Review Coordinator: Group:
Melvin P. Vaz
Robert Mark Lacey Soniya Carvalho IEGPS1

2. Project Objectives and Components:

a. Objectives:
The objectives of the project as stated in the Project Appraisal Document (PAD) (page 5) and Financing Agreement (page 5) are identical: to improve (i) efficiency and effectiveness of some internal processes of the government of Rwanda; and (ii) the delivery of applications and services in selected key sectors including better access to information through the use of technology.

b. Were the project objectives/key associated outcome targets revised during implementation?
No

c. Components:

1. Strengthening government effectiveness and efficiency (US$3.80 million at appraisal, US$3.80 million at closure)

This component's aim was to improve the key internal systems of the government. Its main focus was to support: the harmonization of the basic technology for the Ministries, and Provincial and District Offices. It supported the implementation of a government portal and intranet to link Central Ministries and District Offices and provide availability and easy access to a variety of innovative applications and information services. The component also assisted the government to: develop standards for the deployment of the new system to Central Ministries, Provincial and District Offices; equip Ministries and District Offices; put in place a network infrastructure (WAN) that connects all relevant institutions at the national and sub-national levels; establish cyber-security system to ensure data accuracy, integrity and interoperability; establish a maintenance and support infrastructure for the Information and Communication Technologies (ICT) applications; and ICT training and capacity building activities.

2. Improving service delivery to citizens and the private sector (US$ 3.51 million, US$3.51 million at closure)

This component assisted the government to improve access to information about basic health, education, agriculture, environment and judicial services, and to disseminate this information primarily to vulnerable groups, such as women, children, and farmers through provision of public access points, such as internet, radio, brochures, street theatre, etc. in urban and rural areas. The component also provided assistance for the involvement of communities in content development and dissemination mechanisms to better focus public services to local communities. The project was to fund the development and maintenance of public access points in both rural and urban areas through kiosks and included training for kiosk operators.

3. Improving access to services through enhanced infrastructure in energy and telecommunications (US$ 0.82 million at appraisal, US$ 0.82 million at closure).

This component was to assist the government in promoting access to competitive and affordable telecommunications services in remote areas by: strengthening the Rwanda Utility Regulatory Agency's (RURA's) operational efficiency effectively to regulate and monitor the sector; and improving basic access to infrastructure by financing connectivity to a set of provinces and district offices, and linking public access points in rural areas to effectively disseminate and exchange information.

4. Project management (US$ 1.47 million at appraisal, US$ 1.47 million at closure)

This component supported the Ministry of Infrastructure and its implementing agency (Rwanda Information and Technology Authority (RITA)) to establish a project management system for implementing the above reforms. The component included the establishment of an adequate quality assurance mechanism and development of a monitoring and evaluation system for the eGovernment reforms. This component would also fund technical assistance, analytic studies, audits, training and the salaries of RITA staff dedicated to this project.

d. Comments on Project Cost, Financing, Borrower Contribution, and Dates
Project Cost: The total estimated cost of the project at appraisal was US$ 10 million. Of this amount, US$ 9.6 million was allocated among the 4 project components and US$ 0.38 was reported as unallocated in the PAD (page 8). The actual cost of the project was US$ 9.6 million (ICR page 33). After the Mid-Term Review (MTR) in July 2009, component 3 was dropped because its activities were being funded and managed through another World Bank project (Regional Communication Infrastructure Program). Its activities were replaced by 2 new activities (a Document Tracking and Workflow Management System (DTWMS), and additional technical assistance and consultancy activity for the government).

Financing: Estimated cost at Appraisal of US$ 10 million was to be wholly financed by a Grant from IDA. At closing, the project disbursed 100 percent of its commitments equivalent to US$ 9.6. The difference between the estimated cost at appraisal and actual cost at closing was mainly due to fluctuations in the exchange rate of USD and SDR.

Borrower Cost: There was no contribution from the recipient or from other development partners.

Dates: The project closed on schedule on December 31, 2010.


3. Relevance of Objectives & Design:

a. Relevance of Objectives:
Substantial.

The statement of objectives was not fully clear as it did not specify either the internal processes of the government under sub-objective 1 (efficiency and effectiveness of some internal processes of the government of Rwanda) or the key sectors under sub-objective 2 (delivery of applications and services in selected key sectors including better access to information through the use of technology) of the project.

Country Conditions. In 2005, Rwanda had very low Information and Communications Technology (ICT) penetration rates compared to both Sub-Saharan African and world averages. Only 0.3 percent of the population had fixed phone subscriptions, 2.5% of the population had mobile phone subscriptions, and 0.6 % of the population used the internet (ICR page 3).

Borrower Strategy. In 2000, the government issued a long-term strategy (Vision 2020) with a vision of transforming Rwanda from a largely agriculture-based economy to a knowledge-based economy and becoming a middle income country by 2020. The government believes that investment in ICT would be a key driver for this transformation and a vehicle for improved public and private service delivery, especially for rural areas. Between 2000 and 2005, a 5-year implementation plan called the National Information and Communication Infrastructure (NICI) (aligned with Vision 2020) with a budget of US$ 500 million was implemented and executed (ICR page 1). At the time of appraisal of this project, the NICI II plan was being implemented to address the remaining agenda from NICI I.

World Bank Strategy. According to the FY09-12 CAS, "Support to the ICT sector will focus on strengthening the enabling environment, and improving both domestic and international connectivity to lower the barrier for investment by operators to extend the geographic coverage of broadband access. This support will be provided primarily through the ongoing E-Rwanda Project for eGovernment applications and services and through Rwanda’s participation in the second phase of the Regional ICT Project (RCIP) planned for early FY09." . Also, eGovernment was recognized as an important tool to strengthen the government's internal management and information service delivery to citizens in the 2002 CAS (ICR page 15).

b. Relevance of Design:
Modest.

As noted above, the statement of objectives was not fully clear which also detracted from design relevance. The results framework (PAD, Annex 3, page 28) presents a clear linkage between the activities financed by the project and the outputs and intermediate outcomes which might be expected to lead to achievement of the second objective -- the delivery of applications and services in selected key sectors including better access to information through the use of technology. The financing of connectivity and enhanced infrastructure in energy and telecommunications, as well as the provision of public access points could reasonably be expected to improve the flow of information and the level of services provided to the targeted sectors. The causal chain leading to the achievement of the first objective -- improving the efficiency and effectiveness of some internal processes of the government of Rwanda -- is, however, less clear. The provision of a government portal and intranet links between Ministries, and Provincial and District Offices would enable demonstration of the change in ICT penetration within government, including the extent of the usage of ICT by government employees, and of changes in data availability, as well as whether the internal systems have the capacity to support citizen services. But the final link to demonstrating increased efficiency in internal processes is not specified. It is assumed to be self-evident.


4. Achievement of Objectives (Efficacy) :

(i) To improve the efficiency and effectiveness of some internal process of the Government of Rwanda: Modest.

Outputs:
  • Harmonization of the basic technology by installing Local Area Networks (LANs) in 24 targeted government institutions (such as, Ministries, Districts, and government agencies), slightly below the target of 27.
  • Hardware purchased for the National Data Center (NDC) include: 3 servers, a computing platform called virtualization and an internet bandwidth supply.
  • Features implemented under the eCabinet system include: Administration Portal; Collaboration Portal; Consultation Portal; Agenda Portal; Information Portal; Meeting Portal; and XML Autor.
  • Functions made available under the eParliament system include: Admin Module; Member Module; and Reporting Module.
  • The project supported the Rwanda Development Board (RDB/IT) to conduct the initial study and purchase hardware and Oracle software licenses for DTWMS, a one-stop for eGovernment services.
  • The project supported the purchase of hardware, software licences, and some cost of system implementation of e-forms, an initiative to automate public information and enable on-line application processing.
  • Training was provided to 812 government employees (112 ICT technical staff and 760 end user staff) Cyber security was conducted and a national cyber security framework was established in September 2008.
  • In the case of the government portal, the top navigation page was established, but the portal was under construction to connect to Ministries' websites.

Outcomes:
  • Although the applications financed under the project reportedly reduced the need for paper documentation, streamlined cabinet and parliament sessions and automated work flows, by distributing electronically the agenda, issuing minutes, approvals, and accessing archives, no concrete evidence is offered of increased efficiency and effectiveness of some internal government processes.

(ii) To improve the delivery of services in selected key sectors including better access to information through the use of technology: High.
    Outputs:
    • An agriculture market price information system (eSoko) was launched in June 2009 to provide commodity market prices through mobile phones as well as through a website.
    • A Telemedicine Store and Forward System between King Faisal, Ruhengeri and Kabgayi hospitals was established. Also, internet connectivity to Ruhengeri and Kabgayi hospitals was provided.
    • The creation or redesigning of 94 websites for MDAs was supported.
    • The creation of the following two web based information applications: REMA on-line biodiversity catalogue and Kinyarwanda Digital Library at the National University of Rwanda Library was supported.
    • 18 district telecenters were equipped with a total of 765 PCs, 383 UPS, 18 projectors, 18 scanners and 18 printers. Also, the eContent strategy for the telecenters was developed.
    • Four ICT buses each equipped with 20 laptop computers, internet access, overhead projector, large TV monitor, and power generator were acquired in order to provide opportunity for rural people to become familiar with computers and Internet. Since, the launch in November 2009, ICT buses have covered 33 out of 49 sectors in 4 districts and a total of 1,044 rural people visited the ICT buses.
    • Training sessions through the ICT busses enabled participants to obtain basic computer operation skills (such as, typing, composing emails, and browsing websites).
    Outcomes
    • Provincial and District offices disseminate local news through their websites and receive feedback from citizens.
    • Through the eSoko system, farmers and traders are able to monitor price changes in commodities by mobile phone, assisting them in determining the right time and location to sell their products. The system covers prices of 77 commodities in 50 markets across the country. Total inbound and outbound traffic reached 27,293 text messages since the launch of the system in June, 2009. Also, eSoko won the Technology in Government in Africa (TIGA) award which is considered to be the highest award by the United Nations Economic Commission for Africa in the category of public service delivery to citizens or communities, and other awards according to the local media.
    • Availability of on-line content in local language (Kinyarwanda) enables citizens not speaking a European language to access government information more easily.
    • Websites streamline the immigration process by allowing visitors to obtain visas on-line, and also allow citizens to search and find on-line public information more easily.
    • The Telemedicine system enables doctors in the two regional hospitals to ask for remote diagnosis, treatment and second opinion, and send referrals to the King Faisal Hospital.
    • According to the project team, "the system has been used for regular video conferences among the three hospitals and continuous professional education and knowledge sharing. Based on the promising results of telemedicine driven by the eRwanda project, the Government has been setting up a policy to mandate the use of telemedicine systems in daily diagnosis (starting with radiology and dermatology) (ICR page 38). The eRwanda activities were well recognized by international media and the development community (ICR page 67)."
    • According to the survey results:
        • 80.6 percent of the target population expressed satisfaction with e-services provided by the Government of Rwanda (the target was 100 percent).
        • 74 percent of the target population expressed satisfaction with access to public information supported by the project (again the target was 100 percent).

    5. Efficiency:

    The project closed on time. However, based on the information in the ICR, it is clear that the project was not ready for implementation due to limited PMU capacity in technical, administrative, and operational expertise in ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years. Moreover, the Minister in charge of ICT and the head of Rwandan Information and Technology Authority (RITA) changed 3 times and further delays were caused by the time needed for the new management to understand the scope of the project. Neither the PAD nor the ICR computed the Economic Rate of Return (ERR) or Financial Rate of Return (FRR) of the project. Regarding cost effectiveness analysis, of the five main activities in the project, the ICR/Region's response did not provide any cost analysis on four (eSoko,Telemedicine system, Digital Content creation, Telecenters), but it did for one (ICT buses). On costs related to ICT Buses, the ICR makes a statement on page 21 that the unit costs for the entire training session for participant is estimated at approx. $90 and that is reasonable compared to other similar ICT training (Footnote: Similar ICT skill training for beginners cost is $170 in United States). It is also mentioned in the ICR (page 25) that ICT buses operational costs are relatively high (around $200/month/bus).Efficiency is rated modest.

    a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:


    Rate Available?
    Point Value
    Coverage/Scope*
    Appraisal:
    No
    %
    %
    ICR estimate:
    No
    %
    %

    * Refers to percent of total project cost for which ERR/FRR was calculated

    6. Outcome:

    Relevance of objectives is rated substantial rather than high because of poorly specified objectives. Relevance of design is modest because of a weak linkage between the activities financed and the first development objective. Efficacy of the first objective is rated modest due to lack of evidence of improvements in the effectiveness and efficiency of the government’s internal processes. Efficacy of the project's second objective of contributing to enhanced service delivery through improved public access to information, as well as through special services to key sectors like eSoko and Telemedicine, is rated high. Efficiency is assessed as modest.
    Overall outcome is assessed as Moderately Satisfactory.

    a. Outcome Rating: Moderately Satisfactory

    7. Rationale for Risk to Development Outcome Rating:

    eRwanda is a subset of the NICI II plan which is strongly promoted by the government whose vision is to transform Rwanda from an agriculture-based economy to a knowledge-based economy by investing in ICT. The following transition arrangements were put in place before project closing (ICR page 13): (i) staff involved in implementing the project would continue with the same roles and responsibilities within the government structures to ensure continuous operation and quality of service delivery; (ii) management and maintenance of sector applications and LANs were transferred to the respective ministries and each ministry has allocated a budget for operation and staffing; (iii) the Rwanda Development Board /IT remained the overall IT technical support authority and in charge of managing telecenters and ICT buses; and (iv) UN agencies such as, WFP, FAO and UNECA showed interest in scaling up eSoko to take advantage of the regional market.

    Risk to Development Outcome is, nonetheless, rated as Moderate due to the fact that the sustainability of telecenters and ICT buses would depend on the development of a financially independent and sustainable business model which had not been established at the time of project closing.

    a. Risk to Development Outcome Rating: Moderate

    8. Assessment of Bank Performance:

    a. Quality at entry:
    A quality at entry assessment carried out by the Quality Assurance Group on June 22nd, 2007 assigned an overall rating of Satisfactory to this project. According to this assessment,"the team members possessed skills in ICT and public administration, and by working effectively together, preserved the right balance between these two elements. In addition their sector and country experience were excellent, and drew upon a large staff with an inter sectoral background". The project was prepared rapidly with an elapsed time of about 10 months between concept note and Board approval. Based on the information in the ICR, it is clear that the project was not ready for implementation due to limited PMU capacity in technical, administrative, and operational expertise in ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years.The ICR (page 25) notes that the team could have utilized a Project Preparation Advance to strengthen PMU staff capacity while waiting for the completion of the project ratification process by the Rwandan government prior to effectiveness. Targets for the two user satisfaction indicators (satisfaction of the target population with e-services and satisfaction of the target population with access to public information) were too ambitiously set to 100 percent. Component 3 had to be dropped during the mid-term review because its activities were being funded and managed through another World Bank project (Regional Communication Infrastructure Program). The duplication in effort could have been identified and anticipated at the time of project design.

    Quality-at-Entry Rating: Moderately Satisfactory

    b. Quality of supervision:
    Based on the information in the ICR, it is clear that the project was not ready for implementation due to limited PMU capacity in technical, administrative, and operational expertise in ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years. The ICR (page 26) also points out that there was a change in task management at the time of the mid-term review, but the transition was managed smoothly. The ICR reports that both the project teams demonstrated strong team building and communication skills with staff and stakeholders. On-site support from the procurement specialist and Financial Management specialist, both based in the country office, was critical to the project's completion on schedule. However, the revisions to the results framework and monitoring indicators in the mid-terms review's aide-memoire did not occur until near to the project's closing. Also, there was lack of candor in reporting of ratings during the initial phases in the Implementation Status Reports. Although, there were significant procurement issues during the first two years of implementation, the procurement risk flag was never downgraded to Moderately Unsatisfactory or worse and the implementation progress indicator remained Satisfactory. Also, project implementation was affected during 2007-2008 by the fact that the Task Team and Project Management Unit staff were also working on the preparation of another World Bank project (“Regional Communication Infrastructure Program”). However, the well conducted mid-term review -- which involved several lead experts -- and the intense supervision efforts from the team in the later stages of implementation enabled the project to close on schedule after disbursing 100 percent of its commitments.

    Quality of Supervision Rating: Moderately Satisfactory

    Overall Bank Performance Rating: Moderately Satisfactory

    9. Assessment of Borrower Performance:

    a. Government Performance:
    The government's commitment toward ICT development was strong with support from the President of the Republic. In addition, the government had a clear responsibility matrix with ICT champions in each ministry for the implementation of NICI II. The matrix was also applied to the eRwanda project because it was a subset of NICI II. However, the ratification process in Rwanda's parliament led to later effectiveness than anticipated. Moreover, the Minister in charge of ICT and the head of Rwandan Information and Technology Authority (RITA) changed 3 times and delays were caused by the time needed for the new management to understand the scope of the project. According to the ICR (page 9), these challenges were overcome by the continuous support from the ICT champions in the government and stakeholders in beneficiary agencies.

    Government Performance Rating: Moderately Satisfactory

    b. Implementing Agency Performance:
    Initially, the Rwandan Information and Technology Authority (RITA) was the agency responsible for implementing the project and it housed the Project Management Unit. In 2010, RITA was merged with the Rwanda Development Board (which was established in 2009 and focused on Rwanda's economic development) to became an Information and Technology Unit reporting directly to the Rwanda Development Board's Chief Operating Officer. The project was not ready for implementation due to limited PMU capacity in technical, administrative, and operational expertise in ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years.The difficulties were compounded by the Unit’s involvement, until the mid-term review, in the preparation of another IDA funded project (Regional Communications and Infrastructure Program). Two additional clearance layers for large tenders were added after RITA's merger with the Rwanda Development Board in 2010. Despite the above challenges, according to the ICR (page 28), the Unit staff devoted significant amount of time by closely following up on the clearance process and managed to get procurement back on track.

    Implementing Agency Performance Rating: Moderately Unsatisfactory

    Overall Borrower Performance Rating: Moderately Satisfactory

    10. M&E Design, Implementation, & Utilization:

    a. M&E Design:
    The M&E indicators in the PAD were measurable and included target values, frequency of reporting, data collection instruments and entities responsible for data collection. However, the results framework in the PAD provided only one indicator (Number of targeted government institutions which demonstrated improved internal efficiency in the delivery of services and information to citizens) to measure the progress of the first project development objective. Moreover, this indicator was never tracked in the Implementation Status Reports until the mid-term review. No baseline values for the indicators were established. The PAD (page 12) mentioned that a baseline survey for the ICT industry was initiated during project preparation and would be repeated two years after effectiveness. The PAD (page 12) stated that external evaluations (including a quality assurance mechanism) and impact assessments would be carried out on a regular basis to address progress and impact of eGoverment reforms. Targets for the two outcome indicators were over ambitiously set to 100%. The Kigali Institute of Science and Technology was responsible for monitoring the NICI II Plan and the eRwanda project.

    b. M&E Implementation:
    The following revisions were made to the M&E indicators during the mid-term review: two new development indicators (Percentage of target population expressing satisfaction with supported e-services provided by the government and percentage of target population expressing satisfaction with access to public information provided by the project supported channels) were added to track the progress of the second development objective; four new indicators were added to track the progress of component 1; two to track component 2; all three indicators under component 3 were dropped; and both the indicators under component 4 were dropped. After the mid-term review, the linkages between components, indicators and objectives of the project were clearer. A baseline survey was not conducted until October 2008, and even this did not establish baselines for the indicators reported in the ICR. Two out of three development outcome indicators and five out of the eleven intermediate outcome indicators were based upon the survey results conducted on October 2010 by the eRwanda team in partnership with the Kigali Institute of Science and Technology. On the basis of these results, it was possible to measure the user satisfaction level for eServices provided by the government of Rwanda and for access to public information provided through project supported channels.The ICR does not provide any information about the external evaluations (including a quality assurance mechanism) and impact assessments that were mentioned in the PAD.

    a. M&E Utilization:
    No information is provided in the ICR concerning the utilization of the project’s M&E system.

    M&E Quality Rating: Modest

    11. Other Issues:

    a. Safeguards:
    This Category “C” project triggered no safeguards policies.

    b. Fiduciary Compliance:
    Procurement

    According to the ICR (page 12), the project manager in the Implementation Unit was not fully delegated the authority to approve procurement contracts and therefore had to get clearances from both the Rwanda Information Technology Authority’s (RITA's) senior management as well as from the Rwanda Public Procurement Authority. Initially, it took five stages (3-5 weeks) of internal approval to award a contract which affected project implementation (ICR page 18). Matters improved after RITA was merged with the Rwanda Development Board and internal clearance processes were simplified for small procurements. However, two extra clearance layers were added for large tenders. Dual language requirement for tenders (English and French) also added an extra administrative layer to precede the selection process because technical specifications had to be published and received in the two languages (ICR page 12).

    Financial Management

    There was a system flaw which prevented the production of accounting reports, due to which the financial management officer had to reformat the financial reports manually. This issue was resolved after RITA had merged into the Rwanda Development Board. According to the project team, all the recommendations of the Financial Management Supervision report were implemented. Financial Management was rated satisfactory after the implementation was completed.
    The following issues were identified in the Financial Management Supervision report of July 2009: lack of a surprise petty cash count as of 31st December 2008; failure to deduct withholding taxes from consultant's fees; and lack of a physical asset inventory.
    The ICR contains no information on external project audits. The project team subsequently reported that the audits were unqualified.

    c. Unintended Impacts (positive or negative):

    d. Other:
    The project promoted female participation by mandating 30 percent participation rate of women in every session of IT training in ICT, and by providing basic computer literacy training to approximately 700 women from the National Women’s Council members, the nationwide women’s representative organization. Also, the Ministry of Gender and Family Promotions was an active promoter of telecenters for adolescent girls to acquire ICT skills and knowledge (ICR page 23).



    12. Ratings:

    ICR
    IEG Review
    Reason for Disagreement/Comments
    Outcome:
    Satisfactory
    Moderately Satisfactory
    Relevance of objective was substantial, but that of design was modest because of the statement of objectives was not specific concerning the scope of the project's activities, and a weak linkage between the activities financed and the first development objective. Efficacy of the first objective is rated modest due to lack of evidence of improvements in the effectiveness and efficiency of the government’s internal processes. Efficacy of the project's second objective of contributing to enhanced service delivery through improved public access to information, as well as through special services to key sectors like eSoko and Telemedicine, is rated high. Efficiency is assessed as modest. 
    Risk to Development Outcome:
    Moderate
    Moderate
     
    Bank Performance:
    Satisfactory
    Moderately Satisfactory
    Quality at Entry is rated Moderately Satisfactory because the targets for the two user satisfaction indicators were too ambitious and the fact that the activities under component 3 were to be funded and managed through another World Bank project was not anticipated during project design. 
    Borrower Performance:
    Satisfactory
    Moderately Satisfactory
    Implementation delays were mainly due to: frequent changes of the Minister in charge of ICT and the head of RITA; and the PMU's limited capacity in administrative technical and operational issues and knowledge of Bank procurement process and ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years. The Unit's attention was also distracted by its involvement until the mid-term review in the preparation of another IDA funded project. 
    Quality of ICR:
     
    Unsatisfactory
     
    NOTES:
    - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
    - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

    13. Lessons:
    The following are key lessons emerging from the information in the ICR:
    • Targets for output and outcome indicators should be realistic: In the case of this project, targets for the two user satisfaction indicators (satisfaction of the target population with e-services and satisfaction of the target population with access to public information) that were used to measure the progress of the project's second development objective were over-ambitiously set to 100 percent.
    • Readiness for implementation must be assured to avoid later inefficiencies: Since, procurement in the ICT sector frequently makes special demands in terms of knowledge and experience, the task team needs to ensure that the national counterparts are well trained and ready for project implementation. The eRwanda project was not ready for implementation due to limited capacity in technical, administrative, and operational expertise in ICT procurement which became a major challenge during the first half of implementation resulting in significant disbursement delays in the first two years of implementation.
    • Frequent changes in the senior management staff within the implementing agency can result in delays during implementation. Efforts should be made to retain an institutional memory in order to avoid delays during project implementation.

    14. Assessment Recommended?

    Yes
    Why?
    To verify the ratings and document lessons, including those relating to innovations supported by the project.

    15. Comments on Quality of ICR:

    The ICR presents limited evidence with which to assess achievement of the project's first development objective of improved efficiency and effectiveness of some internal government processes; the ICR's discussion focuses predominantly on outputs. It merely reports on the deficient PDO indicators without evaluating the outcomes embedded in the statement of objectives. Regarding project efficiency, Bank guidelines require that 'value for money' is measured even for capacity building projects. In this case, the ICR should have conducted some quantitative efficiency analysis for at least some of the activities and cost-effectiveness/unit cost analysis for others (beyond training through ICT buses which the ICR did). There are discrepancies in the estimated project cost at appraisal reported in PAD and in the ICR. The project information page in the PAD shows an estimated project cost of USD 10 million, whereas, the Annex 1 in the ICR (page 33) shows an estimated project cost at Appraisal of USD 9.6 million. Subsequently, the project team clarified that the discrepancies were due to exchange rate variations. Although the project team stated that external audits were conducted, the ICR did not discuss them.

    a. Quality of ICR Rating: Unsatisfactory

    (ICRR-Rev6INV-Jun-2011)
    © 2012 The World Bank Group, All Rights Reserved. Terms and Conditions