United States General Accounting Office GAO Report to Congressional Requesters September 1996 INFORMATION TECHNOLOGY INVESTMENT Agencies Can Improve Performance Reduce Costs and Minimize Risks G A O years 1921 - 1996 GAO AIMD-96-64 GAO United States General Accounting Office Washington D C 20548 Accounting and Information Management Division B-271399 September 30 1996 The Honorable William S Cohen Chairman Subcommittee on Oversight of Government Management and the District of Columbia Committee on Governmental Affairs United States Senate The Honorable William F Clinger Jr Chairman The Honorable Cardiss Collins Ranking Minority Member Committee on Government Reform and Oversight House of Representatives For some time now you have expressed concern about the large amount of money spent annually on information technology IT by federal agencies and whether agencies have processes in place to ensure it is being spent on the right projects and is producing meaningful results This report responds to your request that we assess the IT investment practices of a small group of federal agencies and compare them to those used by leading private and public sector organizations This report also highlights the implications of our findings for the Office of Management and Budget OMB as it responds to investment management requirements of the recent Information Technology Management Reform Act We are sending copies of this report to the Secretaries of Commerce and Transportation the Administrators of the National Aeronautics and Space Administration and Environmental Protection Agency the Commissioner of Internal Revenue the Director of the Office of Management and Budget the Ranking Minority Member of the Subcommittee on Oversight of Government Management and the District of Columbia Senate Committee on Governmental Affairs the Chairmen and Ranking Minority Members of the cognizant subcommittees of the Senate and House Appropriations Committees and other interested congressional committees We will also make copies available to others upon request Please contact me at 202 512-6406 if you have any questions about this report Christopher W Hoenig Director Information Resources Management Policies and Issues Executive Summary Purpose In recent years the Congress and the public have increased their demand for a smaller government that provides improved services at a lower cost To keep pace with rising expectations the federal government must focus on dramatically improving operations Such improvement will require strengthened management of three fundamental assets—personnel knowledge and information and capital property fixed assets Investments in information technology IT can have a dramatic impact on all three of these assets However an IT project’s impact comes from how the investment is selected designed and implemented not from the amount of money that is spent In this age of constrained resources federal executives must find ways to spend more wisely not faster The management of IT projects however has long been a significant problem for many federal agencies The federal government obligated more than $23 5 billion towards IT products and services in fiscal year 1994—about 5 percent of the government’s total discretionary spending Yet the impact of this spending on improving agency operations and service delivery has been mixed at best Federal information systems often cost millions more than expected take longer to complete than anticipated and fail to produce significant improvements in the speed quality or cost of federal programs Some private and public sector organizations on the other hand have achieved significant performance improvements by managing their IT resources within an overall management framework that aligns technology with business needs and priorities In a 1994 report 1 GAO identified 11 fundamental management practices found in leading organizations that led to short- and long-term performance improvements One key practice identified by this research was the management of IT projects as investments By following this practice the organizations minimize risks and maximize returns on those IT projects that have the best chance of significantly improving organizational performance In order to better understand how federal managers can reduce risks control costs and use technology to improve performance you asked GAO to compare and contrast the investment management practices and decision processes used by leading private and public sector organizations with a small group of federal agencies Specifically this report compares the IT investment management practices of leading organizations with IT management activities at five agencies—the National Aeronautics and 1 Executive Guide Improving Mission Performance Through Strategic Information Management and Technology GAO AIMD-94-115 May 1994 Page 2 GAO AIMD-96-64 IT Investment Executive Summary Space Administration NASA National Oceanic and Atmospheric Administration NOAA Environmental Protection Agency EPA Coast Guard and Internal Revenue Service IRS The recent passage of the Information Technology Management Reform Act ITMRA which became effective on August 8 of this year introduces new requirements for how IT-related projects will be selected and managed These requirements closely parallel the investment practices followed by leading organizations Though agencies have the primary responsibility for leading the change effort OMB’s specific responsibilities under the act as well as its central oversight role make it a pivotal player at this early stage of implementation As a result this report also examines the challenges and opportunities presented to OMB as it supports and oversees agencies’ efforts to improve IT management and performance Background In May 1994 GAO issued a report based on its work analyzing the information management practices of several leading private and state organizations One of the key practices that was identified was that leading organizations use disciplined processes to manage IT projects as investments rather than as one-time expenditures In general the leading organizations GAO studied follow a three-phased management approach for selecting controlling and evaluating IT-related projects They assess all IT projects—proposed under development and operational—and then prioritize and make funding decisions based on several factors including cost risk and return as well as how well the project meets mission needs Once selected executives monitor the projects throughout their life cycle taking quick actions to mitigate effects of changes in risks and costs to ensure that the investments are providing expected benefits And after a project has been implemented the organizations evaluate actual versus expected results for the project and revise their investment management process based on the lessons learned Over the years the Congress has passed legislation—the Chief Financial Officers CFO Act the Government Performance and Results Act GPRA as well as revisions to the Paperwork Reduction Act PRA —that enhances agencies’ responsibility and accountability for managing projects for results recognizes the value of effectively managing IT projects and emphasizes maintaining reliable accurate financial cost data In addition ITMRA directs agencies to use a comprehensive capital planning and investment approach for maximizing the value and assessing and Page 3 GAO AIMD-96-64 IT Investment Executive Summary managing the risks of IT projects By eliminating the oversight role of the General Services Administration GSA accountability for IT projects has been placed squarely with the individual agencies In addition OMB’s role in overseeing federal agencies’ selection and management of IT has been significantly heightened with the passage of ITMRA The Director of OMB is now responsible for promoting and directing that federal agencies establish capital planning processes for IT investment decisions The Director is also responsible for evaluating the results of agency IT investments and enforcing accountability through the budget process To its credit OMB has taken a proactive role in drafting new policies and procedures to assist agencies in establishing IT investment decision-making approaches For example in November 1995 OMB published a guide designed to assist agency and OMB staff in creating and evaluating a portfolio of IT investments 2 In addition OMB is currently working on revisions to its key management circular regarding strategic information resources management planning its budget submission guidance to federal agencies and its planning guidance for the acquisition of fixed capital assets 3 In examining the IT decision processes at the five case study agencies GAO found elements of an investment approach embedded in some of the agencies’ existing decision-making policies and procedures Among the elements that GAO found were Results in Brief • • • • project funding decision-making processes that used explicit decision criteria to evaluate risks and returns processes to prioritize IT projects in alignment with key strategic mission goals attempts to integrate IT funding decisions with overall strategic business planning and direction and central management processes in place that included both line managers and IT professionals 2 Evaluating Information Technology Investments A Practical Guide Executive Office of the President Office of Management and Budget November 1995 3 Specifically revisions are being drafted for Circular A-130 Management of Federal Information Resources and Circular A-11 Preparation and Submission of Budget Estimates In addition Bulletin 95-03 Planning and Budgeting for the Acquisition of Fixed Assets has been replaced with Circular A-11 Part 3 Planning Budgeting and Acquisition of Fixed Assets Page 4 GAO AIMD-96-64 IT Investment Executive Summary However GAO also found four cross-cutting weaknesses that prevented the agencies from having a complete institutionalized process that would fulfill the intent of PRA and ITMRA While all four weaknesses may not have been present at each agency in general GAO found that the agencies • • • • lacked uniformity in their internal processes for selecting and managing systems investments focused their selection processes on justifying new project funding rather than managing all IT projects as a portfolio of competing investments made funding decisions without giving adequate attention to management control or evaluation processes and made funding decisions using undefined decision criteria and often without up-to-date or accurate cost benefit and risk data to support their IT investment decisions With a complete investment process agencies can gain better control of their IT budgets increase the odds of operational improvements and reduce risks Conversely without one agencies increase the chance of becoming entrapped in a host of difficult problems such as unmanaged development risks higher failure rates low-value or redundant projects and an overemphasis on maintaining old systems at the expense of using technology to redesign outmoded work processes Principal Findings Agencies Need Consistent Processes to Select and Manage IT Projects Leading organizations use selection control and evaluation processes uniformly at an enterprise level and within each business unit of the organization This enables an organization even one that is highly decentralized to systematically identify cross-functional system opportunities and to determine trade-offs between projects both within and across business units By contrast there was little or no uniformity in how risks benefits and costs of various IT projects were evaluated across subunits within the case study agencies Three of the agencies—NASA EPA and NOAA—chose IT projects based on inconsistent or nonexistent investment processes Thus making cross-comparisons between systems of different size or organizational impact was difficult at best More important management Page 5 GAO AIMD-96-64 IT Investment Executive Summary had no assurance that the most important mission objectives of the agency were being met by the suite of system investments that was selected Agencies Need to Manage Their IT Projects as an Investment Portfolio In conducting their selection processes leading organizations assess and manage the different types of IT projects in order to create a complete strategic investment portfolio By analyzing the entire portfolio managers examine the costs of maintaining existing systems versus investing in new ones comparatively rank projects based on expected net returns and can reach decisions based on overall contribution to the most pressing organizational needs This portfolio process is analogous to the capital planning and budgeting process frequently used in both public and private sector organizations At the federal agencies GAO studied some prioritization of projects was conducted but none made managerial trade-offs across all types of projects NOAA and the Coast Guard for instance conducted portfolio analyses but these analyses focused primarily on new or under-development projects and did not consider spending for operations and maintenance enhancements or research and development IRS only included Tax Systems Modernization TSM projects that were under development in its investment portfolio Of all the agencies that were reviewed the Coast Guard had the most comprehensive selection phase However this selection process was still incomplete because it did not include all types of proposed IT spending Consequently the Coast Guard could not make trade-offs between all types of IT investments creating a risk of implementing new systems that duplicate existing systems or of uneconomically maintaining old systems beyond their life cycle Management Control and Evaluation Processes Were Often Absent Once selection has occurred leading organizations continue to manage their investments maintaining a cycle of continual control and evaluation This enables senior executives to 1 identify and focus on managing high-potential or high-risk projects 2 reevaluate investment decisions early in a project’s life cycle if problems arise 3 be responsive to changing external and internal conditions in mission priorities and budgets and 4 learn from past successes and failures in order to make better decisions in the future This focus on evaluating project performance in terms of actual results and mission impact is also consistent with legislative provisions contained in GPRA Page 6 GAO AIMD-96-64 IT Investment Executive Summary Control mechanisms in the five case study agencies were driven primarily by cost and schedule concerns with little focus on quantitative outcome-based performance measures Two of the reviewed agencies—the Coast Guard and EPA—did not use management control processes that focused on IT systems projects The other three agencies had management control processes that focused primarily on schedule and cost concerns but not interim evaluations of performance and results Rarely did GAO find examples in which anticipated benefits were compared to results at critical project milestones Postimplementation reviews PIR of actual versus projected returns were rarely conducted Four of the five federal agencies did not systematically evaluate implemented IT projects to determine actual costs benefits and risks and information and lessons learned in either the control or evaluation phases were not fed back in to the selection phase to improve the project selection process IRS had developed a PIR methodology that was used to conduct five systems postimplementation reviews However the PIR methodology had not been integrated into a cohesive investment process As a result PIRs that were conducted did not meet one of their primary objectives—to ensure continual improvement based on lessons learned—and IRS runs the risk of repeating past mistakes Agency IT Decisions Were Not Based on Adequate Data To help make high-quality decisions on IT investments leading organizations require all projects to have accurate complete and up-to-date project information This information which includes cost and benefit data risk assessments implementation plans and performance measures is used as the basis for decision-making on project selections ongoing monitoring activities and evaluation of completed projects In the federal government sound financial management and cost data are a cornerstone requirement of the CFO Act and are critical to making informed decisions under the performance management approach required by GPRA The agency IT investment decisions GAO examined were largely based on undefined or implicit decision criteria Of the five federal agencies only the Coast Guard had defined decision criteria for cost risk and return IRS had recently begun identifying and using decision criteria but these criteria were not yet complete and there was no evidence that IRS decisions were based on acceptable data on project costs benefits and risks Generally officials in the other agencies stated that they determined which projects to fund based on the judgmental expertise of Page 7 GAO AIMD-96-64 IT Investment Executive Summary decisionmakers involved in the process Also data on a project’s cost schedule risks and returns were not documented defined or kept up-to-date and in many cases were not used to make investment decisions While the agencies conducted analyses to get projects approved little effort was made to ensure that the information was kept accurate and up-to-date and rarely were the data used to manage a project throughout its life cycle Recommendations Maximizing the returns and minimizing the risks on the billions of dollars that are spent each year for IT will require continued efforts on two fronts First federal agencies must develop and implement a structured IT investment approach that encompasses all aspects of the investment process Second oversight attention far beyond current levels must be given to agencies’ management processes and rigorous analysis of actual results Given the critical policy development and oversight role prescribed to it by ITMRA OMB has a significant leadership responsibility in supporting agencies’ efforts In meeting this responsibility GAO is recommending that OMB address four critical challenges OMB’s first challenge is to guide and assist agencies as they establish and improve their IT investment management processes GAO recommends that OMB develop guidance requiring agencies to 1 implement IT investment decision-making processes 2 periodically analyze their entire portfolio of IT investments 3 design control and evaluation processes that include cost schedule and quantitative performance measures and 4 set minimum quality standards for data used to assess cost benefit and risk decisions Second OMB will need to use the results produced by the improved investment processes to develop recommendations for the President’s budget that reflect an agency’s actual track record in delivering mission performance for IT funds expended Third to ensure that the improved investment management processes are effectively implemented and to make the appropriate linkages between agency track records and budget recommendations OMB will need to marshal the resources and skills necessary to be able to make sound investment decisions on agency portfolios Finally GAO recommends that as part of its internal implementation strategy the Director of OMB should consider developing an approach to Page 8 GAO AIMD-96-64 IT Investment Executive Summary assess OMB’s own performance in executing its oversight responsibilities under ITMRA’s capital planning and investment provisions Such a process could focus on whether and how OMB reviews of agency processes and results have an impact on reducing risk or increasing the returns on information technology investments—both within and across federal agencies Agency Comments OMB’s comments on a draft of this report are presented and evaluated in chapter 3 and are reprinted in appendix I OMB stated that the analysis contained in the report made a positive contribution towards understanding the critical elements that agencies will need to have in place to implement the investment and capital planning provisions of the Information Technology Management Reform Act of 1996 OMB also generally supported the report’s findings and four sets of recommendations and said that it would be implementing many aspects of the recommendations as part of the fiscal year 1998 budget review process of fixed capital assets However OMB did state that rather than evaluating each agency’s IT processes and ensuring compliance with OMB guidance it prefers to focus on the results that are occurring from IT investments GAO agrees with this results-oriented approach provided that OMB maintains its key role in supporting and selectively evaluating agency practices as required by statute Improved IT results which are the intent of ITMRA will not occur without agencies developing and implementing discliplined investment processes and practices Comments from EPA NASA NOAA IRS and the Coast Guard have been incorporated in the report where appropriate and a summary of their comments as well as GAO’s response is included at the end of chapter 2 All of the agencies generally agreed with the report’s findings although in some cases they stated that the report did not appropriately recognize recent progress they had made and did not adequately address the inherent difficulties in implementing the investment requirements of ITMRA In updating the agency information GAO allowed the five agencies to provide additional information reflecting changes and modifications they had made in preparation for implementing provisions of ITMRA This updated information has been incorporated into the report where appropriate However many of the process changes and modifications have occurred very recently and it is not yet practical to fully evaluate these changes or determine their effects Page 9 GAO AIMD-96-64 IT Investment Executive Summary In addition while all of the agencies agreed in principle with the IT investment approach several raised concerns regarding how the investment process would work in different organizational environments GAO recognizes that IT investment decision-making must be adaptable to different agency environments and believes that the approach outlined in the report provides such flexibility GAO also agrees that implementing a mature investment decision-making approach is a complex undertaking and will take several years to implement fully Page 10 GAO AIMD-96-64 IT Investment Page 11 GAO AIMD-96-64 IT Investment Contents Executive Summary 2 Chapter 1 Introduction Objectives Scope and Methodology Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Agencies Need Consistent Processes to Select and Manage IT Investments Agencies Need to Manage Their IT Projects as a Portfolio Management Control and Evaluation Processes Were Often Absent Agency IT Decisions Are Not Based on Adequate Data Agency Comments and Our Evaluation Chapter 3 Conclusions and Recommendations Appendixes Tables 16 20 24 25 29 32 38 42 44 44 45 47 Conclusions Recommendations Agency Comments and Our Evaluation Appendix I Comments From the Office of Management and Budget and Our Evaluation Appendix II Agency Investment Technology Profiles Appendix III Description of an Information Technology Investment Process Approach Appendix IV Brief Description of Agency IT Management Processes Appendix V Summary of Investment-Related Provisions of the Information Technology Management Reform Act of 1996 Appendix VI Major Contributors to This Report 50 Table II 1 Total Agency Budget and IT Spending for Fiscal Year 1990 Through Fiscal Year 1994 Table II 2 Agency Total and IT Staffing for Fiscal Year 1990 Through Fiscal Year 1994 Table II 3 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at the Coast Guard Table II 4 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at EPA 53 Page 12 53 62 68 73 75 54 55 56 GAO AIMD-96-64 IT Investment Contents Figures Table II 5 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at IRS Table II 6 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at NASA Table II 7 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at NOAA 57 Figure 1 1 Fundamental Strategic Information Management Practices Figure 1 2 An IT Investment Approach Used in Leading Organizations Figure 2 1 The IT Investment Process Is Uniform Throughout an Organization Figure 2 2 A Comprehensive Approach Includes All Major IT Projects Figure 2 3 IT Investment Is a Continuous and Dynamic Process Figure 2 4 Consistent Well-defined and Up-to-date Data Are Essential Throughout the IT Investment Process Figure III 1 An IT Investment Approach Used in Leading Organizations 17 Page 13 59 60 19 26 30 33 39 63 GAO AIMD-96-64 IT Investment Contents Abbreviations ASOS CFOL CFO CIO EPA ESC GIS GOES-Next GPRA GSA FTE FTS FY IG IRM IRS IT ITMRA NASA NEXRAD NOAA NWS OMB Op PCM PIR PMC PRA R D ROI SCRIPS TSM U D Page 14 Automated Surface Observing System Corporate Files On-Line Chief Financial Officer Chief Information Officer Environmental Protection Agency Executive Steering Committee geographic information system Next Generation Geostationary Operational Environmental Satellite Government Performance and Results Act General Services Administration full-time equivalent Federal Telecommunications System fiscal year Inspector General information resources management Internal Revenue Service information technology Information Technology Management Reform Act National Aeronautics and Space Administration Next Generation Weather Radar National Oceanic and Atmospheric Administration National Weather Service Office of Management and Budget operational program control meeting postimplementation review Program Management Council Paperwork Reduction Act research and development return on investment Service Center Recognition Image Processing System Tax Systems Modernization under development GAO AIMD-96-64 IT Investment Page 15 GAO AIMD-96-64 IT Investment Chapter 1 Introduction There is an increasing demand coming from the Congress and the public for a smaller government that works better and costs less Having valuable accurate and accessible financial and programmatic information is a critical element for any improvement effort to succeed Furthermore increasing the quality and speed of service delivery while reducing costs will require the government to make significant investments in three fundamental assets—personnel knowledge and capital property fixed assets Investments in information technology IT projects can dramatically affect all three of these assets Indeed the government’s ability to improve performance and reduce costs in the information age will depend to a large degree on how well it selects and uses information systems investments to modernize its often outdated operations However the impact of information technology is not necessarily dependent on the amount of money spent but rather on how the investments are selected and managed This in essence is the challenge facing federal executives Increasing the return on money spent on IT projects by spending money wiser not faster projects however are often poorly managed For example one market research group estimates that about a third of all U S IT projects are canceled at a estimated cost in 1995 of over $81 billion 1 In the last 12 years the federal government has obligated at least $200 billion for information management with mixed results at best Yet despite this huge investment government operations continue to be hampered by inaccurate data and inadequate systems Too often IT projects cost much more and produce much less than what was originally envisioned Even worse often these systems do not significantly improve mission performance or they provide only a fraction of the expected benefits Of 18 major federal agencies 7 have an IT effort that has been identified as high risk by either the Office of Management and Budget OMB or us 2 IT Some private and public sector organizations on the other hand have designed and managed IT to improve their organizational performance In a 1994 report we analyzed the information management practices of several leading private and state organizations 3 These leading organizations were 1 Charting the Seas of Information Technology Chaos The Standish Group International Inc 1994 2 See Information Technology Investment A Governmentwide Overview GAO AIMD-95-208 July 31 1995 3 Executive Guide Improving Mission Performance Through Strategic Information Management and Technology GAO AIMD-94-115 May 1994 Page 16 GAO AIMD-96-64 IT Investment Chapter 1 Introduction identified as such by their peers and independent researchers because of their progress in managing information to improve service quality reduce costs and increase work force productivity and effectiveness From this analysis we derived 11 fundamental IT management practices that when taken together provide the basis for the successful outcomes that we found in leading organizations See figure 1 1 Figure 1 1 Fundamental Strategic Information Management Practices Decide to Change Communicate the urgency to change IT practices Get line management involved create ownership Take action and maintain momentum Direct Change Define customer needs and mission goals Measure the performance of mission processes Focus on process improvement Manage IT projects as investments Integrate planning budgeting and evaluation Support Change Establish customer supplier relationships between managers and IT professionals Create a Chief Information Officer Upgrade IT skills and knowledge Page 17 GAO AIMD-96-64 IT Investment Chapter 1 Introduction One of the best practices exhibited by leading organizations was that they manage information systems projects as investments 4 This particular practice offers organizations great potential for gaining better control over their IT expenditures In the short term within 2 years this practice serves as a powerful tool for carefully managing and controlling IT expenditures and better understanding the explicit costs and projected returns for each IT project In the long term from 3 to 5 years this practice serves as an effective process for linking IT projects to organizational goals and objectives However managing IT projects as investments works most effectively when implemented as part of an integrated set of management practices For example project management systems must also be in place reengineering improvements analyzed and planning processes linked to mission goals While the specific processes used to implement an investment approach may vary depending upon the structure of the organization e g centralized versus decentralized operations we nonetheless found that the leading organizations we studied shared several common management practices related to the strategic use of information and information technologies Specifically they maintained a decision-making process consisting of three phases—selection control and evaluation—designed to minimize risks and maximize return on investment See figure 1 2 4 IT investment is defined as an expenditure of money and or resources for IT or IT-related products and services involving managerial technical and organizational risk for which there are expected benefits to the organization’s performance These benefits are defined as improvements either in efficiency of operations or effectiveness in services such as reductions in process cycle time or operational costs increases in speed or quality of customer service or improvements in productivity Page 18 GAO AIMD-96-64 IT Investment Chapter 1 Introduction Figure 1 2 An IT Investment Approach Used in Leading Organizations Control What are you doing to ensure that the projects will deliver the benefits projected Select How do you know you have selected the best projects Evaluate Based on your evaluation did the systems deliver what you expected Process Information The Congress has passed several pieces of legislation that lay the groundwork for agencies to establish a investment approach for managing IT For instance revisions to the Paperwork Reduction Act PRA Public Law 104-13 have put more emphasis on evaluating the operational merits of information technology projects The Chief Financial Officers CFO Act Public Law 101-576 focuses on the need to significantly improve financial management and reporting practices of the federal government Having accurate financial data is critical to establishing performance measures and assessing the returns on IT investments Finally the Page 19 GAO AIMD-96-64 IT Investment Chapter 1 Introduction Government Performance and Results Act GPRA Public Law 103-62 requires agencies to set results-oriented goals measure performance and report on their accomplishments In addition the recently passed Information Technology Management Reform Act ITMRA Division E of Public Law 104-106 requires federal agencies to focus more on the results achieved through IT investments while streamlining the federal IT procurement process Specifically this act which became effective August 8 of this year introduces much more rigor and structure into how agencies approach the selection and management of IT projects Among other things the head of each agency is required to implement a process for maximizing the value and assessing and managing the risks of the agency’s IT acquisitions Appendix V summarizes the primary IT investment provisions contained in ITMRA also heightens the role of OMB in supporting and overseeing agencies’ management activities The Director of OMB is now responsible for promoting and directing that federal agencies establish capital planning processes for IT investment decisions The Director is also responsible for evaluating the results of agency IT investments and enforcing accountability The results of these decisions will be used to develop recommendations for the President’s budget ITMRA IT OMB has begun to take action in these areas In November 1995 OMB with substantial input from GAO published a guide designed to help federal agencies systematically manage and evaluate their IT-related investments 5 This guide was based on the investment processes found at the leading organizations Recent revisions to OMB Circular A-130 on federal information resources management have also placed greater emphasis on managing information system projects as investments And the recently issued Part 3 of OMB Circular A-11 which replaced OMB Bulletin 95-03 “Planning and Budgeting for the Acquisition of Fixed Assets ” provides additional guidance and information requirements for major fixed asset acquisitions Objectives Scope and Methodology The Chairman Senate Subcommittee on Oversight of Government Management and the District of Columbia Committee on Governmental Affairs and the Chairman and Ranking Minority Member House Committee on Government Reform and Oversight requested that we 5 Evaluating Information Technology Investments A Practical Guide Executive Office of the President Office of Management and Budget November 1995 Page 20 GAO AIMD-96-64 IT Investment Chapter 1 Introduction compare and contrast the management practices and decision processes used by leading organizations with a small sample of federal agencies The process used by leading organizations is embodied in OMB’s Evaluating Information Technology Investments A Practical Guide and specific provisions contained in the Information Technology Management Reform Act of 1996 The agencies we examined are the National Aeronautics and Space Administration NASA $1 6 billion spent on IT in FY 1994 National Oceanic and Atmospheric Administration NOAA $296 million spent on IT in FY 1994 Environmental Protection Agency EPA $302 million spent on IT in FY 1994 Coast Guard $157 million spent on IT in FY 1994 and the Internal Revenue Service IRS $1 3 billion spent on IT in FY 1994 We selected the federal agencies for our sample based on one or more of the following characteristics 1 large IT budgets 2 expected IT expenditure growth rates and 3 programmatic risk as assessed by GAO and OMB In addition the Coast Guard was selected because of its progress in implementing an investment process Collectively these agencies spent about $3 7 billion on IT in FY 1994—16 percent of the total spent on IT Our review focused exclusively on how well these five agencies manage information technology as investments one of the 11 practices used by leading organizations to improve mission performance as described in our best practices report 6 As such our evaluation only focused on policies and practices used at the agencywide level we did not evaluate the agencies’ performance in the 10 other practices In addition we did not systematically examine the overall IT track records of each agency During our review of agency IT investment decision-making processes we did the following • • • reviewed agencies’ policies practices and procedures for managing IT investments interviewed senior executives program managers and IRM professionals and determined whether agencies followed practices similar to those used by leading organizations to manage information systems projects as investments We developed the attributes needed to manage information systems projects as investments from the Paperwork Reduction Act the Federal Acquisition Streamlining Act OMB Circular A-130 GAO’s “best practices” report on strategic information management GAO’s strategic information 6 GAO AIMD-94-115 May 1994 Page 21 GAO AIMD-96-64 IT Investment Chapter 1 Introduction management toolkit and OMB’s guide Evaluating Information Technology Investments A Practical Guide Many of the characteristics of this investment approach are contained in the Information Technology Management Reform Act of 1996 as summarized in appendix V However this law was not in effect at the time of our review To identify effects associated with the presence or absence of investment controls we reviewed agencies’ reports and documents related GAO and Inspector General reports and other external reports We also discussed the impact of the agencies’ investment controls with senior executives program managers and IRM professionals to get an agencywide perspective on the controls used to manage IT investments Additionally we reviewed agency documentation dealing with IT selection budgetary development and IT project reviews To determine how much each agency spent on information technology we asked each agency for information on spending staffing and their 10 largest IT systems and projects The agencies used a variety of sources for the same data elements which may make comparisons among agencies unreliable While data submitted by the agencies were validated by agency officials we did not independently verify the accuracy of the data Most of our work was conducted at agencies’ headquarters in Washington D C Similarly we visited NOAA offices in Rockville Maryland and the National Weather Service in Silver Spring Maryland We also visited NASA program financial and IRM officials at Johnson Space Center in Houston Texas and Ames Research Center in San Francisco California to learn how they implement NASA policy on IT management We performed the majority of our work from April 1995 through September 1995 with selected updates through July 1996 in accordance with generally accepted government auditing standards We updated our analyses of IRS and NASA in conjunction with other related audit work 7 In addition several of the agencies provided us with updated information as part of their comments on a draft version of the report Many of these changes have only recently occurred and we have not fully evaluated them to determine their effect on the agency’s IT investment process 7 Tax Systems Modernization Management and Technical Weaknesses Must Be Overcome to Achieve Success GAO T-AIMD-96-75 March 26 1996 Tax Systems Modernization Actions Underway but IRS Has Not Yet Corrected Management and Technical Weaknesses GAO AIMD-96-106 June 7 1996 NASA Chief Information Officer Opportunities to Strengthen Information Resources Management GAO AIMD-96-78 August 15 1996 Page 22 GAO AIMD-96-64 IT Investment Chapter 1 Introduction We provided and discussed a draft of this report with officials from OMB EPA NASA NOAA IRS and the Coast Guard and have incorporated their comments where appropriate OMB’s written comments as well as our evaluation are provided in appendix I Appendix II profiles each agency’s IT spending personnel and major projects Appendix III provides a brief description of an IT investment process approach based on work by GAO and OMB Appendix IV provides a brief overview of each agency’s IT management processes Because of its relevance to this report the investment provisions of the Information Technology Management Reform Act of 1996 are summarized in appendix V Major contributors to this report are listed in appendix VI Page 23 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain All of the agencies we studied—NASA IRS the Coast Guard NOAA and EPA—had at least elements or portions of an IT investment process in place For instance • • • • the Coast Guard had a selection process with decision criteria that included an analysis of cost risk and return data EPA had created a executive management group to address cross-agency IT issues NASA and NOAA utilized program control meetings to ensure senior management involvement in monitoring the progress of important ongoing IT projects and IRS had developed a systems investment evaluation review methodology and used it to conduct postimplementation reviews of some Tax Systems Modernization projects However none of these five agencies had implemented a complete institutionalized investment approach that would fulfill requirements of PRA and ITMRA Consequently IT decision-making at these agencies was often inconsistent or based on the priorities of individual units rather than the organization as a whole Additionally cost-benefit and risk analyses were rarely updated as projects proceeded and were not used for managing project results Also the mission-related benefits of implemented systems were often difficult to determine since agencies rarely collected or compared data on anticipated versus actual costs and benefits In general we found that the IT investment control processes used at the case study agencies at the time of our review contained four main weaknesses While all four weaknesses may not have been present at each agency in comparison to leading organizations the case study agencies • • • • lacked a consistent process used at all levels of the agency for uniformly selecting and managing systems investments focused their selection processes on selected efforts such as justifying new project funding or focusing on projects already under development rather than managing all IT projects—new under development and operational—as a portfolio of competing investments made funding decisions without giving adequate attention to management control or evaluation processes and made funding decisions based on negotiations or undefined decision criteria and did not have the up-to-date accurate data needed to support IT investment decisions Page 24 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Appendix IV provides a brief overview of how each agency’s current processes for selecting controlling and evaluating IT projects worked Agencies Need Consistent Processes to Select and Manage IT Investments Leading organizations use the selection control and evaluation decision-making processes in a consistent manner throughout different units This enables the organization even one that is highly decentralized to make trade-offs between projects both within and across business units Figure 2 1 illustrates how this process can be applied to the federal government where major cabinet departments may have several agencies under their purview IT portfolio investment processes can exist at both the departmental and agency levels As with leading organizations the key factor is being able to determine which IT projects and resources are shared and should be reviewed at the departmental level and which are unique to each agency Three common criteria used by leading organizations are applicable in the federal setting These threshold criteria include 1 high-dollar high-risk IT projects risk and dollar amounts having been already defined 2 cross-functional projects two or more organizational units will benefit from the project and 3 common infrastructure support hardware and telecommunications Projects that meet these particular threshold criteria are discussed reviewed and decided upon at a departmentwide level The key to making this work is having clearly defined roles responsibilities and criteria for determining the types of projects that will be reviewed at the different organizational levels Page 25 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Figure 2 1 The IT Investment Process Is Uniform Throughout an Organization IT project portfolios Departmental management Change in decision-making authority C Projects that are -- high-dollar amount or high-risk -- cross-departmental or -- common infrastructure projects S Select C Control E Evaluate S E Agency 3 management Agency 2 management Agency 1 management C C S S S E E C E As described in ITMRA agency heads are to implement a process for maximizing the value and assessing and managing the risks of IT investments Further this process should be integrated with the agency’s budget financial and program management process es Whether highly centralized or decentralized matrixed or hierarchial agencies can most Page 26 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain effectively reap the benefits of an investment process by developing and maintaining consistent processes within and across their organizations One of the agencies we reviewed—the Coast Guard—used common investment criteria for making cross-agency IT decisions IRS had defined some criteria but was not yet using these criteria to make decisions The three other agencies—NASA EPA and NOAA—chose IT projects based on inconsistent or nonexistent investment processes There was little or no uniformity in how risks benefits and costs of various IT projects across offices and divisions within these three agencies were evaluated Thus cross-comparisons between systems of similar size function or organizational impact were difficult at best More important management had no assurance that the most important mission objectives of the agency were being met by the suite of system investments that was selected NASA for instance allowed its centers and programs to make their own IT funding decisions for mission-critical systems These decisions were made without an agencywide mechanism in place to identify low-value IT projects or costs that could be avoided by capitalizing on opportunities for data sharing and system consolidation across NASA units As a result identifying cross-functional system opportunities was problematic at best The scope of this problem became apparent as a result of a special NASA IT review In response to budget pressures NASA conducted an agencywide internal information systems review to identify cost savings The resulting March 1995 report described numerous instances of duplicate IT resources such as large-scale computing and wide area network services that were providing similar functions 1 A subsequent NASA Inspector General’s IG report also issued in March 1995 substantiated this special review finding that at one center NASA managers had expended resources to purchase or develop information systems that were already available elsewhere either within NASA or in some cases within that center itself 2 While this special review prompted NASA to plan several consolidation efforts such as consolidating its separate wide area networks for a NASA projected savings of $236 million over 5 years the risk of purchasing duplicate IT resources remained because of weaknesses in its current decentralized decision-making process For example NASA created chief information officer CIO positions for NASA headquarters and for each of its 1 Information Systems Cross-Cutting Team Report NASA March 20 1995 2 Audit Report Survey of NASA Information Systems NASA Office of Inspector General March 29 1995 Page 27 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain 23 centers These CIOs have a key role in improving agencywide IT cooperation and coordination However the CIOs have limited formal authority and to date have only exercised control over NASA’s administrative systems—which account for about 10 percent of NASA’s total IT budget With more defined CIO roles responsibility and authority it is likely that additional opportunities for efficiencies will be identified 3 NASA recently established a CIO council to establish high-level policies and standards approve information resources management plans and address issues and initiatives The council will also serve as the IT capital investment advisory group to the proposed NASA Capital Investment Council NASA plans for this Capital Investment Council to have responsibility for looking at all capital investments across NASA including those for IT NASA’s proposed Capital Investment Council may fill this need for identifying cross-functional opportunities however it is too early to evaluate its impact By having consistent quantitative and analytical processes across NASA that address both mission-critical and administrative systems NASA could more easily identify cross-functional opportunities NASA has already demonstrated that savings can be achieved by looking within mission-critical systems for cross-functional opportunities For instance NASA estimated that $74 million was saved by developing a combined Space Station and Space Shuttle control center using largely commercial off-the-shelf software and a modular development approach rather than the original plan of having two separate control centers that used mainframe technology and custom software EPA like NASA followed a decentralized approach for making IT investment decisions Program offices have had control and discretion over their specific IT budgets regardless of project size or possible cross-office impact As we have previously reported 4 this has led to stovepiped systems that do not have standard data definitions or common interfaces making it difficult to share environmental data across the agency This is important because sharing environmental data across the agency is crucial to implementing EPA’s strategic goals In 1994 EPA began to address this problem by creating a senior management Executive Steering Committee ESC charged with ensuring that investments in agencywide information 3 NASA Chief Information Officer Opportunities to Strengthen Information Resources Management GAO AIMD-96-78 August 15 1996 4 Environmental Protection EPA’s Plans to Improve Longstanding Information Resources Management Problems GAO AIMD-93-8 September 1993 and Environmental Enforcement EPA Needs a Better Strategy to Manage Its Cross-Media Information GAO IMTEC-92-14 April 1992 Page 28 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain resources are managed efficiently and effectively This committee comprised of senior EPA executives has the responsibility to 1 recommend funding on major system development efforts and 2 allocate the IT budget reserved for agencywide IRM initiatives such as geographical information systems GIS support and data standards At the time of our review the ESC had not reviewed or made recommendations on any major information system development efforts Instead the ESC focused its activity on spending funds allocated to it for agencywide IRM policy initiatives such as intra-agency data standards The ESC met on June 26 1996 to assess the impact of ITMRA upon EPA’s IT management process Agencies Need to Manage Their IT Projects as a Portfolio In conducting their selection processes leading organizations assess and manage the different types of IT projects such as mission-critical or infrastructure at all different phases of their life cycle in order to create a complete strategic investment portfolio See figure 2 2 By scrutinizing and analyzing their entire IT portfolio managers can examine the costs of maintaining existing systems versus investing in new ones By continually and rigorously reevaluating the entire project portfolio based on mission priorities organizations can reach decisions on systems based on overall contribution to organizational goals Under ITMRA agencies will need to compare and prioritize projects using explicit quantitative and qualitative decision criteria Page 29 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Figure 2 2 A Comprehensive Approach Includes All Major IT Projects Life-cycle Phases New Underdevelopment Proposals Initial Operational Concept Strategic Mix of Projects Infrastructure R and D Administrative Mission Critical Types of IT Projects At the federal agencies we studied some prioritization of projects was conducted but none made managerial trade-offs across all types of projects IRS NOAA and the Coast Guard each conducted some type of portfolio analyses EPA and NASA did not Additionally the portfolio analyses that were performed generally covered projects that were either high dollar new or under development For example in 1995 we reported that IRS executives were consistently maintaining that all 36 TSM projects estimated to cost up to $10 billion through the year 2001 were equally Page 30 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain important and must all be completed for the modernization to succeed 5 This approach as well as the accompanying initial failure to rank the TSM projects according to their prioritized needs and mission performance improvements has meant that IRS could not be sure that the most important projects were being developed first Since our 1995 report IRS has begun to rank and prioritize all of the proposed TSM projects using cost risk and return decision criteria However these decision criteria are largely qualitative the data used for decisions were not validated or reliable and analyses were not based on calculations of expected return on investment 6 In addition according to IRS its investment review board uses a separate process with different criteria for analyzing operational systems IRS also said that the board does not review research and development R D systems or field office systems Using separate processes for some system types and not including all systems prevents IRS from making comparisons and trade-offs as part of a complete IT portfolio Of all the agencies we reviewed the Coast Guard had the most experience using a comprehensive selection phase In 1991 the Coast Guard started a strategic information resources management process and shortly thereafter initiated an IT investment process Under this investment process a Coast Guard working group from the IRM office ranks and prioritizes new IT projects and those under development based on explicit risk and return decision criteria A senior management board meets annually to rank the projects and decide on priorities The Coast Guard has derived benefits from its project selection process During the implementation of its IT investment process the Coast Guard identified opportunities for systems consolidation For example the Coast Guard reported that five separate personnel systems are being incorporated into the Personnel Management Information System Joint Military Pay System II for a cost avoidance of $10 2 million The Coast Guard also identified other systems consolidation opportunities that if implemented could result in a total cost savings of $77 4 million 5 Tax Systems Modernization Management and Technical Weaknesses Must be Corrected if Modernization Is to Succeed GAO AIMD-95-156 July 26 1995 6 Tax Systems Modernization Actions Underway but IRS Has Not Yet Corrected Management and Technical Weaknesses GAO AIMD-96-106 June 7 1996 Page 31 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain However at the time of our review the Coast Guard’s selection process was still incomplete For example R D projects and operational systems were not included in the prioritization process As a result the Coast Guard could not make trade-offs between all types of proposed systems investments creating a risk that new systems would be implemented that duplicate existing systems Additionally the Coast Guard was at risk of overemphasizing investments in one area such as maintenance and enhancements for existing systems at the expense of higher value investments in other areas such as software applications development supporting multiple unit needs Management Control and Evaluation Processes Were Often Absent Leading organizations continue to manage their investments once selection has occurred maintaining a cycle of continual control and evaluation Senior managers review the project at specific milestones as the project moves through its life cycle and as the dollar amounts spent on the project increase See figure 2 3 At these milestones the executives compare the expected costs risks and benefits of earlier phases with the actual costs incurred risks encountered and benefits realized to date This enables senior executives to 1 identify and focus on managing high-potential or high-risk projects 2 reevaluate investment decisions early in a project’s life cycle if problems arise 3 be responsive to changing external and internal conditions in mission priorities and budgets and 4 learn from past success and mistakes in order to make better decisions in the future The level of management attention focused on each of the three investment phases is proportional based on such factors as the relative importance of each project in the portfolio the relative project risks and the relative number of projects in different phases of the system development process Page 32 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Figure 2 3 IT Investment Is a Continuous and Dynamic Process Project milestones C $ C Funding level C C S E S E S E S E Pilot Prototype Limited implementation Production Life-cycle Phases The control phase focuses senior executive attention on ongoing projects to regularly monitor their interim progress against projected risks cost schedule and performance The control phase requires projects to be modified continued accelerated or terminated based on the results of those assessments In the evaluation phase the attention is focused on implemented systems to give a final assessment of risks costs and returns This assessment is then used to improve the selection of future projects Similarly in the federal government GPRA forces a shift in the focus of federal agencies—away from such traditional concerns as staffing and activity levels and towards one overriding issue results GPRA requires agencies to set goals measure performance and report on their Page 33 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain accomplishments Just as in leading organizations GPRA in concert with the CFO Act is intended to bring a more disciplined businesslike approach to the management of federal programs The agencies we reviewed focused most of their resources and attention on selecting projects and gave less attention to controlling or evaluating those projects While IRS NASA and NOAA had implemented control mechanisms and IRS had developed a postimplementation review methodology none of the agencies had complete and comprehensive control and evaluation processes in place Specifically in the five case study agencies we evaluated we found that • • • Management Control Processes Were Focused Primarily on Cost and Schedule control mechanisms were driven primarily by cost and schedule concerns without any focus on quantitative performance measures evaluations of actual versus projected returns were rarely conducted and information and lessons learned in either the control or evaluation phases were not systematically fed back to the selection phase to improve the project selection process Leading organizations maintain control of a project throughout its life cycle by regularly measuring its progress against not only projected cost and schedule estimates but also quantitative performance measures such as benefits realized or demonstrated in pilot projects to date To do this senior executives from the program IRM and financial units continually monitor projects and systems for progress and identify problems When problems are identified they take immediate action to resolve them minimize their impact or alter project expectations Legislation now requires federal executives to conduct this type of rigorous project monitoring With the passage of ITMRA agencies are required to demonstrate through performance measures how well IT projects are improving agency operations and mission effectiveness Senior managers are also to receive independently verifiable information on cost technical and capability requirements timeliness and mission benefit data at project milestones Furthermore pursuant to the Federal Acquisition Streamlining Act of 1994 Public Law 103-355 if a project deviates from cost schedule and performance goals the agency head is required to conduct a timely review of the project and identify appropriate corrective action—to include project termination Page 34 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Two of the agencies we reviewed—the Coast Guard and EPA—did not use management control processes that focused on IT systems projects The other three agencies—IRS NOAA and NASA—had management control processes that focused primarily on schedule and cost concerns but not interim evaluations of performance and results Rarely did we find examples in which anticipated benefits were compared to results at critical project milestones We also found few examples of lessons that were learned during the control phase being cycled back to improve the selection phase To illustrate both IRS and NASA used program control meetings PCMs to keep senior executives informed of the status of their major systems by requiring reports in the form of self-assessments from the project managers However these meetings did not focus on how projects were achieving interim measurable improvement targets for quality speed and service that could form the basis for project decisions about major modifications or termination IRS for instance used an implementation schedule to track different components of each of its major IT projects under TSM Based on our discussions with IRS officials the PCMs focused on factors bearing on real or potential changes in project costs or schedule Actual verified data on interim application or system testing results—compared to projected improvements in operational mission improvements—were not evaluated At NASA senior program executives attended quarterly Program Management Council PMC meetings to be kept informed of major programs and projects and to take action when problems arose While not focused exclusively on IT issues the PCMs were part of a review process that looked at implementation issues of programs and projects that 1 were critical to fulfilling NASA’s mission particularly those that were assigned to two or more field installations 2 involved the allocation of significant resources defined as projects whose life-cycle costs were over $200 million or 3 warranted special management attention including those that required external agency reporting on a regular basis During the PMC meetings senior executives reviewed self-assessments grades of green yellow and red done by the responsible project manager on the cost schedule and technical progress of the project Using this color-coded grading scheme NASA’s control process focused largely on cost schedule and technical concerns but not on assessing improvements to mission performance Additionally the grading scheme was not based on quantitative criteria but instead was largely qualitative Page 35 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain and subjective in nature For instance projects were given a “green” rating if they were “in good shape and on track consistent with the baseline ” A “yellow” rating was defined as a “concern that is expected to be resolved within the schedule and budget margins ” and a “red” rating was defined as “a serious problem that is likely to require a change in the baseline determined at the beginning of the project ” However the lack of quantitative criteria benefit analysis and performance data invited the possibility for widely divergent interpretations and a misunderstanding of the true value of the projects under review As of 1995 three IT systems had met NASA’s review criteria and had been reviewed by the PMC These three systems constituted about 7 percent of NASA’s total fiscal year 1994 IT spending No similar centralized review process existed for lower dollar projects which could have resulted in problem projects and systems that collectively added up to significant costs being overlooked For instance in 1995 NASA terminated an automated accounting system project that had been under development for about 6 years had cost about $45 million to date and had an expected life-cycle cost of over $107 million In responding to a draft of this report the NASA CIO said that the current cost threshold of $200 million is being reduced to a lower level to ensure that most if not all agency IT projects will be subject to PMC reviews In addition the CIO noted that NASA’s internal policy directive on program project management is being revised to 1 include IT evaluation criteria that are aligned with ITMRA and executive-branch guidance and 2 clearly establish the scope and levels of review agency lead center or center for IT investment decisions Evaluations of Actual Versus Projected Returns Are Rarely Conducted Once projects have been implemented and become operational leading organizations evaluate them to determine whether they have achieved the expected benefits such as lowered cost reduced cycle time increased quality or increased the speed of service delivery They do this by conducting project postimplementation reviews PIRs to compare actual to planned cost returns and risks The PIR results are used to calculate a final return on investment determine whether any unanticipated modifications may be necessary to the system and provide “lessons learned” input for changes to the organization’s IT investment processes and strategy ITMRA now requires agencies to report to OMB on the performance benefits achieved by their IT investments and how those benefits support the accomplishment of agency goals Page 36 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Only one of the five federal agencies we reviewed—IRS—systematically evaluated implemented IT projects to determine actual costs benefits and risks Indeed we found that most of the agencies rarely evaluated implemented IT projects at all In general the agency review programs were insufficiently staffed and used poorly defined and inconsistent approaches In addition in cases where evaluations were done the findings were not used to consider improvements or revisions in the IT investment decision-making process NOAA for instance had no systematic process in place to ensure that it was achieving the planned benefits from its annual $300 million IT expenditure For example of the four major IT projects that constitute the $4 5 billion National Weather Service NWS modernization effort only the benefits actually accruing from one of four—the NEXRAD radars—had been analyzed 7 While not the only review mechanism used by the agency NOAA’s central review program was poorly staffed NOAA headquarters with half a staff year devoted to this review program generally conducted reviews in collaboration with other organizational units and had participated in only four IT reviews over the last 3 fiscal years Additionally these reviews generally did not address the systems’ projected versus actual cost performance and benefits had developed a PIR methodology that it used to conduct five systems postimplementation reviews A standardized methodology is important because it makes the reviews consistent and adds rigor to the analytical steps used in the review process The IRS used the June 1994 PIR on the Corporate Files On-Line CFOL system as the model for this standardized methodology In December 1995 IRS used the PIR methodology to complete a review of the Service Center Recognition Image Processing System SCRIPS Subsequently three more PIRs have been completed TAXLINK the Enforcement Revenue Information System and the Integrated Collection System and five more are scheduled IRS estimated that the five completed systems have an aggregate cost of about $845 million IRS However the PIR methodology was not integrated into a cohesive investment process Specifically there were no mechanisms in place to take the lessons learned from the PIRs and apply them to the decision criteria and other tools and techniques used in their investment process 7 For more information on the status of the NWS modernization see Weather Service Modernization Despite Progress Significant Problems and Risks Remain T-AIMD-95-87 Feb 21 1995 Page 37 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain As a result the PIRs that were conducted did not meet one of their primary objectives—to ensure continual improvement based on lessons learned—and IRS ran the risk of repeating past mistakes To help make continual decisions on IT investments leading organizations require all projects to have complete and up-to-date project information This information includes cost and benefit data risk assessments implementation plans and initial performance measures See figure 2 4 Maintaining this information allows senior managers to rigorously evaluate the current status of projects In addition it allows them to compare IT projects across the organization consider continuation delay or cancellation trade-offs and take action accordingly Agency IT Decisions Are Not Based on Adequate Data requires agencies to use quantitative and qualitative criteria to evaluate the risks and the returns of IT investments As such agencies need to collect and maintain accurate and reliable cost benefit risk and performance data to support project selection and control decisions The requirement for accurate reliable and up-to-date financial and programmatic information is also a primary requirement of the CFO Act and is essential to fulfilling agency requirements for evaluating program results and outcomes under GPRA ITMRA At the five case study agencies we evaluated we found that in general • • agency IT investment decisions were based on undefined or implicit decision criteria and data on the project’s cost schedule risks and returns were not documented defined or kept up-to-date and in many cases were not used to make investment decisions Page 38 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain Figure 2 4 Consistent Well-Defined and Up-To-Date Data Are Essential Throughout the IT Investment Process Postimplementation Reviews Proposal Selections Select Evaluate Control Monitoring Data Information on Mission Benefits Explicit Decision Criteria Were Not Defined Schedule Data To ensure that all projects and operational systems are treated consistently leading organizations define explicit risk and return decision criteria These criteria are then used to evaluate every IT project or system Risk criteria involve managerial technical resource skill security and organizational factors such as the size and scope of the project the extent of use of new technology the potential effects on the user organization the project’s technical complexity and the project’s level of dependency on other systems or projects Return criteria are measured in financial and nonfinancial terms Financial measurements can include return on Page 39 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain investment and internal rate of return analyses while nonfinancial assessments can include improvements in operational efficiency reductions in cycle time and progress in better meeting customer needs Of the five agencies in our sample only the Coast Guard used a complete set of decision criteria These decision criteria included 1 risk assessments of schedule cost and technical feasibility dimensions 2 cost-benefit impacts of the investment 3 mission effectiveness measures 4 degree of alignment with strategic goals and high-level interest such as Congress or the President and 5 organizational impact in the areas of personnel training quality of work life and increased scope of service The Coast Guard used these criteria to prioritize IT projects and justify final selections The decision criteria were weighted and scored and projects were evaluated to determine those with the greatest potential to improve mission performance Generally officials in other agencies stated that they determine which projects to fund based on the judgmental expertise of decisionmakers involved in the process NOAA for instance had a board of senior executives that met annually to determine budget decisions across seven strategic goals Working groups for each strategic goal met and each created a prioritized funding list which was then submitted to the executive decision-making board These working groups did not have uniform criteria for selecting projects The executive board accepted the prioritized lists as submitted and made funding threshold decisions based on these lists As a result the executive board could not easily make consistent accurate trade-offs among the projects that were selected by these individual working groups on a repeatable basis In addition to maximize funding for a specific working group project rankings may not have been based on true risk or return According to a NOAA senior manager and the chair of one of the NOAA working groups one group ranked high-visibility projects near the bottom of the list to encourage the senior decision-making board to draw the budgetary cut-off line below these high visibility projects Few of these high-visibility projects were at the top of the list despite being crucial to NOAA and high on the list of the NOAA Administrator’s priorities Explicit decision criteria would eliminate this type of budgetary gamesmanship Data Were Often Not Consistent or Up-To-Date Leading organizations consider project data the foundation by which they select control and evaluate their IT investments Without it participants in Page 40 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain an investment process cannot determine the value of any one project Leading organizations use rigorous and up-to-date cost-benefit analyses risk assessments sensitivity analyses and project specific data including current costs staffing and performance to make funding decisions and project modifications based whenever possible on quantifiable data While the agencies in our sample developed documents in order to get project approvals little effort was made to ensure that the information was kept accurate and up-to-date and rarely were the data used to manage the project throughout its life cycle During our review we asked each agency to supply us with basic data on its largest dollar IT projects However this information was not readily available and gathering it required agency officials to rely on a variety of sometimes incomparable sources for system cost life-cycle phase and staffing levels In addition some of the agencies could not comparatively analyze IT projects because they did not keep a comprehensive accounting of data on all of the IT systems For example EPA had to conduct a special information collection to identify life-cycle cost estimates on its major systems and projects for this report While the individual system managers at EPA did have system life-cycle cost estimates the fact that this information was decentrally maintained made cross-system comparisons unlikely In a 1995 report the NASA IG found that neither NASA headquarters nor any of the NASA centers had a complete inventory of all information systems for which they were responsible 8 All of the agencies we reviewed conducted cost-benefit analyses for their major IT projects However these analyses were generally done to support decisions for project approval and were seldom kept current In addition the cost-benefit projections were rarely used to evaluate actual project results The NWS modernization for instance has a cost-benefit analysis that was done in 1992 This analysis covers the four major systems under the modernization 9 To be effective an analysis should include the costs and benefits of each project alternatives to that project and finally a combined cost-benefit analysis for the entire modernization However the 8 Audit Report Survey of NASA Information Systems NASA Office of Inspector General March 29 1995 Report No JP-95-003 9 Chapman Robert E Benefit-cost Analysis for the Modernization and Associated Restructuring of the National Weather Service July 1992 National Institute of Standards and Technology Department of Commerce Page 41 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain cost-benefit analysis that was conducted only compares the aggregate costs and benefits of the NWS modernization initiative against the current system It does not assess or analyze the costs and benefits of each system nor does it examine alternatives to those systems As a result NWS does not know if each of the modernization projects is cost-beneficial and cannot make trade-offs among them If using only this analysis decision-makers are forced to choose either the status quo or all of the projects proposed under the modernization Without updated cost-benefit data informed management decisions become difficult We reported in April 1995 that NWS was trying to assess user concerns related to the Automated Surface Observing System ASOS one of the NWS modernization projects but that NWS did not have a complete estimate of what it would cost to address these concerns 10 As we concluded in the report without reliable estimates of what an enhanced or supplemented ASOS would cost it would be difficult for NWS to know whether continued investment in ASOS is cost-beneficial Agency Comments and Our Evaluation We provided and discussed a draft of this report with officials from EPA NASA NOAA IRS and the Coast Guard and have incorporated their comments where appropriate Several of the agencies noted that they in response to the issuance of OMB’s guidance on IT investment decision-making11 and the passage of ITMRA have made process changes and organizational modifications affecting IT funding decisions We have incorporated this information into the report where applicable However many of the process changes and modifications have occurred very recently and we have not fully evaluated these changes or determined their effects Officials from NOAA and NASA also had reservations about the applicability of the investment portfolio approach to their organizations because their decentralized operating environments were not conducive to a single agencywide portfolio model with a fixed set of criteria Because any organization whether centralized or decentralized has to operate within the parameters of a finite budget priorities must still be set across the organization about where limited IT dollars will be spent to achieve maximum mission benefits We agree that many IT spending decisions can 10 Weather Forecasting Unmet Needs and Unknown Costs Warrant Reassessment of Observing System Plans GAO AIMD-95-81 April 21 1995 11 Evaluating Information Technology Investments A Practical Guide Executive Office of the President Office of Management and Budget November 1995 Page 42 GAO AIMD-96-64 IT Investment Chapter 2 IT Investment Approaches in Five Case Study Agencies Progress Has Been Made but Challenges Remain be made at the agency or program level However there are some decisions—especially those involving projects that are 1 high-risk high-dollar 2 cross-functional 3 or providing a common infrastructure e g telecommunications —that should be made at a centralized departmental level Establishing a common organizationwide focus while still maintaining a flexible distribution of departmental and agency program site decision-making can be achieved by implementing standard decision criteria These criteria help ensure that projects are assessed and evaluated consistently at lower levels while still maintaining an enterprisewide portfolio of IT investments Page 43 GAO AIMD-96-64 IT Investment Chapter 3 Conclusions and Recommendations Conclusions Buying information technology can be a high-risk high-return undertaking that requires strong management commitment and a systematic process to ensure successful outcomes By using an investment-driven management approach leading organizations have significantly increased the realized return on information technology investments reduced the risk of cost overruns and schedule delays and made better decisions about how their limited IT dollar should be spent Adopting such an investment-driven approach can provide federal agencies with similar opportunities to achieve greater benefits from their IT investments on a more consistent basis However the federal case study agencies we examined used decision-making processes that lacked many essential components associated with an investment approach Critical weaknesses included the absence of reliable quantitative cost figures net return on investment calculations rigorous decision criteria and postimplementation project reviews With sustained management attention and substantive improvements to existing processes these agencies should be able to meet the investment-related provisions of ITMRA Implementing and refining an IT investment process however is not an easy undertaking and cannot be accomplished overnight Maximizing the returns and minimizing the risks on the billions of dollars that are spent each year for IT will require continued efforts on two fronts First agencies must fundamentally change how they select and manage their IT projects They must develop and begin using a structured IT investment approach that encompasses all aspects of the investment process—selection control and evaluation Second oversight attention far beyond current levels must be given to agencies’ management processes and to actual results that are being produced Such attention should include the development of policies and guidance as well as selective evaluations of processes and results These evaluations should have a dual focus They should identify and address deficiencies that are occurring but they should also highlight positive results in order to share lessons learned and speed success OMB’s established leadership role as well as the policy development and oversight responsibilities that it was given under ITMRA place it in a key position to provide such oversight OMB has already initiated several changes to governmentwide guidance to encourage the investment approach to IT decision-making and has drawn upon the assistance of Page 44 GAO AIMD-96-64 IT Investment Chapter 3 Conclusions and Recommendations several key interagency working groups comprised of senior agency officials Such efforts should be continued and expanded to ensure that the federal government gets the most return for its information technology investments Given its significant leadership responsibility in supporting agencies’ improvement efforts and responding to requirements of ITMRA it is imperative that OMB continue to clearly define expectations for agencies and for itself to successfully implement investment decision-making approaches As such we are recommending four specific actions for the Director of OMB to take Recommendations OMB’s first challenge is to help agencies improve their investment management processes With effective processes in place agencies should be in much stronger positions to make informed decisions about the relative benefits and risks of proposed IT spending Without them agencies will continue to be vulnerable to risks associated with excessively costly projects that produce questionable mission-related improvements Under Sections 5112 and 5113 of the Information Technology Management Reform Act the Director of OMB has responsibility for promoting and directing that federal agencies establish capital planning processes for information technology investment decisions In designing governmentwide guidance for this process we recommend that the Director of the Office of Management and Budget require agencies to • • Implement IT investment decision-making processes that use explicitly defined complete and consistent criteria applied to all projects regardless of whether project decisions are made at the departmental bureau or program level With criteria that reflect cost benefit and risk considerations applied consistently agencies should be able to make more reasonable and better informed trade-offs between competing projects in order to achieve the maximum economic impact for their scarce investment dollars Periodically analyze their entire portfolios of IT investments—at a minimum new projects as well as projects in development and operations and maintenance expenditures—to determine which projects to approve cancel or delay With development and maintenance efforts competing directly with one another for funding agencies will be better able to gauge the best proportion of investment in each category of spending to move away from their legacy bases of systems with excessive maintenance costs Page 45 GAO AIMD-96-64 IT Investment Chapter 3 Conclusions and Recommendations • • Design control and evaluation processes that include cost schedule and quantitative performance assessments of projected versus actual improvement in mission outcomes As a result they should increase their capacity to both assess actual project results and learn from their experience which operational areas produce the highest returns and how well they estimate projects and deliver final results Advise agencies in setting minimum quality standards for data used to assess qualitatively and quantitatively cost benefit and risks decisions on IT investments Agencies should demonstrate that all IT funding proposals include only data meeting these quality requirements and that projected versus actual results are assessed at critical project milestones The audited data required by the CFO Act should help produce this accurate reliable cost information Higher quality information should result in better and more consistent decisions on complex information systems investments OMB’s second challenge is to use the results produced by the improved investment processes to develop recommendations for the President’s budget that reflect an agency’s actual track record in delivering mission performance for IT funds expended Under Section 5113 of ITMRA the Director of OMB is charged with evaluating the results of agency IT investments and enforcing accountability—including increases or reductions in agency IT funding proposals—through the annual budget process In carrying out these responsibilities we recommend that the Director of the Office of Management and Budget • • • Evaluate information system project cost benefit and risk data when analyzing the results of agency IT investments Such analyses should produce agency track records that clearly and definitively show what improvements in mission performance have been achieved for the IT dollars expended Ensure that the agency investment control process are in compliance with OMB’s governmentwide guidance and if not assess strengths and weaknesses and recommend actions and timetables for improvements When results are questionable or difficult to determine monitoring agency investment processes will help OMB diagnose problem causes by determining the degree of agency control and the quality of decisions being made Use OMB’s evaluation of each agency’s IT investment control processes and IT performance results as a basis for recommended budget decisions to the President This direct linkage should give agencies a strong much needed Page 46 GAO AIMD-96-64 IT Investment Chapter 3 Conclusions and Recommendations incentive to maximize the returns and minimize the risks of their scarce IT investments To effectively implement improved investment management processes and make the appropriate linkages between agency track records and budget recommendations OMB also has a third challenge It will need to marshal the resources and skills to execute the new types of analysis required to make sound investment decisions on agency portfolios Specifically we recommend that the Director of the Office of Management and Budget • • • Organize an interagency group comprised of budget program financial and IT professionals to develop refine and transfer guidance and knowledge on best practices in IT investment management Such a core group can serve as an ongoing source of practical knowledge and experience on the state of the practice for the federal government Obtain expertise on an advisory basis to assist these professionals in implementing complete and effective investment management systems Agency senior IRM management could benefit greatly from a high quality easily accessible means to solicit advice from capital planning and investment experts outside the federal government Identify the type and amount of skills required for OMB to execute IT portfolio analyses determine the degree to which these needs are currently satisfied specify the gap and both design and implement a plan with timeframes and goals to close the gap Given existing workloads and the resilience of the OMB culture without a determined effort to build the necessary skills OMB will have little impact on the quality of IT investment decision-making If necessary to augment its own staff resources OMB should consider the option of obtaining outside support to help perform such assessments Finally as part of its internal implementation strategy the Director of the Office of Management and Budget should consider developing an approach to assessing OMB’s own performance in executing oversight responsibilities under the ITMRA capital planning and investment provisions Such a process could focus on whether OMB reviews of agency processes and results have an impact on reducing risk or increasing the returns on information technology investments—both within and across federal agencies Agency Comments and Our Evaluation In its written comments on a draft of our report OMB generally supported our recommendations and said that it is working towards implementing Page 47 GAO AIMD-96-64 IT Investment Chapter 3 Conclusions and Recommendations many aspects of the recommendations as part of the fiscal year 1998 budget review process of fixed capital assets OMB also provided observations or suggestions in two additional areas First OMB stated that given ITMRA’s emphasis on agencies being responsible for IT investment results it did not plan to validate or verify that each agency’s investment control process is in compliance with OMB’s guidance contained in its management circulars As discussed in our more detailed evaluation of OMB’s comments in appendix I conducting selective evaluations is an important aspect of an overall oversight and leadership role because it can help identify management deficiencies that are contributing to poor IT investment results Second OMB noted that the relationship of IT investment processes between a Cabinet department and bureaus or agencies within the department was not fully evaluated and that additional attention would be needed as more data on this issue become available We agree that our focus was on assessing agencywide processes and that continued attention to the relationships between departments bureaus and agencies will contribute to increased understanding across the government and will ultimately improve ITMRA’s chances of success This issue is discussed in more detail in our response to comments provided by the five agencies we reviewed summarized at the end of chapter 2 Page 48 GAO AIMD-96-64 IT Investment Page 49 GAO AIMD-96-64 IT Investment Appendix I Comments From the Office of Management and Budget and Our Evaluation Note GAO comments supplementing those in the report text appear at the end of this appendix See comment 1 Page 50 GAO AIMD-96-64 IT Investment Appendix I Comments From the Office of Management and Budget and Our Evaluation See comment 2 Page 51 GAO AIMD-96-64 IT Investment Appendix I Comments From the Office of Management and Budget and Our Evaluation The following are GAO’s comments on the Office of Management and Budget’s letter dated July 26 1996 GAO Comments 1 As stated in the scope and methodology section of the report we focused our analysis on agencywide processes We agree that continued attention to this issue will contribute to increased understanding across the government and will ultimately improve ITMRA’s chances of success As noted in our response to comments received from the agencies we reviewed provided at the end of chapter 2 we believe that a flexible distribution of departmental and agency program site IT decision-making is possible and can best be achieved by implementing standard decision criteria for all projects In addition we note that particular types of IT decisions such as those with unusually high-risk cross-functional impact or that provide common infrastructure needs are more appropriately decided at a centralized departmental level Experience gained during implementation of the Chief Financial Officers CFO Act showed that departmental-level CFOs needed time to build effective working relationships with their agency- or bureau-level counterparts We believe the same will be true for Chief Information Officers CIOs established by ITMRA and that establishing and maintaining this bureau-level focus will be integral for ensuring the act’s success 2 ITMRA does squarely place responsibility and accountability for IT investment results with the head of each agency Nevertheless ITMRA clearly requires that OMB provide a key policy leadership and implementation oversight role While we agree that it may not be feasible to validate and verify every agency’s investment processes it is still essential that selected evaluations be conducted on a regular basis These evaluations can effectively support OMB’s performance and results-based approach They can help to identify and understand problems that are contributing to poor investment outcomes and also help perpetuate success by providing increased learning and sharing about what is and is not working Page 52 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles In order to develop a profile of each agency’s IT environment we asked the agencies to provide us information on the following • • • total IT expenditures for fiscal year 1990 through fiscal year 1994 total number of staff devoted to IRM functions and activities for fiscal year 1990 through fiscal year 1994 and costs for the 10 largest IT projects for fiscal year 1994 as measured by total project life-cycle cost To gather this information we developed a data collection instrument and submitted it to responsible agency officials Information supplied by the agencies is summarized in the following tables We did not independently verify the accuracy of this information Moreover comparison of figures across the agencies is difficult because agency officials used different sources such as budget data IRM strategic plans etc for the same data elements Table II 1 Total Agency Budget and IT Spending for Fiscal Year 1990 Through Fiscal Year 1994 Dollars in millions Agency FY 1990 FY 1991 FY 1992 FY 1993 FY 1994 $3 304 $3 427 $3 571 $3 649 $3 666 $45 $46 $121 $139 $157 1 1 3 4 4 $6 123 $6 584 $6 969 $6 970 $5 782 $262 $291 $281 $282 $302 4 4 4 4 5 $5 500 $6 113 $6 670 $7 100 $7 188 $789 $979 $1 294 $1 479 $1 293 14 16 19 21 18 $13 981 $14 756 $15 181 $14 950 $14 670 $1 513 $1 589 $1 777 $2 002 $1 604 11 11 12 13 11 $1 200 $1 400 $1 600 $1 700 $1 900 $199 $191 $261 $304 $296 17 14 16 18 16 Coast Guard Total Amount spent on IT Percent of total EPA Total Amount spent on IT Percent of total IRS Total Amount spent on IT Percent of Total NASA Total Amount spent on IT Percent of total NOAA Total Amount spent on IT Percent of total Sources Coast Guard EPA IRS NASA and NOAA Page 53 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Table II 2 Agency Total and IT Staffing for Fiscal Year 1990 Through Fiscal Year 1994 Agency FY 1990 FY 1991 FY 1992 FY 1993 FY 1994 43 102 43 645 45 581 45 692 44 546 331 369 398 428 424 1 1 1 1 1 15 272 16 415 17 010 18 351 17 721 886 840 831 863 850 6 5 5 5 5 111 962 115 628 116 673 113 460 10 665 Not identified 9 001 9 881 9 529 9 030 NA 8 8 8 8 23 669 24 692 24 330 23 996 23 685 1 666 1 659 1 700 1 752 1 476 7 7 7 7 6 12 892 13 410 13 829 14 309 13 292 910 844 1 100 944 1 030 7 6 8 7 8 Coast Guard Total FTE’sa IT FTE’s Percent of total EPA Total FTE’s IT FTE’s Percent of total IRS Total FTE’s IT FTE’s Percent of total NASA Total FTE’s IT FTE’s Percent of total NOAA Total FTE’s IT FTE’s Percent of total Sources Coast Guard EPA IRS NASA and NOAA a One FTE is one full-time equivalent employee Page 54 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Table II 3 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at the Coast Guard Dollars in thousands Amount spent in FY 1994 Life-cycle cost $690 $184 070 Op $21 600 $63 800 Op $8 084 $60 079 U D $12 385 $27 594 U D $60 $25 440 U D $760 $25 208 U D $2 000 $25 000 Op $4 114 $24 459 Op $4 032 $22 256 U D $3 125 $22 125 $56 850 $480 031 Project Status Coast Guard Standard Workstation III U D Provides an organizationwide microcomputer infrastructure and is the primary source for acquiring desktop server and portable hardware operating system and office automation system software utilities and peripherals training personnel support and cabling Coast Guard Standard Workstation II Provides continued support for the Coast Guard’s existing microcomputer infrastructure Finance Center Information Resources Management System Provides a consolidated accounting and pay system Vessel Traffic System Upgrade A configuration of sensors communication links personnel and decision support tools that will modernize and expand the systems in three cities by incorporating radar sensor information overlaid on digital nautical charts as well as improved decision support systems Communication System 2000 Provides an automated and consolidated communication system Aviation Logistics Management Information System Merges two maintenance systems for tracking and recording scheduled aviation maintenance actions Coast Guard Standard Workstation Application Conversion Reprograms most of the existing Coast Guard developed applications to comply with the National Institute of Standards and Technology’s Application Portability Profile Marine Safety Information System Provides safety performance histories of vessels and involved parties and is used as a decision support tool for the Commercial Vessel Safety program Aviation Repair and Supply Center Systems Provides aviation technical publications in electronic format Coast Guard Local Information Network Cabling Upgrade Project Consolidated into the Coast Guard Standard Workstation III system Total Source Office of Command Control and Communications management information system Note Categories have been melded to reflect the way Coast Guard tracks systems and projects categories do not include IT facilities central “bill paying” accounts for IT or “Umbrella” Projects Contracts Note Op is operational or being maintained U D is under development Page 55 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Table II 4 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at EPA Dollars in thousands Amount spent in FY 1994 Life-cycle cost Op $5 470 $202 730 Op $10 149 $138 000 Op $8 470 $115 000 Op $4 737 $75 000 Op $2 390 $68 000 Op $3 750 $45 550 Op $3 457 $35 000 Op $2 810 $28 000 Project Status Integrated Financial Management System Performs funds control from commitments through payment updates all ledgers and tables as transactions are processed provides a standard means of data entry edit and inquiry and provides a single set of reference and control files Toxic Release Inventory System Contains data submitted to EPA under the Emergency Planning and Community Right to Know Act for chemicals and chemical categories listed by the agency Data include chemical identity amount of on-site users release and off-site transfers on-site treatment minimization prevention actions Public access is provided by the National Library of Medicine Contract Laboratory Program System Supports management and administration of chemical samples from Superfund sites that are analyzed under agency contracts with chemical laboratories The system schedules and tracks samples from site collection through analysis to delivery to the agency Aerometric Information Retrieval System Stores air quality point source emissions and area mobile source data required by federal regulations from the 50 states Comprehensive Environmental Response Compensation and Liability Information System Superfund’s official source of planning and accomplishment data Serves as the primary basis for strategic decision-making and site-by-site tracking of cleanup activities Certification Fuel Economy Information System Contains a set of computer applications and a major relational database which is used to support regulation development air quality analysis compliance audits investigations assembly line testing in-use compliance legislation development and environmental initiatives Resource Conservation and Recovery Information System Maintains basic data identifying and describing hazardous waste handlers detailed information about hazardous waste treatment storage and disposal processes environmental permitting information on inspections violations and enforcement actions and tracks specific corrective action information needed to regulate facilities with hazardous waste releases Permit Compliance System Supports the National Pollutant Discharge Elimination System a Clean Water Act program that issues permits and tracks facilities that discharge pollutants into our navigable waters continued Page 56 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Dollars in thousands Amount spent in FY 1994 Life-cycle cost U D $1 400 $28 000 Op $2 230 $28 000 $44 863 $763 280 Project Status Comprehensive Environmental Response Compensation and Liability Information System Version III A replacement for the existing Comprehensive Environmental Response Compensation and Liability Information System described above WasteLAN A PC LAN version of the Comprehensive Environmental Response Compensation and Liability Information System database used by EPA regional offices for data input and local analysis needs Total Source System Project manager or the Senior IRM Official for the office Note Op is operational or being maintained U D is under development Imp is being implemented Table II 5 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at IRS Dollars in thousands Amount spent in FY 1994 Life-cycle cost U D $27 404 $2 847 338 U D $108 877 $1 870 980 U D $159 933 $1 661 329 Op $44 469 $1 261 361 Project Status Service Center Support System ISD-08 Acquire and install Tax System Modernization host-tier computers at three computing centers Integrated Case Processing System ISD-03 Integrates five systems that control assign prioritize and track taxpayer inquiries provides office automation case folder review and inventories and display and manipulation of case inquiry folders automates collection cases provide access to current tax return information automates case preparation and closure and provides standardized hardware and custom software to the criminal investigation function on a nationwide basis Integrated Input Processing System ISD-06 Integrates six systems that will receive and control information being transmitted to or from IRS automates remittance processing activities scans paper tax returns and correspondence for processing in an automated database provides automated telephone assistance to customers permits individual and business tax returns to be filed by utilizing a touch-tone phone and provides access to all electronically filed returns that have been scored for potential fraud Integrated Collection System CO-05 Provides case tracking expanded legal research a document management system for briefs an integrated office system time reporting issue tracking litigation support and a decision support system continued Page 57 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Dollars in thousands Project Status Corporate Systems Design ISD-09 U D Amount spent in FY 1994 Life-cycle cost $38 357 $967 835 Integrates three systems that provide application programs to query search update analyze and extract information from a database aggregates tax information into electronic case folders and distributes them to field locations and provides the security infrastructure to support all components of the Tax System Modernization Servicewide Technical Infrastructure ISD-15 U D a $699 338 Provides a variety of workstation models monitors printers operating systems and related equipment provides for standardization of the small and medium-scale computers used by front line programs in the national and field offices and service centers Tax Processing Mainframe Computer System ISD-13 Op $20 890 $671 739 U D $68 056 $578 208 U D $25 316 $424 415 U D $47 258 $408 905 $540 560 $11 391 448 Provides funding for 1 the mainframe and miscellaneous peripherals at each service center 2 magnetic media and ADP supplies for all service centers 3 lease and maintenance for support equipment and 4 on-line access to taxpayer information and account status Corporate Systems Modernization Transition ISD-10 Provides an interim hardware platform at two computing centers to support master file processing and full implementation of the CFOL data retrieval delivery system Software Development Environment ISM-35 Provides upgradable software development workstations and workbench tools including automated analysis and design tools requirements traceability tools construction kits with smart editors compilers animators and debuggers and static analyzers Communications Modernization ISD-21 Integrates four systems that provide for ordering and delivery of telecommunication systems and services for Treasury bureaus serves as a Government Open Systems Interconnection Profile prototype provides centralized network and operations management and will acquire about 14 000 workstations Total Source IRS’ Information Systems Initiative Summaries Note Op is operational or being maintained U D is under development or being implemented a Note ISD-15 did not exist in FY 1994 Page 58 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Table II 6 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at NASA Dollars in thousands Amount spent in FY 1994 Life-cycle cost U D $55 077 $3 394 872 Op $21 075 $1 680 000 Op $67 185 $490 000 Op $4 503 $460 000 Op $7 648 $430 000 Op $17 636 $418 000 Op $56 906 $347 000 Op $24 221 $332 000 Op $26 866 $286 000 Project Status Earth Observing System Data and Information System Core System Receives processes archives and distributes earth science research data from U S European and Japanese polar platforms selected Earth probes the Synthetic Aperture Radar free flyer selected existing databases and other sources of related data Program Information Support Mission Services Provides telecommunications and computation services for Marshall Space Flight Center Information Systems Contract Supports most data systems networks user workstations and telecommunications systems and provides maintenance operations software development engineering and customer support functions at Johnson Space Center Operations Automatic Data Processing Procurement Provides a family of compatible computing systems covering a broad performance range that will provide ground-based mission operations systems support Engineering Test Analysis Provides a contractor to supply over the next 10 years the necessary personnel management equipment and materials to support over 100 laboratories within the Engineering Directorate and other closely related directorates and offices at the Johnson Space Center Base Operations Contract Provides continuity of base operations including federal information processing resources of sustaining engineering computer operations and communications services for Kennedy Space Center Scientific Engineering Workstations Procurement Acquisition of seven classes of scientific and engineering workstations plus supporting equipment Central Computing Resources Project Furnishes installs and tests the Advanced Computer Generated Image System provides direct computational analysis and programming support to specific research disciplines and flight projects provides for the analysis programming engineering and maintenance services for the flight simulation facilities Also provides support for the Central Scientific and Computing Complex operation and systems maintenance as well as Complex-wide communications systems support and system administration of distributed computing and data reduction systems Small Disadvantaged Business Resources Acquisition Provides a wide array of supporting services including computational professional technical administrative engineering and operations at the Lewis Research Center continued Page 59 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Dollars in thousands Project Status Advanced X-Ray Astrophysics Facility U D Amount spent in FY 1994 Life-cycle cost $4 600 $90 000 $285 718 $7 927 872 Total Source NASA IRM Division APR files Note Op is operational or being maintained U D is under development Imp is being implemented Table II 7 Life-Cycle Cost and Fiscal Year 1994 Expenditures for the 10 Largest IT Projects at NOAA Dollars in thousands Amount spent in FY 1994 Life-cycle cost U D $42 954 $657 048 Op $13 335 $181 143 Op $11 100 $128 696 Op $10 925 $120 300 Op $7 396 $97 628 Op $7 545 $65 616 Op $6 501 $65 260 Project Status Advanced Weather Interactive Processing System An information system including workstations associated data processing and communications designed to integrate data from several National Weather Service information systems as well as from field offices regional and national centers and other sources NWS Supercomputer Replacement Project An initiative to acquire supercomputers necessary to run large complex numeric models as a key component of the weather forecast system Central Environmental Satellite Computer System A distributed-processing system architecture designed to acquire process and distribute satellite data and products Information Technology 1995 An effort to replace a variety of obsolete technology in the National Marine Fisheries Service with a common computing infrastructure that supports distributed processing in an open system environment The system stores integrates analyzes and disseminates large quantities of living marine resource data Geophysical Fluid Dynamics Laboratory High-Performance Computer System Procurement of a high-performance computer system to provide support services for climate and weather research activities Geostationary Operational Environmental Satellite GOES I-M Ground system consisting of minicomputers with associated peripherals and satellite-dependent customized applications software to provide the monitoring supervision and data acquisition and processing functions for the GOES-Next satellites WSR-88D Operational Support Facility System Support A system designed to support weather radars and associated display systems continued Page 60 GAO AIMD-96-64 IT Investment Appendix II Agency Investment Technology Profiles Dollars in thousands Amount spent in FY 1994 Life-cycle cost Op $4 145 $45 135 Op $600 $43 982 $4 330 $35 950 $108 831 $1 440 758 Project Status NWS Gateway Upgrade An effort to replace old mainframes as well as the associated channel-connected architecture with an open systems architecture Polar Orbiting Environmental Satellite Ground system consisting of minicomputers with associated peripherals and satellite-dependent customized applications software intended to provide the monitoring supervision and data acquisition and processing functions for the polar satellites Automated Surface Observing System Imp A system of sensors computers display units and communications equipment to automatically collect and process basic data on surface weather conditions including temperature pressure wind visibility clouds and precipitation Total Sources A-11 Reports for FY 90-93 and FY 90-94 FY 94 IT Operating Plan Resource Summary for FY 94-00 FY 95 IT Operating Plan Resource Summary for FY 95-97 Geophysical Fluid Dynamics Laboratory High Performance Computer System Benefit Cost Analysis submitted to OMB October 1993 12 90 National Weather Service Gateway System’s Upgrade Requirements Initiative WSR-88D Operational Support Facility System Support 11 93 Requirements Initiative Advanced Weather Interactive Processing System Acquisition Office Note Op is operational or being maintained U D is under development Imp is being implemented Page 61 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach This appendix is a compilation of work done by OMB and us on how federal agencies should manage information systems using an investment process It is based upon analysis of the IT management best practices found in leading private and public sector organizations and is explained in greater detail in OMB’s Evaluating Information Technology Investments A Practical Guide 1 Manage Information Technology With an Investment Perspective Leading organizations manage IT projects and systems as investments This approach systematically reduces risks while maximizing benefits because it forces the organization to assess the risks and return of each system throughout its entire life cycle While the specific processes and practices used to implement this approach may vary depending upon the structure of the organization e g centralized versus decentralized operations leading organizations follow several common management activities Specifically these organizations maintain a similar decision-making process consisting of three phases—selection control and evaluation See figure II 1 1 Evaluating Information Technology Investments A Practical Guide Office of Management and Budget Executive Office of the President November 1995 Page 62 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach Figure III 1 An IT Investment Approach Used in Leading Organizations Control What are you doing to ensure that the projects will deliver the benefits projected Select How do you know you have selected the best projects Evaluate Based on your evaluation did the systems deliver what you expected Process Information Selection Phase Choosing the Best IT Investments Key Question How can you select the right mix of IT projects that best meets mission needs and improvement priorities The goal of the selection phase is to assess and prioritize current and proposed IT projects and then create a portfolio of IT projects In doing so this phase helps ensure that the organization 1 selects those IT projects that will best support mission needs and 2 identifies and analyzes a project’s risks and returns before spending a significant amount of project funds A critical element of this phase is that a group of senior executives makes project selection and prioritization decisions based on Page 63 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach a consistent set of decision criteria that compares costs benefits risks and potential returns of the various IT projects Steps of the Selection Phase • • • • Management Tools and Techniques Applicable to This Phase • • • • • Initially filter and screen IT projects for explicit links to mission needs and program performance improvement targets using a standard set of decision criteria Analyze the most accurate and up-to-date cost benefit risk and return information in detail for each project Create a ranked list of prioritized projects Determine the most appropriate mix of IT projects new versus operational strategic versus maintenance etc to serve as the portfolio of IT investments An executive management team that makes funding decisions based on comparisons and trade-offs between competing project proposals especially for those projects expected to have organizationwide impact A documented and defined set of decision criteria that examines expected return on investment ROI technical risks improvement to program effectiveness customer impact and project size and scope Predefined dollar thresholds and authority levels that recognize the need to channel project evaluations and decisions to appropriate management levels to accommodate unit-specific versus agency-level needs Minimal acceptable ROI hurdle rates that apply to projets across the organization that must be met for projects to be considered for funding Risk assessments that expose potential technical and managerial weaknesses Page 64 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach Key Question What controls are you using to ensure that the selected projects deliver the projected benefits at the right time and the right price Control Phase Manage the Investments by Monitoring for Results Steps of the Control Phase Once the IT projects have been selected senior executives periodically assess the progress of the projects against their projected cost schedule milestones and expected mission benefits The type and frequency of the reviews associated with this monitoring activity are usually based on the analysis of risk complexity and cost that went into selecting the project and that are performed at critical project milestones If a project is late over cost or not meeting performance expectations senior executives decide whether it should be continued modified or canceled • • Management Tools and Techniques During This Phase • • • • Use a set of performance measures to monitor the developmental progress for each IT project to identify problems Take action to correct discovered problems Established processes that involve senior managers in ongoing reviews and force decisive action steps to address problems early in the process Explicit cost schedule and performance measures to monitor expected versus actual project outcomes An information system to collect project cost schedule and performance data in order to create a record of progress for each project Incentives for exposing and solving project problems Page 65 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach Key Question Based on your evaluation did the system deliver what was expected Evaluation Phase Learn From the Process Steps of the Evaluation Phase The evaluation phase provides a mechanism for constantly improving the organization’s IT investment process The goal of this phase is to measure analyze and record results based on the data collected throughout each phase Senior executives assess the degree to which each project met its planned cost and schedule goals and fulfilled its projected contribution to the organization’s mission The primary tool in this phase is the postimplementation review PIR which should be conducted once a project has been completed PIRs help senior managers assess whether a project’s proposed benefits were achieved and refine the IT selection criteria • • • Management Tools and Techniques During This Phase • Compare actual project costs benefits risks and return information against earlier projections Determine the causes of any differences between planned and actual results For each system in operation decide whether it should continue operating without adjustment be further modified to improve performance or be canceled Modify the organization’s investment process based on lessons learned Postimplementation reviews to determine actual costs benefits risks and return Page 66 GAO AIMD-96-64 IT Investment Appendix III Description of an Information Technology Investment Process Approach • • Modification of decision criteria and investment management processes based on lessons learned to improve the process Maintenance of accountability by measuring actual project performance and creating incentives for even better project management in the future Page 67 GAO AIMD-96-64 IT Investment Appendix IV Brief Description of Agency IT Management Processes The following sections briefly describe the information technology management processes at each of the five agencies we reviewed These descriptions are intended to characterize the general workings of the agency processes at the time of our review We used the selection control evaluation model as summarized in appendix III and described in detail in OMB’s Evaluating Information Technology Investments A Practical Guide as a template for describing each agency’s IT management process Agency Selection Processes Coast Guard The Coast Guard had an IT investment process used to select IT projects for funding IT project proposals were screened evaluated and ranked by a group of senior IRM managers using explicit decision criteria that took into account project costs expected benefits and risk assessments The ranked list with recommended levels of funding for each project was submitted for review to a board of senior Coast Guard officers and then forwarded to the Coast Guard Chief of Staff for final approval Environmental Protection Agency EPA used a decentralized IT project initiation selection and funding process Under this broad process program offices independently selected and funded IT projects on a case-by-case basis as the need for the system was identified EPA had IRM policy and guidance for IT project data and analysis requirements—such as a project-level risk assessment and a cost-benefit study—that the program offices had to identify in order to proceed with system development EPA did not have a consistent set of decision criteria for selecting IT projects Internal Revenue Service selection and funding activities within IRS differed depending on whether the project was part of the Tax System Modernization TSM or an operational system In 1995 IRS created a senior-level board for selecting controlling and evaluating information technology investments and began to rank all of the proposed TSM projects using its cost risk and return decision criteria However these criteria were largely qualitative data used were not validated or reliable and the analyses were not based on calculations of expected return on investment According to IRS its IT Page 68 GAO AIMD-96-64 IT Investment Appendix IV Brief Description of Agency IT Management Processes investment review board used a separate process with different criteria for evaluating operational systems The board did not review research and development systems or field office systems IRS did not compare the results of its different evaluation processes National Aeronautics and Space Administration Within NASA IT project selection and funding decisions were made by domain-specific program managers NASA had two general types of IT funding—program expenditures and administrative spending Most of NASA’s IT funding was embedded within program-specific budgets Managers of these programs had autonomy to make system-level and system support IT selection decisions Administrative IT systems were generally managed by the cognizant NASA program office or center has recently established a CIO council to establish high-level policies and standards approve information resources management plans and address issues and initiatives The council will also serve as the IT capital investment advisory group to the proposed NASA Capital Investment Council NASA plans for this Capital Investment Council to have responsibility for looking at all capital investments across NASA including those for IT While this Capital Investment Council may fill the need for identifying cross-functional opportunities it is not yet operational NASA National Oceanic and Atmospheric Administration project selection and funding decisions at NOAA were made as part of its strategic management and budgeting process NOAA had seven work teams—each supporting a NOAA strategic goal—that prioritized incoming funding requests Managers on these work teams negotiated to determine IT project funding priorities within the scope of their respective strategic goals These prioritization requests were then submitted to NOAA’s Executive Management Board which had final agency decision authority over all expenditures A key decision criterion used by the work teams was the project’s contribution to the agency’s strategic goals however no standard set of decision criteria was used in the prioritization decisions Other data such as cost-benefit analyses were also sometimes used to evaluate IT project proposals although use of these data sources was not mandatory IT Page 69 GAO AIMD-96-64 IT Investment Appendix IV Brief Description of Agency IT Management Processes Agency Control Processes Coast Guard The Coast Guard conducted internal system reviews but these reviews were not used to monitor the progress of IT projects The review efforts were designed to address ways to improve efficiency reduce project cost and reduce project risk Cost benefit and schedule data were also collected annually for some new IT projects but the Coast Guard did not measure mission benefits derived from each of its projects Environmental Protection Agency had a decentralized managerial review process for monitoring IT projects EPA’s IRM policy set requirements for the minimum level of review activity that program offices had to conduct but program offices had primary responsibility for overseeing the progress of their IT projects In an effort to provide a forum for senior managerial review of IT projects EPA in 1994 created the Executive Steering Committee ESC for IRM to guide EPA’s agencywide IRM activities The ESC was chartered to review IRM projects that are large important or cross-organizational The committee’s first major system review was scheduled for some time in 1996 EPA is currently formulating the data submission requirements for the ESC reviews Internal Revenue Service IRS National Aeronautics and Space Administration senior executives regularly reviewed the cost and schedule performance of major programs and projects but they reviewed only the largest IT projects No central IRM review has been conducted since 1993 EPA regularly conducted senior management program control meetings PCM to review the cost and schedule activity of TSM projects IRS had two types of PCMs The four TSM sites—Submission Processing Computing Center Customer Service and District Office—conducted PCMs to monitor the TSM activity under their purview Also IRS could hold “combined PCMs” to resolve issues that spanned across the TSM sites IRS did not conduct PCMs to monitor the performance of operational systems To date 1 working procedures 2 required decision documents 3 reliable cost benefit and return data 4 and explicit quantitative decision criteria needed for an effective investment control process are not in place for the IRS Investment Review Board NASA Page 70 GAO AIMD-96-64 IT Investment Appendix IV Brief Description of Agency IT Management Processes NASA put senior-level CIOs in place for each NASA center but these CIOs exercised limited control over mission-related systems and had limited authority to enforce IT standards or architecture policies NASA’s proposed Capital Investment Council which is intended to supplement the Program Management Council by reviewing major capital investments may address this concern once the Investment Council is operational National Oceanic and Atmospheric Administration conducted quarterly senior-level program status meetings to review the progress and performance of major systems and programs such as those in the NWS modernization NOAA had defined performance measures to gauge the progress toward its strategic goals but did not have specific performance measures for individual IT systems Also while some offices had made limited comparisons of actual to expected IT project benefits NOAA did not require the collection or assessment of mission benefit accrual information on IT projects NOAA Agency Evaluation Processes Coast Guard The Coast Guard did not conduct any postimplementation reviews of IT projects Instead the Coast Guard focused its review activity on systems that were currently under development Environmental Protection Agency did not conduct any centralized postimplementation reviews EPA did conduct postimplementation reviews as part of the General Services Administration’s GSA triennial review requirement but curtailed this activity in 1992 when the GSA requirement was lifted Internal Revenue Service directives required that postimplementation reviews be conducted 6 months after an IT system is implemented At the time of our review IRS had conducted five postimplementation reviews and had developed a standard postimplementation review methodology However no mechanisms were in place to ensure that the results of these IRS investment evaluation reviews were used to modify the IRS selection and control decision-making processes or alter funding decisions for individual projects EPA IRS Page 71 GAO AIMD-96-64 IT Investment Appendix IV Brief Description of Agency IT Management Processes National Aeronautics and Space Administration NASA did not conduct or require any centralized project postimplementation reviews NASA stopped conducting centralized IRM reviews in 1993 and now instead urges programs to conduct IRM self-assessments National Oceanic and Atmospheric Administration While the agency conducted other reviews NOAA’s IRM office has participated in only four IRM reviews over the last 3 years These reviews tended to focus on specific IT problems such as evaluating the merits of electronic bulletin board systems or difficulties being encountered digitizing nautical navigation maps No postimplementation reviews had been conducted over the past 3 years Page 72 GAO AIMD-96-64 IT Investment Appendix V Summary of Investment-Related Provisions of the Information Technology Management Reform Act of 1996 On February 10 1996 the Information Technology Management Reform Act of 1996 Division E of Public Law 104-106 was signed into law This appendix is a summary of the information technology investment-related provisions from this act it is not the actual language contained in the law Provision Summary of Provision Narrative Sec 5002 3 Information technology IT is defined as any equipment or interconnected system or subsystem of equipment that is used in the automatic acquisition storage manipulation management movement control display switching interchange transmission or reception of data or information It may include equipment used by contractors Sec 5112 b The OMB Director is to promote and be responsible for improving the acquisition use and disposal of IT by federal agencies Sec 5112 c The OMB Director is to develop a process as part of the budget process for analyzing tracking and evaluating the risks and results of major capital investments for information systems the process shall include explicit criteria for analyzing the projected and actual costs benefits and risks associated with the investments over the life of each system Sec 5112 c The OMB Director is to report to the Congress at the same time the budget is submitted on the net program performance benefits achieved by major capital investments in information systems and how the benefits relate to the accomplishment of agency goals Sec 5112 e The OMB Director shall designate as appropriate agency heads as executive agents to acquire IT for governmentwide use Sec 5112 f The OMB Director shall encourage agencies to develop and use “best practices” in acquiring IT Sec 5113 b 2 The OMB Director shall direct that agency heads 1 establish effective and efficient capital planning processes for selecting managing and evaluating information systems investments 2 before investing in new information systems determine whether a government function should be performed by the private sector the government or government contractor and 3 analyze their agencys’ missions and revise the mission-related and administrative processes as appropriate before making significant investments in IT Sec 5113 b 4 Through the budget process the OMB Director is to review selected agency IRM activities to determine the efficiency and effectiveness of IT investments in improving agency performance Sec 5122 a Agency heads are to design and implement a process for maximizing the value and assessing and managing the risks of IT investments Sec 5122 b The agency process is to 1 provide for the selection management and evaluation of IT investments 2 be integrated with the processes for making budget financial and program management decisions 3 include minimum criteria for selecting IT investments and specific quantitative and qualitative criteria for comparing and prioritizing projects 4 provide for identifying potential IT investments that would result in shared benefits with other federal state or local governments 5 provide for identifying quantifiable measurements for determining the net benefits and risks of IT investments and 6 provide the means for senior agency managers to obtain timely development progress information including a system of milestones for measuring progress on an independently verifiable basis in terms of cost capability of the system to meet specified requirements timeliness and quality Sec 5123 3 Agency heads are to ensure that performance measurements are prescribed for IT and that the performance measurements measure how well the IT supports agency programs continued Page 73 GAO AIMD-96-64 IT Investment Appendix V Summary of Investment-Related Provisions of the Information Technology Management Reform Act of 1996 Provision Summary of Provision Narrative Sec 5123 4 Where comparable processes and organizations exist in either the public or private sectors agency heads are to quantitatively benchmark agency process performance against such processes in terms of cost speed productivity and quality of outputs and outcomes Sec 5124 a 1 Agency heads may acquire IT as authorized by law the Brooks Act—40 U S C 759—is repealed by sec 5101 except that the GSA Administrator will continue to manage the FTS 2000 and follow-on to that program sec 5124 b Sec 5125 a Agency heads are to designate Chief Information Officers in lieu of designating IRM officials—as a result of amending the Paperwork Reduction Act appointment provision Sec 5125 b Agency Chief Information Officers CIOs are responsible for 1 providing advice and assistance to agency heads and senior management to ensure that IT is acquired and information resources are managed in a manner that implements the policies and procedures of the Information Technology Management Reform Act of 1996 is consistent with the Paperwork Reduction Act and is consistent with the priorities established by the agency head 2 developing maintaining and facilitating the implementation of a sound and integrated agency IT architecture and 3 promoting effective and efficient design and operation of major IRM processes Sec 5126 Agency heads in consultation with the CIO and CFO are to establish policies and procedures that 1 ensure accounting financial and asset management systems and other information systems are designed developed maintained and used effectively to provide financial or program performance data for agency financial statements 2 ensure that financial and related program performance data are provided to agency financial management systems on a reliable consistent and timely basis and 3 ensure that financial statements support the assessment and revision of agency mission-related and administrative processes and the measurement of performance of agency investments in information systems Sec 5127 Agency heads are to identify in their IRM plans required under the Paperwork Reduction Act major IT acquisition programs that have significantly deviated from the cost performance or schedule goals established for the program the goals are to be established under title V of the Federal Acquisition Streamlining Act of 1994 Sec 5141 This section establishes which provisions of the title apply to “national security systems ” Sec 5142 “National security systems” are defined as any telecommunications or information system operated by the United States government that 1 involves intelligence activities 2 involves cryptologic activities related to national security 3 involves command and control of military forces 4 involves equipment that is an integral part of a weapon or weapon system or 5 is critical to the direct fulfillment of military or intelligence missions Sec 5401 This section requires the GSA Administrator to provide through the Federal Acquisition Computer Network established under the Federal Acquisition Streamlining Act of 1994 or another automated system not later than January 1 1998 governmentwide on-line computer access to information on products and services available for ordering under the multiple award schedules Sec 5701 The Information Technology Management Reform Act takes effect 180 days from the date of enactment February 10 1996 Page 74 GAO AIMD-96-64 IT Investment Appendix VI Major Contributors to This Report Accounting and Information Management Division Washington D C David McClure Assistant Director Danny R Latta Adviser Alicia Wright Senior Business Process Analyst Bill Dunahay Senior Evaluator John Rehberger Information Systems Analyst Shane Hartzler Business Process Analyst Eugene Kudla Staff Evaluator 510981 Page 75 GAO AIMD-96-64 IT Investment Ordering Information The first copy of each GAO report and testimony is free Additional copies are $2 each Orders should be sent to the following address accompanied by a check or money order made out to the Superintendent of Documents when necessary VISA and MasterCard credit cards are accepted also Orders for 100 or more copies to be mailed to a single address are discounted 25 percent Orders by mail U S General Accounting Office P O Box 6015 Gaithersburg MD 20884-6015 or visit Room 1100 700 4th St NW corner of 4th and G Sts NW U S General Accounting Office Washington DC Orders may also be placed by calling 202 512-6000 or by using fax number 301 258-4066 or TDD 301 413-0006 Each day GAO issues a list of newly available reports and testimony To receive facsimile copies of the daily list or any list from the past 30 days please call 202 512-6000 using a touchtone phone A recorded menu will provide information on how to obtain these lists For information on how to access GAO reports on the INTERNET send an e-mail message with info in the body to info@www gao gov or visit GAO’s World Wide Web Home Page at http www gao gov PRINTED ON RECYCLED PAPER United States General Accounting Office Washington D C 20548-0001 Official Business Penalty for Private Use $300 Address Correction Requested Bulk Rate Postage Fees Paid GAO Permit No G100