FOIA Search Survey Results and Analysis

By Swetha Kareti, Nate Jones, and Sean Moulton

Download the PDF version of this report here. 

The National Security Archive and the Project on Government Oversight developed and circulated an online survey to both FOIA processors and requesters to better understand how agencies search for records requested under the Freedom of Information Act (FOIA).  The aim of this survey was to establish a foundation of knowledge on how agencies across government complete perhaps the most agency-unique, complicated, and time-consuming aspect of the FOIA process: finding the documents that have been requested under the law.

METHODOLOGY

The National Security Archive and the Project on Government Oversight used Google Forms online software to prepare the survey.  Respondents first identified themselves as either a FOIA requester or a FOIA processor.  Different sets of questions for were given to each group, and the results were kept completely anonymous in order to allow respondents to answer candidly. The survey was conducted over a nearly month-long time frame, during which it received 57 responses. Of the 57 responses in total, 30 were processors and the remaining 27 were FOIA requesters.

The questions we asked were:

For Requesters:

• Please describe in as much detail as possible the FOIA search procedures that you are aware of (from personal knowledge, government reports, and/or court declarations).
• If not described above, please describe the procedures for searching email and other electronic records that you have knowledge of.
• Are there inefficiencies in the FOIA search procedures that you are familiar with? If so, what are they? How could they be improved?
• Please provide links or other references to documentation of FOIA searches that you are aware of.
• Anything else you would like to add?

For Processors:

• Which agency do you currently work for? Your identity will remain anonymous.
• What search software does your agency use? Approximately how old is the software? Are you satisfied with it?  Are there notable limitations with the software that you think should be addressed?
• Please describe in as much detail as possible the FOIA search procedures at your agency or other agencies that you have knowledge of.
• If you have not described it above, please describe the procedures for searching email and other electronic records at your agency or other agencies that you have knowledge of. Do you have the ability to search electronic records?  If not, what is the reason given for this?
• Are there inefficiencies in the FOIA search procedures that you are familiar with? If so, what are they? How could they be improved?
• Anything else you would like to add?

THE SEARCH PROCESS

The Department of Justice Guide to the Freedom of Information Act states that, “The FOIA defines the term ‘search’ as ‘to review, manually or by automated means, agency records for the purpose of locating those records which are responsive to a request’” and that,  “[a]s a general rule, courts require agencies to undertake a search that is ‘reasonably calculated to uncover all relevant documents.’”  But the Guide provides little guidance as to the mechanisms agencies can or must use to “uncover all relevant documents.”  In practice, each of the 100 different agencies subject to the Freedom of Information Act completes searches in its own unique way.  According to the results of this survey, some agencies use state of the art software, while others use no software at all.

Our survey showed in practice –and it is important to remember the high level of variance between each of the 100 agencies subject to FOIA– that the search process goes something like this:

First, an agency receives a FOIA request.  The request goes through an “initial phase” where an agency reviews if the request was sent to the correct agency or component, determines the request’s fee category and/or fee waiver status, and asks and answers other “initial” questions.  The time this phase takes ranges from hours to months, depending on the agency.

Then, the request goes into a queue to be processed once it has been “perfected.”  Generally, the first phase of processing a request is for the requested document or documents to be searched for.

The FOIA Ombudsman, the blog of the Office of Government Information Service (OGIS), states that, “It appears that frequently, Agencies are responsible for a mind-boggling amount of information, and finding a record amid years and years of accumulated documents can feel like searching for the proverbial needle in the haystack.”  This assessment appears to be largely correct.

Moreover, each agency searches their “haystack” in their own way.   Some appear to do so methodically and efficiently, others haphazardly and imprecisely.  Some agencies and agency components have access to state of the art e-discovery tools to search their records for the requested documents (though as was pointed out in the January 2017 FOIA Advisory Committee Meeting, some agencies only use these efficient tools after being sued); other agencies do not even search records electronically at all.

Some FOIA professionals, including those at the Department of State, have centralized search capabilities, meaning that they have the ability to search large portions of their agency’s records themselves.  Because the primary mission of FOIA processors at agencies is to process FOIA requests as quickly as possible, and because FOIA professionals are often well-trained at records management and retrieval, often centralized search is the quickest way to procure the documents requested.

Most agencies, however, appear to not have centralized FOIA search capabilities.  This means that after the FOIA office receives a request, it “tasks it out” to the part of the agency that it believes is likely to have the documents.  Then, in addition to all of their other job obligations, these federal employees –sometimes termed “subject matter experts”–  are tasked to retrieve the requested document and provide it to the FOIA office to review (often with suggestions on what should be withheld or released).  Although the Department of Justice states that, “FOIA is everyone’s responsibility,” often these “subject matter experts” perform slow, unthorough, or otherwise imprecise searches for the requested records.  According to the review of the responses to this survey, this “search black hole” –when “subject matter experts” do not quickly respond to search taskings by the FOIA department– appears to be one of, if not the, primary causes for unacceptably long response times to FOIA requests.

The FOIA office then receives the requested document or documents, reviews them, and sometimes redacts them for release.  Unfortunately, statistics show that an extremely large percentage of FOIA requests (over 16 percent) are denied because an agency claims that “no responsive records were found.”  Certainly, there are cases where requesters send requests to the wrong agency, request documents that do not exist, or where the requested records were properly (or improperly) destroyed.  But many of these “no responsive document” denials are in fact the result of improper or poorly conducted searches.  Administrative appeals often result in a second, more thorough search that finds the documents requested.

Finally, after the FOIA office receives and reviews the requested documents –and if coordination or referral with another agency or component is not required– the last step is mailing or emailing the records to the requester with a letter explaining the number of documents released, the justification for any redaction, and the rights of the requester to appeal or contact OGIS.

ANALYSIS

After reviewing the results of our survey, the National Security Archive and the Project on Government Oversight’s have come to the following conclusions about the FOIA search processes:

• Every single FOIA shop would be wise to reevaluate their search procedures; almost none of the 57 FOIA requesters and processors surveyed stated that search procedures are working perfectly.  Senior agency officials and FOIA managers should ask their teams what can be done –much without spending more money– to improve searches.  Agencies and FOIA professionals must resist the temptation to not improve processes, simply because “they have always been done this way.”

• FOIA search processes are not nearly as streamlined or as optimized as they could be. According to the survey, 25% of FOIA professionals did not use any electronic or searchable databases in order to process FOIA requests. Another 28.6% of respondents used “combination” search methods, which were usually a mix of databases alongside other methods, like email searches, to process these requests. In short, over 50% of the FOIA processors responded that they did not use an agency-wide database/tracker in order to process FOIA requests. Clearly, the lack of standardized software/search methods in FOIA processing can lead to major inefficiencies and delays.

• Agencies must invest in organizational FOIA processing software (if they have not already) that can effectively streamline the process. Moreover, agency decision makers and software purchasers must factor in FOIA-searchability when they purchase other software used by their agency. Much of the poor search situation FOIA shops find themselves in today is due to the fact that their agencies bought software that “does not play well” with FOIA.  FOIA professionals must make this known and advocate for future software purchases that make their jobs (and complying with the law!) easier.

• Agencies should allow FOIA processors to search all digital records agency-wide. Allowing these records to be searched digitally in response to FOIA requests will greatly decrease the time it takes to process FOIA requests.  While there is some notable opposition to this strategy for privacy, security, and other concerns (one respondent wrote FOIA processors should not have a “skeleton key” to all agency records), it should be recognized that allowing non-FOIA employees to search through emails without supervision and possibly delete, or simply not provide requested emails or other records, is also not a viable solution.

• It is true that FOIA searches are often a low priority for departments and agencies, and consequently receive very little funding. While we are sympathetic to this concern and agree that the best FOIA programs consistently have support from the top of their agency, many of the problems with FOIA searches are due the implementation of ingrained, unquestioned inefficiencies.  It is extremely likely that many of these inefficiencies can be fixed with no or minimal funding increases.  Agency heads, senior FOIA officials, and FOIA processors should put their heads together and reevaluate which fixes, small and large, could be implemented to the most inefficient part of the FOIA process: the search for records.

NOTABLE FOIA SOFTWARE/SERVICES

According to our survey and product websites, the following software programs are used by the following agencies to conduct their FOIA searches.

While there may not be a workable solution to allow FOIA professionals to centrally search all paper records (including those, for example, held at agencies’ remote field offices), electronic records –including emails– hold the promise of allowing FOIA professionals to quickly and efficiently search for the records themselves.  Allowing centralized search by FOIA professionals would likely, “cut out the middleman” and greatly reduce the time it takes to process requests.  According to results of this survey, some agencies are moving in this direction with their digital records, others are not.

Clearwell

https://www.veritas.com/product/information-governance/ediscovery-platform

National Marine Fisheries Service, Department of Education.

Eos.web

Bureau of Reclamation UC Region

Email

National Park Service, Corporation for National and Community Service.

FOIA Online

https://foiaonline.regulations.gov/foia/action/public/home

The Environmental Protection Agency, Department of Commerce (except the US Patent and Trademark Office), U.S. Customs and Border Protection, Office of General Counsel of the National Archives and Records Administration, Merit Systems Protection Board, Federal Labor Relations Authority, Pension Benefit Guaranty Corporation, Department of the Navy, General Services Administration, Federal Communications Commission, Small Business Administration, Department of Justice – Office of Information Policy, Defense Logistics Agency and the Office of the Inspector General of the Department of Defense (per website).

FOIAXpress

http://ains.com/foiaxpress/

Court Services and Offender Supervision Agency, United States Forest Service.

Infolinx

http://www.infolinx.com/

Bureau of Reclamation.

Laser Fiche

https://www.laserfiche.com/

Department of Commerce.

SEMS-RM

Environmental Protection Agency  (as contractor)

SORNS

United States Customs and Border Protection

ZL Tech

http://www.zlti.com/

National Archives and Records Administration.

Combination of databases

Department of the Interior, Defense Information Systems Agency, Environmental Protection Agency (Region 4), United States Bureau of Reclamation, Department of Health and Human Services.

None

National Guard Bureau, Navy (Field Activity), US Forest Service, Bureau of Land Management, United States Fish and Wildlife Service, National Institutes of Health, National Oceanic and Atmospheric Administration

RAW DATA

The complete set of full survey responses can be viewed here.