It is hard to have a reasoned conversation in Washington, DC, about the problems associated with government transparency. Criticize some aspect of it and reform groups are quick to label your position as being against transparency generally. The idea that open government equates to better democracy is a core tenet of American political reform doctrine. Another dose of transparency is a popular remedy for nearly everything that ails American democracy.¹ And thanks to high tech and the internet, U.S. citizens have more tools than ever before to monitor public officials if they so choose. On the face of it, all of this seems good for American democracy. But is it really? 

Some threshold level of transparency is indisputably essential to modern democracy. Citizens need to have enough information about their public officials in order to hold them accountable. Transparency also enables checks and balances between different government branches and promotes party competition. But transparency’s meaning and scope have evolved over time well beyond threshold levels. Transparency rules have multiplied, becoming more specific and detailed at the same time. In addition, they have expanded temporally, encompassing not only what public officials have done, but also what they are doing at the moment and what they intend to do in the future. Since the entire transparency framework evolved episodically in reaction to specific scandals, it is reasonable to question its coherence and ask whether it still aligns intelligently with contemporary political conditions.

Transparency opportunities often serve purposes other than democratic accountability—for example, allowing competitors to find out what rival businesses are up to, providing the media with an entertaining narrative of behind-the-scenes negotiations, and giving the losers of legislative battles another chance to reverse or at least modify policy at the implementation stage. These interests are at best unrelated and at worse detrimental to the democratic purpose of making government policies and actions responsive to the majority.

At the same time, the political landscape has changed dramatically since freedom of information laws, open meeting rules, and public comment opportunities were first introduced. Today, politics is highly professionalized, polarized, and pervasive in all phases of policymaking. Operating in the context of a highly fractured U.S. government, this means more opportunities for obstructing electoral mandates and greater special interest influence. 

In addition, government services are often contracted out to private enterprises and nonprofits that operate far less transparently than the official government. American democracy might be better served by making these delegated government functions more transparent than by simply increasing official scrutiny of the public sector. 

Government transparency has evolved into many forms covering many aspects and all stages of policymaking. Candidates for office are subject to formal campaign finance disclosure laws, and increasingly also to the informal presumption that they should reveal their financial assets, medical histories, and other personal information so that voters can better judge their qualifications for office. Conflict-of-interest and lobbying-disclosure rules at the legislative and rule-making phases were put in place to prevent various types of political corruption. Open-meeting laws allow citizens and organizations to observe and give testimony at agency proceedings, enhancing their capacity to monitor and influence the bureaucrats who implement the laws. And after government decisions are made, citizens can ask for and obtain documents about government decisions and actions by making Freedom of Information requests in the interests of accountability. The rationales in each instance make sense on paper. 

 But they also rest on key behavioral assumptions about citizens and the way democratic institutions operate that are unrealistic in practice. Average citizens do not review the candidate’s disclosure statements. Those documents are uncovered and used in campaign advertising by political consultants, often in a distorted way. Average citizens also rarely provide public comment for agencies, commissions, and legislative bodies. Mostly, nonprofit and commercial stakeholders do. FOIA laws were sold as mechanisms of democratic accountability, but in reality most FOIA requests are made either by citizens looking for specific information about themselves or by businesses and other organizations looking for information about their competitors. As a consequence of transparency capture by both interested and self-interested intermediaries, decisions that were seemingly settled at the ballot box or in Congress can be modified or blocked by stakeholders at a later stage in the policymaking process.  

Does this imply that modern transparency tools should be abandoned? No, but it means that we need to make more realistic assumptions about the behavior of average citizens and try to reform transparency opportunities in a way that reduces democratic distortion and excessive obstruction. 

This is particularly important in an era characterized by increasing polarization and professionalization. Polarization makes stakeholders and party activists less willing to accept compromise. Political professionalism enhances a party or organization’s capacity to monitor government and ensure that its interests are being protected adequately. The latter also drives up election costs and makes elected officials more dependent upon donors and independent spenders. The combined effect of polarization and professionalization makes it harder for elected officials to act without the prior consent of key individuals and groups in their party coalitions. 

This is not just a Republican problem with regard to the Tea Party factions or the Koch brothers. Democrats are also constrained by their core supporters. Trade unions, for instance, blocked Trade Adjustment Assistance in 2015 by threatening to withhold PAC contributions from Democrats in Congress who voted the wrong way. 

Congress nonetheless retains more deliberative space for itself than it allows Executive Branch agencies or Federal advisory committees. The Administrative Procedures Act (APA) of 1946 governs the way that agencies enact regulations, requiring them to inform the public and to provide for public participation in the rule-making process. But it exempts Congress from its provisions. The Federal Advisory Committee Act of 1972 (FACA) also applies to the approximately 900 Federal advisory committees that advise Federal agencies, but not to Congress. Administered by the GSA, FACA requires the advisory committees to hold open meetings and allow for public participation. 

Given the technical nature of regulations and permits, Federal agencies and advisory committees rarely hear from average citizens. More often, stakeholders with strong material interests and preferences dominate the public hearings and opportunities for ex parte discussions with agency officials. Consequently, there is no guarantee that these agencies and committees will receive balanced information reflecting broader public interests.  

Agency officials also often have to balance valid competing considerations without clear guidelines and priorities when applying regulations or issuing permits, especially in the areas of energy and the environment. Weighing tradeoffs and different interests is an inherently political task, but bureaucrats rarely have the proper training or necessary political instincts to handle this competently. Attaching democratic procedures to the policy implementation stage has the unintended effect of prolonging political battles and preventing closure. Public comment opportunities can assist sore losers to modify or nullify laws that they could not stop at the electoral or legislative stages.  

Democracies need to be able to achieve policy closure at some reasonable point in order to govern effectively. This is a particularly important consideration in the U.S. political system because the prevalence of divided power and a Federal structure renders it particularly susceptible to gridlock. If interest groups also capture the public comment and participation opportunities, it exacerbates the problems that come with having so many veto points. 

When there are many conflicting interests and laws, it can be hard even under the best of conditions for skilled politicians to achieve a majority consensus. Voting rules at the electoral or legislative stages achieve closure procedurally by foreclosing certain options based on pre-defined levels of threshold support. But when a particular environmental and energy project involves several different laws and multiple agencies, there is often no process for officials to deliberate together or reconcile their differences into a collective decision. Instead, each agency makes separate choices, creating, in effect, a system of sequential vetoes and enabling those who oppose the project to delay it using the public comment process. 

This problem is especially evident in environmental permitting. With multiple relevant laws in play (for example, the Clean Water Act, the Endangered Species Act, NEPA, and so forth), local permitting processes allow interested parties to use public comment opportunities to throw objections, questions, and demands at agencies with no clear guidelines about what needs a response and why. Since the online submission enables and encourages more public input, agencies can be overwhelmed by the amount of public testimony directed their way. 

Consider a pertinent California example. The San Francisquito Creek begins in the Bay Area hills and runs down the Palo Alto, Menlo Park, and East Palo Alto borders. In 1998, it flooded many areas of Palo Alto and East Palo Alto, causing $20 million of damage. The neighboring communities and water district shortly thereafter formed a Joint Powers Authority (JPA) to raise revenue and devise a plan to build floodwalls around the creek to prevent future damage. In order to get approval for this project, the JPA needed to go through state and Federal environmental reviews and obtain approvals from several agencies including the EPA, the Corps of Engineers, the State Water Resources Control Board of California, the state’s Department of Fish and Wildlife, the Bay Conservation and Development Commission, and others. 

As of 2015, the project had still not finished running the gauntlet of necessary approvals after 17 years of effort. The JPA’s floodwall proposal was rejected several times for not providing adequate answers to objections raised by various outside groups. Stakeholders continued to raise questions about non-flood matters such as what Stanford University was planning to do about silt behind the dam where the creek originates, whether an endangered bird or field mouse in an adjoining marshland would be adequately protected, whether the water could be diverted onto a local airport instead of a salt marsh, and the like. 

City officials believed that the public comment process has been used to leverage the bargaining power of groups that sought policy goals beyond flood protection. Stakeholder demands for more information seemed to them a delay tactic intended to build pressure on local officials who were understandably anxious about escalating project costs and the lack of public safety. The lead agency, the State Water Resources Control Board, had no statutory authority to prioritize the flooding problem over these other issues, including whether or not to construct walkways on the levies to enable nature walks in the wetlands area where the creek enters San Francisco Bay. The absence of rules about ex parte discussions was also a problem as it led to suspicions about informal meetings between stakeholders and the Agency staff that excluded JPA officials. 

Leaving aside the merits of whether any or all of these non-flood issues were legitimate, transparency capture of public comment opportunities in the context of a sequential permitting process has created a formidable obstacle for local communities trying to protect their citizens. The whole situation is one major storm away from a political and human catastrophe. And while the process drags on, the community most likely to suffer is the minority community of East Palo Alto living in the dense low-income housing adjacent to the creek. 

Technology also challenges our current transparency framework. In the not-so-distant past, public officials primarily communicated with each other and their constituents by phone, written notes, written memos, and conversation. Now, like most citizens, public officials have a much wider range of options, including email, twitter, Facebook, and other social media. In theory, this change should improve communications between public officials and their constituents since email is a far cheaper and faster way to communicate than sending updates to constituents by regular bulk mail. But it also raises novel problems due to the speed and spread of internet transmission. 

Public officials sometimes react more quickly than they should. Tone-deaf statements about gender or race can drag an official into a career-ending death spiral. While some politicians like Donald Trump or Sarah Palin might relish this kind of controversy as enhancing their brand and rallying their base supporters, most would not. Eventually, political consultants will make sure that all Twitter and Facebook messages are carefully crafted and edited messages. The volume of communication may increase but the value of the content will remain low.

More problematic is the blurring of private and public communication methods in email. In the past, the dividing line between private and public conversation was clearer. You were unlikely to send a memo to your spouse, children or friends. Unless traveling, you would most often talk to them personally or over the phone. But today, a person might text or email friends, colleagues, or family several times a day. Using a work device for private messaging often violates official prohibitions against using public resources for private purposes. At the same time, maintaining separate phones, servers, or computers increases the odds of violating transparency requirements. 

 This issue is vividly illustrated by the controversy over Hillary Clinton’s use of a private email server for some communications with staff while she was Secretary of State. The controversy began because State Department officials were unable to locate some of her emails related to the September 2012 Benghazi terrorist attack. Republicans in the House were convinced that there was more to the story than she had revealed in her testimony before Congress, and immediately suspected that she had deleted potentially damaging emails from her private account. There was no official prohibition against using private email for official business during the time that Clinton was Secretary of State. That changed in 2009 when President Obama signed an amendment to the 1950 Presidential and Federal Records Act expanding Federal records to include electronic communications and requiring that they be copied or preserved within twenty days of transmission. 

While the issue of whether Hillary Clinton complied with the letter and spirit of the State Department email rules dominated most of the ensuing public debate, there was little serious consideration about whether and when the emails of government officials should be part of the public record. Emails are often written in an informal and personal style even when their subject concerns official business. This is good for the press because it make otherwise dull political reporting more fun to read. 

For instance, the National Inquirer wrote a July 9 article with the headline “Hillary Clinton’s Emails Reveal Lesbian Lust”, based on one email that revealed a staffer’s shock that a Wall Street Journal reporter got so close to Secretary Clinton that they were “knee to knee” to the point of practically being “knee in between knee.” Most of the 30,000 emails that Secretary Clinton eventually released contained very little of policy importance. The press was left with only personal anecdotes about Clinton’s troubles using the home fax machine and details about the assurances she obtained about the friendliness of upcoming interviews. 

The press reaction to a proposal to put all FOIA requests online gives some insight into the tension between the private and public uses of FOIA. When the Obama Administration announced a pilot project in which seven agencies would post all FOIA documents simultaneously to the requester and the public, it immediately drew objections from the Washington Post, a heavy FOIA user, which claimed that it would hurt its reporters’ ability to report exclusively on what they found. This is understandable, but insofar as FOIA’s purpose is democratic accountability, it seems logical that other groups and individuals should have the right to view FOIA documents directly as well. 

The core questions, however, are whether emails written as work products during the process of deliberation over government policy or action should be made public, and, if so, when. A Congressional committee doing oversight on the Executive Branch might justifiably need to see materials that relate to an investigation involving potential negligence or culpability. But does that mean the entire email record should be widely published? Perhaps the entire record should be shared initially only with the committee. Perhaps only the emails relevant to culpability or negligence issues should be disclosed publicly at the appropriate moment, and the rest of the record not disclosed. 

But some argue that full democratic accountability requires knowing more than what was decided and the official reasons for it. Citizens, they argue, should know how an agency or public official arrived at a given decision, including whom they consulted, what was discussed about different alternatives, and so forth. This is what I term “in-process” transparency. 

Requiring all emails to be preserved for the public record will inevitably drive most frank discussions between officials out of email. Officials will be acutely aware that anything they write in an email about policy or other players might be revealed publicly. The predictable effect of this will be that they will discuss serious matters in person or over the phone. Email transparency is thus ultimately self-defeating in the sense that it discourages any exchange of the meaningful information that transparency advocates seek to discover. And if email is really a better and faster way to exchange early drafts and analyses, this policy will undermine government efficiency to some degree as well. 

More broadly, in-process transparency can hinder honest discussion and better decision-making. In an ideal world of fully engaged citizens with lots of time on their hands, one might believe that officials are more likely to make decisions in keeping with the majority’s preferences when they are closely watched by the public. But in the real world of American politics, interested individuals and organizations, not average citizens, have the greater incentive and means to monitor the government closely. This can open the door to obstruction and policy distortion as it enables regulatory capture by interested parties who advocate freely for their views without any countervailing public voice. 

Another development that has changed the context in which our transparency framework operates is the growing importance of private commercial and non-profit agents for the delivery of government services. In the traditional conception of the state dating back to Max Weber, the public sector was clearly demarcated from the private sector. Government bureaucrats were paid by the state and assigned specific roles within a hierarchical reporting structure. Civil service reforms inoculated public servants from political influence and introduced a merit basis for promotion. Conflict-of-interest rules kept government employees focused on the public interest, not personal financial gain. Inspector Generals and the GAO provided internal checks, and Congressional committee oversight operated as an external check on agency fraud, waste, and abuse. 

Transparency played an essential role in this traditional story of Federal bureaucracy accountability. Rules and procedures were formalized and published. FOIA laws provided access to the documents agencies produce. GAO reports served a public auditing function. And in the absence of a competitive market, the merit system, public service norms, Congressional budget scrutiny, and the threat of public exposure funneled bureaucratic attitudes and actions in the direction of a democratic mandate.

In recent decades, the U.S. government has contracted out an increasing number of public services to commercial and non-profit organizations, particularly in the period up to 2008. The underlying rationale was partly based on America’s faith in market efficiency and its fundamental suspicion about strong centralized government. The effort to remake government was bipartisan. Vice President Al Gore’s initiative to reinvent government sought to bring private sector efficiency and consumer orientation to the public sector and to make government leaner and smaller. The Bush Administration was innately suspicious of government and preferred the private sector even to carry out certain military functions of the Iraq War. 

Outsourcing public services was an important aspect of this trend. As a consequence, the size of our proxy government is now quite large. Scholar of public service Paul Light estimated that there were 7.5 million Federal contract employees in 2013 as compared to about two million full-time Federal government employees. Contract spending in 2014 constituted 11.8 percent of all Federal spending.

And yet, there is much that we do not know about our government by proxy. As Don Kettl points out, we do not have precise numbers for the size of the contracted employees. We also do not know how much is done by subcontractors, how contract management is split between government employees and contractors, and most importantly, whether outsourced work is really cheaper or better in quality than the insourced work of full-time government employees.

The latter point is of course disputed. A 2012 report by the Project on Government Oversight (POGO) reviewing Department of Defense (DoD) contracting concluded that contractor employees cost 2.94 times more than an average DoD civilian employee performing the same job, and that DoD service contracts cost $253.8 billion versus $108 billion for full-time employees (base plus overhead) in FY 2010. POGO also found that government employees were less expensive than contractors in 33 of the 35 occupational classifications it reviewed and that the contracted labor force is as de facto permanent as DoD civilian employees. The question of whether insourcing is cheaper than outsourcing is disputed, but we should have the facts to determine this accurately. 

To date, transparency advocates have devoted far more attention and effort to the public sector than to private contractors. They have also largely ignored the many nonprofits that deliver many important governmental functions. The number of 501(c)3 organizations—that is, charities and foundations—has grown steadily over the past few decades. They too have benefited from the movement away from the central state model and the shift toward more localized service provision. But we lack good data as to how much is accomplished effectively by this sector and at what cost. The GAO estimates that Federal support for nonprofits increased by over 230 percent over the period 1980-2004. But how they make decisions and who is accountable for them is often unclear because we often lack adequate information on their internal governance.  

With both for-profit and non-profit organizations, it is fair to worry about whether a heavier reporting requirement would strain the budgets of some of the smaller businesses and organizations. But given the increasing importance of these agents of delegated government, we need to increase transparency in turn in order to achieve some degree of accountability for the public money and responsibilities that we hand over to them.

When confronted with these points, reform advocates sometimes claim that their role is to push for more transparency, not to think about its potential limits. After all, they argue, there are many politicians, lobbyists, and others who are anxious to weaken transparency laws. Fair enough, but a dogmatic approach toward transparency may ultimately prove counterproductive to the reform cause if seemingly public opportunities serve the interests of the few more often than the majority.

A useful reform critique should not only explicate the problems with our current transparency rules but also provide possible solutions for discussion. Therefore, let’s consider four suggestions. 

First, we need to prioritize quality over quantity. There is too much focus currently on the number of requests, the length of time for compliance and the number of denials. These metrics are not entirely useless, but we need to go beyond them to give more priority to FOIA items that relate to democratic accountability. 

The right way to do this would require considerable thought, discussion, and experimentation, but for the moment, here are a few potential ideas. Agencies should prioritize certain kinds of FOIA requests (for example, those related to possible waste, fraud, and service delivery failures) over others (such as those made to obtain information about rival companies). This could be accomplished by making the requester designate a reason for a particular FOIA request. All FOIA requests should be posted online, and other groups and individuals should be allowed to join a FOIA request. The urgency of any FOIA request could then be linked to the number of co-sponsors. As for the concern that some journalists have about the need for exclusivity, they could be given some period—say 30 days—for sole use before their FOIA materials are released to others.  

Another necessary reform, second, is to limit “repeated play” input into public comment and participation provisions. In matters as diverse as redistricting hearings and environmental permitting, some actors use public comment opportunities time and again. Often they simply repeat the same concerns and demands, offering little new information or insight. Agencies need germaneness guidelines to help them decide which questions need to be answered in order to resolve a given matter and to minimize delaying tactics disguised as requests for additional information. 

Third, the issue of email and agency work-product transparency needs to be thought out more seriously. Disclosing every policy-related communication will only succeed in driving policy conversations out of email and into phone calls and personal conversations. Monitoring every single conversation officials have about a policy is neither feasible nor desirable. Good deliberation requires that people discuss and change their mind in some protected deliberative space free from outside constraint. Officials should be held accountable for what they decide or which actions they take, not for what they initially thought until they changed their minds or compromised with others. 

Lastly, we need to apply many if not all of our current transparency requirements to both for-profit and nonprofit entities that carry out traditional government functions or deliver public services. Our current transparency laws overlook the important contemporary shift to outsourcing, leaving a serious accountability gap. Knowing more about what for-profit contractors and nonprofit organization do with public money is as important knowing what full-time public employees do. 

In sum, the issue is not whether transparency is essential to American democracy but how much transparency and for what purposes. Democratic opportunities can be hijacked for individual gain and special interests. The naive assumption that citizens will live up to highly idealized expectations of citizen responsibility can lead to democratic capture and distortion. Democracies that cannot govern effectively and inclusively can breed discontent and unrest. A blind defense of current transparency practices obstructs the important task of improving them.