The profession of diplomacy—which is the execution of foreign policy—is in crisis in the United States today. The rise of near peer competitors threatening the U.S.-led global security system, the disappointing results of U.S. Middle Eastern engagement—Afghanistan, Iraq, Syria, even the Iran nuclear agreement—and the Trump Administration’s cuts to State Department funding (and tardiness in nominating Department leaders below the level of Secretary), have all contributed to this crisis.

But at its core the present crisis is not the result of a bad conjuncture of disparate elements; rather, at the core lies a failure of the primary practitioner of diplomacy, the Department of State, and of the larger American foreign policy community, to understand what diplomacy does. Getting this right is as important as any other aspect of foreign policy. Fortunately, the threats aimed at America and the reform that budget cuts will require of State provide twin incentives to get diplomacy right—or, to put it in the contemporary vernacular, to make American diplomacy great again. But reform must start by understanding the problem.

The Drug of Internal Engagement

Foreign policy circles throughout Washington rejoiced when the Trump Administration seemed to shift over the past several months from a rhetoric suggestive of a military-heavy, “America First” foreign policy to a more traditional “maintain the global order” posture. Over the same time, officials like Defense Secretary James Mattis, National Security Advisor H.R. McMaster, and NSC staff chief strategist Dina Powell have appeared to become the most influential figures in Trump’s foreign policy dramatis personae. These officials are steeped in the lore of “diplomacy first,” with the use of military force being a last choice, best summed up by General Mattis’s 2013 quote: “If you don’t fund the State Department fully, then I need to buy more ammunition ultimately.”

This apparent shift is welcome, but the idea of diplomacy envisaged in the new approach may end up having little to do with what diplomacy can actually accomplish. The Trump team’s reference point, shared by many military leaders and encouraged institutionally by the Department of State and USAID, sees “diplomacy” ever more as societal reconciliation, along with economic and governance outreach, to select foreign populations with the mission of either transforming their societies into “Denmark”—as Michael Mandelbaum put it recently in Mission Failure—or “winning hearts and minds” to support American policy goals. Not only is this approach rarely successful; it also obscures how diplomacy advances American interests, which, to put not too fine a point on it, it does by helping to avoid the use of force or, when necessary, by exploiting its use to achieve specific policy goals.

How did U.S. “diplomacy” get turned around wrong this way? The answer is twofold: first, a post-Cold War loss of focus in the government, military, and foreign policy community on the ends and means of foreign policy; and second, a Department of State that has played down, in its institutional structures and culture, traditional diplomacy while advancing an outreach-to-populations, or social engineering model, of U.S. international action.

The starting point for understanding all of this is definitions. If foreign policy establishes the ends of a nation’s involvement in the world, then diplomacy is the conduct of that policy through the interaction of a state with foreign actors—usually but not always states (international organizations, sub-state actors, trans-state political movements, and other non-state actors do matter)—to advance the interests of all sides. In conducting diplomacy an actor wields the carrots and sticks available to it, even as it often enough seeks to acquire more of both.

These carrot and sticks, in turn, can be military, economic, or political (UN votes, state visits, and so on). In certain contexts (such as the security dilemmas in the greater Middle East) military tools are predominant. In others, say U.S. relations with the Organization of American States, there is very little military element. But in the end these are all tools, and even the military one comes under diplomatic overview, since the underlying issues that provoke a resort to arms are between the states or other political actors that deploy them.

Once the definitions of goals (foreign policy), their operational execution (diplomacy), and the specific tools are clear, the confusion in U.S. thinking between diplomacy, the overall implementation of foreign policy, and the diplomatic “tool” of outreach becomes apparent. Economic assistance, governance coaching, and reconciliation of squabbling groups are all forms of the outreach tool; they are not and cannot be ends in and of themselves. Yet for practical purposes that is how many State Department officials have come to think of them.

This outreach tool is a reasonable if exceptional element of diplomacy when dealing with crises or longer-term problems within states, although its efficacy is limited. The U.S. government and the so-called international community generally succeed with emergency assistance—food aid, disaster relief, refugee programs, stemming contagious diseases, and related crisis response. But longer-term success of development assistance typically is limited to the micro level or specific sectors. For example, the international community has advanced education in Afghanistan, and the U.S. government helped build a competent Iraqi central bank. But experience in a hundred countries over seven decades has yielded few examples of transformational political change by means of outside assistance, as development assistance expert Tom Dichter has just pointed out in Quartz.¹

Despite this experience, particularly since American engagement in Afghanistan and Iraq began, this exceptional tool has taken on huge significance, having been “tasked” with missions it could not accomplish.² For the past several decades America’s foreign policy has focused on internal conflicts, and a tool that was once thought to be able to “transform” conflict-producing societies obviously was welcomed as a secret sauce to avoid the usual choices: retrenchment, all-in commitment of U.S. forces for decades, or distasteful compromises to achieve “half-a-loaf.”

The belief in this tool is ubiquitous in the policy community, especially the military. There, it is doctrinally anchored in the joint Army Marine Counterinsurgency Field Manual 3-24, published in 2006, drafted under the supervision of then Generals Petraeus and Mattis, and the game plan for the militarily successful “surge” in Iraq. It emphasizes “Phase IV” or stability operations, the “build” phase (civil security and control; governance; essential services; economic/infrastructure development) of the counterinsurgency “clear-hold-build” triad. It acknowledges that in internal conflicts political factors are primary, and the main objective is the establishment of legitimacy. In theory, these are largely tasks for a local government, but the manual bears an extraordinarily broad definition of those tasks: “Counterinsurgents take upon themselves responsibility for the people’s well-being in all its manifestations. “

If the host government cannot accomplish this responsibility (and typically the U.S. government is engaged only when it cannot), then that becomes the U.S. objective at the heart of “victory” in a counterinsurgency, a counter-terror operation, or an internal conflict-resolution mission. That mission is typically the responsibility of the State Department and the local embassy-led country team. (Under certain circumstances the military, with its robust “services” and ability to “self protect,” takes on Phase IV, but this is a deviation from doctrine.) But 3-24, written by soldiers and marines, did not consider how civilian agencies long focused on government-to-government relations could transform themselves into fundamental change agents at grassroots levels, often under fire. As it turns out, this is very hard to do—or at least hard to do well.

Studies of America’s recent wars highlight stability operations but mainly document the failures. For example, “Lessons Encountered: Learning from the Long War,” an official National Defense University study of Afghanistan and Iraq, argues that “when it became clear State could not do (stability operations) . . . DoD argued it should assume responsibility for some . . . civil military duties.” But the study concluded: “We seem incapable of solving the problem.” The Rand Corporation’s November 2013 study on the transition of stability operations in Iraq following the 2011 troop withdrawal, “Ending the U.S. War in Iraq,” states that despite the military efforts in Phase IV and a massive post-2011 State Department-led stability operation, Iraq was left with “enduring challenges . . . to its political and democratic development.” Voluminous reporting by the congressionally mandated Special Inspector General for Iraq Reconstruction (SIGIR) and the counterpart for Afghanistan (SIGAR), on the $60 billion Iraq program and $115 billion Afghanistan one, documents countless failures, along with some successes. The conclusion from the SIGAR January 2017 update to Congress is damning: “Much of the reconstruction mission is at risk” and “could ultimately fail” despite the huge investment made.

In spite of these sobering and consistent analyses of stability operations gone wrong, the view among many military and policy community organs seems to be, “the mission is still right, but we got the execution wrong somehow.” For example, the influential former House Intelligence Committee Chair, Mike Rogers, writing in the Washington Post on April 21, argued: “The State Department was found wanting in Iraq and Afghanistan, unable to . . . support counterinsurgency programs.” The above-cited Rand study concluded, among other things, that in ending any military operation in an internal conflict the U.S. government should “reassess the campaign goals and objective . . . recognizing that previously established goals likely will not be achieved by the end of the transition.” While that sounds like healthy skepticism toward stability missions, the report then provides a list of recommendations to make mission execution finally work properly. Yet another example is the congressionally funded United States Institute for Peace, created to reflect faith in transformational engagement in societies, whose current program includes: “We recommit ourselves to prevent, mitigate and resolve violent conflicts around the globe.” The Department of Defense has its own Peacekeeping and Stability Operations Institute, focused on how to get execution right. One of its recent publications advocated the creation of a “civilian surge” capacity to ensure “robust civil­ian participation” in future (reconstruction and stability) operations.

These various peacekeeping, reconstruction, and stability-oriented institutions and task forces within the U.S. government, think tank, and academic communities, as well as in international institutions and other countries, are well-funded publicly or privately, netted to each other, and extraordinarily influential in espousing the “transformation is feasible” message. And to be sure, the international community and the U.S. government as its leader cannot ignore internal conflicts, nor can they ignore the internal drivers of aggressive behavior, civil wars, and outbreaks of terrorism. Programs that effectively reinforce local political forces arrayed against such drivers make sense, as do reconciliation and peacemaking efforts between political forces. Certain assistance of proven effectiveness as noted above—disaster assistance, food aid, and technical programs by governmental or corporate entities to improve specific sectors—are tools that can also make a difference.

But that is not the argument of those who have pushed this line in government, in institutions, in journals, and on the ground for a broad strategic mission to operate inside states. One does not justify spending almost $200 billion dollars in ten-plus years in Iraq and Afghanistan to achieve improvements “at the margins” but rather to have a transformational effect on a state. Max Boot, a well-respected and prolific writer on the “small wars” theme, is a leading proponent of the strategic use of this tool. In his November-December 2014 Foreign Affairs piece, “Counterinsurgency Is Here to Stay,” he advocated nation-building, arguing that the U.S. military finally got the strategic execution right with the Iraq surge. As with other commentators in this vein, he reacts to the dismal record of such efforts by focusing on failure of execution, stressing the inability of civilian agencies to respond. His solution: Transform USAID “into an organization focused not on development for its own sake but on state building” in the service of global stability.

Yet there has been little “state building” success since the 1940s. The closest has been Plan Colombia, which took decades to achieve modest success. With no real track record to justify turning a tool into a goal, a means into an end, where then did the idea of transformational American diplomacy arise?

The answer is rooted in American history and values. To twist slightly Henry Kissinger’s quote concerning Iran, the U.S. government has never decided whether it is a country or a cause. Furthermore even as an international “cause” it is bifurcated—between an international order of law and collective security among sovereign, “equal” nations (Wilson’s concept) and the even more far-reaching mission to transform societies and liberate individuals within states.

This combined (and contradictory) mindset is etched into the 1945 Preamble to the UN Charter, now the lodestone of international law. That document reiterates the League of Nations worldview (“treaties and other sources of international law can be maintained . . . equal rights . . . for nations large and small . . . maintain international peace and security”), and a call to arms for liberal values to peoples within their states (“faith in fundamental human rights, the dignity and worth of the human person, . . . promote social progress . . . practice tolerance . . . promotion of the economic and social advancement of all peoples.”)

This double definition of “liberal values,” as order and collective security among states and as advancement of individuals and peoples as a global responsibility, has characterized U.S. foreign policy since World War II. Moreover, the more far-reaching element—transformation within states—was reinforced by advancing democracy and individual freedom in Germany and Japan, then the Marshall Plan’s success transforming economies in Europe, and the conduct of the Cold War itself. The discipline of a peer competitor required Washington to focus primarily on realpolitik containment, but the ideological conflict between liberalism and collectivism at the heart of the confrontation encouraged both camps to appeal to liberal values.

In the post-Cold War period the United States, as Mandelbaum documents, maintained its leadership of a global security coalition now focused not on near-peer challenges but on fixing the internal order of states under stress. Each problem tackled was seen as separate, not linked to a common strategy beyond the idea that “the West” had to lead the international community against disturbances that troubled moral consciences and security at regional levels. Thus the U.S. government and its allies intervened in internal conflicts in Colombia, Somalia, Rwanda, Bosnia, Kosovo, northern Iraq, Libya, and (in a desultory way) Syria, and tried to resolve “frozen conflicts” from Northern Ireland and Kashmir to the West Bank, Georgia, Moldova, Nagorno-Karabakh, and Cyprus. And as country after country following 1989 invited the West and its institutions to help transform themselves into liberal states, interest grew in the policy community and especially in Congress to extend this “nation-building” and “values” agenda to recalcitrant states that balked at the American world order—up to and including, in some cases, regime change.

After 9/11, this seemingly “on the shelf” capability was seized to respond to the terrorist threat, and then in Iraq and Afghanistan. The formula was (1) totalitarian or failed states (or the terrorists they generate) are threats to international order; (2) America has a diplomatic tool which, combined with the universal values peoples everywhere share, can transform these states and thereby end their threat to global security. To quote from President George W. Bush’s Second Inaugural: “America’s vital interests and our deepest beliefs are now one. . . . Advancing human rights and self-government is the mission that created our Nation. . . . Now it is the urgent requirement of our nation’s security.”

State’s Complicity

The State Department itself has contributed to the problem of mis-defining U.S. diplomacy since World War II, beginning as noted above with the rebuilding of Europe and Japan and the ideological battles with Communism. Both of these endeavors elevated advancement of abstract ideals and appeals to populations as complements to traditional diplomacy’s realpolitik focus on state-to-state communication in the service of avoiding war or, failing that, to direct the use of force to achieve U.S. political goals.

The Department has also been hampered over many years by internal tinkering and congressional interference. Unlike any other diplomatic service, the U.S. State Department expanded its corps of generalist diplomatic personnel (Foreign Service Officers or FSOs, the group that provides most Ambassadors and senior State officials) into a specialized or “cone” system by integrating officers doing labor-intensive “services” jobs in consular affairs, internal administration, and later cultural programs. This massive broadening of what constituted FSO duties reduced diplomacy to essentially anything someone in an FSO career path was doing. Thus the Foreign Service could not follow the usual pattern of successful professional cadre in focusing relentlessly on core competencies (for example, “every Marine a rifleman”).

Meanwhile, congressional mandates, motivated by ambivalence about funding Department activities in the first place, or else by support of foreign social engineering projects, added enormously to State’s workload, particularly by way of new and usually unfunded reporting requirements. In particular, congressional “pet rock” priorities competed for attention and resources with traditional diplomatic operations, further muddling any core concept of diplomacy.

But the greater impulse in this direction within the Department was the end of the Cold War. As diplomatic engagement shifted ever more to humanitarian or values-driven issues, institutional emphasis on tools to implement such social engineering gained ground within State. One major step was the incorporation totally of the United States Information Agency, responsible for overseas messaging, and partially of USAID as well, into the Department of State.

At the time staffers in those agencies feared that their ameliorant culture would be pushed aside by the realpolitik-centric Department. Instead, the opposite occurred, as seen by the dominance of articles related to outreach to populations after integration in the official Department organ, State Magazine, and the American Foreign Service Association’s Foreign Service Journal.

Faced after 2000 with internal situations generating security threats—first 9/11, then Afghanistan and Iraq, the Bush Administration’s State Department embraced bottom-up societal transformation, in part to compete with the Defense Department, which increasingly was taking on civil affairs and governance functions in Afghanistan, and particularly in Iraq with the Coalition Provisional Authority. Colin Powell, a believer in Embassy and State Department-leadership in civilian stability operations, assigned two of his most trusted subordinates, John Negroponte and Richard Armitage, to oversee these activities in Iraq.

Powell’s successor Condoleezza Rice even more enthusiastically embraced the idea of civilian-led transformation of troubled societies, arguing before Congress for the “clear-hold-build” concept, and strongly backing civilian-military “provincial reconstruction teams” (PRTs) to undertake grassroots transformation in Iraq and Afghanistan.

Hillary Clinton took the transformation even further. While Powell and Rice focused State’s internal engagement on security threats in the greater Middle East, Secretary Clinton pursued a global transformational agenda complete with special emissaries, from women’s rights to LGBT and small business. Moreover, she introduced the Department’s Quadrennial Diplomacy and Development Review (QDDR), modeled on the Defense Department’s Quadrennial Defense Review, which establishes operational and budget priorities and institutional focus.

The State product, published in 2010 and updated in 2015 under John Kerry, took a further step in turning diplomacy on its head.³ Intervening in foreign societies for transformational goals seemingly became the Department’s core mission. Some of its goals were reasonable, including combating terror and climate change, but state-to-state relations, alliance building, and containment of classic security threats were all played down as “so 19^th century,” to quote a certain former Secretary of State.

Moreover, the preferred means for any State success was engagement with populations to achieve transformational successes, essentially over the heads of their governments. As State organizes its operating bureaus and embassies and even the work requirements (and evaluations) of its officers along QDDR lines, its influence has proved profound in substituting this new concept of State Department purpose—in which governments are targets—for the more traditional one in which governments are business partners.

Impact

Much of the controversy about the State Department’s mission and relative lack of success, and particularly its friction with the Defense Department, is generated by the perversion of diplomacy’s primary focus toward engagement at the sub-state level for supposed transformational results.

In the worst-case scenario (which has actually materialized twice), this focus encourages large-scale military commitments under the assumption that only regime change by force of American arms could generate the social petri dish in which our transformational agenda could work its magic. But such engagements ignore local realities, the interests of neighbors (Pakistan, Iran, and Syria, respectively, in the Afghanistan and Iraq cases), and the engrained impatience of the American people. So while the cause may appear just, the remedy at hand temptingly complete, and the values and institutions to be implanted unarguably admirable, the “secret sauce” ultimately and inevitably fails to achieve far-reaching U.S. goals. As costs approach $200 billion in a given case—that seems to be the magic threshold number—the Department of State is accused (especially by the military) of failing to achieve this allegedly “core” mission. If there is one theme in the various studies of stability operations, including those cited above, it’s motivating State to “figure out” how to execute stability operations.

But this is a category error of the first order. The Department of State has never in all its history been equipped, trained, focused or (except the Marshall Plan) funded, on transformational missions of other countries’ political cultures. Making the State Department the lead agency in a theoretically unlimited quest to bring a secularized messianic age into being does not magically make it capable of doing any such thing. But the outreach mindset behind it is dismayingly absolutist: The U.S. government, if it uses the right techniques, can win total success on the cheap—transformation that advances universal moral values while simultaneously eliminating security threats. So the whole world becomes in effect a 1989-period Eastern Europe.

And diverting an institution to this peripheral “mission impossible” comes at a cost. As Walter Russell Mead first noted in these pages in 2013, the United States now faces near-peer competitors intent on tearing apart the U.S. global security system.⁴ This means, first, a return of classical high-stakes balance-of-power diplomacy at the state-to-state level. Given America’s recent neglect of this threat, diplomacy properly understood now has to do some heavy lifting, but a Department of State oriented institutionally toward internal population outreach has lost the proper muscle tone as both resources and attitudes have been diverted away from traditional diplomacy.

Meanwhile, back in the realpolitik world, in dealing with China, Russia, or their allies North Korea and Iran, there are no easy victories. The two diplomatic tools here are major military or economic engagement, at great risk and cost, or international diplomatic initiatives for compromise outcomes distasteful to many, such as the Iran nuclear deal. Under such circumstances, traditional diplomacy appears nearly illegitimate as it promises only limited gains but considerable pains.

In addition, the great power competition now facing us is often played out (as it was throughout the Cold War) within weak or failed states wracked by civil wars and internal unrest, such as Syria, Ukraine, and Yemen. There is a place for stability tools in such conflicts, to achieve—as discussed above—micro successes and sector improvements, and respond to emergencies. But the priority has to be on traditional diplomacy. Vladimir Putin turned a low-cost intervention in the Syrian internal conflict into a significant strategic victory not because he focused on (to quote FM 3-24) “responsibility for the people’s well-being” for transformational ends, but rather because he chose a decidedly “people unfriendly” savage bombing campaign to achieve very limited but feasible ends. The U.S. government cannot emulate him, but we can’t counter him with transformational schemes that don’t work, especially when they come at the expense of (more humanely) using our own power

Transforming State

The issues raised over the past twenty years about intervention inside societies, now complemented by “near peer” challenges we all recognize, are still being debated within the foreign affairs communities of the United States and the West. But one initial step can be taken with the Department of State. The Department’s leadership advocates significant reform, and personnel and budget cuts will necessitate it in any case (with, by the way, development assistance and thus the “sinews” of transformational diplomacy being on the chopping block). If done correctly, such reform can position State to lead the U.S. government in dealing with the range of challenges from classic state-to-state or alliance-to-alliance confrontation to conflicts within states, and would encompass three areas: mission, organization, and personnel.

Reforming State’s mission is the simplest to describe and the most vital to implement. The key is to put classic diplomacy—state-to-state relations to advance American and universal security and other interests—in pride of place. Much can be done by public statements by senior Department officials. But a new QDDR or a successor document should be issued once the Administration releases its National Security Strategy document, or comparable “Trump Worldview.” The Department document should track with the White House product, but elaborate on the principles and priorities on which the Department will focus. They include the building and maintenance of alliance relationships, developing common international positions, and mobilization of all elements of national power. While engagement within states is one of those elements, it should not be oversold.

Finally, the Department must assert formally and in every way possible bureaucratically, that under the National Security Council, State has the overall coordination responsibility in government for international security, including military as well as non-military tools.

The Department is currently a rabbit-warren of Under Secretaries, special emissaries, bureaus, officers, and single purpose units, many of the latter urged or thrust onto it by Congress. It also is drowning in paper, much of it also congressionally mandated. The Trump Administration did the State Department a service by cutting two of the four true subcabinet positions (the second Deputy Secretary and the Counselor). This allows the Deputy Secretary to concentrate on classic Deputy duties—stand in for Secretary, monitor those disparate Department programs not under the policy Under Secretary, and promote internal operations, essentially a chief operating officer position. The Under Secretary for Policy should focus on diplomatic operations through the geographic and UN bureaus and their embassies. Various hard-power elements within the rest of the State bureaucracy—police and justice programs, counter-terrorism, stability programs, energy diplomacy, and political-military operations—should be placed under him/her and integrated with the bureaus.

Special emissaries should be eliminated. Mandatory reporting and other taskings that torment the Department should be placed in an “intensive care unit” headed by the Deputy Secretary to radically reduce the number, and length, of Department products. Many such are not formally mandated; even those that are, typically by Congress, can be drastically shortened and simplified. Even better, Department leadership can lobby Congress to retract dysfunctional mandates and properly fund the valuable ones.

While this is in the interest of most everyone in the Department, in bureaucracies general benefits cannot compete with entrenched special interests: the combination of congressional players, a few advocates within the Department, and true-believers in the NGO world who create this mountain of work. In a Department facing a dramatic cut in personnel, this must be first area to rationalize.

State-Defense relations could be most effectively improved by clarity over what State can and cannot do overseas, and certain organizational changes would help the clarification process along. State’s operating arms—the geographic bureaus, should at long last be aligned with Defense’s combatant commands. This would dramatically improve coordination at that crucial level.

To that end, State’s nearly one hundred POLADs (political advisors) to military headquarters should be largely discarded. We should retain only POLADs with whom Defense Department leaders coordinate with State or embassies daily. This would include the combatant and functional military commands, a total of ten, along with a handful of Pentagon leaders. POLADs should not be Ambassador rank but senior mid-level officers (colonel equivalent), integrated into the commanders’ personal command team, and reporting back to the bureau “twinned” with that command or agency. If POLADs are lower in rank and number, State can find truly exceptional officers for these critical jobs. Finally, POLADs and other State-Defense mechanisms should monitor Defense Department assistance programs that inevitably overlap with State programs.

Embassies typically perform well, including coordination with other civilian agencies, the intelligence community, and regional commanders (COCOMs). The latter could be further improved if the COCOM and the chief of mission would inventory each specific military element’s reporting responsibility in a given country. (Such an inventory is already done for security.) A directive from the Secretary of Defense to each COCOM on coordination and cooperation with chiefs of mission that parallels language the latter have in their letter from the President would improve an already healthy relationship.

Personnel change should focus on the Foreign Service generalist officer corps, central to executing diplomacy. As discussed above, its functions have been watered down. Internal structuring on a quasi-voluntary or agreed basis with staffers and unions is virtually impossible. But under the pressure of budget cuts, the Department could employ a top-ranked consulting firm, with a mandate drawn up not by the personnel system but by the Secretary and his Deputy, to examine personnel issues such as the cone system, recruitment, advancement, and discipline. Adoption of the “national security agencies” standard, as at CIA, at least for the Foreign Service, should be considered.

Such reorganization of State does not guarantee a more realistic approach to what diplomacy can do in support of foreign policy. But no such approach is possible as long as State is institutionally confused about its core mission, as it has become since the end of the Cold War. It will take fully engaged leadership to set that straight.