poldis
{poldis}: an R package that provides tools for analyzing political
discourses over time!
Why {poldis}?
{poldis} provides the tools necessary to work with political discourse
texts, making it easier to analyse political discourses that are less
structured and frequently involve multiple actors and topics. Political
discourse data come in many different forms, ranging from single-speaker
speeches to debates. While other R packages allow us to analyse data
from single-speaker speeches easily, this only reveals a portion of the
what, how, and where politics “gets done”. For most other settings where
political discourse appears as debates, interviews, news conferences, or
campaign rallies, working with text programmatically becomes tricky
where there may be multiple speakers, questions, different languages,
and other types of “noise” in text data. Texts might need to be
separated by speakers, have questions removed for analysis, or retain
only the context around specific topics and keywords. The package
contains the tools required for wrangling and annotating these different
types of political discourse texts within R.
Beyond these basic tools for text analysis, {poldis} also introduces
tools for Urgency Analysis (UA), a new method for the analysis of
urgency in political texts. Urgency is an expression of how critical or
immediate problems or solutions are in political discourse. UA rests
upon a multidimensional, weighted, and survey-validated conception of
how urgency can be expressed in political discourses. It combines
Natural Language Processes (NLP) and dictionary approaches to provide a
contextualized, comparable, and scalable new method for the analysis of
political texts.
Installing the package
The easiest way to install {poldis} is directly from CRAN:
install.packages("poldis")
library(poldis)However, you may also install the development version from GitHub.
# install.packages("remotes")
remotes::install_github("henriquesposito/poldis")
library(poldis)Getting started
{poldis} offers several text tools for wrangling texts within R. For
example, users can extract the context surrounding a specific string in
the text.
set.seed(1234)
text <- sample(US_News_Conferences_1960_1980$text, 2, replace = TRUE)
extract_context(" war ", text, level = "sentences", n = 1)## [[1]]
## [1] " We do a good many things.,Now, coming down to the question which is rather sidestepped, that is, if the United States should go to war in order to remove Castro. That nettle is not grasped, and it would seem to me that we have pretty much done all of those things that can be done to demonstrate hostility to the concept of a Soviet satellite in the Caribbean except take these other steps which bring in their wake violence, and may bring a good deal of worldwide difficulty."
##
## [[2]]
## [1] " President, did you limit yourself to this global theme?,THE PRESIDENT. Well, because the issue of war and peace is involved, and the survival of perhaps the planet, possibly our system. And, therefore, this is a matter of primary concern to the people of the United States and the people of the world."Other text functions in the ‘extract’ family of functions include
extract_names(), that uses NLP to extract names of persons refereed to
in the text, and extract_text_similarities(), that compares textual
similarities for multiple texts.
Annotating text
With annotate_text(), text annotations are offered at the word (token)
and sentence levels. The annotated syntax metadata helps to code words
or sentences (e.g. code adverbs or adjectives where/when present) and
avoid false positive matches. Please see the help page
(?annotate_text) for details on the algorithm used to annotate texts.
annotate_text(text, level = "words")
annotate_text(text, level = "sentences")Selecting political priorities
select_priorities() allows political priorities (i.e. statements in
which actors express their intent or commitment to take political action
in the future) to be extracted once texts have been annotated.
priorities are captured by identifying sentences containing modal verbs
(e.g. must) and other indications of future-oriented sentences with our
custom-built dictionaries (e.g. need to, going to).
select_priorities(text)Gathering topics
We automate topic generation for political texts by constructing a custom-built dictionary of terms for capturing 20 major political topics based on the Comparative Agendas Projects (Jones et al., 2023). Topics are assigned when words related to subjects appear in texts. Alternatively, users can also declare their own dictionary for topics.
gather_topics(text)## [1] "Macroeconomics, Civil Rights Minority Issues and Civil Liberties, Health, Labor and Employment, Education, Energy, Immigration, Transportation, Law Crime and Family Issues, Community Development and Housing Issues, Banking Finance and Domestic Commerce, Defense, Space Science Technology and Communications, Foreign Trade, International Affairs and Foreign Aid, Government Operations, Public Lands and Water Management"
## [2] "Macroeconomics, Health, Agriculture, Labor and Employment, Education, Immigration, Law Crime and Family Issues, Social Welfare, Community Development and Housing Issues, Defense, Space Science Technology and Communications, Foreign Trade, International Affairs and Foreign Aid, Government Operations"
## attr(,"class")
## [1] "topics"Measuring urgency
Urgency is an expression of how necessary and/or how soon a political action should be undertaken. We code four dimensions of urgency in political discourses using purpose-built dictionaries of terms: frequency, timing, intensity, and commitment. For more details on conceptualising and measuring urgency, please contact us for the working paper introducing Urgency Analysis.
get_urgency(text)## .data
## 1 THE PRESIDENT. Good afternoon.,[1.] Prime Minister Pearson of Canada and I have agreed to meet at Hyannis Port, Mass., on May 10 and 11 for a first discussion of the many important questions that are of common interest to the two countries.,[2.] Secondly, Under Secretary of State for Political Affairs W. Averell Harriman, having consulted in Paris with French Foreign Minister Couve de Murville and in London with Foreign Secretary Lord Home, British Co-chairman of the International Control Commission for Laos, will proceed to Moscow tomorrow to discuss the Laotian situation with the Soviet Co-chairman, Foreign Minister Andrei Gromyko. He does have a short message for Premier Khrushchev from me, explaining the purposes of his trip.,[3.] Q. Mr. President, with Laos boiling up, could you assess for us the relative threats posed to the United States by the Soviet Union and Red China?,THE PRESIDENT. No, I think it would be a mistake to attempt to make that assessment on this occasion. We have difficult problems in Southeast Asia. They directly involve, of course, the Soviet Union, as the Soviet Union is the co-chairman and is also, as I have already said, a signatory to the Geneva accord. It has assumed in the past a special responsibility for the maintenance of a neutral and independent Laos, in the Vienna statement which the Chairman and I made in June 1961,1 committing ourselves to that result. We have also of course been conscious of the threat to the security of independent countries of Asia and Southeast Asia, which has been made quite clear by the Chinese. So I would say that we have serious problems with them both. We would hope that the Soviet Union would make an effort to fulfill its commitments under the Geneva accord as the United States is attempting to do.,1 See 1961 volume, this series, Item 225.,[4.] Q. Mr. President, there were reports from Moscow earlier today that the British and American Ambassadors during their meeting with Chairman Khrushchev had presented a new proposal on inspection in an effort to break the deadlock on the nuclear test ban treaty negotiations. Is it correct that the United States has presented such a proposal, and is there anything you can tell us about prospects now on this issue?,THE PRESIDENT. The United States made proposals for intensification of the negotiations and suggested some procedures by which those negotiations might be speeded up. I am not overly sanguine about the prospects for an accord. We have been caught, really, since December, on the disagreement between the number of tests that should take place in any one year--the United States discussing seven and the Soviet Union three. No movement from the Soviet Union has taken place. In addition there are other details which are still unresolved, not so much the matter of tests but the area of inspection, the means by which the inspection will be carried out, the freedom of the teams, and what will be the composition of the inspection teams; all these questions are still unresolved.,As we feel time is running out, the Prime Minister and I wrote to Chairman Khrushchev in an effort to see if we could develop some means by which we could bring this matter to a climax and see if we could reach an accord, which we feel to be in the interest of the nuclear powers, the present nuclear powers, to prevent diffusion. But, as I say, I am not sanguine and this represents not a last effort but a very determined effort to see if we can prevent failure from coming upon us this spring.,[5.] Q. Mr. President, back on Laos, it has been more of a testing ground for coexistence since the Geneva accord than perhaps any other place in the world. Would you interpret a Soviet refusal to go along with efforts to maintain peace in the government of national union there as a shift toward a hard line by the Soviet Union?,THE PRESIDENT. Well, I don't want to say anything that will prejudice Secretary Harriman's trip. I think we will know a good deal more about the prospects after he has visited Moscow. Quite obviously, we regard the maintenance of the Geneva accord as very essential to the security of Laos itself, and also, as you quite rightly say, as a test of whether it is possible for an accord to be reached between countries which have serious differences, an accord to be reached and maintained.,If we fail in Laos, then I would think the prospects for accords on matters which may be geographically closer to us would be substantially lessened. But I think we will have an idea as to whether the Soviet Union is prepared to meet its commitments and whether the other countries who are also signatories--which include the Communist Chinese and the North Vietnamese, and others--are prepared to really see a neutral and independent Laos, or determined to try a military takeover. I think we should have a clearer idea of that after Governor Harriman's return.,Q. Could I ask just one more question on Laos? Do we have any evidence that the Soviet Union is not in control of the ground in Laos, as they seemed to be in control in 1961 and last year, when the Geneva agreement was signed?,THE PRESIDENT. Well, that, I think, is a matter which I think time will tell us. There was a direct control because of the supply lines which were being maintained by the Soviet airlift. Whether the Soviets maintain the same degree of control now, whether they desire to maintain their influence, and whether their influence will be thrown in the direction of a maintenance of the Geneva settlement are the questions which I think we should find answered in the next 3 or 4 weeks.,What, of course, is happening in Laos is a struggle between the neutralist forces of Kong Le, who were allied with the Communist forces in 1961. So that it seems to me that the very nature of the struggle and the forces that are involved in the struggle are the best answer to the charges that have been made in the last 24 hours, that it is the United States which has disturbed the status quo. The struggle is not between the forces of Phoumi and the neutralists, but between the Pathet Lao and the Kong Le forces which, of course, are the army of Souvanna Phouma, whom the Communists themselves supported in 1961. So I think we have a very clear idea of where the responsibility lies, and it would be a distortion to attempt to place the burden for the breakdown upon the United States.,I think the world can tell very clearly who is struggling in the Plaine des Jarres and who, therefore, must bear the responsibility. Now, the solution is not to engage in polemics or debate, but to bring about a cease-fire, and to see if we can maintain what is a very fragile structure today.,[6.] Q. Mr. President, how do you feel about the recommendations of the National Academy of Sciences and also of Professor John Rock of Harvard, that the Federal Government should participate actively in an attack on uncontrolled population growth?,THE PRESIDENT. Well, I don't know--I am familiar with the general thesis of Professor Rock. As you know, the United States Government today, through the National Institutes of Health, gives assistance to research in the whole area of fertility, biological studies, reproduction, and all the rest, which I think are important studies, and there are several millions of dollars of Federal funds involved, and I think they are very useful and should be continued.,Q. I think the recommendations are that our Government should take the lead and should participate much more actively and strongly than it has done before. You, sir, have never taken a position on this, I believe.,THE PRESIDENT. Well, what is your question?,Q. The question is: Will you accept the recommendations of the National Academy that we should participate in international birth control studies--supply funds?,THE PRESIDENT. Well, we are participating in the study of fertility and reproduction in the United Nations, which is an international study, at the present time. Now, if your question is: Can we do more, should we know more about the whole reproduction cycle, and should this information be made more available to the world so that everyone can make their own judgment, I would think that it would be a matter which we could certainly support. Whether we are going to support Dr. Rock's proposal, which is somewhat different, is another question.,[7.] Q. Mr. President, do you see any prospect for a meeting between yourself and Mr. Khrushchev any time in the next couple of months, in Europe, for example?,THE PRESIDENT. No, I haven't heard any, and there is none planned.,Q. The British, according to reports from London, are hoping for a three-way summit perhaps on the test ban.,THE PRESIDENT. There is none planned, and it doesn't seem to me that it would be useful unless we were in agreement upon a test ban, which we are not now.,[8.] Q. Mr. President, would you care to address yourself to criticism expressed by some Republicans, including Mr. Nixon recently, about the administration's attitude toward Cuba, and suggesting, perhaps, that we are not taking as firm a stand toward them as we should? Would you care to speak to that, sir?,THE PRESIDENT. No. I know there is a good deal of concern in the United States because Castro is still there. I think it is unfortunate that he was permitted to assume control in the 1950's, and perhaps it would have been easier to take an action then than it is now. But those who were in positions of responsibility did not make that judgment.,Now, as to what the present situation--we have, as you know, without going through the entire list, we have--and the other countries of the free world have--cut free world trade in the last 2 years from $800 million to $80 million. We are working with the OAS to set up an organization which will limit the movement of potential guerrillas in and out of Cuba. We have--the OAS have almost diplomatically isolated Castro in this hemisphere. I think the members of the OAS have made it very clear that Marxist-Leninism and the Soviet presence is not a matter which is acceptable to the people of the hemisphere. We have been working through the Alliance for Progress to prevent a repetition of the Cuba incident. We have made it very clear that we would not accept a Hungary in Cuba. We have made it very clear we would not permit the movement of troops from Cuba to another country for offensive purposes. We maintain surveillance. We do a good many things.,Now, coming down to the question which is rather sidestepped, that is, if the United States should go to war in order to remove Castro. That nettle is not grasped, and it would seem to me that we have pretty much done all of those things that can be done to demonstrate hostility to the concept of a Soviet satellite in the Caribbean except take these other steps which bring in their wake violence, and may bring a good deal of worldwide difficulty. If they are advocating that, then I recognize that as an alternate policy, but if it is merely a policy which says that we should do something without defining it, except perhaps as I have said, unleashing the exiles, which cannot do the job, it seems to me that we deserve in a question of this importance a good deal more precision in our prescriptions for its solution.,Q. Mr. President, now that the 21 Americans who were imprisoned in Cuba have been released, what do you think that the U.S. policy will be toward exile raids in the future if no U.S. laws are violated, and if these raids may have some military value, perhaps done in conjunction with the underground within Cuba?,THE PRESIDENT. Well, I would think a discussion of that kind of a question, if the question is as you put it, is really not very useful to the exiles, or to the cause of Cuba. It does not seem to me that public discussion of these sorts of activities is worthwhile at this time, or beneficial.,[9.] Q. Mr. President, I understand that at the request of the Defense Department, the United States Information Agency is now supplying two 5-minute commentaries daily on international affairs which are being broadcast by Armed Forces Radio Service transmitters on both the East and West Coasts, and in Germany. And I would like to respectfully ask you whether you feel it is the business of an official Federal agency to be disseminating comment and opinion to our citizen soldiers and their families overseas.,THE PRESIDENT. What did these programs consist of that is objectionable?,Q. Comment and opinion on international affairs.,THE PRESIDENT. Well, is there anything about the comment that is at all objectionable or slanted?,Q. I am not overseas, and so I haven't heard them.,THE PRESIDENT. Well, I'll be glad-[laughter]--you and I share--[laughter]-I would be glad to check into it and find out if there is anything that is improper about it.,[10.] Q. Mr. President, France is not on your itinerary for this summer, and apparently no invitation has been extended, and certainly you have not solicited one, but I wonder in the light of Secretary Rusk's talk with President de Gaulle if you think a talk between yourself and President de Gaulle would be useful this summer?,THE PRESIDENT. Well, I went to France last year. We are going to go to Italy and Germany and Ireland for good reasons in every case. We have not--I think actually according to protocol, which need not stand in our way, it would be the time for the French President to come to the United States. I think General de Gaulle would be glad to come or, protocol aside, I would be glad to go to France if there were some matters which we felt an exchange, a personal exchange would solve. I think that perhaps both of us feel that on those matters which concern us in common, France and the United States, that they can be best discussed at the diplomatic level.,[11.] Q. Another point on the exile problem, sir, rather in line with an admonition that you yourself made last December; the Attorney General suggested the other day that the Cuban exiles should compose their differences and speak with more of one voice, particularly in terms of their relationship with the Government. Is there an implication here, sir, of an approval or enthusiastic approval on the part of your administration toward the setting up of an exile government, a government in exile?,THE PRESIDENT. No, we supported the arrangement of the Revolutionary Council in order to give the exiles a voice which we hoped would be speaking for the exile community in all those matters which affect their relations with the United States and the United States Government. For us to agree and support a government in exile, however, is an entirely different question, because you have to--we would want to support a government which would strike a responsive chord in Cuba itself. The experience with governments in exile have not been particularly felicitous, historically speaking. There is no evidence that exiles themselves could develop a government which would necessarily be the government which the people of Cuba would freely choose.,It would seem to me what would be most valuable now would be a greater degree of cohesion among the exiles regardless of their political view, and there are substantial differences among them, so that they can negotiate with us, if that is the proper word, and bring their case before other Latin American countries, in the OAS, so that we can talk to someone about the many problems which we face and the exiles face with 200,000--250,000 people coming into our country- But a government in exile, I think that is a different question, and in my view it would be imprudent today and I don't think it would help the struggle.,[12.] Q. Mr. President, Senator Keating says that according to his information there has been no reduction in the number of Russian troops in Cuba. He said several thousand have left, several thousand have arrived there, with no change in the overall number since November. Would you care to comment?,THE PRESIDENT. Yes. I have already said that the best information we have from the intelligence community--and I rely upon the Director of the Central Intelligence Agency as chairman of the intergovernmental intelligence community for the information which I have given publicly. We attempt to ask any Congressman or Senator who has information to the contrary for his sources so they can be evaluated. I have stated that our information was, I think the last time we met, that 4,000, we thought, left in March, and that no substantial number had come in this winter. There is some evidence that some have left in April, but not a large number. Of course, the equipment itself seems to still be there, however, so that I would think there has been some reduction this winter in the number of Russian personnel on the island. There has not been a substantial reduction in the equipment. There has been no evidence, however, of any substantial introduction.,It is not, in my opinion, a grave question as to whether there's 17,000, 15,000, 13,000. There are still important elements on the island, and there's still Soviet equipment on the island. So I don't think Senator Keating and I are debating a serious question, unless there is a challenge on one side or the other of good faith, and I am sure there isn't. It is our best information that 4,000 or 5,000 have left since January and that there has not been an equal number come in. In fact, much, much less--300 or 400 at the most. That's our best evidence and I repeat it as it has been gathered by our intelligence sources.,[13.] Q. Mr. President, you have rejected the Civil Rights Commission's proposal for the withholding of funds from the State of Mississippi in particular; yet Negroes and other persons in some Southern States are encountering violence and the withholding of some of their rights. Could you discuss with us what alternative steps the Federal Government might be able to take to bring some of these States into line with the law of the land?,THE PRESIDENT. Well, in every case that the Civil Rights Commission described, the United States Government has instituted legal action in order to provide a remedy. The Civil Rights Commission gave a number of cases, the dogs, of a denial of equal rights at the airline terminal, and all of the rest. We are attempting through the established procedures set out by the United States Constitution to give protection, through lawsuits, through decisions by the courts, and a good deal of action has been taken in all of these cases.,Now, it is very difficult. We had outrageous crime, from all accounts, in the State of Alabama, in the shooting of the postman who was attempting in a very traditional way to dramatize the plight of some of our citizens, being assassinated on the road. We have offered to the State of Alabama the services of the FBI in the solution of the crime. We do not have direct jurisdiction, but we are working with every legislative, legal tool at our command to insure protection for the rights of our citizens, and we shall continue to do so.,We shall also continue not to spend Federal funds in such a way as to encourage discrimination. What they were suggesting was something different, which was a blanket withdrawal of Federal expenditures from a State. I said that I didn't have the power to do so, and I do not think the President should be given that power, because it could be used in other ways differently.,But I can just say to you that the Federal Government has been extremely active in the State of Mississippi, from before Oxford and since, in an attempt to provide for constitutional guarantees. We hope the State of Mississippi will do it, we hope the local police will do it, we hope the mayors will do it. Where they don't do it, the Federal Government will do it within the limits of our authority.,[14.] Q. Mr. President, Budget Director Gordon says there are fewer Federal employees for every hundred people today than in 1952 or 1957. Much of the press has always given the opposite impression. Hasn't the administration been making correct information available, or do you think this is an instance, perhaps, of the press managing the news?,THE PRESIDENT. Well, I would not ever suggest that anyone would manage the news. You have two kinds of statistics. One, you have Federal employment rising and therefore that's printed. That's news. Federal employment is rising. Then you have the question of whether Federal employment is rising in relation to the population, and it isn't. It is, as you suggested, declining. Federal expenditures in relationship to the population--nondefense expenditures--are declining. The Federal debt in relationship to the gross national product is declining. The Federal debt has gone up in the last 15 years, but in relationship to the gross national product it is declining. It seems to me this is the framework in which these statistics should best be put. If the population increases 3 or 4 million a year it's quite obvious you are going to have to have additional services. But the question is whether this increase is excessive. And, in nearly every case, in percentage of expenditures and in employment we have gone down.,I hoped the budget would make that point, because otherwise the people get an impression that there are excessive expenditures by the National Government; that we are in a very difficult economic position, when the fact of the matter is our national debt was 120 percent of our gross national product 15 years ago, and today it's 53 percent. So we are far stronger economically than we were 15 years ago. We are far stronger economically than we were 10 years ago or 5 years ago. And we have every chance to be far stronger through this decade if we will follow monetary and fiscal policies that encourage the growth of this country instead of stifling it.,And one of the reasons why I think we have such difficulty getting an acceptance of our expenditures and our tax policies is because people misread the statistics or are misled.,[15.] Q. Mr. President, this has to do with the Wall Street Journal survey on grassroots apathy which has just been published. Do you agree, sir, that such apathy actually exists, and if so, how do you account for it, and if it does exist, what do you plan to do about eliminating it?,THE PRESIDENT. Every April the Wall Street Journal writes a story on the left-hand side of the paper, reporting that Congressmen who have come back find great apathy about the President's programs. [Laughter] The fact of the matter is t