Introduction to Programming with gdeltr2

What Is It

This parameter lets you search GDELT’s entire Full Text API for up to the last 3 months for ANY term your heart desires.

This can be a person, place, thing, name, phrase, literally ANYTHING you can imagine and when you think about the fact that GDELT monitors millions of websites from around the world every second you quickly realize that you can use this to find some truly obscure information.

How It Works

GDELT indexes and sucks in all the text from page it processes. The V2 Full Text API lets users search through this text and if there is a match returns any matches.

Tidbits to Remember

If you want to find exact terms you need to use a quoted string.

To GDELT if you search for "Brooklyn Nets" it will go through every article and return anything that contains the words Brooklyn AND Nets. The words don’t have to be next to each other to return a positive match.

For example, if there was an article in a publication that discussed a fundraiser in Brooklyn that nets a large donation from an unknown benefactor it would return this as a match. This is not something I want to see if I am only looking for articles about my favorite NBA team the Brooklyn Nets. To find those matches I want want to search '"Brookyln Nets"'.

In some cases this won’t matter too much, but depending on the terms you are looking for it may.

You can also combine phrases like '"Brooklyn Nets" playoffs' which would search for “Brooklyn Nets” AND the word playoffs or '"New York City" "City Council"' which would look for “New York City” AND City Council.

Example: Terms to Search

Now that we have covered what this parameter is lets build a vector of terms to use for our analysis

sports_terms <-
    c('"Brooklyn Nets"', "Caris LeVert", '"Kyrie Irving" Trade', '"Luka Doncic"',
      'NBA "Draft Prospect"', '"Jarrett Allen"')

political_terms <-
    c('"Bill Perkins"', '"New York City" "City Counsel"')

finance_real_estate_terms <-
  c("Eastdil", "Condo Bubble", '"JBG Smith"', '"CPPIB"', "Anbang",
    "WeWork", '"Goldman Sachs"' , 'Blackstone "Real Estate"')

other_terms <-
    c("Supergoop", '"LNG"', 'Maryland "High School Football"',
      '"Jared Kushner"', '"Eddie Huang"')

my_terms <-
  c(sports_terms, political_terms, finance_real_estate_terms, other_terms)

What Is It

Indexing and parsing web-domains is the lifeblood of GDELT. If there is a website you read that publishes information the odds are it is tracked by GDELT.

The V2 Full Text API lets you provide a set of web domains and if they are indexed by GDELT it will return the articles it indexed over the user specified time period.

This parameter makes GDELT an easy way to continuously keep track of the websites you enjoy reading.

How It Works

Each second GDELT scours millions of websites from around the world for new content. When a new article is discovered the contents of the article get processed through GDELT’s APIs.

The APIs do a variety of things from extracting out numeric references, people, places, to generating sentiment scores. They even extract out media content and process the videos and photos through all of Google Cloud Vision’s APIs which can do everything from identifying what is in the photo, to predicting the sentiment of the any person in the photo, they even attempt to identify whether the photo contains a brand!

Tidbits to Remember

Don’t use http:// or https:// when searching for domains.

Remember to use the type of website that it is ie don’t search espn if you are looking for espn.com

Example: Domains to Search

Now lets build a vector of some of the domains I enjoy from various aspects of what I like to monitor in the world.

news_domains <-
  c("nypost.com", "washingtonpost.com", "wsj.com", "gothamgazette.com")

sports_domains <-
  c("espn.com", "netsdaily.com")

finance_real_estate_domains <-
  c("realdeal.com", "zerohedge.com", "institutionalinvestor.com", 'pionline.com',
    "curbed.com", "archdaily.com")

random_domains <-
  c("tmz.com", "snopes.com", "alphr.com", "oilprice.com")

my_domains <-
  c(news_domains, sports_domains, finance_real_estate_domains, random_domains)

What Is It

This is one of my favorite features!!

GDELT has created a list of nearly 21,000 themes it actively seeks to tag whenever it process an article.

Some of the themes are taken from other organizations like the World Bank while others internally developed. The themes cover just about every imaginable topic in world affairs. They range from obscurities like dandruff to outlandish topics like perverted actions to serious topics like drug overdoses.

They track economic events, political events, scientific events, corporate actions, just to name a few categories!

How It Works

Every time GDELT processes new content it performs Natural Language Processing [NLP] on the text.

As part of this they use the processed text assign any number of themes to the article.

For example if GDELT reads in an article and processes text to find the phrase “white stuff falling to the ground from the man’s head from a dry scalp” it would assign the GKG theme corresponding to dandruff from above.

One article can have many themes as well.

Tidbits to Remember

When entering in a theme you must be exact.

If you were looking for articles about dandruff you must enter the exact code of "TAX_DISEASE_DANDRUFF". The codes are not case sensitive "tax_disease_dandruff" or Tax_Disease_Dandruff would also work.

Example: Themes to Find

In this example in addition to the themes I specifically define, I am going to include 3 additional random themes.

The reason you will see the set.seed function is to ensure that your random 3 themes are the same as mine.

Here is how we define them:

df_gkg <-
  gdeltr2::get_gdelt_codebook_ft_api(code_book = "gkg")

my_themes <-
  c("ECON_WORLDCURRENCIES_CHINESE_YUAN", # stories about china's currency -- god way to find stories about china's economy
    "ECON_BUBBLE", # articles about economic bubble
    "TAX_FNCACT_BROKER", # articles about brokers of things
    "ECON_HOUSING_PRICES", # articls about housing prices
    "ECON_BITCOIN", # articles about bitcoin
    "ELECTION_FRAUD", # articles about election fraud
    "SOC_POINTSOFINTEREST_GOVERNMENT_BUILDINGS", # articles about government buildings
    "WB_1277_BANKRUPTCY_AND_LIQUIDATION", # articles about bankruptcy
    "WB_639_REPRODUCTIVE_MATERNAL_AND_CHILD_HEALTH", # articles about pregnancy and child health
    "WB_2151_CHILD_DEVELOPMENT", # articles about child development
    "TAX_FNCACT_BUILDER" # articles about builders
    )

set.seed(1234)

random_themes <-
  df_gkg %>% pull(idGKGTheme) %>% sample(3)

my_themes <- 
  c(my_themes, random_themes)

What Is It

This parameter enables you use GDELT to search for text extracted from media content.

This is a great way to keep track of brands and find pictures containing certain text that may be of interest to you.

How It Works

One of the many really cool things Google Cloud Vision does is extract out text from media content.

Since GDELT uses Google Cloud Vision for every website it monitors it takes each piece of media content it finds and uses Google Cloud Visions OCR API to extract out any text it finds.

GDELT’s V2 Full Text API will then let you search through the results for any matching user defined text!

Tidbits to Remember

Consider spelling, abbreviations and logo names. Google’s OCR technology isn’t perfect so there may be some inaccuracy.

Example: Words to OCR

Here we are going to define a vector of some random things I want to look for in photos as part of our analysis.

my_ocr <-
  c(
  "Brooklyn Nets",
  "Panerai",
  "Four Seasons",
  "NBA",
  "Goldman Sachs",
  "Philadelphia Eagles",
  "Supergoop",
  "Boston Celtics",
  "Big Baller Brand",
  "BBB",
  "Boston Properties"
  )

What Is It

This parameter lets you harness to the power of machine learning to search for “things”.

When I say things think of it how you would describe something in our universe. For example in the photo of Theo and Chase from before we could come up with a few “things” that describe what is in the picture.

There is a dog, specifically a toy poodle. There is a human being, specifically a baby. There is a chair. We could probably come up with a bunch more if we wanted to. GDELT lets you search for things and those “things” consist of all the “things” that are understood and tagged by the google’s imagetag framework.

How It Works

Similar to the OCR mechanism, GDELT uses Google Cloud Vision and for every piece of media content processes.

Whenever it processes media it will try to identify “things” that it thinks match a tag in its imagetag memory. To do this it processes the extracted photo through what is called a Neural Network to try to find possible matches. For each tag it finds the Neural Network assigns a probability it is something in the Neural Network’s memory with a maximum of 1.

The GDELT V2 Full Text API only returns image tag matches with a high probability of being a match.

If you have ever heard about the concept of machine learning this is it in action and in a way that you can use easily!

Tidbits to Remember

Use the specific code without quotes. These are also not case sensitive. The tags also won’t always be accurate.

Example: Imagetags to find

Now lets define some imagetags we want GDELT to search for.

my_image_tags <-
  c("Toy Poodle", # looks for toy poodles
    "poodle", # looks for any form of poodle
    "commercial building", # looks for an office building
    "basketball player", # looks for a professional basketball player
    "supermodel" # take a guess :)s
    )

What Is It

Imageweb tags are very similar to imagetags except they are more specific.

Imagewebtags seek to find exact matches to something.

For example in the photo of my family it would not only look to identify the baby and toy poodle in the picture, it would try to figure out WHO exactly was in the photo, ie that really cute poodle is Chase Bresler aka littlechasebigworld.

For this to work the match has to be contained in the list of imagewebtags Google Cloud Vision knows.

How It Works

As is the case with imagetags and OCR, when GDELT finds media content in article it extracts it and runs it through Google Cloud Vision.

Google Cloud Vision has on an going list of exact people, places, things, ideas, and concepts that it knows and tries to identify. GDELT will ask Google Cloud Vision if the media content matches that and if it returns a match with a high probability score it will return the corresponding webtag contained in the photo.

Tidbits to Remember

As is the case with OCR, GKG, and Imagetags don’t use quotes and give the exact term in which ever case you desire. With these I encourage you to monitor them over time as Google is constantly learning new things that you can search for and also remember that Google isn’t always accurate in its predictions.

Example: Imagewebtags to Find

Now lets define some imagewebtags we want GDELT to search for.

my_image_web <-
  c(
  "Jared Kushner",
  "Empire State Building",
  "Serena Williams",
  "New York City",
  "Ivanka Trump",
  "Tesla Model 3",
  "Jeremy Lin",
  "NBA",
  "Brooklyn Nets"
  )

#1: Timespan

As I mentioned before, GDELTs V2 Full Text API gives you access to information over continuous 3 month periods.

You can ask the API for information as far back as 12 weeks to as recent as 1 minute ago.

In gdeltr2 you must define time-spans, if you don’t it reverts the default of anything in the last 24 hours.

When defining a timespan it must be given as a string containing either minutes, hours, days, or weeks ie: "24 hours", "97 minutes", "17 days", or "12 weeks".

For our initial example lets define the timespan as anything published within the last 5 days.

my_timespan <- 
  "5 days"

#2 Maximum Records

This parameter defines the maximum amount of results you want to return for any API call with results.

The default and maximum allowed per API call is 250 which at times may mean you aren’t getting the full set of results.

There is a way to circumvent this a bit by defining specific dates instead of time-spans for the time horizon but that is much more complicated and outside the scope of this tutorial. Any advanced user who needs to know more about this can reach out to me directly.

For our example we can leave this parameter empty as we want to use the default of 250 results.

Image Panels

Parameter: modes = 'ArtList'

This is the default parameter and my personal go to for how I interact with GDELT on a daily basis.

In order to create an image panel trelliscope all we do is pass along each of the parameters we defined and in no time you should have an interactive trelliscope with clickable links for your view into the world.

Here is how to do it:

get_data_ft_v2_api(terms = my_terms, domains = my_domains, images_web_tag = my_image_web, 
    images_tag = my_image_tags, images_ocr = my_ocr, gkg_themes = my_themes, 
    modes = c("Artlist"), timespans = my_timespan, trelliscope_parameters = my_trelliscope_parameters)

trelliscopeImage

Click here if you wish to explore in full screen mode

Timeline Volume

Parameter: modes = "TimelineVolInfo"

This mode will create an interactive chart with the date on the x axis and the volume score on the y axis.

The volume score is essentially how much your specified parameter was mentioned.

In these trelliscopes if you click on the line it will return a tooltip containing clickable links to your specified search parameter.

When using this mode I highly recommend changing the timespan to 12 weeks in order to give you the maximum view on how often your parameter was discussed.

Here is how to create this:

get_data_ft_v2_api(terms = my_terms, domains = my_domains, images_web_tag = my_image_web, 
    images_tag = my_image_tags, images_ocr = my_ocr, gkg_themes = my_themes, 
    modes = c("TimelineVolInfo"), timespans = "12 weeks", trelliscope_parameters = my_trelliscope_parameters)

trelliscopeHighcharter

Click here if you wish to explore in full screen mode

Wordclouds

Parameter: modes = c("WordCloudEnglish", "WordCloudTheme", "WordCloudImageTags", "WordCloudImageWebTags")

These modes allow you to create various wordclouds for each specified search parameter.

WordCloudEnglish returns a word cloud of the most commonly used English words for the specified sarch parameter.

WordCloudTheme returns a wordcloud of the most identified GKG themes.

WordCloudImageTags returns the most commonly found Imagetags and WordCloudImageWebTags returns the most commonly found ImageWebTags.

These wordclouds can be a great way to quickly understand what is going on with regards to specific search parameter.

Here is how we create a trelliscope that displays each of these wordcloud types. For this analysis we are going to slightly modify the panels to only show 1 column and 1 row and we will also expand the time frame to 2 weeks.

get_data_ft_v2_api(terms = my_terms, domains = my_domains, images_web_tag = my_image_web, 
    images_tag = my_image_tags, images_ocr = my_ocr, gkg_themes = my_themes, 
    modes = c("WordCloudEnglish", "WordCloudTheme", "WordCloudImageTags", "WordCloudImageWebTags"), 
    timespans = "2 weeks", trelliscope_parameters = list(rows = 1, columns = 1, 
        path = NULL))

trelliscopeWordcloud

Click here if you wish to explore in full screen mode

Interacting with the Trelliscope

After you execute the code you will notice the trelliscope in your viewer pane. To best interact with the Trelliscope I like to open it in my browser of choice. To do that click the button to the right of the broom button.

If you always want to open interactive content in your browser you can run the following code:

options(viewer =  NULL)

Navigating Interactive Trelliscopes

Now that we’ve created a bunch of interactive trelliscopes I wanted to take a moment and help you better understand how to use them. These trelliscopes will help you quickly navigate through vasts amounts of information in large part due to their interactive filter capabilities.

Shortcuts

You case use the f key to call up all the filters. You can modify the grid by pressing g. You can modify the labels by pressing l. You can change the sorting parameters by pressing s.
Instead of clicking the left and right arrows you can use the left or right keys, or if you are on a mobile device or tablet and it is a trelliscope that lives on the web you can swipe left or right.

Filtering

Parting Notes

I hope everyone enjoyed this tutorial and if you made it to the end that you were able to get these trelliscopes showcasing the GDELT V2 Full Text API data up and running on your own computer.

If not, please feel free to reach out to to me and I will try to assist you in getting this to work.

You can access the raw R code here if you want a quick way to try to reproduce the code from this tutorial.

A special thank you to Kalev Leeatru and the entire GDELT team for all the truly remarkable work they have done. I also want to thank Google for their support of GDELT.

A major shout out is owed to Ryan Hafen for his truly beautiful and incredibly powerful trelliscopeJS package. As well as DARPA for funding Ryan and his vision for trelliscopeJS.

This tutorial’s interactive visualizations wouldn’t be possible without the contributions of Joshua Kunst and his work on the wordcloud2 and highcharter packages. Josh you are an all-star!!

I also want to thank Kenton Russell and Ramnath Vaidyanathan for laying the ground work for nearly all the interactive packages that make R such a dynamic programming language with their revolutionary htmlwidgets package.

Remember, GDELT is the coolest thing you now HAVE heard of and you just completed some pretty advanced programming so you cannot say that learning to code is too difficult for you!

Until next time.