Feel free to try the exercises below at your leisure. Solutions will be posted later in the week! Note: as usual, the answers below are just one way of solving the prompts!

Data Scraping

  1. Using rvest::html_table, scrape the table of City Council members in Washington D.C. from Wikipedia
wiki_url <- 'https://en.wikipedia.org/wiki/Council_of_the_District_of_Columbia'
council_outputs <- rvest::read_html(wiki_url) %>%
  rvest::html_table() %>%
  .[[3]]
council_outputs %>% head
## # A tibble: 6 × 7
##   Councillor          Position Party Party   Committee chaired[21…¹ `Term starts`
##   <chr>               <chr>    <lgl> <chr>   <chr>                          <int>
## 1 Phil Mendelson      Chairman NA    Democr… "The Whole"                     1999
## 2 Anita Bonds         At-large NA    Democr… "Executive Administra…          2012
## 3 Doni Crawford       At-large NA    Indepe… ""                              2026
## 4 Christina Henderson At-large NA    Indepe… "Health"                        2021
## 5 Robert White        At-large NA    Democr… "Housing"                       2016
## 6 Brianne Nadeau      Ward 1   NA    Democr… "Public Works and Ope…          2015
## # ℹ abbreviated name: ¹​`Committee chaired[21]`
## # ℹ 1 more variable: `Term ends` <int>
  1. Using the inspector gadget or similar tool, web scrape the news article titles and links from the Climate Change page from the AP News.
url <- 'https://apnews.com/hub/climate-change'
item <- 'h3'

titles <- rvest::read_html(url) %>% 
  rvest::html_elements(item) %>%
  rvest::html_text2()

hyperlinks <- rvest::read_html(url) %>% 
  rvest::html_elements(item) %>%
  rvest::html_elements('a') %>% 
  rvest::html_attr("href") 

data.frame(titles, hyperlinks) %>%
  head
##                                                                                               titles
## 1          EPA watchdog finds nation’s most contaminated sites are vulnerable to flooding, wildfires
## 2  AI’s arrival complicates Big Tech climate goals, and some worry it’s locking in more fossil fuels
## 3 Records shattered as summer heat hits Southwest in March; ‘This is what climate change looks like’
## 4 Two dozen states, 10 cities sue EPA over repeal of ‘endangerment’ finding central to climate fight
## 5               American Airlines and Google say AI helped airplanes reduce contrails that trap heat
## 6       Scientists train to dive beneath polar ice as climate change warms the Arctic and Antarctica
##                                                                                                                                               hyperlinks
## 1                                          https://apnews.com/article/epa-superfund-sites-toxic-risk-flooding-wildfires-4c7ed2ab7b9d53335b86b75ae6cb9374
## 2                             https://apnews.com/article/technology-artificial-intelligence-climate-change-data-centers-ef3a9c264bd6376d77e2c81ab266fb38
## 3                                    https://apnews.com/article/heat-southwest-warming-climate-disasters-extreme-deadly-0c3ef415241d3275fd9c260d57ccc3e5
## 4                                               https://apnews.com/article/trump-climate-change-epa-states-endangerment-6b1b5b38140c76a5cc55e17ae5f3b99b
## 5                                https://apnews.com/article/american-airlines-google-contrails-climate-warming-aviation-05934141f00fab931b9875c97f6fcdba
## 6 https://apnews.com/video/scientists-train-to-dive-beneath-polar-ice-as-climate-change-warms-the-arctic-and-antarctica-5f7fdbf5bf4f4bbda3098049979e8379

Working with APIs

  1. Register for an API key with the U.S. Census Bureau. Once it is received, download any data point of interest from the American Community Survey or Decennial Census. (Documentation here)
#api key not printed here, register at link above
#arbitrarily deciding to get the number of citizen voting age population 
#in washington DC according to the 2021 American Community Survey (ACS)
url <- stringr::str_c('api.census.gov/data/2024/acs/acs1/profile?get=NAME,DP02_0001E&for=state:11&key=', api_key) 

api_call <- httr::GET(url)

api_call %>% 
  httr::content(type = 'text') %>%
  jsonlite::fromJSON()  #convert jumbled text to matrix 
##      [,1]                   [,2]         [,3]   
## [1,] "NAME"                 "DP02_0001E" "state"
## [2,] "District of Columbia" "329687"     "11"
  1. Try to replicate #1 using the tidycensus package, which is an API wrapper.
#replicating the same estimate as above
#2021 ACS, 1 year estimate of CVAP population in DC
tidycensus::get_acs(geography = 'state',
                    variables = 'DP02_0001', 
                    year = '2024', 
                    key = api_key, 
                    survey = 'acs1',
                    state = 'DC'
                    )
## # A tibble: 1 × 5
##   GEOID NAME                 variable  estimate   moe
##   <chr> <chr>                <chr>        <dbl> <dbl>
## 1 11    District of Columbia DP02_0001   329687  4510