1 Introduction

In this module, we will discuss the following concepts:

  1. How to bring your own datasets into GEE.
  2. How to associate values from remotely sensed data with your own data.
  3. How to export features from GEE.


2 Background

Knowing how animals react to their environment is critical in understanding how to manage those species. While animals are forced to make choices to meet their basic needs it is likely that their choices are also influenced by dynamic factors such as local weather conditions. Outside of direct observation, it can be difficult to connect animal behavior to weather conditions. Within this module we are going to integrate the GPS collar data collected from a cougar with daily temperature estimates from the Daymet climatological dataset accessed via GEE.

This will require us to bring in our own data into GEE, connect the weather values to the point locations, and bring those value added data back out of GEE for further analysis.

Camera trap photo of a Mountain Lion near one of the top tourist destinations in Los Angeles, California. Photo: Earth Island Journal

2.1 GPS location data

A 2016 study by Mahoney et al used GPS collars to track the movement of the two cougars and 16 coyotes in central Utah. These data were used to understand some of the behavioral patterns of the individuals from the two species. Anyone can access the data that these researchers collected during the study on the website Movebank. This website hosts animal movement dataset from all over the world. While some Movebank datasets only list the contact information of the authors, others allow you to display the information on their webmap, and a few allow you to download the data.


Example of the interaction map on Movebank.com that allows you to search for data on animal movement.

2.2 Daymet Weather Data

The Daymet dataset provides gridded estimates of daily weather parameters. Seven surface weather parameters are available at a daily time step, 1 km x 1 km spatial resolution, with a North American spatial extent. Access to the Daymet dataset is available from the ORNL DAAC through a variety of tools and formats allowing a rich resource of daily surface meteorology. Source: Daymet/NASA

With data for every day at a 1 km x 1 km spatial resolution, the Daymet data are a great resource for the temporal and spatial scale at which a cougar would interact with the landscape. There are seven measured values in total. This allows us to check multiple aspects of the weather to assess how it may be effecting behavior.

The metadata associated with the Daymet imagery within GEE.

If you are interested in learning more about climatic data that is available across the globe look to Module 6.

3 Bringing Your own Data Into Earth Engine

In this exercise, we will discuss how to move your own data into GEE, extract values from a dataset, and export those values out of GEE. The processes by which you can bring data into GEE have been changing rapidly and as with most things, it is best to go directly to the documentation to view the latest updates. That information can be found here.

3.1 Cleaning the Data

The data on animal movement were downloaded as a csv file. To bring them into GEE, we need to convert them to a shapefile. While there are many ways to convert a csv files to a shapefile, we are going to use R. The code below contains all that is needed to make this conversion. Details on how to convert a csv file to shapefile in R can be found here.

Some of the complexity of the code comes from renaming columns to remove the “.”. This is necessary in order to match GEE requirements for naming conventions. While this specific detail is not in the documentation it was described in a post on the help forums.

You do not need to run this code, it is here for your future reference

# Load necessary libraries
library(sp)
library(rgdal)
library(dplyr)

# read in CSV of data
baseDir <- "The folder your csv is held in"
data <- read.csv(paste0( baseDir, "/Site fidelity in cougars and coyotes, Utah_Idaho USA (data from Mahoney et al. 2016).csv"))

# convert to spatial points data frame
# remove all NA values from lat and long columns
names(data)
noNAs <-  data[complete.cases(data[ , 4:5]),]
# filter to select animal of interest
glimpse(noNAs)
cougarF53 <- noNAs %>%
  filter(individual.local.identifier == "F53") %>%
  dplyr::select("event.id", "timestamp", "location.long","location.lat")

# Unique GEE issue
# GEE does not accept column names with dots So we will rename our columsn
colnames(cougarF53) <- c("id", "timestamp", "lon", "lat")
# check the time line of data collection so we can match those dates in GEE
timeframe <- sort(cougarF53$timestamp)

print(paste0("The first time stamp is ", timeframe[1], " the last collection is ", timeframe[length(timeframe)] ))


# Create a spatial feature with the sp package
# only keep unique id as data
cougarF53Spatial <- sp::SpatialPointsDataFrame(coords = cougarF53[,3:4], data = cougarF53[,1])
# set coordinate reference system to WGS84, using the proj4str
crs(cougarF53Spatial) <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs "

# Export as shapefile
# write a shapefile
writeOGR(cougarF53Spatial, baseDir, "/cougarF53Locations", driver="ESRI Shapefile")

We wrote out the shapefile with only a single column for each row, a unique id. We did this because we plan on doing most of our analysis outside of GEE so loading in all the extra data is not necessary. The unique ID will allow us to join the value added data from GEE back with the original dataset.

3.2 Bringing in an Asset

  • Coordinate Reference System: First, it is important to note that the projection that GEE uses is WGS 1984 EPSG: 4326. Thus, all data you would like to bring into GEE will need that same coordinate reference system. Keep in mind WGS1984 is a geographic coordinate system. You do not want a projected coordinate system on your data.

  • Uploading a shapefile: In the R code above, we converted a csv file of data into a shapefile and defined the coordinate reference system (CRS) to match what GEE is expecting (WGS 1984). When you load a feature into Google Earth Engine, you are adding a personal asset which will be associated with your GEE account.





You will be able to monitor the upload progress on the task pane.





Once uploaded, you can edit the asset via the asset pane on the left side of the code editor. This allows you to set the sharing parameters. For this example, anyone can read the asset. This means whomever runs the code will be able to work with the dataset even though they do not own it or have it downloaded.



Example of sharing a personal asset.

The process to upload a shapefile can take a while so rather than have you go through the process we have provide a link to the script that has the data needed for this lesson already loaded. Code with preloaded dataset. Please, use this script as a starting point for the rest of the lesson.

After you have run though this content, we suggest loading a shapefile of your own. This can be your data or if you want something quick and easy try a shapefile from the Natural Earth Data. This is a great site for geographic data at various map scales. The 1:110m Physical Vectors that the link above will take you to are very generalized and, as a result, will load much quicker than a more data rich layer.

Every asset has sharing preferences similar to the those of other features you might have on your Google Drive.

import Allows you to add your newly acquired asset to the script. This works much the same as importing an imageCollection to your script.

share Allows you to define who can see and edit the asset.

delete Use this to clean up space but remember that gone is forever.

Once you asset loads, import it to your script by either double clicking on the asset name in the asset panel or by pressing the little arrow icon that appear to the right of the feature when you hover your mouse over the name. Rename the feature something that is descriptive. Then visualize it on a map to make sure the feature looks as you would expect.

In the preloaded script, you can see we have completed these steps already. We also added a print statement to access the data structure.


// Imported the data and not add it to the map and print.
Map.addLayer(cougarF53, {}, "cougar presence data");
print(cougarF53, "cougar data");


You can use the inspector tool to look at the attribute data associated with the new asset.

With the points visualized, make a geometry feature that encompasses our area of interest. We will use the geometry feature to filter our climate data.

You can do this by selecting the square geometry feature and drawing a box that encompasses the points.

Draw a geometry feature around the points to filter the climate data.

3.2.1 Uploading a Raster

Bringing in rasters follows the same process as what we just walked through with our shapefile. Image collections (sets of rasters), being more complicated data types, have a few other requirements that you can read about here.

3.3 Defining Weather Variables

In this lesson were are using Google Earth Engine as a means of associating remotely sensed data (i.e. our rasters) with our point locations. While the process is conceptually straight forward it does take some work to accomplish. With our points loaded, the next step is importing the Daymet weather variables.

3.3.1 Call in Climate Data Daymet

We are using the NASA derived dataset, Daymet V3, due to the 1 km spatial resolution and the fact that it measures the environmental conditions experienced by cougars. We will import it by calling the unique ID of the dataset and filtering it to our bounding box geometry.

// Call in image and filter.
var Daymet = ee.ImageCollection("NASA/ORNL/DAYMET_V3")
.filterDate("2014-01-01", "2017-08-01")
.filterBounds(geometry)
.select('tmin')
.map(function(image){return image.clip(geometry)});

print(Daymet,"Daymet");
Map.addLayer(Daymet, {}, "Daymet");


A view of the structure of the Daymet data from the print statement.

From the print statement, we can see that this is an image collection with 267 images (though your total number of images may be different as the dataset changes over time). Each image has seven bands relating to specific weather measurements. Now that both datasets are loaded in, we are going to associate the occurrences data from the cougar with the weather data.

3.4 Extract Values

With our points and imagery loaded, we can call a function to extract values from the underlying raster based on the known locations of the cougar. We will do this using the ee.Image.sampleRegions function. Search for the ee.Image.sampleRegions() function under the Docs tab to familiarize yourself with the parameters it requires.

ee.Image.sampleRegions() is a function of an image so if we tried to call it on the Daymet ImageCollection we would get an error. To get around this, we are going to convert the Daymet image collection into a multiple band image. Each of the seven measurements for every day will become a specific band in our multiband image. This process will help us in the end because each band is defined by the date it was collected and the variable it is showing. We can use this information to determine which data connects to the positions of the cougar on a specific day.

Important note: With many images in the image collection we are going to create a single image with large number of bands. Because GEE is so good at data manipulations, it can handle this type of request.

// Convert to a multiband image and clip.
var DaymetImage = Daymet
  .toBands()
  .filterBounds(geometry);

print(DaymetImage, "DaymetImage");


A print statement showing the results of converting from image collection to multiband image.

Now that we have a single multiband image, we can use the sampleRegions function. There are three parameters of this function that you need to think about.

Collection The vector dataset that the sampled data will be associated with.

Properties Defines what columns of the vector dataset will remain. In this case, we want to keep the “id” column because that is what we will use to join this dataset back to the original outside of GEE.

Scale: This refers to the spatial scale (cell size) of the dataset. The scale should always match the resolution of your raster data. If you are not sure what the resolution of the raster is, search for the dataset using the search bar and that information will be present documentation.


// Call the sample regions function.
var samples = DaymetImage.sampleRegions({
  collection: cougarF53,
  properties: ['id'],
  scale: 1000 });
print(samples,'samples');


From the print statement we can see that our point locations now have weather measurements associated with them. Again, your results may look slightly different.

3.5 Exporting

3.5.1 Export Points as a Shapefile

We have a series of daily weather data associated to the known locations of cougar 57. While we could work more with these data in GEE, it will be easy to bring them into R or Excel. There are a few options to define where the exported data should end up. Generally speaking, saving the data to a Google Drive account is a safe bet. We are going to use a dictionary (denoted by the curly brackets) to define the parameters of the export.table.toDrive() function.

Shapefile field limit: A shapefile can only contain 255 fields; these data have 1869. Due to this, we are going to export the data as a csv file.

// Export value added data to your Google Drive.
Export.table.toDrive({
  collection: samples,
  description:'cougarDaymetToDriveExample',
  fileFormat: 'csv'
});

When you export something, your task pane will light up. You need to individually run the task by selecting the run button.



Example of the task bar once a script with an export function has been run.

When you select the run button, the following pop-up will appear. This allows you to edit the details of the export.



An example of the user defined parameters present when exporting a feature from GEE.

3.5.2 Exporting a Raster

While working with all these spatial data, you may have realized that a raster showing the median values over the time period which data was collected on the cougar could be a pretty useful bit of information to have. For more information about working with rasters look to Module 5

To do this, we will apply a median() reducer function to the Daymet image collection to generate a median value for each parameter in each cell. Just like with the tabular data, we are going to export this multiband image to Google Drive. Once we convert the image collection to an image using the median() function, we can clip it to the geometry feature object. This feature will be exported as a multiband raster.

// Apply a median reducer to the dataset.
var Daymet1 = Daymet
  .median()
  .clip(geometry);
  
print(Daymet1);

// Export the image to drive.
Export.image.toDrive({
  image: Daymet1,
  description: 'MedianValueForStudyArea',
  scale: 1000,
  region: geometry,
  maxPixels: 1e9
});

Like most things in GEE, there are many options for exporting images. One of the most important options is the max.pixels setting. Generally speaking GEE will not allow you to export raster with more then 10^9 pixels. With the max.pixels parameter, you can bump this up to around 10^12 pixels per image. If your exporting data for an area larger than 10^12 pixels you will need to be creative about how to get the information out of GEE. Sometimes this involves splitting the image into smaller piece or reevaluating the usefulness of such a large image outside of GEE.

4 Conclusion

While Google Earth Engine can be used for planetary scale analyses, it is also an effective resource for quickly accessing and analyzing large amounts of information across time with your own data. The method presented in this module is a great way to add value to your own datasets. In this example, we worked with weather data, but that is by no means the only option! You can connect your data to many other datasets within Google Earth Engine. It is up to you to determine what is important and why.

4.1 Complete Code

// Imported the data and not add it to the map and print.
Map.addLayer(cougarF53, {}, "cougar presence data");
print(cougarF53, "cougar data");

// Call in image and filter.
var Daymet = ee.ImageCollection("NASA/ORNL/DAYMET_V3")
.filterDate("2014-01-01", "2017-08-01")
.filterBounds(geometry)
.select('tmin')
.map(function(image){return image.clip(geometry)});

print(Daymet,"Daymet");
Map.addLayer(Daymet, {}, "Daymet");

// Convert to a multiband image and clip.
var DaymetImage = Daymet
  .toBands()
  .clip(geometry);

print(DaymetImage, "DaymetImage");

// Call the sample regions function.
var samples = DaymetImage.sampleRegions({
  collection: cougarF53,
  properties: ['id'],
  scale: 1000 });
print(samples,'samples');

// Export value added data to your Google Drive.
Export.table.toDrive({
  collection: samples,
  description:'cougarDaymetToDriveExample',
  fileFormat: 'csv'
});

// Apply a median reducer to the dataset.
var Daymet1 = Daymet
  .median()
  .clip(geometry);
print(Daymet1);

// Export the image to drive.
Export.image.toDrive({
  image: Daymet1,
  description: 'MedianValueForStudyArea',
  scale: 1000,
  region: geometry,
  maxPixels: 1e9
});