C3S Soil Moisture Data Access and Anomaly Analysis Notebook¶

This notebook can be run on free online platforms, such as Binder, Kaggle and Colab, or they can be accessed from GitHub. The links to run this notebook in these environments are provided here, but please note they are not supported by ECMWF.

Introduction¶

This notebook-tutorial provides a practical introduction to the Soil moisture gridded data from 1978 to present dataset available in the CDS. We give a short introduction to satellite soil moisture products, followed by a demonstrative use case where we download some data and perform a simple analyses of the observed soil moisture over selected study regions.

Please visit the CDS Soil moisture page to get an overview over the available C3S soil moisture datasets and related documentation.

Satellite sensors are capable of observing the water content in the top soil layer from space. Various satellite missions from different space agencies provide measurements of radiation from the Earth’s surface across different (microwave) frequency domains (e.g. Ku-, X-, C-, and L-band). These (raw) measurements are closely linked to the amount of water stored in the soil. There are two types of sensors that are used to measure this information: passive systems (radiometer) and active systems (radar).

For a detailed description, please see the C3S Soil Moisture Algorithm Theoretical Baseline Document, which is available along with the data on the Copernicus Climate Data Store.

Soil moisture from radiometer measurements (PASSIVE)¶

Brightness temperature is the observable radiance measured by passive sensors (expressed in $K$). It is a function of kinetic temperature (the actual physical temperature of the soil surface) and emissivity. Wet soils have a higher emissivity than dry soils and therefore a lower brightness temperature. Passive soil moisture retrieval uses this difference between the kinetic temperature and brightness temperature to model the amount of water available in the soil of the observed area, while taking into account factors such as the water held by vegetation.

NASA’s SMAP and ESA’s SMOS satellites are examples for L-band radiometer missions. They are suitable for retrieving soil moisture globally, even when vegetation is present in a scene.

Different models to retrieve Soil Moisture from brightness temperature measurements exist. One of the them is the Land Parameter Retrieval Model (Owe et al., 2008, Owe et al., 2001, and van der Schalie et al., 2016). This model is used to derive soil moisture for all passive sensors in C3S.

The PASSIVE product of C3S Soil Moisture contains merged observations from passive systems only. It is given in volumetric units $[m^3 / m^3]$.

Soil moisture from scatterometer measurements (ACTIVE)¶

Active systems emit radiation in the microwave domain (C-band for the data available via the CDS). As the energy pulses emitted by the radar hit the Earth’s surface, a scattering effect occurs and part of the energy is reflected back, containing information on the surface state of the observed scene. The received energy is called “backscatter”, with rough and wet surfaces producing stronger signals than smooth or dry surfaces. Backscatter comprises reflections from the soil surface layer (“surface scatter”), vegetation (“volume scatter”) and interactions of the two.

ESA’s ERS-1 and ERS-2, as well as EUMETSAT’s Metop ASCAT sensors are active systems used in C3S soil moisture. In the case of Metop ASCAT, C3S Soil Moisture uses the Surface Soil Moisture products directly provided by H SAF, based on the WARP algorithm (Wagner et al., 1999, Wagner et al., 2013).

The ACTIVE product of C3S Soil Moisture contains merged observations from active systems only. It is given in relative units $[\%$ $saturation]$.

Merged product (COMBINED)¶

Single-sensor products are limited by the life time of the satellite sensors. Climate change assessments, however, require the use of long-term data records, that span over multiple decades and provide consistent and comparable observations. The C3S Soil Moisture record therefore merges the observations from more than 15 sensors into one harmonized record. The main two steps of the product generation include scaling all sensors to a common reference, and subsequently merging them by applying a weighted average, where sensor with a lower error are assigned a higher weight.

The following figure shows all satellite sensors merged in the PASSIVE (only radiometers), ACTIVE (only scatterometers) and COMBINED (scatterometers and radiometers) C3S product (data set version v202212).

C3S Soil Moisture is based on the ESA CCI SM algorithm, which is described in Dorigo et al., 2017 and Gruber et al., 2019.

The COMBINED product is also given in volumetric units $[m^3 / m^3]$. However, the absolute values depend on the scaling reference, which is used to bring all sensors into the same dynamic range. In this case we use soil moisture simulations for the first 10 cm from the GLDAS Noah model (Rodell et al., 2004).

Prepare your environment¶

The first thing we need to do, is make sure you are ready to run the code while working through this notebook. There are some simple steps you need to do to get yourself set up.

Setup the CDSAPI and your credentials¶

The code below will ensure that the cdsapi package is installed. If you have not setup your ~/.cdsapirc file with your credentials, you can replace None with your credentials that can be found on the how to api page (you will need to log in to see your credentials).

(Install and) Import libraries¶

In the next cell we import all libraries necessary to run code in this notebook. Some of them are python standard libraries, that are installed by default. If you encounter any errors during import, you can install missing packages using the conda package manager via:

!conda install -y -c conda-forge <PACKAGE>

We also define a color map that we will use later for plotting the soil moisture data.

Specify data directory¶

In the next cells we create a variable DATADIR which points to a .zip archive (either newly downloaded or the one provided with this tutorial) containing the selected data from CDS as individual images.

Explore data¶

Different products and versions for C3S Soil Moisture are available on the Copernicus Climate Data Store. In general, there are 2 types of data records:

• CDR: The long term Climate Data Record, processed every 1-2 years, contains data for more than 40 years, but not up-to-date.

• ICDR: Interim Climate Data Record, updated every 10-20 days, extends the CDR, contains up-to-date (harmonised) observations to append to the CDR.

Search for the data¶

There are different options to specify a valid C3S Soil Moisture CDS data download request. You can use the CDS GUI to generate a valid request. Having selected the correct dataset, we now need to specify what parameters we are interested in. These parameters can all be selected in the “Download data” tab. In this tab a form appears in which we will select the following parameters to download:

At the end of the download form, select “Show API request”. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below). Having copied the API request into the cell below, running this will retrieve and download the data you requested into your local directory.

Download the data¶

In the next cell we extract all files from the downloaded .zip archive into a new folder. We do this using standard python libraries. This step might take up to 1 minute to process during first execution!

Inspect data¶

We will use the library xarray to read the data. We can use the function xarray.open_mfdataset to load all extracted files and concatenate them along the ‘time’ dimension automatically. This way we get a 3-dimensional (longitude, latitude, time) data cube, that we store in a global variable DS. In addition, we extract the unit and valid range of the soil moisture variable from the netCDF metadata (SM_UNIT and SM_RANGE).

Finally we plot a table that shows the contents of DS. You can see the dimensions of the data cube, coordinates assigned to each dimension, and data variables (each data point has coordinates assigned for each dimension). You can also explore the ‘Attributes’ field (i.e. metadata assigned to the data cube) by clicking on it.

Use case 1: Data Visualization¶

Now that we have a data cube (DS) to work with, we can start by visualizing some of the monthly soil moisture values. Our data cube DS is an xarray.Dataset, which comes with a lot of functionalities. For example we can create a simple map visualization of:

• soil moisture for a certain date.

• the number of observations for a certain date (note that nobs is only available for monthly and 10-daily averaged data downloaded from CDS). Here we extract an image from the data cube using the xarray.Dataset.sel method.

In the next cell, you can select a date to plot soil moisture data. We then select the correct soil moisture (and observation count) data and create a map visualization.

Try:

• Choosing a different month and year in the first line, to plot the data for a different time stamp.

When looking at images of different dates, you can see data gaps (grey areas; especially in winter, mountainous regions and high latitude) and areas of high (blue) and low (brown) soil moisture content (left map). When changing the date to plot in the cell above, you can see that the number of observations is much larger in later periods of the record than in earlier ones due to the larger number of available satellites (right map).

Use case 2: Time Series Extraction and Analysis¶

In the following two examples we will need to set a study area and a time range. We will use a chosen ‘focus point’ assigned to the study area to extract a time series from the loaded stack at this location.

Study areas¶

First we define some potential study areas that we can use in the following applications. Below you can find a list of bounding boxes, plus one ‘focus point’ in each bounding box. You can add your own study area to the end of the list. If you do, make sure to pass the coordinates in the correct order: Each line consists of:

• A name for the study area.

• WGS84 coordinates of corner points of a bounding box around the study area.

• WGS84 coordinates of a single point in the study area (focus area).

(<STUDY_AREA_NAME>, ([<BBOX min. Lon.>, <BBOX max. Lon.>, <BBOX min. Lat.>, <BBOX max. Lat.>], [<POINT Lon.>, <POINT Lat.>]))

Visual representation study area¶

We will visualize the absolute monthly soil moisture values for a certain date, as we did in ‘Application 1: data visualization’. In addition, we will show the selected study area together with the data on a map. We will also show a chosen ‘focus point’ assigned to the study area, marked by the red X.

Try:

• Choosing a different study area by changing the STUDY_AREA variable.

Extracting the time series data¶

We use the chosen ‘focus point’ assigned to the study area (marked by the red X in the map above) to extract a time series from the loaded stack at this location. We then compute the climatological mean (and standard deviation) for the chosen time series using the selected reference period (baseline). Finally, we subtract the climatology from the absolute soil moisture to derive a time series of anomalies. Anomalies therefore indicate the deviation of a single observation from the average (normal) conditions. A positive anomaly can be interpreted as “wetter than usual” soil moisture conditions, while a negative anomaly indicates “drier than usual” states.

Expressing anomalies¶

There are different ways to express anomalies. For example:

1. Absolute Anomalies: Derived using the difference between the absolute values for a month ($i$) and year ($k$) ($SM_{k,i}$) and the climatology for the same month ($SM_i$). Same unit as the input data.

   $\Large SMA_{k,i}^{abs} = SM_{k,i} - \overline{SM_i}$

   
   

2. Relative Anomalies: The anomalies are expressed relative to the climatological mean for a month ($SM_i$), i.e. in % above / below the expected conditions.

   $\Large SMA_{k,i}^{rel}=\frac{SM_{k,i} - \overline{SM_i}}{\overline{SM_i}} \cdot 100\% $

   
   

3. Z-Scores: Z-scores are a way of standardizing values from different normal distributions. In the case of soil moisture, the spread of values between months can vary, e.g. due to differences in data coverage (i.e. a different distribution for values in summer than for those in winter is expected). This can affect the computed climatology and therefore the derived anomalies. By taking into account the standard deviation ($\sigma$) of samples used to compute the climatologies, we can normalize the anomalies and exclude this effect. Z-scores therefore express the number of standard deviations a value is above / below the expected value for a month.

   $\Large Z_{k,i} = \frac{SM_{k,i} - \overline{SM_i}}{\sigma_{i}}$

   
   

Note: The anomaly formulas are presented in Jupyter Notebook format and may not be viewable in a standard web browser.

Application extracting the time series data¶

First we extract a time series for the chosen ‘focus point’ of the study area (see location in the map of the previous example) and a chosen reference period.

Try:

• Choosing a different study area by changing the STUDY_AREA variable. See Study areas defined above for your options.

• Choosing a different reference period by changing the BASELINE_YEARS variable. The first value is the start year of the reference period, the second value is the last year that is considered when computing the climatological reference.

We plot the extracted soil moisture time series, for our chosen reference years and study area:

We use the time series data to compute the climatological mean and standard deviation for the chosen time series, using the selected reference period (baseline):

Create a plot for the climatology - the mean and standard deviation for each month in the chosen baseline period:

Using the above formulas (section 5.2.1 Expressing anomalies), we compute the three anomaly metrics for the extracted time series and store them in a pandas DataFrame (columns ‘abs_anomaly’, ‘rel_anomaly’ and ‘z-score’):

Now we create a plot for each anomaly metric derived from the extracted time series and climatology:

Choosing a different study area will change the input data and plot anomalies for a different ‘focus point’. You can also change the baseline period and see how it affects the climatology (and anomaly) computation. While the 3 time series look the same at first glance, there are some differences depending on the magnitude of the climatology and the intra-annual soil moisture variance in a point.

Use case 3: Vectorized Anomaly Computation¶

We will now compute the anomalies for the whole image stack (not only on a time series basis as in the previous example). For this we use some of the functions provided by xarray to group data. For the climatology (reference) we use all data from 1991 to 2020 (BASELINE_YEARS). This is the standard baseline period, but you can of course try a different period.

We will select all (absolute) soil moisture values in the reference period, group them by month (i.e. all January, February, ... values for all years in the baseline period) and compute the mean for each group. This way we get the CLIM stack, which consists of 12 images (one for each month):

We can now use the climatology stack and compute the difference between each image of absolute soil moisture and the climatological mean of the same month. This way we compute the anomalies for the whole data cube, without a time-consuming loop over all individual locations. We assign the result to a new variable in the DS stack called sm_anomaly:

We can use the bounding box of the chosen study area to extract all soil moisture values in the area of interest. We then compute the mean over all locations in the bounding box to get a single time series for the study area (MEAN_TS). Note that the coverage of C3S Soil Moisture varies over time (see the previous examples), which can also affect the value range of computed anomalies (standard deviation $\sigma$ is usually larger for earlier periods).

We will resample the monthly soil moisture values in MEAN_TS with an annual frequency (“A”):

Now we can visualize the monthly anomalies from the processed anomaly stack as a bar plot. We will also create a map of annual mean values using the created time series for the whole study area bounding box. This is done for the year assigned to the variable YEAR in the next cell.

Try:

• Plotting the annual anomalies for a different year (assign a different value to YEAR in the next cell).

• Changing BASELINE_YEARS in this application and see how the chosen period affects values in the map and time series.

We see an overall trend towards drier conditions in most regions. You can look at a different study area by selecting it in the first application. Don’t forget to re-run all relevant cells in this chapter to use the new study area and accordingly update the plots!

References¶

Dorigo, W., Wagner, W., Albergel, C., Albrecht, F., Balsamo, G., Brocca, L., Chung, D., Ertl, M., Forkel, M., Gruber, A., Haas, E., Hamer, P.D., Hirschi, M., Ikonen, J., De Jeu, R., Kidd, R., Lahoz, W., Liu, Y.Y., Miralles, D., Mistelbauer, T. Nicolai-Shaw, N., Parinussa, R., Pratola, C., Reimer, C., van der Schalie, R., Seneviratne, S.I., Smolander, T., Lecomte, P. (2017) ESA CCI Soil Moisture for improved Earth system understanding: State-of-the-art and future directions. Remote Sensing of the Environment, 203, 185-215. Doi: 10.1016/j.rse.2017.07.001.

Gruber, A., Scanlon, T., van der Schalie, R., Wagner, W., Dorigo, W. (2019) Evolution of the ESA CCI Soil Moisture climate data records and their underlying merging methodology. Earth System Science Data, 11(2) 717-739. Doi: 10.5194/essd-11-717-2019.

Rodell, M., Houser, P.R., Jambor, U., Gottschalck, J., Mitchell, K., Meng, C.-J., Arsenault, K., Cosgrove, B., Radakovich, J., Bosilovich, M., Entin, J.K., Walker, J.P., Lohmann, D., Toll, D. (2004) The Global Land Data Assimilation System. Bulletin of the American Meteorological Society, 85(3), 381-394. Doi: 10.1175/BAMS-85-3-381.