GitHub

NetCDFTools.jl

Documentation for NetCDFTools.jl, a Julia package for loading/writing NetCDF data on the top of NCDatasets.jl.

Installation

Inside the Julia shell, you can download and install using the following commands:

using Pkg
Pkg.add("NetCDFTools")
Base.getindex — Function
getindex(
    dims::Vector{NcDim},
    name::Union{Regex, AbstractString}
) -> Any

Examples

f = "test/data/temp_HI.nc"
dims = nc_dims(f)
dims[["lon", "lat"]]
source
NetCDFTools.NcDim — Type
NcDim(name::AbstractString, vals::AbstractArray, atts::Dict = Dict())

Represents a NetCDF dimension of name name optionally holding the dimension values.

source
NetCDFTools.ncdim_def — Function
ncdim_def(ds, name, val, attrib = Dict())
ncdim_def(ds, dim::NcDim)
ncdim_def(ds, dims::Vector{NcDim})

Examples

## 1. by existing nc file
# f = "/mnt/j/Researches/Obs/Obs_ChinaHW_cluster/ncell_16/clusterId_HItasmax_movTRS_Obs.nc"
dims = nc_dims(f)
ds = nc_open("f2.nc", "c")
ncdim_def(ds, dims);
println(ds)
close(ds)

## 2. by assigned variables
# dates = nc_date(ds)
ds = nc_open("f3.nc", "c")
dim_t = NcDim_time(dates)
ncdim_def(ds, dim_t)
close(ds)
source
NetCDFTools.ncvar_def — Function
ncvar_def(ds, name, val, dims::Vector{<:AbstractString}, attrib = Dict();
    compress = 1, kwargs...)
ncvar_def(ds, name, val, dims::NcDim, attrib = Dict();
    compress = 1, kwargs...)

Arguments

  • val: can be data or vtype

  • compress: Compression level: 0 (default) means no compression and 9 means maximum compression. Each chunk will be compressed individually.

  • type: which type to save? Julia variable types (not string), e.g., Float32.

  • options: Dictionary object,

    • fillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute.

    • chunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.

    • shuffle: If true, the shuffle filter is activated which can improve the compression ratio.

    • checksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)

    • attrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)

Examples

f = "f1.nc"
isfile(f) && rm(f)
ds = nc_open(f, "c")
compress = 1

ncdim_def(ds, "lon", lon, Dict("longname" => "Longitude", "units" => "degrees east"))
ncdim_def(ds, "lat", lat, Dict("longname" => "Latitude", "units" => "degrees north"))
ncdim_def(ds, "time", 1:ntime)

ncvar_def(ds, "HI", dat, ["lon", "lat", "time"], Dict("longname" => "hello"); compress = compress)
close(ds)

# append a variable
ds = nc_open(f, "a")
ncvar_def(ds, "HI2", dat, ["lon", "lat", "time"]; compress = compress)
print(ds)
close(ds)

nc_info(f)

@seealso ncdim_def

source
NetCDFTools.ncatt_put — Function
ncatt_get(f::NCfiles, key)
ncatt_put(f::AbstractString, attrib = Dict())
ncatt_del(f::AbstractString, keys::Vector{<:AbstractString})

add or delete global attributes

source
NetCDFTools.nc_read — Function
nc_read(file) -> Any
nc_read(
    file,
    band;
    type,
    period,
    ind,
    raw,
    nodata,
    verbose
) -> Any

Arguments

  • band : string (variable name) or int (band id). If int provided, bandName = nc_bands(file)[band]

  • type : returned data type, e.g. Float32

  • period: [year_start, year_end] or year, time should be in the 3rd dimension.

  • ind : If ind provided, period will be ignored.

  • raw : Boolean.

    • true: raw data.
    • false (default): missing will be replaced with nodata

  • verbose: Boolean. It true, data and ind will be printed on the console.

source
NetCDFTools.nc_write — Function
nc_write(
    f::AbstractString,
    varname::AbstractString,
    val,
    dims::Vector{NcDim}
) -> Any
nc_write(
    f::AbstractString,
    varname::AbstractString,
    val,
    dims::Vector{NcDim},
    attrib::Dict;
    compress,
    overwrite,
    mode,
    global_attrib,
    kw...
) -> Any

Arguments

  • attrib: An iterable of attribute name and attribute value pairs, for example a Dict, DataStructures.OrderedDict or simply a vector of pairs (see example below)

  • type: which type to save? Julia variable types.

  • kwargs: Parameters passed to ncvar_def, and further to [NCDatasets.defVar]. For examples:

    • fillvalue: A value filled in the NetCDF file to indicate missing data. It will be stored in the _FillValue attribute.

    • chunksizes: Vector integers setting the chunk size. The total size of a chunk must be less than 4 GiB.

    • shuffle: If true, the shuffle filter is activated which can improve the compression ratio.

    • checksum: The checksum method can be :fletcher32 or :nochecksum (checksumming is disabled, which is the default)

  • mode(not used): one of "r", "c", "w", see [NCDatasets.nc_open()] for details

Examples:

dims = [
    NcDim("lon", lon, Dict("longname" => "Longitude", "units" => "degrees east"))
    NcDim("lat", lat, Dict("longname" => "Latitude", "units" => "degrees north"))
    NcDim_time(dates)
]
dims = make_dims(range, cellsize, dates) # another option

nc_write(f, varname, val, dims)
nc_write(f, varname, val, dims)
nc_write(
    f,
    varname,
    val,
    dims,
    attrib;
    compress,
    overwrite,
    mode,
    global_attrib,
    kw...
)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/nc_write.jl:45.

@seealso ncvar_def

source
NetCDFTools.nc_write! — Function
nc_write!(
    f::AbstractString,
    varname::AbstractString,
    val,
    dims::Vector{<:Union{AbstractString, NcDim}}
) -> Any
nc_write!(
    f::AbstractString,
    varname::AbstractString,
    val,
    dims::Vector{<:Union{AbstractString, NcDim}},
    attrib::Dict;
    compress,
    kw...
) -> Any

nc_write!(f, varname, val, dims)
nc_write!(f, varname, val, dims, attrib; compress, kw...)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/nc_write.jl:85.

source
nc_write!(
    val::AbstractArray,
    f::AbstractString,
    dims::Vector{<:Union{AbstractString, NcDim}}
) -> Any
nc_write!(
    val::AbstractArray,
    f::AbstractString,
    dims::Vector{<:Union{AbstractString, NcDim}},
    attrib;
    varname,
    kw...
) -> Any

nc_write!(val, f, dims)
nc_write!(val, f, dims, attrib; varname, kw...)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/nc_write.jl:100.

source

Utilities

NetCDFTools.QDM — Function
QDM(
    y_obs::AbstractArray{T<:Real, 1},
    y_calib::AbstractArray{T<:Real, 1},
    y_pred::AbstractArray{T<:Real, 1};
    type_adj,
    kw...
) -> Any

Arguments

  • type_adj:

    • add: 高斯分布的变量，如Tair
    • mul: 降水

Notes

QDM: 需要使用一个滑动窗口。

  • ClimDown使用的是10年的窗口，一次处理10年；

  • climQMBC使用的是nobs的窗口，一次处理1年。

References

  1. Cannon, A. J., Sobie, S. R., & Murdock, T. Q. (2015). Bias Correction of GCM Precipitation by Quantile Mapping: How Well Do Methods Preserve Changes in Quantiles and Extremes? Journal of Climate, 28(17), 6938–6959. https://doi.org/10.1175/JCLI-D-14-00754.1

  2. https://github.com/pacificclimate/ClimDown/blob/master/R/QDM.R#L126

  3. https://github.com/rpkgs/climQMBC/blob/master/R/map_QDM.R#L43

source
NetCDFTools.bilinear — Function
bilinear(
    x,
    y,
    z::AbstractArray{T<:Real, 3},
    xx,
    yy;
    na_rm
) -> Any

Suppose that the location, (locx, locy) lies in between the first two grid points in both x an y. That is locx is between x1 and x2 and locy is between y1 and y2. Let ex= (l1-x1)/(x2-x1) ey= (l2-y1)/(y2-y1). The interpolant is

( 1-ex)(1-ey)z11 + (1- ex)(ey)z12 + ( ex)(1-ey)z21 + ( ex)(ey)z22

Where the z's are the corresponding elements of the Z matrix.

bilinear(x, y, z, xx, yy; na_rm)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/Interpolation/bilinear.jl:30.

References

  1. https://github.com/cran/fields/blob/master/R/interp.surface.R

  2. https://en.wikipedia.org/wiki/Bilinear_interpolation

Examples

lon = 70:5:140
lat = 15:5:55

Lon = 70:2.5:140
Lat = 15:2.5:55
Z = rand(T, length(lon), length(lat), 2)
r = bilinear(lon, lat, Z, Lon, Lat; na_rm=true)
source
NetCDFTools.nc_subset — Function
nc_subset(
    f,
    range::Vector
) -> Union{Nothing, NCDatasets.NCDataset{Nothing}}
nc_subset(
    f,
    range::Vector,
    fout;
    delta,
    check_vals,
    verbose,
    big,
    plevs,
    band,
    outdir,
    overwrite
) -> Any

Arguments

  • check_vals: If download failed, length(unique(vals)) = 1. Default check the length of data unique values

Note

! 目前只保存其中一个变量

Example

# 4-d array
url = "http://esgf3.dkrz.de/thredds/dodsC/cmip6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/zg/gn/v20191108/zg_day_ACCESS-CM2_historical_r1i1p1f1_gn_19500101-19541231.nc"

nc_open(url)

# 3-d array
url = "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/huss/gn/v20191108/huss_day_ACCESS-CM2_historical_r1i1p1f1_gn_18500101-18991231.nc"

range = [70, 140, 15, 55]
delta = 5

@time nc_subset(url, range)
nc_subset(f, range)
nc_subset(
    f,
    range,
    fout;
    delta,
    check_vals,
    verbose,
    big,
    plevs,
    band,
    outdir,
    overwrite
)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/utilize/nc_subset.jl:32.

source

CMIP functions

NetCDFTools.CMIP.heat_index — Function
heat_index(t::Union{<:Real,Missing}, rh::Union{<:Real,Missing})
heat_index(t::AbstractArray{T}, rh::AbstractArray{T}) where {T<:Union{Missing,Real}}
heat_index(f_tair::AbstractString, f_rh::AbstractString, outfile::AbstractString;
    raw=true, offset = -273.15,
    varname="HI", type = Float32, compress=1, 
    overwrite=false)

Arguments

  • t: degC
  • rh: %

Examples

heat_index(f_tair, f_rh, outfile; overwrite = false, raw = true, compress = 1)

References

  1. https://www.wpc.ncep.noaa.gov/html/heatindex_equationbody.html
  2. Kong, D., Gu, X., Li, J., Ren, G., & Liu, J. (2020). Contributions of Global Warming and Urbanization to the Intensification of Human‐Perceived Heatwaves Over China. Journal of Geophysical Research: Atmospheres, 125(18), 1–16. https://doi.org/10.1029/2019JD032175.
source
NetCDFTools.CMIP.CMIPFiles_info — Function
CMIPFiles_info(
    files;
    detailed,
    include_year,
    include_nmiss
) -> DataFrames.DataFrame

Note: currently, only works for daily scale

Return

  • model:
  • ensemble:
  • date_begin, date_end:
  • file:
CMIPFiles_info(files; detailed, include_year, include_nmiss)

defined at /home/runner/work/NetCDFTools.jl/NetCDFTools.jl/src/CMIP/CMIPFiles_info.jl:82.

Example

fs = [
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/huss/gn/v20191108/huss_day_ACCESS-CM2_historical_r1i1p1f1_gn_18500101-18991231.nc",
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/huss/gn/v20191108/huss_day_ACCESS-CM2_historical_r1i1p1f1_gn_19000101-19491231.nc",
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/huss/gn/v20191108/huss_day_ACCESS-CM2_historical_r1i1p1f1_gn_19500101-19991231.nc",
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/CMIP/CSIRO-ARCCSS/ACCESS-CM2/historical/r1i1p1f1/day/huss/gn/v20191108/huss_day_ACCESS-CM2_historical_r1i1p1f1_gn_20000101-20141231.nc",
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/CSIRO-ARCCSS/ACCESS-CM2/ssp126/r1i1p1f1/day/huss/gn/v20210317/huss_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_20150101-20641231.nc",
  "http://esgf-data04.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/CSIRO-ARCCSS/ACCESS-CM2/ssp126/r1i1p1f1/day/huss/gn/v20210317/huss_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_20650101-21001231.nc"
]
info = CMIP.CMIPFiles_info(fs; detailed=false)
source