NetCDF data and datafiles

This is intended as a short introduction to netCDF data. The complete netCDF documentation can be found at the UNIDATA web site.

Contents

• What is NetCDF?
• Why use NetCDF?
• NetCDF and the Climate and Forecast (CF) Metadata Convention
• What's in a NetCDF file? – Example.
• Reading and writing NetCDF files – Including links to downloadable software packages.
• Further information – On NetCDF and the Climate Forecast (CF) Metadata Convention.

NetCDF (network Common Data Form) is an interface for array-orientated data access and a library that provides an implementation of that interface. Many groups have adopted netCDF as a standard way to represent their scientific data. The netCDF software was developed at the Unidata Program Center in Boulder Colorado USA.

• NetCDF is used extensively in the atmospheric and oceanic science communities.
• NetCDF is a portable self-describing binary data format.
• NetCDF is network-transparent, meaning that it can be accessed by computers that store integers, characters and floating-point numbers in different ways.
• NetCDF provides direct-access: a small subset of a large dataset may be accessed efficiently, without first reading through all the preceding data.
• NetCDF is appendable: data can be appended to a netCDF dataset along one dimension without copying the dataset or redefining its structure.
• NetCDF datasets can be read and written in a number of languages, these include C, C++, FORTRAN, IDL, Python, Perl, and Java.
• The different language implementations are freely available from the UNIDATA ftp area or from other mirror sites.
• Several graphics packages support netCDF input, making it very easy to display and analyse netCDF datasets. For instance FERRET and CDAT provide both command line and graphical user interfaces for displaying and analysing gridded data.
• NetCDF is completely and methodically documented in UNIDATA's NetCDF User's Guide.
• Several groups have defined conventions for netCDF files, to enable the exchange of data. BADC has adopted the Climate and Forecasting (CF) conventions for netCDF data.

The BADC supports and strongly recommends the compliance with the Climate and Forecast (CF) Metadata Convention.

CF conventions are guidelines and recommendations as to where to put information within a netCDF file, and they provide advice as to what type of information you might want to include. CF conventions allow the creator of the dataset to include information about the data and the dataset itself (metadata) in a structured way, which makes it easier for other users to retrieve the information. Global attributes describe the general properties and origins of the dataset while local attributes are used to characterise the recorded variables.

We have available a very simple example of a netCDF file with CF conventions applied. A netCDF dataset (the filename ends in .nc) is made up of three basic components:

• variables
• dimensions
• attributes

The variables store the actual data, the dimensions give the relevant dimension information for the variables, and the attributes provide auxiliary information about the variables or the dataset itself. You can study an example of a netCDF file and see more details of the components in a netCDF dataset. The global attributes can also be added to the netCDF file, following CF conventions.

The popularity of netCDF stems from its portability and relative ease of use. Although it is a binary data format the netCDF distribution comes complete with the ncdump utility which produces an ASCII dump of the dataset and thereby provides a quick-look facility. You can also generate netCDF files from ASCII data files using the ncgen utility.

The datasets are commonly written and read using library routines. The library routines are sensibly named and relatively consistently named across the different language implementations.

1. Guidance and tools provided by the BADC:
   
   
   • Installing the netCDF package provided by Unidata and trying out ncgen and ncdump. – Converting NetCDF to ASCII.
     
     
   • XCONV/CONVSH – Allows visualisation, conversion and subsetting of NetCDF data on Unix/Linux systems.
     
     
   • NetCDF format checker – If you wish to check that files are correctly formatted before uploading them to the BADC, please use this online tool.
     
     
2. Selected links to available online software packages to write, read and handle NetCDF files:
   
   
   • Software for Manipulating or Displaying NetCDF Data – An near-exhaustive list of links to a variety of freely available and commercial or licensed software packages to create or handle NetCDF files, provided by Unidata at UCAR (USA). Most links below appear in this list.
     
     
   • AEROS – Available from UCAR (USA), in two versions: one for Linux, one for Windows.
     
     
   • ANTS – Provided by the Atmospheric Radiation Measurement (ARM) Program (USA), the ARM NetCDF Tool Suite (ANTS) is a collection of netcdf tools and utilities providing various means of creating and modifying netcdf files. ANTS is based on NCTOOLS written by Chuck Denham.
     
     
   • CSIRO NetCDF interface to matlab – Available from the Marine and Atmospheric Research Department of the Commonwealth Scientific and Industrial Research Organisation (Australia). The interface has options for automatically handling missing values, scalefactors, and the permutation of hyperslabs. It also has a simple syntax.
     
     
   • FERRET – Interactive computer visualization and analysis environment designed to meet the needs of oceanographers and meteorologists. It takes NetCDF files as input and runs on most Unix systems, and on Windows XP/NT/9x using X windows for display. The FERRET site includes FAQs on NetCDF.
     
     
   • HIPHOP – Initially developed by Dominik Brunner (Switzerland) for HDF files, the Handy IDL Program for HDF Output Plotting (HIPHOP) has been expanded to accept NetCDF and GRIB files.
     
     
   • IGOR – Available from the University of Osaka (Japan), in two versions: one for MacIntosh, one for Windows.
     The BADC also holds routines provided by Jonathan Crosier (University of Manchester) to assist users of FAAM data in reading the FAAM core data files with the Igor software. These are freely available from the FAAM software archive.
     
     
   • MEXNC, SNCTOOLS and NetCDF Toolbox – Created by Chuck Denham to read and write NetCDF files. Provided through SourceForge. Will cease to be supported (but will still be available) from January 1^st, 2007 onwards.
     
     
   • CDAT – Climate Data Analysis Tools available from the Program for Climate Model Diagnosis and Intercomparison (PCMDI) (USA).
     
     
   • NCL– NCAR Command Language (NCL), an interpreted language designed for scientific data analysis and visualisation. It supports netCDF-3/4, GRIB-1/2, HDF4-SDS, HDF4-EOS, HDF5-EOS and shapefiles.

For a more extensive description you are urged to read the UNIDATA documentation on netCDF.

The UNIDATA documentation also provides the File Format Specification for netCDF and Best Practice recommendations on writing netCDF files.

The NetCDF Climate and Forecast (CF) Metadata Convention defines metadata that provide a definitive description of what the data in each variable represents, and of the spatial and temporal properties of the data. The CF conventions generalize and extend the COARDS conventions developed under the sponsoring of NOAA.