Changes to Nimrod Rain Radar data holdings

This page details important and necessary changes to the Nimrod archive holdings that will come into effect in January 2011, explaining the reasons for the changes and a schedule for the implementation.

Please see the " Progress " section at the bottom of this page for latest information about this implementation of these changes. This will be updated as progress is made.

Why are changes being made?

The present delivery mechanism of the Nimrod rain radar products from the Met Office to the BADC suffers from occasional drop outs due to a variety of reasons - e.g. site connection issues at the BADC, ftp transfer issues from the Met Office.

While a small back log of data are possible at the Met Office which can then be resent once a connection has been re-established (and this means most delayed data arrives ok) as a consequence of the file naming convention used in the delivery mechanism lacking a date string and are re-cycled every 24 hours drop outs longer than 24 hours result in loss of data. (files are checked on arrival at the BADC and given the appropriate date string when they are re-named for inclusion in our archive).

In addition to the delivery mechanism issues there are archive management issues at the BADC given the sheer number of files within the Nimrod dataset (over 17 million files, the single largest number of files in any one dataset at the BADC). The large number of files result in performance issues with the back-up and file corruption checking systems. To address this necessary changes to the archive structure will be instigated.

Finally, the changes required to meet the issues above provide an opportunity to bring the Nimrod file naming convention inline with the BADC file naming convention. This will make the file names more explicit, allowing users to identify the products with greater ease.

The content and format of the files will not be changed - only the archive structure and file names will be. Therefore, the files should remain compatible with existing scripts to read in and plot data.

What changes will be made?

To address the issues above two changes are being made. The first is to switch the delivery mechanism to use an agreed route that the BADC has much greater control over. This will give us an improved ability to cope with Met Office- BADC connection issues in the future. Coupled with this the file naming convention used by the Met Office to deliver data to the BADC will be changed to include a date-string. Although some delays in delivery may still occur from time to time users should no longer suffer from gaps in archive for future data (sadly, there is still no mechanism to back fill existing gaps).

Finally, once files have arrived at the BADC they will be re-named to meet BADC file naming conventions, compressed and "tar"-ed together with files for the same day. ("tarring" creates one archive file containing multiple files)

To ensure consistency of the archive and address archive management issues, the existing data will then be re-processed to rename, compress and "tar" together files.

How will the old archive structure and file names map to the new structure and names?

The directory structure will remain largely the same, with the only change being that the need to have month and day splitting will be removed, thus making for easier and quicker navigation of the archive.

File Name Mapping

The table below shows how the old filenames map over to the new file naming convention which is consistent with the BADC file naming convention

Progress

2011-02-22 : Reprocessing of existing archive has been completed and new version of the archive are being ingested. New filenaming convention and delivery route for data has been enabled at the Met Office. Operational ingest process has been deployed and tests will continue over the next week to ensure that all data are arriving and ingesting ok. During this period data will continue to be delivered in the old filenaming convention and delivery root.

2011-01-11 : Reprocessing of existing archive initiated. This will be background operation until products are ready.

2011-01-10 : Change request has been submitted to Met Office for change in data supply route and file naming convention used for supplied files. Ingest scripts have been prepared for new ingest and start of back reprocessing of data.

Who to Contact

If you have queries on this page or about obtaining the data then you should contact BADC Support.