Daily site check process
The daily site check process
UAF is tasked with checking the status of the instruments at the sites on the North Slope every working day. There are a number of resources available from ARM with which to make these checks. To make the daily review easier a script on nanuna fetches plots from Plot Browser and puts the plots into a single file which can be scrolled. This saves quite a bit of time and mouse-clicking. This page describes the using these pages for the daily checks.
The script output of the script is consistently written to these site-specific URLs:
At the top of these pages are some basics:
- How to regenerate this report (which you may want to do if data ingest is delayed, for example).
- Links to other tools that should be checked daily.
- Note that DSView in its various forms has limitations. In particular, while the DSView Data Stream Status is supposed to be more authoritative than Plot Browser, regarding data ingest. However, experience has shown there are times when DSView indicates periods where no data has been ingested for an instrument while Plot Browser indicates data for the same period. It is important to cross-check if there is any doubt.
- The past seven days of these plots. These are helpful, for example, on Monday morning, when the previous Friday, Saturday and Sunday should also be checked.
Some instruments have only a single plot for their data streams. Other instruments have many data streams. The heading structure of the page indicates the grouping of the instruments and the data streams from the instruments. The grouping and order corresponds to that in the NSA weekly operations status reports, which you probably want to update as you make your way through these pages.
For each plot, the URL to go directly to that plot on the Plot Browser server is above the plot. This is so that you can immediately go to the latest version of the given plot. Some plots are updated continuously. If no plot is available, the failure will be noted on the page.
Note there are several cases where available plots and/or data will not be presented in directly in this page:
- Movies: mp4 files are not yet embedded in the file. Links are provided to follow to the corresponding movie. The movie will pop up in a new tab/window.
- Directories: Sonde launch data goes into a directories with known dates in the name. However, the individual plot files have launch time to the minute embedded. Sonde launches do not always happen at the exact time they are scheduled. Equipment can have problems, the weather can delay or cancel launches. Best we can presently do is link to the directory and then you can click on the file you wish to view.
Maintaining the script
- The script is a relatively simple Bash script with embedded comments.
- The script is owned by the nsa account on nanuna. You can sign onto nanuna and used sudo to become the nsa user.
- The script is located in ~/nsa/nsaaao/scripts. It is named URL_gen_get_sites.bash.
- The script can be run at any time with no arguments to regenerate the current day's plots with the latest plots from Plot Browser.
- There is one input file per site, defined in the SITES array, currently:
- The XXX_instruments_datastreams.txt file is a csv file with these fields:
- Instrument reporting groups per the weekly reports
- Instrument subgroup, i.e., the short name of the instrument
- ABLE: "enabled" or "disabled": disabled instruments will have the ABLE_REASON printed in the plot file with an extra line for disabled instruments.
- ABLE_REASON: Free text explaining the why the instrument is disabled, noting that it is disabled. If an instrument is enabled, and the ABLE_REASON is not blank, the text will be printed.
- URL part 1: URL derived from Plot Browser URLs to which the "Thumb" thumbnail images point. The script uses URL part 1, URL part 2 and F_TYPE together with the calculated date (see LAG below) to build the URL of interest. This field is the first part of URL, before the first date. If zero length, URL will not be added to output file. This is a way to leverage putting an html href into the ABLE_REASON field without junking up the output with an invalid URL or failed wget attempt.
- URL part 2: Second part of URL, after the first date.
- F_TYPE: The file type of the linked plot. Known types include jpg, png, mp4.
- LAG: Some plots on Plot Browser are up to date within an hour or so of the present. Other data streams lag behind, by days in some cases. The value in this field is fed to a C program, day_offset, and day_offset calculates the date for which data would be expected. That is, day_offset with an argument of 4 yields a date four days ago, adjusting for all the months, including leap years.
- Output from the script is written to nanuna:/products/nsadb/htdocs/uafdr/XXX where XXX is the site ID in the SITES array.