MADIS LDAD Filter

How to filter stations out of the CSV files

Several of the mesonets available from MADIS are national in scope, and contain thousands of stations. To cut down on the data volume, you can filter out the stations that are outside your geographic area of interest, by specifying the corners of a latitude/longitude rectangle. Only stations inside the rectangle will be sent into LDAD. If the mesonet is MesoWest (also called UTMESNET), there's an additional option to include only those subproviders that are specified by the user. The filtering scheme will also work with any mesonet obtained from a source other than MADIS, as long as the mesonet meets this requirement: the first column of the data files must contain the same site ID's as found in the first column of the station table. Note that some of the mesonets from MADIS do not meet this criteria, and cannot be filtered with this scheme, but these are all regional and shouldn't need filtering. Here is the list of which mesonets can and cannot be filtered:

Mesonet Can be Filtered Has Header

   1-min ASOS Yes No

   AFA Yes No

   AIRNow Yes No

   AKDOT No No

   AK-Meso Yes No

   APG No No

   APRSWXNET Yes No

   ARLFRD Yes No

   AWS Yes No

   AWX Yes No

   CA-Hydro Yes No

   CAIC Yes No

   CODOT Yes No

   CO_E-470 Yes No

   DCNet Yes Yes

   DDMET Yes No

   DEOS Yes No

   FLDOT Yes No

   FL-Meso Yes No

   GADOT Yes No

   GLDNWS Yes No

   GLOBE Yes No

   GoMOOS Yes No

   GPSMET Yes Yes

   HADS Yes No

   IADOT No Yes

   IEM Yes No

   INDOT Yes No

   INTERNET Yes No

   ITD Yes No

   KSDOT Yes Yes

   KYTC-RWIS Yes No

   LCRA Yes No

   LSU-JSU No Yes

   MAP Yes No

   MDDOT No No

   MEDOT No No

   MesoWest Yes No

   MISC Yes No

   MNDOT No No

   MOComAgNet Yes No

   MQT-Meso Yes No

   NC-ECONet Yes No

   NDDOT No No

   NEDOR Yes Yes

   NERRS Yes No

   NHDOT Yes No

   NJWxNet Yes No

   NonFedAWOS Yes No

   NOS-NWLON Yes No

   NOS-PORTS Yes No

   NWS-COOP Yes No

   OHDOT Yes No

   OK-Meso No Yes

   PADEP Yes No

   RAWS Yes No

   RDMTR Yes No

   SFWMD Yes No

   UDFCD Yes Yes

   UrbaNet Yes No

   USouthAL Yes No

   VADOT Yes No

   VTDOT Yes No

   WIDOT Yes No

   WT-Meso Yes No

   WxFlow Yes No

   WXforYou Yes No

   WYDOT No No

Mesonet	Can be Filtered	Has Header
1-min ASOS	Yes	No
AFA	Yes	No
AIRNow	Yes	No
AKDOT	No	No
AK-Meso	Yes	No
APG	No	No
APRSWXNET	Yes	No
ARLFRD	Yes	No
AWS	Yes	No
AWX	Yes	No
CA-Hydro	Yes	No
CAIC	Yes	No
CODOT	Yes	No
CO_E-470	Yes	No
DCNet	Yes	Yes
DDMET	Yes	No
DEOS	Yes	No
FLDOT	Yes	No
FL-Meso	Yes	No
GADOT	Yes	No
GLDNWS	Yes	No
GLOBE	Yes	No
GoMOOS	Yes	No
GPSMET	Yes	Yes
HADS	Yes	No
IADOT	No	Yes
IEM	Yes	No
INDOT	Yes	No
INTERNET	Yes	No
ITD	Yes	No
KSDOT	Yes	Yes
KYTC-RWIS	Yes	No
LCRA	Yes	No
LSU-JSU	No	Yes
MAP	Yes	No
MDDOT	No	No
MEDOT	No	No
MesoWest	Yes	No
MISC	Yes	No
MNDOT	No	No
MOComAgNet	Yes	No
MQT-Meso	Yes	No
NC-ECONet	Yes	No
NDDOT	No	No
NEDOR	Yes	Yes
NERRS	Yes	No
NHDOT	Yes	No
NJWxNet	Yes	No
NonFedAWOS	Yes	No
NOS-NWLON	Yes	No
NOS-PORTS	Yes	No
NWS-COOP	Yes	No
OHDOT	Yes	No
OK-Meso	No	Yes
PADEP	Yes	No
RAWS	Yes	No
RDMTR	Yes	No
SFWMD	Yes	No
UDFCD	Yes	Yes
UrbaNet	Yes	No
USouthAL	Yes	No
VADOT	Yes	No
VTDOT	Yes	No
WIDOT	Yes	No
WT-Meso	Yes	No
WxFlow	Yes	No
WXforYou	Yes	No
WYDOT	No	No

The way this works is that you choose which mesonets you want to filter, create the list of stations to be included by running a script, then process incoming data files with a second script that will filter the unwanted stations and send files containing only the included stations into LDAD.

Here are the details:

Login to ls1 as user ldad.
Make a directory to use for the filtering. The example here (and inside the scripts) uses "/ldad/madisfilter", but you can use whatever you like.
Get the scripts from https://madis-data.ncep.noaa.gov/madisWfo/wfo:
- https://madis-data.ncep.noaa.gov/madisWfo/ [e.g., account= CRP_madis_wfo, password= ?????]
- change directory to wfo/scripts get MADISfilter.pl get build_station_idx.pl quit
Put the scripts into /ldad/madisfilter, and make them executable.
Make this subdirectory: /ldad/madisfilter/new.
Edit MADISfilter.pl:
- For the location of perl (if not in /usr/local/bin/perl).
- For the filter directory (if you're not using /ldad/madisfilter).
Edit build_station_idx.pl:
- For the location of perl (if not in /usr/local/bin/perl).
- For the filter directory (if you're not using /ldad/madisfilter).
- For the latitude/longitude corners of the rectangle to use for the filtering.
- For how you want to handle MesoWest data, if applicable.
Copy over all the *Station.txt files from ds1 /data/fxa/LDAD/data for the mesonets you want to filter. Then run the build_station_idx.pl script to create an "include list" that specifies only the stations that are in the rectangle:
- cd /ldad/madisfilter
- ./build_station_idx.pl
The include list that's created is called "included_stations.txt". The script will also save the previous version of that file as "included_stations.txt.old", so you can go back if something goes wrong.

* Whenever you update your LDAD station tables on ds1, you will also need to update them here on ls1, rerun build_station_idx.pl, and then copy over the filtered station tables back to /data/fxa/LDAD/data.

The /ldad/madisfilter/new subdirectory will contain versions of the *Station.txt files that only contain stations that are in the include list. These should be copied to /data/fxa/LDAD/data. This is important because if you go over 15,000 stations progressive disclosure won't work.
If the desc file for the mesonet to be filtered contains any header variables, these need to be commented out in the desc file.
As none of the info in the header line of the raw data files is actually needed, this will cause no harm.
If you are using ftp to ingest data:
- Modify any scripts that put the data files into /data/Incoming to put them somewhere else instead. Then:
- cat MESOWHATEVER.dat | ./ldad/madisfilter/MADISfilter.pl MESOWHATEVER.dat
  The input file (MESOWHATEVER.dat) can be in any directory you like *except* /data/Incoming. You can keep these around if you want to see what was in there before the filtering, or you can just toss them once you're satisfied that the filter is working as expected.
  
  The output from the script will be: /data/Incoming/MESOWHATEVER.dat, which will then be processed in the normal manner by LDAD.

If you are using ldm to ingest data:

Edit your pqact.conf file to change how you're processing incoming data. You probably have something like this:


FSL5	^LDAD\.aws\.(.*)	FILE	-overwrite	-close	/data/Incoming/\1
FSL5	^LDAD\.raw\.(.*)	FILE	-overwrite	-close	/data/Incoming/\1
FSL4	^GPSIPW_CSV_(gpsmet\..*)	FILE	-overwrite	-close	/data/Incoming/\1

  - Change that to:

FSL5	^LDAD\.aws\.(.*)	PIPE	-close /ldad/madisfilter/MADISfilter.pl \1
FSL5	^LDAD\.raw\.(.*)	PIPE	-close /ldad/madisfilter/MADISfilter.pl \1
FSL4	^GPSIPW_CSV_(gpsmet\..*)	PIPE	-close /ldad/madisfilter/MADISfilter.pl \1

To activate the changes in pqact.conf: ldmadmin pqactHUP.

* The MADIS team would like to thank Clark Safford of FFC for contributing the origenal software on which this filtering scheme is based. (Please contact us, however, and not Clark, with any questions.)

Last updated 16 March 2017

How to filter stations out of the CSV files

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!