Weatherlink to Cumulus log converter

[steve]

Where my humidity results were --- in my WLK file I get an output of -3276 and whether I have had any rain or not, a figure of -6553.6 for my rainfall rate. This also I assume affects the dayfile text file as I have some dewpoint values of -1838.

As noted above, it doesn't currently check for all of the 'no reading' values. The Davis Weatherlink document doesn't actually define them; I think they must be as defined in the serial protocol spec.

Also, once the monthly files are included and the dayfile updated, will Cumulus automatically recalculate to display all the high / lows etc in all the various display options, or are additional steps needed to achieve this.

You will need to use the various high/low editors to scan the new logs and update the highs and lows as required.

[ramsded]

Steve

Thanks for your reply.

Presumably if I were to replace all the affected values with 0, this would affect the high/lows and averages, is there a value that can be used so as not to do this, ie outside a range depending on the parameter that Cumulus would then ignore ?

Dave

[steve]

It's tricky, and it's one of the reasons (apart from being too busy) that I haven't done any more to the converter. What Cumulus does when it can't get new data for whatever reason is to re-use the last value that it had. While this has its disadvantages, it does have a number of advantages (like not affecting highs and lows and having minimal affect on averages). It means that Cumulus doesn't need and doesn't have 'null' values, so there is no figure you can insert which will get ignored.

So the converter will probably have to behave in the same way as Cumulus (it already does for UV and solar).

[steve]

The Davis Weatherlink document doesn't actually define them; I think they must be as defined in the serial protocol spec.

Actually, it's clear that the .wlk file 'dash values' are not the same as the serial protocol dash values - they can't be. For example, in the archive records in the serial protocol, the dash value for humidity is 255 and humidity is held as a whole number. In the wlk files, humidity is held as tenths of a percent, and the dash value is apparently, but not actually documented anywhere, minus 32768.

[mcrossley]

the dash value is apparently, but not actually documented anywhere, minus 32768.

= 0xFFFF = -1 = false ???

[steve]

-32768 is one of the 'dash values' in the serial protocol; but they also use +32767, 255 and 0. So the wlk files could use any of those for various fields, but from the post above it appears that it also uses -65536.

0xFFFF

Ah hold on, yes - maybe the 'dash value' fields just have all of the bits set to 1.

[ramsded]

Having now used the converter to convert all my WLK files back to Jan 2000, I am experiencing some problems ( no doubt of my own making ) whilst editing the high / low data. Initially this worked fine as I converted the files and edited the data month by month and then year by year. But eventually I realised I could do it all and paste all the monthly files and update the dayfile.txt in one go, when I then tried to fetch the files to edit, it only gets some data for Jan and none for the other months from the dayfile and only populates approx. 50% of that from the log data.

All the monthly files are there and the dayfile.txt contains all the data from 01/01/2000 ?

Other than when viewing this data from this period, all other options - view month, year and NOAA only allow me to select back to 2003 from the drop down pick list, I believed you could go back to 2000?

Is it possible to edit the NOAA headers for the UK ie be able to change State to County ?

[steve]

Having now used the converter to convert all my WLK files back to Jan 2000, I am experiencing some problems ( no doubt of my own making ) whilst editing the high / low data. Initially this worked fine as I converted the files and edited the data month by month and then year by year. But eventually I realised I could do it all and paste all the monthly files and update the dayfile.txt in one go, when I then tried to fetch the files to edit, it only gets some data for Jan and none for the other months from the dayfile and only populates approx. 50% of that from the log data.

All the monthly files are there and the dayfile.txt contains all the data from 01/01/2000 ?

Sorry, but I can't follow that. Are you saying that you are using Edit -> Monthly Records? If you zip up your dayfile.txt and attach it, I'll take a look.

Other than when viewing this data from this period, all other options - view month, year and NOAA only allow me to select back to 2003 from the drop down pick list, I believed you could go back to 2000?

It's because I started writing Cumulus in 2003, so there was no point in allowing selection of earlier years. I can change that fairly easily.

Is it possible to edit the NOAA headers for the UK ie be able to change State to County ?

No; I followed the 'standard' format so that the files would be the same as produced by other software - Weatherlink in particular. Does WL allow you to change that?

[ramsded]

Thank you for the offer of checking my files, these are hopefully attached and go back to 01/01/2000 and Yes, I am using Edit - monthly records.

It would be good if you can allow selection back to 2000 in a future update.

Regards the NOAA setup, as you say you cannot change the headers in WL, but your program is streets ahead so I thought I would ask.

Dave

[steve]

It's because of the problems mentioned above with the converter; it doesn't currently ignore all of the 'dash' values. Your Weatherlink files have data missing, so your converted Cumulus logs have invalid data (values and timestamps) in them, and hence so does your dayfile.txt, from the very first line:

01/01/00,11.0,0,01:00,-0.2,23:59,7.9,00:15,-1109.65,34:07,-1109.65,34:07,-8323.1,34:07,0.2,3.4,57.3,4.0,01:00,-3276,34:07,-3276,34:07,621.79,0.0,-1838.0,34:08,-70.8,00:00,-110.9,00:15,0.2,07:30,-0.3,08:15,-1838.0,34:07,-1838.0,34:07,112,14.9,0.0,0,00:00,0,00:00

I'm working on it, as time permits. But it's hard to know what to do in some cases.

[oman4eva]

Hey Steve,

I'm getting lots of "Floating Point Division by Zero" errors upon trying to convert some of my old data. The partially converted file ends here

20/01/03,19:30,-9.6,1,-56.0,7.2,16.1,292,-6553.6,0.0,989.44,0.0,20.4,6,16.1,-18.3,-9.6,0,0,252.60,0.00,-18.6,0,0.0,292,0.0,0.0

While an original export of the jan 2003 data shows no issues here

03/20/2003 7:30 p 6.1 6.1 2.4 6.4 5.7 100 6.1 11.3 27.4 E 0.0 990.5 22.1 23 30
03/20/2003 8:00 p 5.7 5.7 4.3 6.2 5.6 100 5.7 8.0 22.5 E 0.0 990.3 22.2 22 30

the same january exported file does have dashes down the end of the file

03/31/2003 5:00 p -- -- -- -2.3 -2.3 -- -- 12.9 22.5 S 0.0 992.9 23.2 12 30
03/31/2003 5:30 p -1.1 -1.1 -2.8 -1.1 -2.2 100 -1.1 8.0 25.7 SSW 0.0 992.3 23.3 12 30
03/31/2003 6:00 p -- -- -- -- -- -- -- 0.0 0.0 --- 0.0 991.9 23.2 12 30
03/31/2003 6:30 p -- -- -- -- -- -- -- 0.0 0.0 --- 0.0 991.8 23.2 11 30

etc


any ideas if there's a fix here.

cheers
oman

[steve]

If you attach that Jan 03 wlk file (you may need to zip it so the forum doesn't object to the extension), I'll run it here so I can see what the problem is. The converter does clearly need work, though, and I don't know when I'll find time to do it.

[Safety]

Steve,
First of all thank you for taking the time to produce the converter utility programme.
In my case I have a number of years of data to convert (2003-2010). Back in 2003 I had reception problems from my ISS so each month I have a few erroneous values in my archive files.
Therefore when I run the converter I get errors as well documented within this group topic. In my case my objective is to use the data so I can compare each year of weather data using the graphs from “Select-A-Graph”.
To get the converter working all I needed to do is delete records that were missing data so I used a programme call “WLKReaderWriter” in my case I can live with losing a few 15 minute data grabs. This got me thinking, if you are looking for any ideas on new features for your program, I would like to suggest the following features:-
To be able to undertake a batch Load of a number of WLK files with the intention of detecting invalid records.
Each invalid record would then be flagged so that they could be highlighted for deletion.
The criteria for detecting if a record is invalid or not could be based on if the first temperature field (the field after the date / time fields in your txt file) has an erroneous value. In my case when I checked all the records that I needed to delete this is the field that always displays a value of -1838

[oman4eva]

Hi Steve

I've attached the March 2003 wlk file which was one of the month's having issues.


I've gone through and copied the monthly log files to the data directory and updated the dayfile.txt by copying the newly generated one into the original. I had a few questions.

a) Does the order of the dayfile matter? the entries appear to be a bit out of order, perhaps cause i didn't convert them in chronological order.

b) While Cumulus can display the past records if you load them or look at monthly records etc. However if i go to my website http://www.atmosp.physics.utoronto.ca/wstat/monthlyrecord.htm it doesn't show any records from prior to when i set cumulus up initially. I.e. i've been swapping between cumulus and weatherlink since October 2013 but have just converted data back to 2001. I assume there must be a way to update this monthlyrecord.htm file automatically?

cheers
oman

[steve]

I've attached the March 2003 wlk file which was one of the month's having issues.

I'll have a play with it when I get chance.

Does the order of the dayfile matter?

Probably not; for most things (e.g. the 'This Month' etc displays) it scans the entire file looking for the entries it needs. I can't guarantee that will always be the case, and it may be that some of the code does already assume the correct order.

I assume there must be a way to update this monthlyrecord.htm file automatically?

Use the high/low editors to fetch the values from the dayfile, as mentioned earlier in the thread. But the off-scale values generated by the current version of the converter will probably render that ineffective.