Create/Cleanup Monthly Log Files

[sperry russ]

I started using Cumulus 1 in 2012 and used a Fine Offset station. Then I changed to the Windblown Station (emulates a Fine Offset) that Charlie Gale built. I have data from 2012 till the present time (2017). Over the years I had quite a few inaccurate reading due to different problems especially with the Fine Offset station.

Anytime I had a inaccurate readings in the monthly log files I would try to find them and usually I would delete those entries. But I would always correct any readings in the dayfile.txt. At that time in my thinking, was that for statistics, all I needed was an accurate dayfile.txt. Which has worked fine for me.

But, now I am changing to a Davis Pro 2 and going to use Meteotemplate for my website. I am able to use the monthly files and import them into Meteotemplate. However as I said a lot of the monthly log files have inaccurate reading or deleted entries.

I know there is some way to use the dayfile.txt to somehow fix the monthly log files. Apparently it will create missing entries. But in this process will it place the proper highs, lows etc. into the monthly log files? If an existing reading is too low or high in the monthly log files will it recognize that and place the proper reading in them? Also some of my log files are in 10 minute intervals and some in 5 minutes intervals.

I have read the wiki and searched the forum but it has not answered all of these questions for me.

I am still using Cumulus 1. But I will possibly in the future switch to CumulusMX.

Thanks

[PaulMy]

I know there is some way to use the dayfile.txt to somehow fix the monthly log files. Apparently it will create missing entries. But in this process will it place the proper highs, lows etc. into the monthly log files? If an existing reading is too low or high in the monthly log files will it recognize that and place the proper reading in them? Also some of my log files are in 10 minute intervals and some in 5 minutes intervals.

You can create "missing" for dayfile but not aware that you can for monthly log.

I also had differences in logging from 30 min initially, then 15 min and now 10 minutes. I've just used those log files as they are to populate Meteotemplate MySQL alldata table and no issues, but my monthly log files were in pretty good shape.
http://www.komokaweather.com/j-template

Enjoy,
Paul

[sperry russ]

You can create "missing" for dayfile but not aware that you can for monthly log.

After I read your post I went back to the wiki and looked at it again. So it does look like the dayfile.txt will not create or adjust the monthly logs.

I wonder if there is any other way to somehow automate the corrections?

[steve]

The monthly log files are the only place that the detailed data is held, so there is nowhere else to get data to correct those files. The only way that I can think of to automate part of the process would be to write a program to adjust/remove values which are wrong.

Identifying wrong entries is quite difficult in itself, but Cumulus does the best it can by ignoring obviously bad values, and providing the spike removal facility, which ignores a lot of the typical bad data supplied by Fine Offset stations. You presumably have bad readings which weren't catered for by either of those two processes; they are probably going to be difficult to remove other than manually.

[jachym]

Hi guys,
I looked at Sperry´s files previously too. The problem is that Meteotemplate import script can also filter out obvious nonsense, but in this case this won´t work because for example, the one Sperry sent me had a max of 91 instead of 95, which obviously is both realistic and there is absolutely no way a script can identify this. What I suggested was using the Data Check plugin in my template where you can set certain maximum rate of change per time and it will look for spikes in data.
However in cases like the one mentioned above, where the temperature is simply slightly off, Im afraid the only way would be correcting the monthly logs manually...

[sfws]

I once spent a lot of time working on monthly logs and dayfile trying to improve my data quality, and indeed I rewrote some of the Wiki entries re those Cumulus files based on my experiences. I now just accept that I have a hobby that does not have high quality data, but at least I do have around 9 years of electronic held statistics and I can use that for experimenting with different analysis.

Basically, I agree with everything Steve says, you now have to manually do the best you can.

One partly automated but basically manual approach, you might find worth trying is:
1. Make a back up copy of your dayfile and all the monthly logs, so you keep what you already have safe just in case you do an edit that goes wrong perhaps by deleting lines you want to keep!
2. Rename your live dayfile.txt and create an empty file called dayfile.txt in its place in the 'data' directory.
3. Using your Cumulus 1 edit functionality, let it "create missing day" dayfile.txt records for all your days from your monthly logs. This will take a lot of time!
4. Use a file compare program, and let it look at both your renamed dayfile and this newly populated dayfile.txt. The program should match up all the lines that are same in both files, and highlight where there are differences.
5. For each highlighted difference, go to the relevent monthly log, and delete the whole line in the log (or edit the relevent individual entry if you can guess a better value) to resolve the entry that caused the newly populated dayfile.txt to differ from the dayfile you previously corrected.
6. Delete that day (only) from the newly populated dayfile, for each highlighted difference as you deal with it. The file compare program should change from highlighting differences in particular items to indicate a whole line is now different.
7. When you finish dealing with all the differences, let Cumulus "create missing days" for the days you have processed.
8. Repeat steps 4 to 7 as required to be sure you have made all necessary corrections to your monthly logs.
9. Finally rename the dayfile.txt you have been working with, so you have a record of what you did. Now you can rename your original dayfile.txt back and continue working with that old file you are happy with.

[sperry russ]

Okay. Thanks for all the input everybody. I may try what sfws suggested.