Missing Data

[Chris Attkins]

At irregular intervals, my WH 1080 from Maplin appears to stop receiving data from the outdoor sensors for periods ranging from a few minutes to several hours. In the absence of fresh data, Cumulus seems to invent startling readings, which it then holds until the arrival of fresh data. Is anyone else experiencing this anomaly - and can anyone suggest a cause please?
The batteries in the transmitter unit are fresh and the system has only been up for a month, so it hardly seems likely to be corroded battery terminals.

[MarkB]

data from the outdoor sensors

Your 'trends' page shows your inside sensors 'stop' at the same time.
I think this probably indicates a communication problems between your base unit and your PC...

[Chris Attkins]

Well spotted Mark! This would indicate as you suggest, but either way, the weather station panel has dotted lines instead of data during the blank period recorded by Cumulus. Is it possible that a PC problem is causing the wall panel to lock up? I suppose I could try unplugging the computer for a day or two...

Cheers,

Chris

[MarkB]

panel has dotted lines

I've never seen that on mine (same model, just rebadged). Possible faulty base unit? Try unplugging the USB next time you see it - it might clear it.

Cheers
Mark

[steve]

It looks like there was a glitch in some of the data, which Cumulus duly recorded, immediately followed by the station going offline (presumably both - and the dashed lines on the console - are symptoms of the same problem with the console), so Cumulus continued logging the same data. A work around for the latter might be the version of Cumulus which restarts when it detects 'unplugged' (see link elsewhere in the forum, but I will also be making it the beta version soon), assuming the 'unplugged' condition does get signalled in this situation.

Steve

[steve]

panel has dotted lines

I've never seen that on mine (same model, just rebadged). Possible faulty base unit? Try unplugging the USB next time you see it - it might clear it.

Cheers
Mark

On both my Oregon and Davis stations, the dotted lines are an indication that contact with the remote sensors has been lost.

Steve

[Chris Attkins]

Thanks Steve. I will download and install your "re-starting" version and see if that solves the issue.

[steve]

Just for clarification - that version won't really solve anything - it looks like you had some sort of hardware problem. But what it might do is prevent the situation where Cumulus continues to log the same value repeatedly when the station stops sending new data. The theory being that the 'unplugged' status might be signalled, Cumulus will restart and be unable to reconnect, so will stop logging data. Probably slightly better than the situation where it logs the last data it received over and over again.

Steve

[Chris Attkins]

Thanks for clarifying that Steve. I've got the new version running now, so next time there's a glitch we'll see what happens.

I've just exported all the data from the logger and intend to compare this with that exported from Cumulus. A quick glance suggests that Cumulus 'invented' the extreme data, such as six degrees of frost last night (never on Arran!), for the lowest recorded temperature seems to be +6.3 just before the station went dumb.

We do sometimes get very powerful bursts of radar from ships moored in Brodick Bay (strong enough to produce audible pulsing sounds from a television loudspeaker that's switched off!) so I am going to keep an eye on shipping movements to see whether this correlates with my data blanking out.

If I understand correctly from snippets elsewhere on this forum, instead of manually editing errant data on various Cumulus files, it's possible to force the program to reload all data from the logger, which simply has --- in place of missing data. Will Cumulus cope with such data, or should I enter approximate mean values before reloading?

[Chris Attkins]

Ooer! I have just realised that Cumulus will read from the logger, so there would be no point in editing the exported data!

So my question ought to be: Will Cumulus correctly interpret --- as missing data and connect historic graphs accordingly? Or should I get busy editing the Cumulus data files manually?

[steve]

my question ought to be: Will Cumulus correctly interpret --- as missing data and connect historic graphs accordingly? Or should I get busy editing the Cumulus data files manually?

I don't know what the station does so I can't really say. There's no such thing as '---' as far as the logger is concerned, it holds binary data, so I don't know what actual values will have been logged.

Steve

[Chris Attkins]

OK, what I really wanted to know was whether Cumulus could cope with whatever 'null' data was represented by these '---' markers.
Having backed up the original data, I edited the .ini file as recommended to force Cumulus to reload all data directly from the logger. This resulted in even more peculiar results! (see current website trends online).

Here is a short section of data exported from the logger by EasyWeather:

1409 04-04-2009 22:32 30 42 21.3 77 6.5 1013.9 0.0 0.7 S 1016.0 2.8 6.5 0.0 22.5 52.8 275.1 275.1 0 1
1410 04-04-2009 23:02 30 41 21.3 78 6.7 1013.8 0.0 0.7 NW 1015.9 3.1 6.7 0.0 22.5 52.8 275.1 275.1 0 1
1411 04-04-2009 23:32 30 44 21.4 79 6.3 1013.6 0.0 0.7 NW 1015.7 2.9 6.3 0.0 22.5 52.8 275.1 275.1 0 1
1412 05-04-2009 00:02 30 44 21.5 --- --- 1013.7 --- --- --- 1015.8 --- --- 9.6 32.1 62.4 284.7 284.7 --- ---
1413 05-04-2009 00:32 30 44 21.5 --- --- 1013.5 --- --- --- 1015.6 --- --- 9.6 31.8 62.4 284.7 284.7 --- ---
1414 05-04-2009 01:02 30 43 21.4 --- --- 1013.8 --- --- --- 1015.9 --- --- 0.0 31.8 62.4 284.7 284.7 --- ---
1415 05-04-2009 01:32 30 44 21.3 --- --- 1013.9 --- --- --- 1016.0 --- --- 0.0 31.8 62.4 284.7 284.7 --- ---
1416 05-04-2009 02:02 30 43 21.2 --- --- 1014.1 --- --- --- 1016.2 --- --- 0.0 22.2 62.4 284.4 284.7 --- ---
1417 05-04-2009 02:32 30 44 21.1 --- --- 1014.2 --- --- --- 1016.3 --- --- 0.0 21.9 62.4 284.4 284.7 --- ---
1418 05-04-2009 03:02 30 44 21.0 --- --- 1014.1 --- --- --- 1016.2 --- --- 0.0 21.9 62.4 284.4 284.7 --- ---
1419 05-04-2009 03:32 30 44 20.9 --- --- 1014.1 --- --- --- 1016.2 --- --- 0.0 21.9 62.4 284.4 284.7 --- ---
1420 05-04-2009 04:02 30 44 20.7 --- --- 1014.2 --- --- --- 1016.3 --- --- 0.0 21.6 62.4 284.4 284.7 --- ---
1421 05-04-2009 04:32 30 44 20.6 --- --- 1014.1 --- --- --- 1016.2 --- --- 0.0 21.3 62.4 284.4 284.7 --- ---
1422 05-04-2009 05:02 30 43 20.5 --- --- 1013.9 --- --- --- 1016.0 --- --- 0.0 20.7 62.4 284.4 284.7 --- ---
1423 05-04-2009 05:32 30 44 20.3 --- --- 1013.7 1.0 13.6 S 1015.8 --- --- 0.0 20.1 62.4 284.4 284.7 1 6
1424 05-04-2009 06:02 30 43 20.2 --- --- 1013.5 --- --- --- 1015.6 --- --- 0.0 19.8 62.4 284.4 284.7 --- ---
1425 05-04-2009 06:32 30 44 20.0 --- --- 1013.8 --- --- --- 1015.9 --- --- 0.0 19.5 62.4 284.4 284.7 --- ---
1426 05-04-2009 07:02 30 44 19.9 --- --- 1013.5 --- --- --- 1015.6 --- --- 0.0 19.5 62.4 284.4 284.7 --- ---
1427 05-04-2009 07:32 30 43 19.7 85 6.0 1013.6 1.0 2.0 NW 1015.7 3.7 6.0 0.0 19.5 62.4 284.4 284.7 1 2
1428 05-04-2009 08:02 30 44 19.6 --- --- 1013.7 --- --- --- 1015.8 --- --- 0.0 19.2 62.4 284.4 284.7 --- ---
1429 05-04-2009 08:32 30 44 19.7 --- --- 1013.8 --- --- --- 1015.9 --- --- 0.0 19.2 62.4 284.4 284.7 --- ---
1430 05-04-2009 09:02 30 44 19.9 77 7.8 1013.9 1.0 2.4 NW 1016.0 4.0 7.8 0.0 19.2 62.4 284.4 284.7 1 2
1431 05-04-2009 09:32 30 47 20.0 73 9.5 1014.2 1.0 3.4 SW 1016.3 4.9 9.5 0.0 19.2 62.4 284.4 284.7 1 3

You will notice that the interior temperature (5th column) was recorded correctly, but is evidently not loading into Cumulus.

Steve, you seem to suggest that these '---' data are an unknown quantity, rather than nulls. Is my station the only one dropping data from time to time? If so I guess it's faulty and I will have to return it to Maplin.

I would really appreciate hearing from anyone else with a similar problem.

Cheers,

Chris

[steve]

I think there are two issues here as far as Cumulus is concerned. There's a status bit with each set of data which indicates when contact has been lost with the remote sensors. I've just realised that it's inconsistent in the way it handles the status bit while reading from the logger compared to 'real time'. When reading from the logger it ignores the status bit. In 'real time' it discards all of the data if the status bit is set.

Clearly, it shouldn't ignore the status bit when using logger data. I'll fix that.

But should it discard all of the data when the status bit is set, or just the external data? Probably just then external data, but I think that I did this for safety when we were going through the long and difficult process of trying to eliminate the bad data that the stations sends form time to time. Note that Cumulus has no concept of 'null' data, so ignoring any data means that the last valid reading continues to be used and logged.

But as a separate issue, it does look as though you have a reception problem of some sort.

Steve

[Chris Attkins]

Let no one doubt your dedication to this project Steve! I was amazed to receive your latest post late on a Sunday evening and slept on the various points you made. As usual, you pointed towards solutions, for which I am once again grateful.

When reading from the logger it ignores the status bit. In 'real time' it discards all of the data if the status bit is set.
Clearly, it shouldn't ignore the status bit when using logger data. I'll fix that.

I realised that my first attempt to reload data from the logger merely added to the existing data held by Cumulus. Today, having deleted the Cumulus datafiles, it imported the logger data perfectly. And it did ignore any data blocks with missing elements, so I think it is responding correctly to the status bit.

But should it discard all of the data when the status bit is set, or just the external data?

I think you are right to ignore all data when some if it is dodgy, for the very reasons you state. Having now reloaded the data, Cumulus has very neatly joined the ends of the graphs to achieve a smooth transition between available data - exactly as I had hoped.

But as a separate issue, it does look as though you have a reception problem of some sort.

Spot on Steve! At 4am I woke with the realisation that the intermittent data loss from the outdoor sensors began after attaching the receiver panel to the wall. This only involved moving it two feet from where it had been sitting on a table, but evidently the signal is quite susceptible to position, as indicated by temporarily locating the receiver in various places this morning. So a little experimentation should resolve this issue too.

Thanks again,

Chris

[steve]

And it did ignore any data blocks with missing elements, so I think it is responding correctly to the status bit.

Ah, that's good - I must not have looked at the code carefully enough

Steve