BST to GMT - am I missing something?

[Nottub]

Just a question regarding the clock change, both BST to GMT and GMT to BST.

When the clocks change from BST to GMT I end up with two 02:00 entries, two 02:15 entries, two 02:30 entries and two 02:45 entries. These additional entries throw the 'Recent Charts' out with a defined change in the measurements.

I have deleted these 'extra' log files today so i get a single entry for each 15 minute period (and a more progressive change in the source values).

I think in the GMT to BST change I end up with a gap in the records.

Have I got my settings right?

Thanks
Martyn

[freddie]

Sounds correct - duplicate records when the clocks go back, and missing records when the clocks go forward.

As you have deleted your dupes you now have a gap in your readings but a continuous data series. Whereas when the clocks go back you have no gap in your readings but a discontinuous data series.

It is much cleaner to stay on one time zone all year. Ideally your presentation client will handle the changeover, presenting the data as a continuous series with the graph axis labelling catering for the timeline discontinuities. Not possible at the moment with the default Cumulus web pages, but definitely do-able if your web site reads data from a database that MX populates, rather than the web site reading data files produced by MX.

[mcrossley]

Even if CMX stored the data in UTC, when displayed in local time the graph will still jump back an hour or have an hour's gap. I don't see how you can display data in local time any other way?

[freddie]

Even if CMX stored the data in UTC, when displayed in local time the graph will still jump back an hour or have an hour's gap. I don't see how you can display data in local time any other way?

By making your graph axis label display local time. So your data is continuous, but you either don't have an 0200 label on the axis (when clocks go forward) or you have duplicate 0100 labels on the axis (clocks going back). So it's the labelling of the timeline on the graph that varies with the clock change. We do this at work. I'll post a screenshot later (on my phone at the moment).

[Mapantz]

I think there's a UTC flag that can be set in highcharts.. whether that solves it, I don't know?!

[freddie]

I'll post a screenshot later (on my phone at the moment).

See https://www.hosiene.co.uk/capture.mp4
Watch the time values displayed above the radar animation (it's a manual animation, so a bit jerky). Although not a graph axis timeline (I don't have access to that one from home) it is the same principle - local time values used on a timeline.

[mcrossley]

You could configure highcharts to use a data axis rather than a timeline, bit then if you have gaps in the data it will still show as a contiguous series which looks a bit odd.

[sfws]

The issue is the way Cumulus is designed, as Freddie has hinted in all his posts.

The main issue is that data stored by Cumulus MX, in recent history, CSV files, and into database tables, all use clock times, and those overwrite or have gaps at any DST changeovers. Operating your Cumulus computer at a fixed timezone does avoid such discontinuities in the data when stored, but a better solution, working with any timezones, would be to store UNIX times in CSV files as an additional field*, and in database tables to replace primary keys (and time-stamp fields) storing local times^.

A secondary issue is the use by Cumulus of datetime as x-axis type for charts. The json formatted data supplied to the charts is using "milliseconds since midnight 1 Jan 1970 UTC" for the x values, but in Cumulus "charts.js" these x values get converted into local time for viewing charts on any device. That conversion potentially reintroduces discontinuities. As Freddie says "presentation" labelling overcomes any DST complications, and I believe that is why highcharts offers "TimezoneOffsetCallbackFunction(timestamp)" and to answer Mark's recent post I also believe that could be made to cope with data gaps better than treating x axis as simple data.

* There was an announcement in this forum that at MX 3.9.0 (b3095) a UNIX timestamp field would be added to the "monthly" CSV logs. Further reference to that has appeared periodically in posts since then. Perhaps this could actually be implemented at next release. In meantime people could store data that way using the new custom log functionality, but that does not solve the need to be skilled to write code for reading from that custom log and making charts!

^ Freddie mentions use of database as source for plotting. The internal SQL for "Monthly log file upload" could use UNIX time for primary key instead of local date/time, so at least that database table would not have any discontinuities. Theoretically in meantime people could use a custom minutes SQL command that used the UNIX tag as primary key. If the custom minutes was set to 1 minute interval it would effectively replace use of recent history data, and could produce similar looking plots to those provided with MX. As Freddie hints, anybody capable of rewriting the chart plotting code could use such a revised table and avoid discontinuity. This is an imposition that would be possible for only the most skilled developers! There is also a need to control size of this custom table, perhaps only retaining 7 days?

[Nottub]

Thanks for all your considered responses. It seems like I'm not missing anything after all, which I'm happy with.

I'll probably still tweak my records to remove the duplicated times. It's easy for me to do.


Thanks as ever for a helpful forum of knowledgable people.


Martyn

[mcrossley]

I had a play with CMX4 which does store all data in UTC, eventually I persuaded HighCharts to plot like this...
.

Screenshot 2022-10-31 115252.png

[freddie]

I had a play with CMX4 which does store all data in UTC, eventually I persuaded HighCharts to plot like this...
.
Screenshot 2022-10-31 115252.png

That's what I had in mind, along with TZ labels.