Welcome to the Cumulus Support forum.
Latest Cumulus MX V4 release 4.0.1 (build 4023) - 16 May 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V4 release 4.0.1 (build 4023) - 16 May 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
BST to GMT - am I missing something?
Moderator: mcrossley
-
- Posts: 202
- Joined: Fri 04 Dec 2020 4:35 pm
- Weather Station: Davis VP2 (Cabled)
- Operating System: RPi 4 (Bookworm)
- Contact:
BST to GMT - am I missing something?
Just a question regarding the clock change, both BST to GMT and GMT to BST.
When the clocks change from BST to GMT I end up with two 02:00 entries, two 02:15 entries, two 02:30 entries and two 02:45 entries. These additional entries throw the 'Recent Charts' out with a defined change in the measurements.
I have deleted these 'extra' log files today so i get a single entry for each 15 minute period (and a more progressive change in the source values).
I think in the GMT to BST change I end up with a gap in the records.
Have I got my settings right?
Thanks
Martyn
When the clocks change from BST to GMT I end up with two 02:00 entries, two 02:15 entries, two 02:30 entries and two 02:45 entries. These additional entries throw the 'Recent Charts' out with a defined change in the measurements.
I have deleted these 'extra' log files today so i get a single entry for each 15 minute period (and a more progressive change in the source values).
I think in the GMT to BST change I end up with a gap in the records.
Have I got my settings right?
Thanks
Martyn
-
- Posts: 2518
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: BST to GMT - am I missing something?
Sounds correct - duplicate records when the clocks go back, and missing records when the clocks go forward.
As you have deleted your dupes you now have a gap in your readings but a continuous data series. Whereas when the clocks go back you have no gap in your readings but a discontinuous data series.
It is much cleaner to stay on one time zone all year. Ideally your presentation client will handle the changeover, presenting the data as a continuous series with the graph axis labelling catering for the timeline discontinuities. Not possible at the moment with the default Cumulus web pages, but definitely do-able if your web site reads data from a database that MX populates, rather than the web site reading data files produced by MX.
As you have deleted your dupes you now have a gap in your readings but a continuous data series. Whereas when the clocks go back you have no gap in your readings but a discontinuous data series.
It is much cleaner to stay on one time zone all year. Ideally your presentation client will handle the changeover, presenting the data as a continuous series with the graph axis labelling catering for the timeline discontinuities. Not possible at the moment with the default Cumulus web pages, but definitely do-able if your web site reads data from a database that MX populates, rather than the web site reading data files produced by MX.
- mcrossley
- Posts: 12951
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: BST to GMT - am I missing something?
Even if CMX stored the data in UTC, when displayed in local time the graph will still jump back an hour or have an hour's gap. I don't see how you can display data in local time any other way?
-
- Posts: 2518
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: BST to GMT - am I missing something?
By making your graph axis label display local time. So your data is continuous, but you either don't have an 0200 label on the axis (when clocks go forward) or you have duplicate 0100 labels on the axis (clocks going back). So it's the labelling of the timeline on the graph that varies with the clock change. We do this at work. I'll post a screenshot later (on my phone at the moment).
-
- Posts: 1832
- Joined: Sat 17 Dec 2011 11:55 am
- Weather Station: Davis Vantage Pro2
- Operating System: Windows 11 x64
- Location: Dorset - UK
- Contact:
Re: BST to GMT - am I missing something?
I think there's a UTC flag that can be set in highcharts.. whether that solves it, I don't know?!
-
- Posts: 2518
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: BST to GMT - am I missing something?
See https://www.hosiene.co.uk/capture.mp4
Watch the time values displayed above the radar animation (it's a manual animation, so a bit jerky). Although not a graph axis timeline (I don't have access to that one from home) it is the same principle - local time values used on a timeline.
- mcrossley
- Posts: 12951
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: BST to GMT - am I missing something?
You could configure highcharts to use a data axis rather than a timeline, bit then if you have gaps in the data it will still show as a contiguous series which looks a bit odd.
-
- Posts: 1183
- Joined: Fri 27 Jul 2012 11:29 am
- Weather Station: Chas O, Maplin N96FY, N25FR
- Operating System: rPi 3B+ with Buster (full)
Re: BST to GMT - am I missing something?
The issue is the way Cumulus is designed, as Freddie has hinted in all his posts.
The main issue is that data stored by Cumulus MX, in recent history, CSV files, and into database tables, all use clock times, and those overwrite or have gaps at any DST changeovers. Operating your Cumulus computer at a fixed timezone does avoid such discontinuities in the data when stored, but a better solution, working with any timezones, would be to store UNIX times in CSV files as an additional field*, and in database tables to replace primary keys (and time-stamp fields) storing local times^.
A secondary issue is the use by Cumulus of datetime as x-axis type for charts. The json formatted data supplied to the charts is using "milliseconds since midnight 1 Jan 1970 UTC" for the x values, but in Cumulus "charts.js" these x values get converted into local time for viewing charts on any device. That conversion potentially reintroduces discontinuities. As Freddie says "presentation" labelling overcomes any DST complications, and I believe that is why highcharts offers "TimezoneOffsetCallbackFunction(timestamp)" and to answer Mark's recent post I also believe that could be made to cope with data gaps better than treating x axis as simple data.
* There was an announcement in this forum that at MX 3.9.0 (b3095) a UNIX timestamp field would be added to the "monthly" CSV logs. Further reference to that has appeared periodically in posts since then. Perhaps this could actually be implemented at next release. In meantime people could store data that way using the new custom log functionality, but that does not solve the need to be skilled to write code for reading from that custom log and making charts!
^ Freddie mentions use of database as source for plotting. The internal SQL for "Monthly log file upload" could use UNIX time for primary key instead of local date/time, so at least that database table would not have any discontinuities. Theoretically in meantime people could use a custom minutes SQL command that used the UNIX tag as primary key. If the custom minutes was set to 1 minute interval it would effectively replace use of recent history data, and could produce similar looking plots to those provided with MX. As Freddie hints, anybody capable of rewriting the chart plotting code could use such a revised table and avoid discontinuity. This is an imposition that would be possible for only the most skilled developers! There is also a need to control size of this custom table, perhaps only retaining 7 days?
The main issue is that data stored by Cumulus MX, in recent history, CSV files, and into database tables, all use clock times, and those overwrite or have gaps at any DST changeovers. Operating your Cumulus computer at a fixed timezone does avoid such discontinuities in the data when stored, but a better solution, working with any timezones, would be to store UNIX times in CSV files as an additional field*, and in database tables to replace primary keys (and time-stamp fields) storing local times^.
A secondary issue is the use by Cumulus of datetime as x-axis type for charts. The json formatted data supplied to the charts is using "milliseconds since midnight 1 Jan 1970 UTC" for the x values, but in Cumulus "charts.js" these x values get converted into local time for viewing charts on any device. That conversion potentially reintroduces discontinuities. As Freddie says "presentation" labelling overcomes any DST complications, and I believe that is why highcharts offers "TimezoneOffsetCallbackFunction(timestamp)" and to answer Mark's recent post I also believe that could be made to cope with data gaps better than treating x axis as simple data.
* There was an announcement in this forum that at MX 3.9.0 (b3095) a UNIX timestamp field would be added to the "monthly" CSV logs. Further reference to that has appeared periodically in posts since then. Perhaps this could actually be implemented at next release. In meantime people could store data that way using the new custom log functionality, but that does not solve the need to be skilled to write code for reading from that custom log and making charts!
^ Freddie mentions use of database as source for plotting. The internal SQL for "Monthly log file upload" could use UNIX time for primary key instead of local date/time, so at least that database table would not have any discontinuities. Theoretically in meantime people could use a custom minutes SQL command that used the UNIX tag as primary key. If the custom minutes was set to 1 minute interval it would effectively replace use of recent history data, and could produce similar looking plots to those provided with MX. As Freddie hints, anybody capable of rewriting the chart plotting code could use such a revised table and avoid discontinuity. This is an imposition that would be possible for only the most skilled developers! There is also a need to control size of this custom table, perhaps only retaining 7 days?
-
- Posts: 202
- Joined: Fri 04 Dec 2020 4:35 pm
- Weather Station: Davis VP2 (Cabled)
- Operating System: RPi 4 (Bookworm)
- Contact:
Re: BST to GMT - am I missing something?
Thanks for all your considered responses. It seems like I'm not missing anything after all, which I'm happy with.
I'll probably still tweak my records to remove the duplicated times. It's easy for me to do.
Thanks as ever for a helpful forum of knowledgable people.
Martyn
I'll probably still tweak my records to remove the duplicated times. It's easy for me to do.
Thanks as ever for a helpful forum of knowledgable people.
Martyn
- mcrossley
- Posts: 12951
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: BST to GMT - am I missing something?
I had a play with CMX4 which does store all data in UTC, eventually I persuaded HighCharts to plot like this...
.
.
You do not have the required permissions to view the files attached to this post.
-
- Posts: 2518
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: BST to GMT - am I missing something?
That's what I had in mind, along with TZ labels.