C2 Today High Pressure time incorrect

RayProudfoot · **Posted:** Sun Aug 22, 2010 2:25 pm

Steve,

C2 'Today' page reports high pressure of 1014.8 at 23:55. We haven't got there yet.

C1 reports today's high pressure was 1014.8 at 00:00. Other 'high' readings for today look fine. I imagine it's to do with all time stamps in UT.

Attachment:

C2_PressureTime.jpg

steve · **Posted:** Sun Aug 22, 2010 2:49 pm

From the graph on your web site, it looks like your recent high pressure was indeed just before midnight, so it's correct in that respect, if you see what I mean; I agree that it's probably to do with UTC and deciding what 'today' actually is. A day at the moment is 2300 to 2300 UTC, I may be doing 2300 to 2300 local instead. I assume your C2 data for around midnight came from the VP2 logger - ie. C2 wasn't actually running at the time?

Can you have a look in the database and see what timestamp in UTC that 1014.8 value corresponds to?

Edit: I've looked at the code and I think it's OK, i.e. I'm taking the right 24-hour period as far as the timestamps in the main table are concerned. But this new 'offset' business is screwing it up because I don't take that into account when extracting today's entries. So your high pressure timestamp probably is in the 'today' range, but subtracting the offset takes it into yesterday. This is somewhat tricky, it's looking like another rewrite

To confirm that this is the problem, I could do with seeing your 'offset' table values, but that's not easy for you to look at, so maybe you could email me your database to steve@sandaysoft.com? If you zip it the size should come down quite a bit.

RayProudfoot · **Posted:** Sun Aug 22, 2010 3:46 pm

steve wrote:

From the graph on your web site, it looks like your recent high pressure was indeed just before midnight, so it's correct in that respect, if you see what I mean;

Indeed.

steve wrote:

I assume your C2 data for around midnight came from the VP2 logger - ie. C2 wasn't actually running at the time?

C2 Viewer wasn't running but CumulusService was. It's installed on a netbook that runs 24/7.

steve wrote:

Can you have a look in the database and see what timestamp in UTC that 1014.8 value corresponds to?

I opened the database using CumulusControl but it is very slow navigating to records. The Atom2 processor is probably not up to it. I've sent it privately so hopefully you can see the entry.

steve wrote:

Edit: I've looked at the code and I think it's OK, i.e. I'm taking the right 24-hour period as far as the timestamps in the main table are concerned. But this new 'offset' business is screwing it up because I don't take that into account when extracting today's entries. So your high pressure timestamp probably is in the 'today' range, but subtracting the offset takes it into yesterday. This is somewhat tricky, it's looking like another rewrite

Oh dear. Sorry to hear that.

steve wrote:

To confirm that this is the problem, I could do with seeing your 'offset' table values, but that's not easy for you to look at, so maybe you could email me your database to steve@sandaysoft.com? If you zip it the size should come down quite a bit.

Sent privately.

steve · **Posted:** Sun Aug 22, 2010 4:40 pm

RayProudfoot wrote:

Sent privately.

Thanks - it's as I thought. The entry with the high value is timestamped midnight BST, but the offset is 5, which says it actually occurred 5 minutes previously, which makes sense as the pressure was falling.

This may not be the disaster I thought; I think I just need to take the logging interval into account when working out what is today/yesterday/this month etc, as all of the highs and lows for a particular timestamp occurred up to N minutes before the timestamp. But this all getting rather complicated. Perhaps this offset thing wasn't such a good idea after all, and I should have just stuck the actual timestamps in the main table. But if accessing the table is already slow...

RayProudfoot · **Posted:** Sun Aug 22, 2010 4:58 pm

Thanks for analysing it Steve. Have a sleep on it and maybe tomorrow you'll have a solution. I certainly can't think of one right now.

But complicated is better than a disaster.

As for database access wait until you have some more feedback on this. I would not expect to have to edit any data except in very rare circumstances. The original C2 database wasn't too bad to access so it's strange this one seems slower.

Here's the C1 data from Aug10Log.txt for the period:-

21/08/10,23:35,13.7,93,12.6,2.0,5.0,186,0.00,0.06,1014.70,16.01,25.2,58,3.0,13.7,13.7,0.0,0
21/08/10,23:40,13.6,92,12.3,2.0,5.0,187,0.00,0.06,1014.73,16.01,25.2,58,1.0,13.6,13.6,0.0,0
21/08/10,23:45,13.5,92,12.2,1.0,3.0,197,0.00,0.06,1014.73,16.01,25.2,58,0.0,13.5,13.5,0.0,0
21/08/10,23:50,13.4,92,12.1,1.0,4.0,187,0.00,0.06,1014.84,16.01,25.2,58,3.0,13.4,13.4,0.0,0
21/08/10,23:55,13.3,92,12.0,1.0,4.0,165,0.00,0.06,1014.77,16.01,25.2,58,2.0,13.3,13.3,0.0,0
22/08/10,00:00,13.2,93,12.1,1.0,3.0,156,0.00,0.06,1014.73,16.01,25.2,58,0.0,13.2,13.2,0.0,0
22/08/10,00:05,13.2,93,12.1,2.0,6.0,149,0.00,0.00,1014.67,16.01,25.2,58,3.0,13.2,13.2,0.0,0

steve · **Posted:** Sun Aug 22, 2010 5:19 pm

Quote:

Have a sleep on it and maybe tomorrow you'll have a solution.

Yes, I think a bit of a break is needed and see what comes to mind. At the moment I'm ready to give up on the whole thing, so now is not the time to be making any decisions

RayProudfoot wrote:

But complicated is better than a disaster.

The problem is that complicated means that I am likely to introduce bugs... And Cumulus 2 was supposed to be the fresh start that was more easily enhanced.

Quote:

The original C2 database wasn't too bad to access so it's strange this one seems slower.

Different grid component - this one is straight Windows Forms, the old one was WPF. Also, I think the old one only displayed a month at a time? I find this new one takes a while to open, but then it's quite quick to move around and edit. Not the same for you?

RayProudfoot · **Posted:** Sun Aug 22, 2010 6:11 pm

steve wrote:

Quote:

Have a sleep on it and maybe tomorrow you'll have a solution.

Yes, I think a bit of a break is needed and see what comes to mind. At the moment I'm ready to give up on the whole thing, so now is not the time to be making any decisions

I'm sorry to hear you feel like that after so much hard work. A well -needed break is essential. I'll not post any more bug reports for now.

steve wrote:

The problem is that complicated means that I am likely to introduce bugs... And Cumulus 2 was supposed to be the fresh start that was more easily enhanced.

It perhaps shows that C1 was better than you thought. To me, having C1 data and C2 looks would be the perfect combination. But if you're trying to extract all sorts of data from the files perhaps C1 is not the way to go.

steve wrote:

Different grid component - this one is straight Windows Forms, the old one was WPF. Also, I think the old one only displayed a month at a time? I find this new one takes a while to open, but then it's quite quick to move around and edit. Not the same for you?

After stopping the Service and pressing Edit it takes 31 secs for the database records to be visible. I can then press CTRL+END to go to the latest record quicky. I haven't tried editing a record. The main problem is the column titles which are not wide enough to read. It's only a 10" screen and not ideal of this kind of thing. It's not possible to edit the database from another PC with a larger screen unfortunately.

Is it possible to create a database from C1 data and for it to be updated every 24 hours? That way you have the ease of editing plain text files to remove inaccuracies, accurate time stamps for records and a database for any record you want to analyse. Something to chew over.

steve · **Posted:** Sun Aug 22, 2010 6:29 pm

RayProudfoot wrote:

The main problem is the column titles which are not wide enough to read.

You do know that you can expand the headings by double-clicking, like in Excel?

Quote:

It's only a 10" screen and not ideal of this kind of thing

I'm not sure there's any solution to that, however the data is stored. It's the same issue you get with any spreadsheet with a large number of columns. Even on my 22" widescreen there's still a lot of moving left and right required. I suppose some sort of mechanism for showing/ hiding some of the columns...

Quote:

Is it possible to create a database from C1 data and for it to be updated every 24 hours? That way you have the ease of editing plain text files to remove inaccuracies, accurate time stamps for records and a database for any record you want to analyse. Something to chew over.

I'm not sure I follow you - can you explain a bit more what you mean? Do you mean maintain the data in text files (like C1), then save it to the database every 24 hours?

RayProudfoot · **Posted:** Sun Aug 22, 2010 6:52 pm

steve wrote:

You do know that you can expand the headings by double-clicking, like in Excel?

Yes. It's just a little fiddly stood up and having to use a small touchpad.

steve wrote:

I'm not sure there's any solution to that, however the data is stored. It's the same issue you get with any spreadsheet with a large number of columns. Even on my 22" widescreen there's still a lot of moving left and right required. I suppose some sort of mechanism for showing/ hiding some of the columns...

This is the downside of running C2 on a netbook. A filtering option just to show the columns you're interested in would help. But that's for another day.

steve wrote:

I'm not sure I follow you - can you explain a bit more what you mean? Do you mean maintain the data in text files (like C1), then save it to the database every 24 hours?

Got it in one. If older data in the text files had been edited then that needs thinking through. Maybe use the modified date of the files to check if any had been changed.

It's just something that I thought of and wondered if it might be feasible as it might overcome these time offsets problem. But being able to view C2 interface from another PC is really good so I hope that will remain. Perhaps C1 just needs a little tweaking Steve?

steve · **Posted:** Sun Aug 22, 2010 8:12 pm

RayProudfoot wrote:

Got it in one. If older data in the text files had been edited then that needs thinking through. Maybe use the modified date of the files to check if any had been changed.

I think I would rather have all of the data go straight into the database. But your suggestion does have merit, so I'm not dismissing it. But I've had a thought:

The entries in the 'standard data' table in C2 are similar to the Weatherlink .wlk file entries (by design). In addition, WL has "daily summary" entries, containing the days highs and lows, with their timestamps. I want to get away from the 'daily summary' concept to make it more flexible, and the current C2 database design basically take the summary down to the 'logger interval' level. Hence the large amount of data to be stored. So perhaps, as a compromise, C2 has a table of 'hourly summaries' with an entry for each hour for the highs and lows and timestamps for each. Each entry would have a timestamp for the start of the hour, which C2 can use in an SQL query, and each high and low value has a 'minute' value associated with it, so that the to-the-minute timestamp can be constructed by adding that to the entry timestamp. It's not as flexible as the current design, but it allows most of the queries that someone might want to do.

This doesn't resolve the issue of the large size of the standard table, and its large number of columns, but I'm sure a bit of enhancement to the editor can sort that out, even if it means resorting to the month-at-a-time approach.

I'll drink some more beer and see if anything else comes to mind...

RayProudfoot · **Posted:** Sun Aug 22, 2010 8:39 pm

steve wrote:

I think I would rather have all of the data go straight into the database. But your suggestion does have merit, so I'm not dismissing it. But I've had a thought:

The entries in the 'standard data' table in C2 are similar to the Weatherlink .wlk file entries (by design). In addition, WL has "daily summary" entries, containing the days highs and lows, with their timestamps.I want to get away from the 'daily summary' concept to make it more flexible, and the current C2 database design basically take the summary down to the 'logger interval' level. Hence the large amount of data to be stored.

C1 has these daily summary entries in dayfile.txt and there isn't an equivalent in the current C2 - hence the problems you've been having with these offsets. Although it means you're storing data in more than one table there is a lot of merit in having the contents of dayfile.txt in a table. Paul (synewave) recently discussed his web pages where he makes extensive use of the dayfile.txt. This is what he's achieved.

steve wrote:

So perhaps, as a compromise, C2 has a table of 'hourly summaries' with an entry for each hour for the highs and lows and timestamps for each. Each entry would have a timestamp for the start of the hour, which C2 can use in an SQL query, and each high and low value has a 'minute' value associated with it, so that the to-the-minute timestamp can be constructed by adding that to the entry timestamp. It's not as flexible as the current design, but it allows most of the queries that someone might want to do.

Sounds good and would have 1/12 of the current number of entries.

steve wrote:

This doesn't resolve the issue of the large size of the standard table, and its large number of columns, but I'm sure a bit of enhancement to the editor can sort that out, even if it means resorting to the month-at-a-time approach.

I'll drink some more beer and see if anything else comes to mind...

It's not the number of columns as such but the sheer number of records especially when most of them are not really required. Do you need to create a 24 hour graph of weather data from 6 months ago?

If you have an hourly set of records maybe these could eventually be imported into a daily set of records replicating the current dayfile.txt?

And the All-Time records? Given the small number you hold would a separate table make sense for those instead of calculating the values on-the-fly every time? Enjoy your beer.

steve · **Posted:** Sun Aug 22, 2010 9:20 pm

RayProudfoot wrote:

It's not the number of columns as such but the sheer number of records especially when most of them are not really required. Do you need to create a 24 hour graph of weather data from 6 months ago?

If you have an hourly set of records maybe these could eventually be imported into a daily set of records replicating the current dayfile.txt?

But I don't like throwing data away

I suppose a 'compress' option might be a possibility at some point...

Quote:

And the All-Time records? Given the small number you hold would a separate table make sense for those instead of calculating the values on-the-fly every time?

I'm trying to resist doing this, but I may have to concede eventually :lol:

The benefit of not storing them in a fixed table means that all of the data is a candidate for an all-time record, so adding missing ones (like the ones you pointed out earlier) is just a case of finding space for them on a display somewhere. But for performance reasons it may become necessary to 'cache' them in a separate table at some point.

steve · **Posted:** Mon Aug 23, 2010 5:22 pm

Ray, how long does it take for all of the all-time records to appear for you after you open the window?

RayProudfoot · **Posted:** Mon Aug 23, 2010 5:35 pm

steve wrote:

Ray, how long does it take for all of the all-time records to appear for you after you open the window?

From my laptop across a wireless network between 25-30 secs. It's the same on the netbook running the service. The last 2 records take the longest.

steve · **Posted:** Mon Aug 23, 2010 7:23 pm

RayProudfoot wrote:

From my laptop across a wireless network between 25-30 secs. It's the same on the netbook running the service. The last 2 records take the longest.

On my P4 it takes about a minute to do all but the last two, and then another 45 seconds to do those :-(

This design clearly doesn't work. I think it's time to call it a day, I don't think I can do another rewrite.

	It is currently Tue May 21, 2013 7:00 am
Please click here before posting. Help me to help you!
Useful Links: Cumulus FAQ • Enhancement requests • Wiki (documentation)
Please put your approximate location into your profile
Add your web site to the Cumulus user map
Vantage Pro2 users with firmware 3.00 should upgrade to fw 3.12 and Cumulus 1.9.4
Latest Cumulus release v1.9.3 (build 1059) - Feb 09 2013 Latest Cumulus beta v1.9.4 build 1076 - May 12 2013 --> Latest Cumulus Realtime v1.0.0.5 - Sep 05 2009

Sandaysoft

C2 Today High Pressure time incorrect

Who is online