Ecowitt Data Access API

[RussellNZ]

Recently setup an Aercus Weathermaster (I believe is essentially an HP2551?) and could only get it to communicate with CMX using the HTTP (Ecowitt) method. No worries. Then subsequently created account at Ecowitt.net and generated application and api keys so that CMX could recover missing data from the net for situations where CMX service was down for a period. All good.
However, I did some testing the other day by stopping CMX service for about 15-20 minutes then restarted. It grabbed data ok from Ecowitt.net but appears to have grabbed from the wrong timestamps. It is effectively out by 5mins i.e. the first row of data expected was for 11:45 since my last good CMX log entry was 11:40. However the first data row grabbed was 11:40. It grabbed 3 rows as expected and slotted them into CMX monthly log file for times 11:45, 11:50 & 11:55 which are the correct times wanted however the data values against them appear to be for 11:40, 11:45 & 11:50.
Also, some of the updates to monthly log file don't correlate with actual data downloaded from Ecowitt nor what is showing in MXDiags\yyyymmdd-hhmmss.txt.
I've attached the various files which hopefully illustrate the event. I have perhaps missed something in my assumptions about how this piece should work but I understand the Ecowitt Data Access API coding is relatively new so maybe this info will be helpful in its evolution.

[SamiS]

Intrestingly the version history shows, that Mark has implemented a five minute offset to ecowitt catch-up, and I think it was because of the api was returning wrong data/timestamps. I wonder if Ecowitt has later changed / repaired their end of the api.

viewtopic.php?p=162294#p162294

[mcrossley]

I have no reason to think it has changed, but I can check. Ecowitt timestamps seem to refer to the start of the log interval, all other software I know of uses the end of the log interval.

[SamiS]

I checked my own logs and did not see similar behaviour than RussellNZ. However there is another inconsistency regarding logging interval. For historical reasons I have 10min logging interval. But for the timeframe of catch-up from ecowitt cloud, my logfiles contain data with 5 minute interval and the 10 minute logging resumes when CMX is running normally. This does not seem to cause any notable issues though.

[mcrossley]

Hmm, yes it will create historic records at the 5-minute log interval that Ecowitt use.

It would require quite a bit of work for CMX to combine these into 10-minute or larger records - considering peaks and averages etc.

It will not cause CMX any issues, only for users if they expect all log entries to be at their specified log interval. Of course, it works the other way too, people with 1-minute log intervals will get catch-up data at longer 5-minute intervals.

[HansR]

Interval inconsistency has always been part of CMX though recently (somewhere the last two years), when Mark improved startup/shutdown significantly, it has much improved and now has become rare. It is kind of regression that with the Ecowitt catch up is has become unavoidable again. This is the reason I changed my logging interval to 5 minutes (logfilesize was an argument too btw). But also when a station has no catchup you will get irregular intervals for some reason or another so we have to deal with it.

It does not create issues when just displaying, but it will when you start using the logged data for calculations.

When using the original logging for data analysis (or aggregates) one needs to reflect on this and account for it in the algorithms. I borrowed from discrete integration techniques to use moving interval size: e.g. to calculate solar energy using each entry in the logfiles I take from start time (sun up) to end time (sun set) the average solar radiation for each interval and multiply that with the interval width. Adding all those values no matter from what actual interval they are I get the total solar radiation or energy per day simply by calculating for each interval and sum those over the total sun up time.

Assuming a constant interval over a whole day, to me, has proven to be an error.

[SamiS]

Hmm, yes it will create historic records at the 5-minute log interval that Ecowitt use.

It would require quite a bit of work for CMX to combine these into 10-minute or larger records - considering peaks and averages etc.

It will not cause CMX any issues, only for users if they expect all log entries to be at their specified log interval. Of course, it works the other way too, people with 1-minute log intervals will get catch-up data at longer 5-minute intervals.

Ok, thank you, this and Hans’s post make things much more clear to me again. So basically for me and most of people who are not making any own calculations from the data, the log interval inconsistencies during catch-up do not cause a real problem. I had not noted that this behaviour existed, since I was running a good old old wh1080 from 2010 until this August.

The thread’s original question about the ecowitt.net api catch-up time offset however still remains unanswered.

[mcrossley]

The thread’s original question about the ecowitt.net api catch-up time offset however still remains unanswered.

See...

I have no reason to think it has changed, but I can check. Ecowitt timestamps seem to refer to the start of the log interval, all other software I know of uses the end of the log interval.

[SamiS]

I have no reason to think it has changed, but I can check. Ecowitt timestamps seem to refer to the start of the log interval, all other software I know of uses the end of the log interval.

Sorry for being inaccurate. I was referring to the part ”but I can check” that suggested me to wait for you to confirm it being one way or the other, since my and RussellNZ’s observations were different.

[RussellNZ]

Hi team, yes, I'm still really interested in whether there is something awry regarding whether values being extracted from Ecowitt are for the correct matching times. Specifically in my case, where CMX was trying to fill data for a time slot, say 11:15 for example, the values taken from Ecowitt were from 11:20. And for next slot, 11:20, it was taking the Ecowitt value from 11:25 and so on. Essentially getting the values from the slot immediately prior to the one actually wanted.
I have no issues with any inconsistency of logging time interval between CMX and Ecowitt. I recently changed CMX log interval to 10min solely to reduce the amount of data! I also take all the CMX data out and into Excel for data analysis for trends and other types of min/max calcs not available already in CMX and whilst shifting time intervals is a tad awkward it's not a major problem to overcome.
Thanks heaps though for all the interest.

[mcrossley]

Afaiks the 5 minute offset still applies.

I just looked at my data on ecowitt.net for today, and at that time...

My max gust was at 01:03, it is in the logged data for the period 01:00, not 01:05 as you would expect.
Same for my max solar today which was at 09:13, and is in the logged data as 09:10, not 09:15 as you would expect.

Also, the last available data is never for the end of the last 5 minute period, but the period before that.
I.e. at 09:34 the last archive data available is 09:25, not 09:30 as you might expect.

[RussellNZ]

Ah! I see now
I should have thought to check against the data directly on my console device and I might have then seen that the resulting log file data was in fact correctly aligned with reality (not the time-shifted reality of Ecowitt! )
Thanks Mark for all your great work.