at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e,

[radilly]

See also viewtopic.php?f=27&t=16311

Mostly FYI
I was looking at the most recent log in /MXdiags I noticed a caught exception that shows up periodically. It doesn't seem to be causing issues on the Cumulus MX side. Weather Underground (the WU prefix I assume) looks OK, although I only see a record every 5 minutes (though CMX is configured to update every minute).

Code: Select all

2017-09-20 08:50:00.914 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
   at System.Net.WebConnection.ReadDone(IAsyncResult result)


I do note that based on the timing, this is where I'd expect to see "WU Response: OK: success". So I'd guess there some response or something from WU that's being handled as an exception.

Here's the end of the log grepped for "Exception" showing the timestamps. On 09-19 I noticed 7 in a row, every minute... That shows up in WU as a gap. (Interesting around that time, on the WU side, the entries are at different time intervals, as short as a minute instead of the usual 5).

Code: Select all

2017-09-19 23:17:00.246 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-19 23:18:00.249 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 01:38:00.689 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 03:20:01.006 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 03:54:01.104 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 04:07:01.133 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 06:34:00.527 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
2017-09-20 08:50:00.914 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
   at System.Net.WebConnection.ReadDone(IAsyncResult result)


Cheers,
Bob

[steve]

Yes - it's not a particularly helpful exception message as it doesn't actually say what the exception is!

[radilly]

[radilly]

I've been tracking various error / warning conditions from my rig, and I noticed an increase over the past few days of this error. Here's a plot by hour, i.e. exceptions / hour.

20180105_WU_Exceptions.jpg

I couldn't get the x-axis the way I wanted, but the shape is what's important, and this plot covers about 6 1/2 days of data overall. I think the jump is pretty clear, easily 3-4 X visually, likely more.

Maybe time to start spying on the http data with Weather Underground (which I have yet to dig into).

Bob

[jlmr731]

I believe WU has been having problems with rapid fire and probably its other services
Nothing new but keep your eye on it and see if it clears up

[GlueGuy]

This may have been happening to my MX for a while, but I didn't happen to notice it until today. I thought something was going south with MX, and rebooted the Raspberry pi. It's been running MX for at least a couple of years, so I have pretty good confidence in it.

Well, the reboot didn't help, so I started poking around the forum, and found this thread. Guess I will wait for it to "go away".

I will say, the symptom from the WU side is that cumulus updates stop while this is happening. I have my WU updates set to 5 minutes, and when I happened to be checking WU, it was telling me that it had not been updated for 17 minutes. Kind of caught me off guard.

[radilly]

Somewhere in my notes I did note some odd gaps in the WU Weather History Table. If I recall in those notes (which I can't find) I did seem to see a correlation between a group of these exceptions in Diags, and a gap in the WU data.

At the moment I'm seeing this CMX exception so rapidly that if dropouts in the transmission are a result, I'm surprised anything is getting through. Just to see if it changes anything, I went from 1 minute updates to every 3 minutes. So far ... not so much. Indeed, with a third as many attempts, it appears less data is getting through.

I finally started to tinker with fiddler (https://www.telerik.com/blogs/fiddler-f ... ta-is-here) but not having much success so far at getting it running. I don't run X on the Pi so I was trying to use Xming (https://sourceforge.net/projects/xming/) which may be asking too much. I just started to look around for solutions, but would welcome any tips - indeed since the tool is mentioned a number of time, including by Steve, I thought it might be worthy of a separate discussion thread?

I tried a restart of CMX, which I'm running as a service but that didn't seem to affect this. It did free up some other resources, though memory usage is still a little higher than after a reboot. A reboot will likely be the next thing to try. I've been watching some resource consumption issue which seem to ramp up after a few weeks. I'm rebooting around once a month.

Cheers,
Bob

[GlueGuy]

I've got a tail running on the CMX log file, and sometimes as much as 50% of the WU updates get lost. I've wondered sometimes why my weather station doesn't show up in the local Wundermap, and I'm suspicious that this may be the reason.

I've considered changing the WU updates to once a minute to see if that changes anything. If I still lose 50% of the updates, at least it will be > than once every 5 minutes.

Would be nice if there was more information in the exception. Has anyone upped the debug level to see if it captures more information?

[radilly]

Steve commented above, and perhaps elsewhere. I got the impression the next logical step was to investigate the HTTP traffic - hence Fiddler. I've yet to get that working on my Pi.

Apologies for the detail that follows, but I wanted to document what I'm seeing - which is a failure rate as high as 80%.

Over the past hour it looks like around 12 updates succeeded and 48 failed. Here I just pulled out (and truncated) the WU lines.

Code: Select all

2018-01-08 15:24:01.028 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:25:22.802 WU Response: OK: success
2018-01-08 15:26:01.039 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:27:01.037 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:28:13.305 WU Response: OK: success
2018-01-08 15:29:01.042 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:30:01.056 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:31:54.445 WU Response: OK: success
2018-01-08 15:32:01.052 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:33:01.057 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:34:01.056 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:35:53.705 WU Response: OK: success
2018-01-08 15:36:01.061 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:37:01.064 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:38:01.074 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:39:01.078 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:40:26.116 WU Response: OK: success
2018-01-08 15:41:01.091 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:42:01.093 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:43:01.095 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:44:53.497 WU Response: OK: success
2018-01-08 15:45:01.105 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:46:01.111 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:47:01.117 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:49:01.678 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:50:01.127 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:51:01.124 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:52:01.119 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:53:01.124 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:54:01.132 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:55:01.138 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:56:01.131 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:57:01.139 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:58:01.134 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 15:59:01.139 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:01:01.683 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:02:01.153 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:03:01.158 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:05:01.681 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:06:01.165 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:07:01.168 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:08:01.171 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:09:01.177 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:10:01.181 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:11:01.174 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:12:01.257 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:13:01.162 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:14:15.424 WU Response: OK: success
2018-01-08 16:16:01.719 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:17:01.179 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:18:11.691 WU Response: OK: success
2018-01-08 16:19:01.194 WU update:    at System.Net.WebConnection.HandleError
2018-01-08 16:20:15.443 WU Response: OK: success
2018-01-08 16:21:40.789 WU Response: OK: success
2018-01-08 16:22:50.826 WU Response: OK: success
2018-01-08 16:23:20.496 WU Response: OK: success
2018-01-08 16:24:00.221 WU update:    at System.Net.WebConnection.HandleError

Looks like between 15:45:01 and 16:13:01 every call to WU failed. Still, when I look at the Weather History Table in WU I find a record from 16:04

20180108_WU_Table_Fragment.jpg

Drilling into the Diags log in more detail around that time, there's a gap that aligns exactly with that data point at 16:04 - I inserted the "<<<<<<<<<" row to point to the gap...

Code: Select all

2018-01-08 16:02:01.153 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
   at System.Net.WebConnection.ReadDone(IAsyncResult result)
2018-01-08 16:03:01.158 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
   at System.Net.WebConnection.ReadDone(IAsyncResult result)
       <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018-01-08 16:05:00.963 Writing log entry for 1/8/2018 4:05:00 PM
2018-01-08 16:05:00.964 Written log entry for 1/8/2018 4:05:00 PM
2018-01-08 16:05:00.968 Writing today.ini, LastUpdateTime = 1/8/2018 4:05:00 PM raindaystart = 36.4960629549 rain counter = 36.54330704934
2018-01-08 16:05:00.968 Latest reading: E300: 07 1D C5 00 54 13 00 3A 26 0E 14 00 0A 16 0C 80
2018-01-08 16:05:01.681 WU update:    at System.Net.WebConnection.HandleError(WebExceptionStatus st, System.Exception e, System.String where)
   at System.Net.WebConnection.ReadDone(IAsyncResult result)

Finally, in the first, distilled log fragment there are a few similar log gaps at
15:48
16:00
16:04
16:15

I'm not quite sure what it means, but there's some doc on the situation I'm seeing.

Bob

[steve]

It’s not clear to me why you think this isn’t simply yet another problem at Weather Underground, given their history of problems, and the fact that people on other forums who aren’t using Cumulus (e.g wxforum) are also currently experiencing problems uploading? As mentioned by Jeff, above. Cumulus is just passing on the message that there was a problem with the upload, as apparently reported by WU in a not very helpful way.

[radilly]

Hi Steve-
In fact, for the past few hours or so I've been looking for WU alternatives - under the assumption that the primary issue is Weather Underground. Having started with Ambient Weather some years back with our first PWS, I've just stuck with it. But I'm thinking the time has come...

OPEN QUESTION: Is there a discussion here (or on https://www.wxforum.net/ or elsewhere) anyone can recommend on the pros and cons of the various sharing site options, e.g. for Cumulus:

• Twitter
• Weather Underground
• PWS Weather
• Met Office WOW
• Weatherbug
• APRS/CWOP
• Awekas
• WeatherCloud

I've yet to find a comparison of these option. I want to stick with Cumulus because, WU idiosyncrasies notwithstanding, its been reliable and it is capable of way more than I'm using. Besides my own site, I would like to share my observations to a network of PWSs - though, admittedly, the siting of our sensors is poor given the huge trees on our property. I've also been very interested in a Raspberry Pi-compatible solution, and in the course of retiring myself it keeps my fingers in software and hardware a bit. I was delighted to see MX support for the Pi.

My intent was to support the Cumulus community by sharing my observations in some detail because, I'm suspecting, Weather Underground is slowly dying. BTW - I reported a webcam issue to WU a couple days ago. Little has changed on my side, but WU is reporting the camera as down. Even the portion of their site for configuring a PWS or webcam only works sporadically. I've heard nothing back, not even an acknowledgement of the report. So as I wrote, I'm concluding the time has come... for me to dump WU.

Cheers, and Thanks!
Bob


...Steve, I know you've retired but keep you fingers in this - and experience has taught me repeatedly that no one can fix what they can't see. I'd love to collect enough information to report the apparent issue to WU, even though that may be mute. I've not been able to get Fiddler to work on the Pi (in fact when it failed yesterday it left behind some proxy stuff in the ~/.bashrc that caused my watchdog scripting to fail to contact my hosted webserver - which took some digging to find and fix). If I have any (minor) criticism of Cumulus, it's that I'd love to have more information logged to report to WU about the issue ... without have to tangle with Fiddler. I've done a lot of software development and support and understand the challenge of "defensive programming" when you have to interact with systems outside your control, or even influence.

[PaulMy]

OPEN QUESTION: Is there a discussion here (or on https://www.wxforum.net/ or elsewhere) anyone can recommend on the pros and cons of the various sharing site options, e.g. for Cumulus:

Twitter
Weather Underground
PWS Weather
Met Office WOW
Weatherbug
APRS/CWOP
Awekas
WeatherCloud

I do all of those with Cumulus 1 and none are quite as informative as WU for my data and the extra that WU offers, but as is well known WU is quite unreliable currently for several of those additional extras. I also admit that I don't check these various sites very often as my own websites are much more reliable and most of those sites are linked to at least one of my sites.

Twitter through Cumulus works perfect for me.
PWS and AWEKAS work very good but there have been no updates to their layout for years.
WOW is quite nice and has the occassional update but not as common usage here in North America
Weatherbug is a bug
APRS/CWOP is good for data info and that data pops up at several other sites such as http://weatherobs.com/?homelat=43&homel ... homezoom=8
Weathercloud is good and promises to be adding features but nothing much I've seen.

Enjoy,
Paul

[steve]

Bob, if you are not already uploading to PWS Weather, then it would be worthwhile doing that, as the mechanism and protocol are virtually identical to WU, and if nothing else it would serve to confirm that the problems you are having with WU lie solely with WU.

[radilly]

Certainly seemed easy enough to add PWS Weather. They say "Your data will appear with local forecasts on WeatherForYou.com within 48 hours of when data is first sent and is received by our servers." So I may have to run for a while to see anything.

Meanwhile WU almost seems to have gotten worse. My station page says "This PWS is not reporting" while the plot below shows current data. Current data shows on Wundermap though.

Thanks guys!
Bob

[Toxic17]

plenty of WU issues, nothing new really just they havent fixed it yet.