Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

Degraded allocation warning messages

Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.
senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Degraded allocation warning messages

Postby senapsys » Thu Dec 03, 2015 10:23 pm

Hi there

I’ve been running CumulusMX on a RPi 2 for about 2 weeks now. I had no issues with installing it and it’s been running fine.

I am however seeing error messages from time to time as per below:

Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Repeated degraded allocation. Consider increasing nursery-size.

CumulusMX still appears to be running ok despite these error messages. Typically they start appearing 2-3 days after the application has been started.

My version information is as follows:

Raspberry Pi 2 running the Jessie release of Raspbian, patched with the latest updates as of about 1 week ago.
CumulusMX - B3034
Mono – 3.2.8

I was previously running B3033 and I also noticed the same message every now and then.

I did a quick search through the latest log file within the MXdiags folder, and I’m not seeing any log entries that match this.

In terms of CPU usage, it’s typically sitting at less than 5% with the occasional spike that ranges between 15 and 35% or thereabouts

Any thoughts on whether this is an error that I should be concerned about, or whether I need to tweak any configuration files would be appreciated.

Regards

Richard

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Fri Dec 04, 2015 10:10 am

It seems to be caused when large numbers of threads are being used, but I don't know why it happens to you.

http://stackoverflow.com/questions/1846 ... g-persists

I think someone else had the same message, and they were having problems with their SD card. Are you using an SD card for MX? SD card corruption is a common problem with the Pi.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Fri Dec 04, 2015 9:15 pm

Hi Steve

I'm aware of the SD card corruption issue so from the very beginning I built this RPi using Berryboot so that I could boot the entire system from a USB connected SSD Drive.

As a precaution I did a "fsck" but everything came through clean and no issues were highlighted.

Regards

Richard

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Sat Dec 05, 2015 4:40 pm

You could have a look at the thread usage on your system, something is apparently using lots of threads, quite possibly MX. If it's MX we would need to work out why it's doing that on your system.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Sat Dec 05, 2015 9:37 pm

Hi Steve

I've attached two files, one is a dump of all of the threads associated with the Mono process that is running CumulusMX after the Degraded Allocation messages start appearing. The other is a dump of the threads associated with the Mono process that is running CumulusMX after a restart of Mono. In the degraded state, Mono has 166 threads associated with it whereas just after Normal running appears, there are just 13.

Note that I'm not using Mono for anything else and a search through the process list has confirmed that there are no other instances of Mono running.

I'm reasonably familiar with Linux and there are a bunch of other stats and info I can pull out of the system. Let me know what else might be of use to you and I'll grab them for you.

Regards

Richard
You do not have the required permissions to view the files attached to this post.

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Sat Dec 05, 2015 10:03 pm

It does use threads for several things, but 166 seems like quite a lot. I wonder if you are having ftp problems, and the threads which handle ftp are hanging up? Is there any way to find out what those threads are actually doing? I suspect that you know far more about Linux than I do!
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Sat Dec 05, 2015 10:14 pm

Hi Steve

I'll do some digging and see what I can find. It might be a day or so before I post again as I have just restarted the application so I'm not seeing the accumulation of threads yet.

One thing I did notice is that there are a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?

Regards

Richard

User avatar
rogerthn
Posts: 152
Joined: Thu Apr 11, 2013 6:31 pm
Weather Station: Oregon Scientific WMR88
Operating System: Raspbian GNU/Linux 8
Location: Trollhättan Sweden
Contact:

Re: Degraded allocation warning messages

Postby rogerthn » Sun Dec 06, 2015 8:50 am

One of my favorites :P

Code: Select all

netstat -taupeen

It could return a lot but sort -kX and grep will help
Image

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Sun Dec 06, 2015 9:12 am

senapsys wrote:a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?
Nothing comes to mind. I had a look on my Pi and MX has 13 threads, just as you do at startup, and mine has been running for about 18 hours.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Tue Dec 15, 2015 3:17 am

Hi Steve

An update for you.

After my last post I deployed some scripts to monitor the number of threads as well as a bunch of other system performance data. As is the case with any fault finding, the moment I put these scripts in place the Mono/CumulusMX app got stage fright and the error didn't manifest itself after the usual 2-3 days :-) Like you, after the app displayed the "Normal running" message and I also observed 13 threads. Sometimes the number of threads increased up to about 17 threads for a brief moment and then returned to 13 or 15 threads. This "quiescent" level of thread variation appeared to be the normal behaviour that I observed over the days whilst waiting for the error to manifest itself again. I'm not sure if this is consistent with what you are seeing?

Anyway, after 9 days I finally started to notice something happening. I got lucky in that I was looking at the system as the thread count started to slowly increase. The increase in threads wasn't dramatic but rather happened at a much more pedestrian pace with an extra one being spawned every 5-10 minutes or thereabouts although this wasn't entirely consistent.

As this was happening I started to look at the various system stats and the one that stood out was an unusually high number of TCP sessions between the RPi system and the FTP server that hadn't been closed properly after a reasonable period of time. As an experiment I logged into the system which was running the FTP server and restarted the FTP service. As soon as I did this the slow increase in threads stopped and it went back to a scenario where the threads would briefly increase and then reduce, albeit with a higher base number of threads.

It's not a definitive answer and I'll need to put in place some more specific monitoring on both the RPi and the FTP server but the initial observations point to an issue with FTP. Whether the fault is with Mono/CumulusMX or the FTP server is impossible to say at this point. I'll do some more monitoring and see whether I'm on the right track.

Regards

Richard

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Tue Dec 15, 2015 8:20 am

I did suspect that it would turn out to be FTP connections not closing down. Turning on the ftp log may give some useful info from the Cumulus end - note that realtime ftp gets logged to the same file, so it can get large quickly.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Fri Dec 18, 2015 5:07 am

Hi Steve

I turned on FTP Logging in CumulusMX and have attached a log file from the last 24 hours during which time the thread count started to increase. To my untrained eye the only thing that really stands out is a number of instances where the multiple consecutive messages of "Disposing FtpSocketStream..." appear in the file.

The server FTP log didn't reveal anything that stood out so I have increased the level of logging to see if I pick anything up next time.

I hope this is of use to you :-)

Regards

Richard
You do not have the required permissions to view the files attached to this post.

User avatar
steve
Cumulus Author
Posts: 26626
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Postby steve » Fri Dec 18, 2015 12:58 pm

I can't see anything there that explains the problem, I'm afraid.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Fri Dec 18, 2015 8:09 pm

Hi Steve

Thanks for looking.

Noting my previous experience where restarting the FTP Server stopped Mono/CumulusMX from spawning additional threads it does point to the root cause being the FTP Server.

I'll put the microscope on the FTP Sever and see whether I can find out what the issue is and report back in due course.

Have a great Christmas/New Years :-)

Regards

Richard

senapsys
Posts: 11
Joined: Sun Nov 22, 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: RPi
Location: Wolumla

Re: Degraded allocation warning messages

Postby senapsys » Wed Dec 23, 2015 10:46 am

Hi Steve

I've been looking over various logs and one thing out of the ordinary which I have noticed in the CumulusMX log in the MXdiags directory is a few messages along the lines of:

Error uploading realtime.txt to cudata/realtime.txt : Access is denied.

When I look at the FTP Log within CumulusMX for the corresponding time I can't seem to see anything that correlates with the time stamp of the access denied message so I'm not sure if this is significant or not. I thought I'd mention it in case it's useful.

Unfortunately it's a bit random as to when the issue manifests but as soon as it happens again I'll go hunting through the server FTP logs and see whether I can see anything at that end.

Regards

Richard


Return to “Cumulus MX”

Who is online

Users browsing this forum: No registered users and 5 guests