Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

Jump in Number of Mono Threads

Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.
User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Jump in Number of Mono Threads

Postby radilly » Fri Dec 08, 2017 7:48 pm

I've been monitoring various aspects of the "health" of my Raspberry Pi 3 running Cumulus MX for a couple of months, adding things as I notice them. One of the things I've noticed is that in time I seem to observe a large increase in the number mono threads running. I've been looking around for information in this, but haven't found much. Wondering if anyone has come across, or better solved it. I'm counting mono threads more or less using:

Code: Select all

/usr/bin/top -H -w 125 -n 1 -b -o +RES | egrep " mono$" | wc -l


I'm running a variant of this in a Python script. What I observe is that on starting up Cumulus MX I have something like 15 threads running (using the above to count). A little under 2 weeks since a restart and I noticed a sudden jump in the number of mono threads. I check every ~ 24 seconds, and over the course of about 4 or 5 minutes it went from around 15 mono threads to 80 - that's about 5X. It's been holding around 80 for several days and otherwise things seem OK.

Anyone else notice this behavior?

In the past I've seen this jump in a number of steps, of a similar order of magnitude, i.e. adding 40-60 mono threads within a few minutes - over time up to ~ 260 threads. In that case a small amount of swap was being used so I restarted Cumulus, even though I didn't really note any functional performance issues. (I'm trying to avoid hitting the SD card; most I/O goes to an SSD.)

This is part of an effort to restart Cumulus or even reboot the Pi before performance suffers (via a watchdog).

Cheers,
Bob


A portion of the log...

Code: Select all

2017/12/06 21:00:53, 16
2017/12/06 21:01:18, 19
2017/12/06 21:01:43, 26
2017/12/06 21:02:07, 34
2017/12/06 21:02:32, 41
2017/12/06 21:02:57, 47
2017/12/06 21:03:22, 55
2017/12/06 21:03:47, 62
2017/12/06 21:04:11, 70
2017/12/06 21:04:36, 78
2017/12/06 21:05:01, 82
2017/12/06 21:05:26, 80



I've been tinkering with posting some Pi / Cumulus MX "health metrics" at http://raspberrypi_02:8998/status.html - Hope to eventually add some highcharts...

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Fri Dec 08, 2017 8:34 pm

What's the MX memory usage like?

I haven't tracked the mono threads but I have noticed that when WU is misbehaving MX memory usage steadily climbs. It normally runs around 65-70MB, currently mine is sat at 165MB - it climbed up there last time WU played up and stopped increasing when it came back. It can live with current usage, but I'll restart MX if it increases much more.

My mono thread count is currently 91.

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Fri Dec 08, 2017 10:44 pm

I don't really see MX directly under top, but mono is pretty much the only thing with any memory and CPU usage:

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8958 root      20   0  228808 129816  18364 S  13.2 13.7 738:27.88 mono


When I started MX, this is what I saw in top for mono:

Code: Select all

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  541 root      20   0   87944  57388  17716 S   0.0  6.1   0:11.86 mono


So the virtual footprint is 87M or so - in the ballpark of what you see. At 228M now, that's nearly 3 x after 2 weeks. Although tiny, I am seeing a little swap usage - 80 bytes. This also creeps up a word or two at a time, which seems a bit odd. (If the LED for the SD card is blinking, I don't see it.) Since I have the GUI disabled there is very little running outside of MX on the Pi. I'll have to look for any correlation between WU issues and changes in loading. I'm also logging the odd exception from Diags which seems to result from a WU exchange - that typically dominates the sort of issues I see in Diags by a good margin.

Cheers!
Bob

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Fri Dec 08, 2017 11:52 pm

Do you have rapid fire enabled (I don't) then if the WU updates are failing you are going to clock up failures very quickly.

MX ran for over 6 months before the last restart, now on 55 days. The increase from 70 MB to 165 happened a week or so ago.

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Sat Dec 09, 2017 2:52 am

I don't use Rapid Fire either. My PWS only transmits every 48 seconds so there's not much point in updating much faster than that. I've been running realtime with an interval of 24 secs - twice the rate from the transmitter, and indeed the data does seem to change - perhaps the indoor data has enough jitter to keep it most always changing.

Thanks-


I think my longest run without a restart was around 4 weeks. I may just let it run until I see an observable performance issue. My swap usage is very slowly creeping up, but it doesn't even register as 1% yet.

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Sun Dec 10, 2017 12:13 pm

I've restarted MX as it seemed it was using a fair bit of CPU with that high thread count.
I have also 'enhanced' my System Info page to add a thread count and change the avg CPU to a 2 second value and an average.

This does add a delay to the page load time, but as it is mainly for my information I can live with that.

I find it really hard to get CPU stats out of Linux, I ended up using...
mpstat for a 2 second system CPU usage
pidstat for a 2 second and lifetime average CMX CPU usage, this also gives the thread count.
psaxo for CMX memory usage and uptime

Code: Select all

mpstat 2 1 | tail -1 | awk '$12 ~ /[0-9.]+/ { printf("%0.1f%%\n", 100 - $12"%") }'

ps axo pid,cmd | grep "CumulusMX" | egrep -v "(sudo|grep)" | awk "{print $1}"

pidstat -h -r -u -v -p <PID> 2 1 | tail -1 | awk '{printf("%0.1f,%d",$7,$14)}'

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Sun Dec 10, 2017 9:01 pm

Hi Mark-
Nice system info page ... and site for that matter! I'm hosting through GoDaddy, so I'm uploading lots of data to that virtual server. (Getting a static IP looks expensive here.) I haven't spent much time on customization, and my little sys info page is admittedly pretty lame as I figure out what is "interesting".

My system doesn't have mpstat or pidstat - which I don't think I've come across. In general I've been trying to keep my system as "vanilla" as possible. I didn't do as good a job of keeping track what packages I've added as I would have liked, but there aren't many. I've poked around /proc and other pseudo-filesystems in the past to get detailed metrics, but I've not tried that on the Pi yet. I supposed using a command interface may insulate your code a little better, but usually accessing the pseudos is pretty low overhead, and thus attractive. So far I'm using uptime, free, top and systemctl (since I've set up CMX and a watchdog as services), but I've been thinking about using trying /proc for some of it - especially number of threads. (man proc might be helpful if interested.)

I see you are running Jesse Light which I wanted to tinker with - well at this point maybe Stretch Lite. I turned off the GUI and actually do most of my work via ssh, though I noticed that part of VNC and some other GUI-related services were still trying to run. I cleaned that up but I don't think I've run X since disabling it for startup. I suspect the Lite versions are "cleaner" if running headless.

Cheers,
Bob

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Sun Dec 10, 2017 9:10 pm

Yeah, I always use the lite builds, they are a lot cleaner, though there still lots of packages that need removing.

mpstat and pidstat are both part of the sysstat package.

The /proc CPU info is mainly about "load" rather than percentage use, as is the info from ps. I'm more accustomed to % use than load.

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Fri Dec 15, 2017 4:19 pm

I've update my status page to now update the Cumulus stats every 6 seconds or so - the page will not work properly with IE/Edge/Opera Mini now, but all other browsers should be OK.

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Fri Dec 15, 2017 9:26 pm

Hey Mark-

You're obviously good on the web side - much better than I am. That's a nice page. I'd love to dig into some of the Java scripting because my (mostly stock) gauges are skewed, I suspect, due to some old data points that were effectively way outside the normal range. (My status page still needs to be refreshed manually, but it's OK for me for now. Only my local subnet it gets updated more often than on the web server, which is pushed by CMX every 5 minutes.)

For what it's worth, I was curious about getting % CPU utilization from /proc/stat. If the top "%Cpu(s): " line is accurate, I think I'm interpreting that pseudo-file correctly. I have a little demo script at https://github.com/radilly/rpi_hacks/bl ... pu_util.py. I believe it would work on a pretty "vanilla" system. Ain't very pretty but it seems to give reasonable numbers.

Interestingly the utilization on my CPU is lower than yours, but it bounces around similarly. The memory load looks similar, but it bothers me swap usage keeps creeping up - now 6%. It has grown slowly, and oddly I don't see the green LED on the SD card flicker. (I've only moved data to the SSD so far, while the O/S is still on the card.)

Bob


One thing I find interested is that when running this script, sampling every 15 sec, I can definitely notice when the 5 minute update runs. Also, once in a while it reports near 0% usage - twice in this example...

Code: Select all

Since last boot:  4220235 * 100 / 750586316
 0.562%
========
2017-12-15 21:18:10  0.087%
2017-12-15 21:18:25  0.958%
2017-12-15 21:18:40  0.402%
2017-12-15 21:18:55  0.267%
2017-12-15 21:19:10  0.703%
2017-12-15 21:19:25  0.000%
2017-12-15 21:19:41  0.666%
2017-12-15 21:19:56  0.646%
2017-12-15 21:20:11  3.675%
2017-12-15 21:20:26  1.694%
2017-12-15 21:20:41  0.000%
2017-12-15 21:20:56  0.526%
2017-12-15 21:21:11  0.455%
2017-12-15 21:21:26  0.018%

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Fri Dec 15, 2017 9:39 pm

My pi is also running my database (MariaDB) and web server (nginx) hosting my two web sites: weather and astro. The weather data is updated every 5 seconds so I'd expect the CPU usage to be fairly high. I'm surprised the pi copes as well as it does.

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Sat Dec 16, 2017 1:10 am

Well it is a quad core, and it sounds like you have it pretty stripped down.

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Fri Dec 29, 2017 9:45 pm

I have just updated my pi to stretch and mono 4.6.2.
The base usage for MX had gone up from 70-92 MB and 12 threads to 90-111 MB and 15 threads. The MX cpu usage has also increased average 1.5 up to 2.1%. Overall memory usage by all processes has dropped markedly.

I think I may try reverting the mono version back to 3.8.2 and see if the memory and CPU usage cross back to previous levels.

Update, thread count now 16 and memory increased a bit.

User avatar
radilly
Posts: 43
Joined: Fri Jul 17, 2015 11:01 am
Weather Station: Ambient WS-2080
Operating System: Raspian Jessie
Location: McMurray, PA, US
Contact:

Re: Jump in Number of Mono Threads

Postby radilly » Sat Dec 30, 2017 5:52 pm

Hi Mark-
Happy new year!

I have a second Pi and as time allows I want to try Stretch (without the GUI). Boy I'd love to be able to clone the PWS data stream and run the 2 Pis side by side for a time. I am interested in, perhaps, stripping down the OS even further as I think you've done. I'm running way less on my Pi than you are!

I continue to track a number of metrics on the Pi and Cumulus. On my last run of about 4 weeks, I captured enough data to get a couple of plots related to memory usage. I'm not entirely sure what it means yet. There's a very pronounced knee in the Swap usage. When I rebooted on the 21st, the real-time updates were still timely, even though memory usage was at 69% and swap at 24%. I've added CPU utilization toward the end of the log which averaged a very light 0.7% for the last couple of hours.

Another interesting observation is that I never saw the green LED flicker even with swap at 24% - which I'd expect. About 3 hours ago swap usage went from 0 to 4 bytes! 4 bytes 8.8 days since reboot! Now I see the green LED is flickering roughly once a second periodically (which seems meaningful).

I'll continue to share observations in case anyone is interested. If I connect the dots, it seems like the Pi would have been in trouble in another week or two. I know others report comparatively long stretches between reboots. From what I observe, I continue to think a periodic reboot will be necessary.

Regarding WU complaints - In the diags file I regularly see " ... System.Net.WebConnection.ReadDone exceptions". I'm trying to track that in a form that I can look at some summary stats. Seems like these occur around 4-5/hour on average (with considerable variation).

Be well!
Bob
You do not have the required permissions to view the files attached to this post.

User avatar
mcrossley
Posts: 5037
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Jump in Number of Mono Threads

Postby mcrossley » Sun Dec 31, 2017 4:58 pm

Rather than downgrade Mono, I decided to try upgrading to the latest release, I'm now running version 5.4.1.6.

The good news is the CMX CPU utilisation has now dropped back to similar or even lower levels to running under version 3.8.2.

The "base" thread count is the same as 4.6.2 at 16 for me. The memory usage is slightly higher again - I guess mono bloats a bit with each release! :lol:

Regarding Raspbian run-time - my Pi was on 222 days when I rebooted it to upgrade from Jessie to Stretch (I did an in-place upgrade without any issues), I hope Stretch is just as stable.


Return to “Cumulus MX”

Who is online

Users browsing this forum: No registered users and 1 guest