Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

HDD or SD Card for CumulusMX

Topics about the Beta trials up to Build 3043, the last build by Cumulus's founder Steve Loft. It was by this time way out of Beta but Steve wanted to keep it that way until he made a decision on his and Cumulus's future.

Moderator: mcrossley

Locked
TechnoDribble
Posts: 22
Joined: Tue 16 Feb 2016 8:36 am
Weather Station: Maplin /FO; Ecowitt HP2551
Operating System: Win 10 / Pi
Location: Guildford

HDD or SD Card for CumulusMX

Post by TechnoDribble »

Hi Chaps,

Firstly let me thank Steve for his awesome software.
I have managed to get CumulusMX working with my new Pi2 B, however being new to Linux and Pi I have been devouring theses pages for tips.
One thing that worries me is the suggestion that running a Pi on just a SD card is unwise?
I have started to read all about corruptions etc and given the effort it took me to get it all working I am not sure I would want to keep repeating that nor lose the weather data.
I have noticed that the LED on the Pi seems to be constantly flashing so I am worried about how long the SD card will last!

I have some spare old laptop HDD and caddies, should/would this be a wise thing to do to move the CumulusMX software folder to the HDD and would that reduce/remove the risk of the SD Card failing? I am perhaps naively hoping that the Pi would be more set and forget.
Any advice would be gratefully received.
User avatar
mcrossley
Posts: 12695
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: HDD or SD Card for CumulusMX

Post by mcrossley »

From what I have seen the SD card corruptions have two sources. The first was the Pi/OS itself, this as far as I can tell has been pretty much eliminated. Ith other source is the SD 'wearing out'. If you use a good quality card it should last for years - in theory, the bigger it is the longer it will last as each memory location gets written to less often.

If you go the HDD route, then personally I would go for a small SSD rather than spinning hardware - less power hungry and fragile.

I was on a HDD, until it got corrupted last week (PSU failure), now I am on SD only while I think about what to do.
TechnoDribble
Posts: 22
Joined: Tue 16 Feb 2016 8:36 am
Weather Station: Maplin /FO; Ecowitt HP2551
Operating System: Win 10 / Pi
Location: Guildford

Re: HDD or SD Card for CumulusMX

Post by TechnoDribble »

OK, my boot disk is a 16Gb SanDisk Class 10. So hopefully this will prove robust. Perhaps I will refresh the SD card every few years to be on the safe side.
I plan to use the start/stop script to make a backup of CumulusMX to a thumb drive each night or perhaps weekly as a belt and braces safety measure!
jank
Posts: 239
Joined: Sat 13 Jun 2015 5:57 pm
Weather Station: FineOffset WS3080
Operating System: Debian Buster - Raspbian - Raspb
Location: Germany - nearby Kassel

Re: HDD or SD Card for CumulusMX

Post by jank »

TechnoDribble wrote:OK, my boot disk is a 16Gb SanDisk Class 10. So hopefully this will prove robust. Perhaps I will refresh the SD card every few years to be on the safe side.
I plan to use the start/stop script to make a backup of CumulusMX to a thumb drive each night or perhaps weekly as a belt and braces safety measure!
Mark is absolutely right......I have at least 5 RaspberryPi's running + 2 BananaPi's and I never had a problem with a damaged SD Card. Sandisk is a good quality card....so you seem to be safe for the next weeks.
However, you should have frequent Backups of your data.
Here is what I have and this is possible with Windows workstations/Servers or also NAS
1. Create a Backup folder on your windows Workstation/Server.
2. Create a Windows share for this Backup folder on your workstation/Server or NAS (On the NAS it could also be a NFS Share)
3. Mount this share on your RaspberryPi (NFS or CIFS Mount)
4. Use the Linux build in "rsync" command and make a backup each hour of the data hwich has changed sine last backup from the Raspberrypi /home/pi/CumulusMX folder to the Backup folder.

Also possible connect an USB Drive to your Raspberrypi.....and install cumulusMX to this USB Drive.
Do want a Managing Dashboard for CumulusMX on RaspberryPi? cumulusmx.sh
viewtopic.php?f=40&t=17907
Image
thecivvie
Posts: 39
Joined: Mon 02 Apr 2012 6:48 pm
Weather Station: Davis Vantage Vue
Operating System: Raspberry Pi
Location: Renvyle, Connemara, Ireland
Contact:

Re: HDD or SD Card for CumulusMX

Post by thecivvie »

As others say, I have a similar setup. Mine boots from the best SD card I could afford and then I use a SSD to hold the main partition with the /home etc on, I have as much on it as possible without it interfering with the boot up process. I maintain a backup of the contents of the SD card on a weekly backup and the Cumulus folder on a daily backup stored onto Dropbox using a script
Image
mauro
Posts: 6
Joined: Sun 27 Dec 2015 5:13 pm
Weather Station: Davis vp2
Operating System: raspberry
Location: Biancade, Italia

Re: HDD or SD Card for CumulusMX

Post by mauro »

Hi,
can you give me the script? And can you explain me how to install it?
Thank you
jank
Posts: 239
Joined: Sat 13 Jun 2015 5:57 pm
Weather Station: FineOffset WS3080
Operating System: Debian Buster - Raspbian - Raspb
Location: Germany - nearby Kassel

Re: HDD or SD Card for CumulusMX

Post by jank »

mauro wrote:Hi,
can you give me the script? And can you explain me how to install it?Thank you
without anticipating thecivvie's answer I would say,
It is not so easy to backup files from RaspberryPi to Dropbox. There is an installation guide available using the API function of dropbox.
It is much easier to backup CumulusMX to any other Cloud Space which is able to handle WEBDAV. (Unfortunately Dropbox does not)
In the newest version of my "CumulusMX Start Stop Script" I added support for WEBDAV Backup.
Supported and tested Cloud Space so far: T-Online Magenta Cloud, 1und1 Cloud , Owncloud
Supported because WEBDAV: Box, Google Drive and more

Another option would be to mount a Cloud storage like a normal network Drive and backup/sync to the mounted drived.
If you have a Windows box, create a share and mount it via CIFS to the Raspberrypi (/mnt/windowsshare)
If you have a Linux server/NAS you can create a NFS Folder (/mnt/NASshare)

This is similar to your windows system where you mounted a remote sahre as an additional drive letter.

A Sync Script which backups all files in the CumulusMX Folder each hour for example would look like this:

00 */1 * * * sudo rsync -rtagu /home/pi/CumulusMX/ /mnt/windowsshare/CumulusMX_Sync/ --exclude=MXdiags --exclude backup

If you want to create a Backup of the CumulusMX folder using ZIP or TGZ I would suggest to use my CumulusMX Start/Stop Script like
sudo /home/pi/cumulusmx.sh -b -z (-b -t)
This will create a BAckup of the CumulusMX folder. advantage, you can tell how many Backup Files should remain. the oldest one will be deleted.
Another desaster Recovery Option in case you fear that the SD Card is completely unusable you need to make a Complete Backup of the SD Card.

the easiest way would be to rmeove the SDCard form the RaspberryPi, insert it into your windows box and use Win32DiskImager.exe to create an image of the SD Card.
In case it is damaged, you can buy a new one and Rebuild the SDCard form the image (similar to Acronis Backup)
The disadavanate is, that all data is lost since the Image back was created but in conjunction with the RSYNC or daily backup you will always have a complete Backup where in worst case 1 hour is lost. (Image + hourly rsync)

By the way, you can also use the CumulusMX Start Stop Script to create such an Image without removing the SD Card from the RaspberryPi by using sudo /home/pi/cumulusmx.sh -b -d
this will use dd to create such an image on a mounted Network share on the fly.

But the option how to backup/sync data to dropbox is very interesting since many people use Dropbox....I will test it when I have more time.
Jan
Do want a Managing Dashboard for CumulusMX on RaspberryPi? cumulusmx.sh
viewtopic.php?f=40&t=17907
Image
mauro
Posts: 6
Joined: Sun 27 Dec 2015 5:13 pm
Weather Station: Davis vp2
Operating System: raspberry
Location: Biancade, Italia

Re: HDD or SD Card for CumulusMX

Post by mauro »

ok jan, thank you.
wxstormer
Posts: 5
Joined: Fri 13 Mar 2015 8:57 pm
Weather Station: Vantage Vue (with VP2 Console)
Operating System: All
Location: New Braunfels TX

Re: HDD or SD Card for CumulusMX

Post by wxstormer »

I have a synology box (but any NAS or NAS-drive should work) and NFS mount the root filesystem on the PI and run that way. The SD Card only has the boot information to mount my partition via NFS and run from there. The great thing is I can take my NAS offline for a reasonable amount of time (I've only ever pushed it to a hour) and I don't lose data... So I power the Pi by battery (which is constantly kept in charge by a trickle charger) so I don't lose data in a power failure. :)
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: HDD or SD Card for CumulusMX

Post by philpugh »

I have recently moved CumulusMX from Raspberry Pi to Banana Pi Pro. The system still boots from a small SD card but runs the main file system (root partition) on an attached SATA/SSD. (The SATA port was the main reason to move to Banana Pi).

Within CumulusMX I have moved to a total MySQL / PHP solution - that is CumulusMX writes data into a MySQL database which runs on a NAS box (which is replicated to a second NAS box - both are RAID 1 ). The BananaPi runs the Apache2 web server and I use PHP5 to provide a series of web pages using data from the MySQL database. The CumulusMX directory is rsync'd to a share on the NAS system.

I also run a POSTFIX/Dovecot mail system and I have just installed a home energy monitoring system on the BananaPi. The BananaPi Pro is running at about 8% average utilisation at present - half of which is the energy monitoring s/w (eagle-owl using the OWL CM160 hardware).

The BananaPi/SSD/OWL Monitor are all powered from the one 5v/2A supply. It certainly looks like a pretty neat solution to me at present - need to find more things to do with it!
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
slashmusic
Posts: 143
Joined: Sun 10 Jan 2010 10:43 am
Weather Station: WS2350
Operating System: Raspberry Pi2 - Debian
Location: Germany
Contact:

Re: HDD or SD Card for CumulusMX

Post by slashmusic »

Phill
I Know this has nothing to so with Cumulus but are you Running postfix/dovecot on your Banana Pi? If yes, how is the performance? Is there a way to push Mails to mobile phones?
Image
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: HDD or SD Card for CumulusMX

Post by philpugh »

Yes both Postfix and dovecot on the BPi Pro. Performance is good. I don't push mail to my phone but use Maildroid over a TLS/SSL secure link. I have a number of portable devices and prefer to use IMAP for mail retrieval.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
jrhilton
Posts: 17
Joined: Thu 31 Jul 2014 4:25 pm
Weather Station: Custom build + old Oregon WMR88
Operating System: Windows 10 / Linux
Location: London

Re: HDD or SD Card for CumulusMX

Post by jrhilton »

For what it is worth I have been running my original PI for almost four years now and I've never needed to change the SD card (16GB Sandisk). It has been running a shell script ever 5 minutes to collect data from some one wire temperature sensors and saving into a MySQL database and then updating a logtxt file report which summaries data by day etc (so quite a but of work to calculate it).

The key is to disable swapping, turn logging off and specify “noatime" so the system doesn't log the last time each file was accessed.

No issues so far.
jank
Posts: 239
Joined: Sat 13 Jun 2015 5:57 pm
Weather Station: FineOffset WS3080
Operating System: Debian Buster - Raspbian - Raspb
Location: Germany - nearby Kassel

Re: HDD or SD Card for CumulusMX

Post by jank »

:D 'never seen that mine took 1 byte swap so far. No Idea why
swap.jpg
cat fstab says:

# 512 MB swapfile
/swapfile1 swap swap defaults 0 0
You do not have the required permissions to view the files attached to this post.
Do want a Managing Dashboard for CumulusMX on RaspberryPi? cumulusmx.sh
viewtopic.php?f=40&t=17907
Image
Locked