Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.5 (build 3282) - 23 February 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Future data formats

Discussion and questions about Cumulus weather station software version 1. This section is the main place to get help with Cumulus 1 software developed by Steve Loft that ceased development in November 2014.

How should a future version of Cumulus store its data?

In a local SQL database
17
11%
In 'flat' text files like the current version
41
27%
In both simultaneously
94
62%
 
Total votes: 152

peterh
Posts: 150
Joined: Fri 21 Dec 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Post by peterh »

In order to update a MySQL database, Cumulus would have to be linked with a MySQL client library,
Which, regarding to this discourse, is fairly unimportant. I think MySql came into this thread because I suggested it :oops: , but I have since learnt that SqLite is a much better and more user-friendly idea to bundle than MySql.

But if you would not distribute said client library, you'd be ok offering MySql as a non-default option. If you would want to choose SqLite as the default, but would want to offer other DB engines as an alternative, and I would choose MySql, I would have to install the client myself. I would then have the dbProvider section in my machine.config, which negates the need for you to put it in your app.config. The only thing you'd have to come up with in your app.config is a valid connection string.
Now, of course, when there would be a difference in SQL syntax between the database engines (Obstacle comes to mind, but something simple like data type stuff can also upset us), the plot, she would thicken. Enter Entity Framework, which abstracts the database layer... and even when the storage model (and thereby also the mapping model) would change because some database engine only wants field names in uppercase, or insists on using NCLOB instead of NVARCHAR, we could generate storage models and mappings at design-time, rather than run time which is the default behaviour of EF...

For me, it would be ideal if the software would offer the option of using SqLite, or use a SQL or MySql database engine that I already have. But if Steve decides to keep it as it is, I'm not going to complain profusely. ;)
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/
BCJKiwi
Posts: 1255
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Future data formats

Post by BCJKiwi »

Well in that case Steve, I return to my earlier option,
Don't use mySQL in Cumulus but provide for table structures and a 'friendly' mechanism that can be used to place that data into SQL on the website. These could be similar to (if not) the existing file structures, or, the more limited SQlite (not sure what those limitations are).

DAJ/Mark already have this working with the existing Cumulus file structures. I understand PHP has all the handlers required to read/write SQL and as far as I can see, as previously suggested, SQL is more important at the Website.
Most Hosters will have mySQL support on board.
In order to test the changes I make to scripts (and ensure I don't create problems at my hoster), I have installed XAMPP which includes Apache webserver, mySQL, PHP, and a host of other stuff all for free and can run on your Windows PC along side everything else you run. So if a user really wants to run mySQL locally then they could do the same.

I don't consider myself a programmer in this area but I do seem to be able to get by and am more comfortable doing so now that I have this test bed.
However I do recognise that many users simply want a turnkey easy to use system and this is a major 'selling point' for Cumulus. I think it is important for the Cumulus user base and for Steve's sanity that the target user base is properly catered for with any change. The excellent reputation that Cumulus has gained in this regard should not be jeopardised.

You may be aware that that I wanted to add some features that Cumulus did not provide for and so looked at WD. I found that to be a very salutory experience and just proved beyond any doubt the value of a clean system that does not step out of its 'comfort' zone. These features are now being developed separately from Cumulus and will be combined into the website in due course.
peterh
Posts: 150
Joined: Fri 21 Dec 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Post by peterh »

BCJKiwi wrote:the more limited SQlite (not sure what those limitations are).
The limitations of SqLite are a minor concern. It is not as scalable or secure as MySql, but for the vast majority of users, this qualifies as Not A Problem At All.
BCJKiwi wrote:the values of a clean system that does not step out of its 'comfort' zone.
:clap:
That may be just the most valuable remark in this entire discourse.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/
Karv
Posts: 40
Joined: Sun 16 Dec 2012 11:19 pm
Weather Station: WH-1080
Operating System: Windows XP / 7, Ubuntu, Centos
Location: South Gloucestershire

Re: Future data formats

Post by Karv »

If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....

However from an actual development / realease and manageability point of view, its probably much more sensible to go with something which can be handled as total solution :)
peterh
Posts: 150
Joined: Fri 21 Dec 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Post by peterh »

Karv wrote:If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....
Nope.

However, my suggestion to use MySql, or SqLite, did not come from my desire as a user to have stuff in a database. I could do that myself just fine, and if necessary, I could release that under a GPL or LGPL license. Or I could just throw the source code out with support for as long as it takes you to download it and be damned with it... which is probably what I would do anyway, because I cannot rely on the realtime.txt schema in the first place. ;)

My idea is that, if today, I were to build, and support, something like Cumulus, I would use a database as a storage repository. My main reason for this would be that this would make maintenance and updates and extensions soooooooooooooooooooooooo much easier.

Disclaimer: I am not bashing Steve's architectural choices. I think Steve has done, and is still doing, a stellar job with Cumulus. While I might have made different choices, it is likely that I wouldn't have done such a stellar job.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/
davidrg
Posts: 3
Joined: Fri 09 Nov 2012 10:36 am
Weather Station: Vantage Vue
Operating System: Windows 7; Linux
Location: New Zealand

Re: Future data formats

Post by davidrg »

peterh wrote:I not the local license specialist, but if I remember correctly this time, the GPL says that if you distribute MySQL with your software, your software should be GPL as well.
LGPL doesn't have this restriction.
Almost. You can modify distribute GPL software as much as you like as long as you distribute the source code too and don't change the license. What you can't do is integrate GPL code (including libraries, etc) into proprietary software. I think the original idea was to make writing Free Software easier and cheaper than proprietary software making it the more attractive option.

Anyway, ---> PostgreSQL<---. Better reliability, a lot more features and a permissive MIT-style license that wouldn't conflict with Cumulus. SQLite is probably still easier and more appropriate than both of them for this particular embedded purpose though.

I'd think the easiest option would be to log all data to a SQLite database and provide an option (perhaps with a schedule?) to export data to CSV or whatever.
gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: Future data formats

Post by gemini06720 »

I have been reading some interesting articles about organizations moving away from Oracle's MySQL to Monty Program's MariaDB, such as, for example, Wikipedia - there are also 'rumors' that Fedora Linux and OpenSUSE might also consider switching from MySQL to MariaDB.

I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).

MariaDB might be an interesting solution for storing the Cumulus data while keeping compatibility with the MySQL database format...
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Future data formats

Post by steve »

That's interesting; the MariaDB client library is LGPL, and while it's written in C and Cumulus is not, I would still be able to use it with Cumulus (in the same way that I already use C DLLs like the Davis one), thus giving access to MySQL (and MariaDB) databases from Cumulus without any licensing issues.

I think I'd still want to use SQLite for local Cumulus storage, though (as MariaDB cannot be embedded).
Steve
peterh
Posts: 150
Joined: Fri 21 Dec 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Post by peterh »

Are you planning to use Entity Framework for database access?
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Future data formats

Post by steve »

peterh wrote:Are you planning to use Entity Framework for database access?
I'm not really planning anything. But not in the current version of Cumulus, no; Cumulus is not a .NET application. If there was future version of Cumulus that was a .NET application, and had database support, I might consider using EF. And Mono has EF, so that's good. I think Cumulus 2 uses it, although ISTR there were some things for which I had to talk SQL directly to SQLite, for some reason. I think there were performance issues.
Steve
DigbyDave
Posts: 47
Joined: Sun 09 Jan 2011 11:55 am
Weather Station: Vantage Pro 2
Operating System: XP SP3
Location: AU

Re: Future data formats

Post by DigbyDave »

gemini06720 wrote:I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).
I for one have been using MariaDB as a mySQL replacement for about 2 years, there is just something I don't like about Oracle. So far, I have not found any incompatibilities between MariaDB and mySQL, not that I'm a power user, just basic help desk, mail server, wordpress etc and testing public domain accounting software and such, no code changes have been required and all apps think they are talking to mySQL. Documentation, I fall back onto mySQL documentation if required.

Anyway, back to the subject. I personally would like to see some development towards some form of DB, not really fussy which, SQLite used to have table lock issues, but I think they have been resolved for some time.

Ultimately it's Steve's call.
Dave

Braitling, NT, Australia
INTBRAIT2
Metanges
Posts: 5
Joined: Tue 06 Mar 2012 8:19 am
Weather Station: WS-2355
Operating System: Windows 7 64 bits
Location: Rouen

Re: Future data formats

Post by Metanges »

I appriciate very much SQLite databases.

I use it on some of my project, it is very easy to use with Visual Studio.

Thks
martinu
Posts: 85
Joined: Wed 01 Jan 2014 7:54 pm
Weather Station: Davis Vantage Vue
Operating System: Raspberry Pi: Raspbian Stretch
Location: Rudston, North Yorkshire

Re: Future data formats

Post by martinu »

I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.
RayProudfoot
Posts: 3372
Joined: Wed 06 May 2009 6:29 pm
Weather Station: Davis VP2 with Daytime FARS
Operating System: Windows XP SP3
Location: Cheadle Hulme, Cheshire, England
Contact:

Re: Future data formats

Post by RayProudfoot »

martinu wrote:I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.
You're a bit late to the party. ;) Topic was launched over 4 years ago. No further development to Cumulus 1 I'm afraid.
Cheers,
Ray, Cheshire.

Image
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Future data formats

Post by steve »

Just to add to that - MX uses the same flat file format as Cumulus 1, and will possibly add local SQL logging in addition if I ever find the time to finish writing the code.
Steve
Post Reply