Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

Future data formats

Discussion and questions about Cumulus weather station software version 1. This section and its subforums are the main place to get help with Cumulus. Anything which is not specific to the type of weather station goes in here; for anything specific to a type of weather station, please use the appropriate subforum. Use the 'website development' section for any questions relating to creating or running a web site for Cumulus data. Discussion of the stations themselves in these sections is fine.

How should a future version of Cumulus store its data?

In a local SQL database
16
11%
In 'flat' text files like the current version
39
27%
In both simultaneously
90
62%
 
Total votes: 145

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sat Jan 05, 2013 10:37 pm

While I admire your desire to cater for the desires of your users, I think that, from a code maintenance point of view, you'd be better off ditching the flat-file-format storage thingy.
I mean, srsly. We are storing structured data here... and when you're storing structured data in a flat-file-system, you are essentially emulating a database anyway.

With Entity Framework (available to all of us, except for those who run on Win2k, which is dead anyway), you can support MySql (free), SQL Server, SQL Express (free, I think) and Obstacle, oops, sorry, I meant to say Oracle, officah... with all these engines using the same code. You'd have to write some code to facilitate the generation of storage models and mapping models for each database engine, but I have that code right here on the shelf, at least for SQL Server and Obstacle. I think that MySql would be capable of using the same storage models and mapping files as SQL Server... but if they don't, that's easy to solve.

I cannot see how anyone would have to worry about running MySql. If you only use it for Cumulus, the system load would be extremely small.

The main rationale for dumping the flat file system storage model is that it makes a huge difference in terms of software maintenance. Plus: using a database for storage is not going to make the software less stable, or slower. Less code to maintain means less moving parts, and thus more stability.

Last but not least:
If you give us something that is as good as Cumulus, and you are providing it as donationware, I think that justifies the fact that you, as a software architect, decide how you architect it.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

Karv
Posts: 38
Joined: Sun Dec 16, 2012 11:19 pm
Weather Station: WH-1080
Operating System: Windows XP / 7, Ubuntu, Centos
Location: South Gloucestershire
Contact:

Re: Future data formats

Postby Karv » Sat Jan 05, 2013 10:53 pm

What peterh said!

MySQL does for flat files what the petrol engine did for the horse and cart.
Image

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: Future data formats

Postby steve » Sun Jan 06, 2013 10:50 am

The last time I looked, the MySQL client was very expensive - in fact it's quite difficult/impossible to get an actual price from Oracle without setting up an account. Is there an alternative that isn't from Oracle that's free for closed source applications?

Note that if I do ever produce a 'new' version of Cumulus, I intend it to be cross-platform. And I will have to re-use a lot of my existing code, because I simply can't start from scratch again, so it's almost certain that the data storage will be as in my previous post. And the SQL option will almost certainly be SQLite. With possibly an option to upload to a remote MySQL server, if a free client is available.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 10:56 am

I beg thee pardon?

From the MySql website:
"MySQL Community Edition is the freely downloadable version of the world's most popular open source database."

http://www.mysql.com/products/community/
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: Future data formats

Postby steve » Sun Jan 06, 2013 11:02 am

And the sentence which follows the one you quoted?
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 11:04 am

Looking at MySql versus SQLite, it looks like SQLite might be a better option if you're going to package a db engine.

The only issue with SQLite is that it doesn't support "database creation" for an Entity Framewodk CodeFirst approach, but that seems like a minor issue.

Personally, I would prefer an approach that allows me to either install SQLite, or use an existing SQL or MySql database engine. With EF, that's a piece of cake. Plus, EF makes coding against a database a lot easier.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 11:06 am

steve wrote:And the sentence which follows the one you quoted?


You mean this one: "It is available under the GPL license and is supported by a huge and active community of open source developers."?

That, imho, is even better news. What do you think is wrong with it"?
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 11:08 am

Some info from StackOverflow:
Stackoverflow wrote:SQLite:

easier to setup
great for temporary (testing databases)
great for rapid development
great for embedding in an application
doesn't have user management
doesn't have many performance features
doesn't scale well.

MySQL:
far more difficult/complex to set up
better options for performance tuning
can scale well if tuned properly
can manage users, permissions, etc.


http://stackoverflow.com/questions/3630/sqlite-vs-mysql
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: Future data formats

Postby steve » Sun Jan 06, 2013 11:10 am

Cumulus isn't GPL, and it's unlikely that a future version would be either - sorry.

SQLite is a great database for this purpose. It's what Cumulus 2 uses/used.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 11:32 am

Aargh. I keep confusing GPL and LGPL... :-(

In that case, SQLite is the better option. But we had already learnt that this is the case anyway.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

BCJKiwi
Posts: 847
Joined: Mon Jul 09, 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Future data formats

Postby BCJKiwi » Sun Jan 06, 2013 10:33 pm

I'm no expert in this so perhaps I should not be posting, however,
Since the development of Cumulus would not in fact be altering in any way the actual mySQL code, and mySQL would not actually need to be distributed with Cumulus then I don't see the problem.

Steve, you could build the Cumulus system, distribute Cumulus and the end user could download and instal mySQL themselves. Then the GPL license would be between the end user and Cumulus would not have modified nor redistributed mySQL.

I have been thinking about this whole issue and am not entirely sure that Cumulus itself needs to have a SQL file store although I do think it would be better if it did.

The real benefit of a proper database comes when data extraction/reporting/manipulation is done. For the weather applications this is typically via a website.
So if the website ideal database schema is planned for, then the local files could mirror that structure but not necessarily be SQL. A schema in which different elements are in different files provides better long term flexibility and ease of ongoing maintenance.

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Sun Jan 06, 2013 10:38 pm

This would be true... but unfortunately for the majority of users, a database would need to be bundled in a default install. That, or a default install would use the flat-file system storage mechanism.
I can see my wife downloading and installing mySql (she manages a 150+ server park), but not my son-in-law.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

BCJKiwi
Posts: 847
Joined: Mon Jul 09, 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Future data formats

Postby BCJKiwi » Mon Jan 07, 2013 12:41 am

peterh wrote:This would be true... but unfortunately for the majority of users, a database would need to be bundled in a default install. That, or a default install would use the flat-file system storage mechanism.
I can see my wife downloading and installing mySql (she manages a 150+ server park), but not my son-in-law.


Does the GPL actually 'forbid' distribution if mySQL itself is unmodified? Surely the schema and table(s) in the Cumulus database would not be considered part of the source covered by the GPL.

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Mon Jan 07, 2013 6:13 am

I not the local license specialist, but if I remember correctly this time, the GPL says that if you distribute MySQL with your software, your software should be GPL as well.
LGPL doesn't have this restriction.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: Future data formats

Postby steve » Mon Jan 07, 2013 8:44 am

In order to update a MySQL database, Cumulus would have to be linked with a MySQL client library, e.g. libmysql.dll as provided by Oracle. This DLL is dual-licensed, so any code using it has to be GPL (or FOSS) or have a commercial licence.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job


Return to “Cumulus 1”

Who is online

Users browsing this forum: No registered users and 3 guests