Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

Future data formats

Discussion and questions about Cumulus weather station software version 1. This section and its subforums are the main place to get help with Cumulus. Anything which is not specific to the type of weather station goes in here; for anything specific to a type of weather station, please use the appropriate subforum. Use the 'website development' section for any questions relating to creating or running a web site for Cumulus data. Discussion of the stations themselves in these sections is fine.

How should a future version of Cumulus store its data?

In a local SQL database
16
11%
In 'flat' text files like the current version
39
27%
In both simultaneously
90
62%
 
Total votes: 145

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Mon Jan 07, 2013 8:51 am

In order to update a MySQL database, Cumulus would have to be linked with a MySQL client library,


Which, regarding to this discourse, is fairly unimportant. I think MySql came into this thread because I suggested it :oops: , but I have since learnt that SqLite is a much better and more user-friendly idea to bundle than MySql.

But if you would not distribute said client library, you'd be ok offering MySql as a non-default option. If you would want to choose SqLite as the default, but would want to offer other DB engines as an alternative, and I would choose MySql, I would have to install the client myself. I would then have the dbProvider section in my machine.config, which negates the need for you to put it in your app.config. The only thing you'd have to come up with in your app.config is a valid connection string.
Now, of course, when there would be a difference in SQL syntax between the database engines (Obstacle comes to mind, but something simple like data type stuff can also upset us), the plot, she would thicken. Enter Entity Framework, which abstracts the database layer... and even when the storage model (and thereby also the mapping model) would change because some database engine only wants field names in uppercase, or insists on using NCLOB instead of NVARCHAR, we could generate storage models and mappings at design-time, rather than run time which is the default behaviour of EF...

For me, it would be ideal if the software would offer the option of using SqLite, or use a SQL or MySql database engine that I already have. But if Steve decides to keep it as it is, I'm not going to complain profusely. ;)
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

BCJKiwi
Posts: 845
Joined: Mon Jul 09, 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Future data formats

Postby BCJKiwi » Mon Jan 07, 2013 10:05 am

Well in that case Steve, I return to my earlier option,
Don't use mySQL in Cumulus but provide for table structures and a 'friendly' mechanism that can be used to place that data into SQL on the website. These could be similar to (if not) the existing file structures, or, the more limited SQlite (not sure what those limitations are).

DAJ/Mark already have this working with the existing Cumulus file structures. I understand PHP has all the handlers required to read/write SQL and as far as I can see, as previously suggested, SQL is more important at the Website.
Most Hosters will have mySQL support on board.
In order to test the changes I make to scripts (and ensure I don't create problems at my hoster), I have installed XAMPP which includes Apache webserver, mySQL, PHP, and a host of other stuff all for free and can run on your Windows PC along side everything else you run. So if a user really wants to run mySQL locally then they could do the same.

I don't consider myself a programmer in this area but I do seem to be able to get by and am more comfortable doing so now that I have this test bed.
However I do recognise that many users simply want a turnkey easy to use system and this is a major 'selling point' for Cumulus. I think it is important for the Cumulus user base and for Steve's sanity that the target user base is properly catered for with any change. The excellent reputation that Cumulus has gained in this regard should not be jeopardised.

You may be aware that that I wanted to add some features that Cumulus did not provide for and so looked at WD. I found that to be a very salutory experience and just proved beyond any doubt the value of a clean system that does not step out of its 'comfort' zone. These features are now being developed separately from Cumulus and will be combined into the website in due course.

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Mon Jan 07, 2013 10:11 am

BCJKiwi wrote:the more limited SQlite (not sure what those limitations are).

The limitations of SqLite are a minor concern. It is not as scalable or secure as MySql, but for the vast majority of users, this qualifies as Not A Problem At All.
BCJKiwi wrote:the values of a clean system that does not step out of its 'comfort' zone.

:clap:
That may be just the most valuable remark in this entire discourse.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

Karv
Posts: 38
Joined: Sun Dec 16, 2012 11:19 pm
Weather Station: WH-1080
Operating System: Windows XP / 7, Ubuntu, Centos
Location: South Gloucestershire
Contact:

Re: Future data formats

Postby Karv » Thu Jan 10, 2013 9:13 pm

If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....

However from an actual development / realease and manageability point of view, its probably much more sensible to go with something which can be handled as total solution :)
Image

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Thu Jan 10, 2013 9:31 pm

Karv wrote:If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....


Nope.

However, my suggestion to use MySql, or SqLite, did not come from my desire as a user to have stuff in a database. I could do that myself just fine, and if necessary, I could release that under a GPL or LGPL license. Or I could just throw the source code out with support for as long as it takes you to download it and be damned with it... which is probably what I would do anyway, because I cannot rely on the realtime.txt schema in the first place. ;)

My idea is that, if today, I were to build, and support, something like Cumulus, I would use a database as a storage repository. My main reason for this would be that this would make maintenance and updates and extensions soooooooooooooooooooooooo much easier.

Disclaimer: I am not bashing Steve's architectural choices. I think Steve has done, and is still doing, a stellar job with Cumulus. While I might have made different choices, it is likely that I wouldn't have done such a stellar job.
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

davidrg
Posts: 3
Joined: Fri Nov 09, 2012 10:36 am
Weather Station: Vantage Vue
Operating System: Windows 7; Linux
Location: New Zealand

Re: Future data formats

Postby davidrg » Wed Jan 23, 2013 6:35 am

peterh wrote:I not the local license specialist, but if I remember correctly this time, the GPL says that if you distribute MySQL with your software, your software should be GPL as well.
LGPL doesn't have this restriction.


Almost. You can modify distribute GPL software as much as you like as long as you distribute the source code too and don't change the license. What you can't do is integrate GPL code (including libraries, etc) into proprietary software. I think the original idea was to make writing Free Software easier and cheaper than proprietary software making it the more attractive option.

Anyway, ---> PostgreSQL<---. Better reliability, a lot more features and a permissive MIT-style license that wouldn't conflict with Cumulus. SQLite is probably still easier and more appropriate than both of them for this particular embedded purpose though.

I'd think the easiest option would be to log all data to a SQLite database and provide an option (perhaps with a schedule?) to export data to CSV or whatever.

gemini06720
Posts: 1700
Joined: Mon Aug 10, 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: Future data formats

Postby gemini06720 » Thu Jan 24, 2013 10:38 am

I have been reading some interesting articles about organizations moving away from Oracle's MySQL to Monty Program's MariaDB, such as, for example, Wikipedia - there are also 'rumors' that Fedora Linux and OpenSUSE might also consider switching from MySQL to MariaDB.

I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).

MariaDB might be an interesting solution for storing the Cumulus data while keeping compatibility with the MySQL database format...

User avatar
steve
Cumulus Author
Posts: 26247
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in France
Contact:

Re: Future data formats

Postby steve » Thu Jan 24, 2013 10:57 am

That's interesting; the MariaDB client library is LGPL, and while it's written in C and Cumulus is not, I would still be able to use it with Cumulus (in the same way that I already use C DLLs like the Davis one), thus giving access to MySQL (and MariaDB) databases from Cumulus without any licensing issues.

I think I'd still want to use SQLite for local Cumulus storage, though (as MariaDB cannot be embedded).
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

peterh
Posts: 150
Joined: Fri Dec 21, 2012 1:08 pm
Weather Station: Alecto WS-5000 rebadged FO 3081
Operating System: Windows server 2008R2
Location: Nederland

Re: Future data formats

Postby peterh » Thu Jan 24, 2013 4:21 pm

Are you planning to use Entity Framework for database access?
An opinion should be the result of a thought process, not a substitution.
http://www.dnl-core.net/CothenWeather/

User avatar
steve
Cumulus Author
Posts: 26247
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in France
Contact:

Re: Future data formats

Postby steve » Thu Jan 24, 2013 5:14 pm

peterh wrote:Are you planning to use Entity Framework for database access?
I'm not really planning anything. But not in the current version of Cumulus, no; Cumulus is not a .NET application. If there was future version of Cumulus that was a .NET application, and had database support, I might consider using EF. And Mono has EF, so that's good. I think Cumulus 2 uses it, although ISTR there were some things for which I had to talk SQL directly to SQLite, for some reason. I think there were performance issues.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

User avatar
DigbyDave
Posts: 47
Joined: Sun Jan 09, 2011 11:55 am
Weather Station: Vantage Pro 2
Operating System: XP SP3
Location: AU

Re: Future data formats

Postby DigbyDave » Sat Mar 16, 2013 2:41 pm

gemini06720 wrote:I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).


I for one have been using MariaDB as a mySQL replacement for about 2 years, there is just something I don't like about Oracle. So far, I have not found any incompatibilities between MariaDB and mySQL, not that I'm a power user, just basic help desk, mail server, wordpress etc and testing public domain accounting software and such, no code changes have been required and all apps think they are talking to mySQL. Documentation, I fall back onto mySQL documentation if required.

Anyway, back to the subject. I personally would like to see some development towards some form of DB, not really fussy which, SQLite used to have table lock issues, but I think they have been resolved for some time.

Ultimately it's Steve's call.
Dave

Braitling, NT, Australia
INTBRAIT2

Metanges
Posts: 5
Joined: Tue Mar 06, 2012 8:19 am
Weather Station: WS-2355
Operating System: Windows 7 64 bits
Location: Rouen

Re: Future data formats

Postby Metanges » Tue Jan 21, 2014 8:15 pm

I appriciate very much SQLite databases.

I use it on some of my project, it is very easy to use with Visual Studio.

Thks

martinu
Posts: 35
Joined: Wed Jan 01, 2014 7:54 pm
Weather Station: Davis Vantage Vue
Operating System: Windows 7
Location: Welburn, North Yorkshire

Re: Future data formats

Postby martinu » Wed Apr 12, 2017 3:50 pm

I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.

RayProudfoot
Posts: 2404
Joined: Wed May 06, 2009 6:29 pm
Weather Station: Davis VP2 with Daytime FARS
Operating System: Windows XP SP3
Location: Cheadle Hulme, Cheshire, England
Contact:

Re: Future data formats

Postby RayProudfoot » Wed Apr 12, 2017 4:36 pm

martinu wrote:I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.


You're a bit late to the party. ;) Topic was launched over 4 years ago. No further development to Cumulus 1 I'm afraid.
Cheers,
Ray, Cheshire.

Image

User avatar
steve
Cumulus Author
Posts: 26247
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in France
Contact:

Re: Future data formats

Postby steve » Fri Apr 14, 2017 3:05 pm

Just to add to that - MX uses the same flat file format as Cumulus 1, and will possibly add local SQL logging in addition if I ever find the time to finish writing the code.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired


Return to “Cumulus 1”

Who is online

Users browsing this forum: No registered users and 7 guests