Page 3 of 4

Re: Future data formats

Posted: Mon 07 Jan 2013 8:51 am
by peterh
In order to update a MySQL database, Cumulus would have to be linked with a MySQL client library,
Which, regarding to this discourse, is fairly unimportant. I think MySql came into this thread because I suggested it :oops: , but I have since learnt that SqLite is a much better and more user-friendly idea to bundle than MySql.

But if you would not distribute said client library, you'd be ok offering MySql as a non-default option. If you would want to choose SqLite as the default, but would want to offer other DB engines as an alternative, and I would choose MySql, I would have to install the client myself. I would then have the dbProvider section in my machine.config, which negates the need for you to put it in your app.config. The only thing you'd have to come up with in your app.config is a valid connection string.
Now, of course, when there would be a difference in SQL syntax between the database engines (Obstacle comes to mind, but something simple like data type stuff can also upset us), the plot, she would thicken. Enter Entity Framework, which abstracts the database layer... and even when the storage model (and thereby also the mapping model) would change because some database engine only wants field names in uppercase, or insists on using NCLOB instead of NVARCHAR, we could generate storage models and mappings at design-time, rather than run time which is the default behaviour of EF...

For me, it would be ideal if the software would offer the option of using SqLite, or use a SQL or MySql database engine that I already have. But if Steve decides to keep it as it is, I'm not going to complain profusely. ;)

Re: Future data formats

Posted: Mon 07 Jan 2013 10:05 am
by BCJKiwi
Well in that case Steve, I return to my earlier option,
Don't use mySQL in Cumulus but provide for table structures and a 'friendly' mechanism that can be used to place that data into SQL on the website. These could be similar to (if not) the existing file structures, or, the more limited SQlite (not sure what those limitations are).

DAJ/Mark already have this working with the existing Cumulus file structures. I understand PHP has all the handlers required to read/write SQL and as far as I can see, as previously suggested, SQL is more important at the Website.
Most Hosters will have mySQL support on board.
In order to test the changes I make to scripts (and ensure I don't create problems at my hoster), I have installed XAMPP which includes Apache webserver, mySQL, PHP, and a host of other stuff all for free and can run on your Windows PC along side everything else you run. So if a user really wants to run mySQL locally then they could do the same.

I don't consider myself a programmer in this area but I do seem to be able to get by and am more comfortable doing so now that I have this test bed.
However I do recognise that many users simply want a turnkey easy to use system and this is a major 'selling point' for Cumulus. I think it is important for the Cumulus user base and for Steve's sanity that the target user base is properly catered for with any change. The excellent reputation that Cumulus has gained in this regard should not be jeopardised.

You may be aware that that I wanted to add some features that Cumulus did not provide for and so looked at WD. I found that to be a very salutory experience and just proved beyond any doubt the value of a clean system that does not step out of its 'comfort' zone. These features are now being developed separately from Cumulus and will be combined into the website in due course.

Re: Future data formats

Posted: Mon 07 Jan 2013 10:11 am
by peterh
BCJKiwi wrote:the more limited SQlite (not sure what those limitations are).
The limitations of SqLite are a minor concern. It is not as scalable or secure as MySql, but for the vast majority of users, this qualifies as Not A Problem At All.
BCJKiwi wrote:the values of a clean system that does not step out of its 'comfort' zone.
:clap:
That may be just the most valuable remark in this entire discourse.

Re: Future data formats

Posted: Thu 10 Jan 2013 9:13 pm
by Karv
If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....

However from an actual development / realease and manageability point of view, its probably much more sensible to go with something which can be handled as total solution :)

Re: Future data formats

Posted: Thu 10 Jan 2013 9:31 pm
by peterh
Karv wrote:If someone were to write a program which took say `realtime.txt` and updated a MySQL database, and published said program with a GNU GPL license, that wouldn't be in contravention of the original MySQL GPL license....
Nope.

However, my suggestion to use MySql, or SqLite, did not come from my desire as a user to have stuff in a database. I could do that myself just fine, and if necessary, I could release that under a GPL or LGPL license. Or I could just throw the source code out with support for as long as it takes you to download it and be damned with it... which is probably what I would do anyway, because I cannot rely on the realtime.txt schema in the first place. ;)

My idea is that, if today, I were to build, and support, something like Cumulus, I would use a database as a storage repository. My main reason for this would be that this would make maintenance and updates and extensions soooooooooooooooooooooooo much easier.

Disclaimer: I am not bashing Steve's architectural choices. I think Steve has done, and is still doing, a stellar job with Cumulus. While I might have made different choices, it is likely that I wouldn't have done such a stellar job.

Re: Future data formats

Posted: Wed 23 Jan 2013 6:35 am
by davidrg
peterh wrote:I not the local license specialist, but if I remember correctly this time, the GPL says that if you distribute MySQL with your software, your software should be GPL as well.
LGPL doesn't have this restriction.
Almost. You can modify distribute GPL software as much as you like as long as you distribute the source code too and don't change the license. What you can't do is integrate GPL code (including libraries, etc) into proprietary software. I think the original idea was to make writing Free Software easier and cheaper than proprietary software making it the more attractive option.

Anyway, ---> PostgreSQL<---. Better reliability, a lot more features and a permissive MIT-style license that wouldn't conflict with Cumulus. SQLite is probably still easier and more appropriate than both of them for this particular embedded purpose though.

I'd think the easiest option would be to log all data to a SQLite database and provide an option (perhaps with a schedule?) to export data to CSV or whatever.

Re: Future data formats

Posted: Thu 24 Jan 2013 10:38 am
by gemini06720
I have been reading some interesting articles about organizations moving away from Oracle's MySQL to Monty Program's MariaDB, such as, for example, Wikipedia - there are also 'rumors' that Fedora Linux and OpenSUSE might also consider switching from MySQL to MariaDB.

I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).

MariaDB might be an interesting solution for storing the Cumulus data while keeping compatibility with the MySQL database format...

Re: Future data formats

Posted: Thu 24 Jan 2013 10:57 am
by steve
That's interesting; the MariaDB client library is LGPL, and while it's written in C and Cumulus is not, I would still be able to use it with Cumulus (in the same way that I already use C DLLs like the Davis one), thus giving access to MySQL (and MariaDB) databases from Cumulus without any licensing issues.

I think I'd still want to use SQLite for local Cumulus storage, though (as MariaDB cannot be embedded).

Re: Future data formats

Posted: Thu 24 Jan 2013 4:21 pm
by peterh
Are you planning to use Entity Framework for database access?

Re: Future data formats

Posted: Thu 24 Jan 2013 5:14 pm
by steve
peterh wrote:Are you planning to use Entity Framework for database access?
I'm not really planning anything. But not in the current version of Cumulus, no; Cumulus is not a .NET application. If there was future version of Cumulus that was a .NET application, and had database support, I might consider using EF. And Mono has EF, so that's good. I think Cumulus 2 uses it, although ISTR there were some things for which I had to talk SQL directly to SQLite, for some reason. I think there were performance issues.

Re: Future data formats

Posted: Sat 16 Mar 2013 2:41 pm
by DigbyDave
gemini06720 wrote:I was going to try and install MariaDB, a drop-in replacement for MySQL (a MySQL fork), but, after going through the maze of 'geek' documentation (not much different than the Oracle's MySQL documentation), I postponed my installation ... until I could find some clear and concise offline documentation (such as in the PDF format - I really hate online knowledge bases).
I for one have been using MariaDB as a mySQL replacement for about 2 years, there is just something I don't like about Oracle. So far, I have not found any incompatibilities between MariaDB and mySQL, not that I'm a power user, just basic help desk, mail server, wordpress etc and testing public domain accounting software and such, no code changes have been required and all apps think they are talking to mySQL. Documentation, I fall back onto mySQL documentation if required.

Anyway, back to the subject. I personally would like to see some development towards some form of DB, not really fussy which, SQLite used to have table lock issues, but I think they have been resolved for some time.

Ultimately it's Steve's call.

Re: Future data formats

Posted: Tue 21 Jan 2014 8:15 pm
by Metanges
I appriciate very much SQLite databases.

I use it on some of my project, it is very easy to use with Visual Studio.

Thks

Re: Future data formats

Posted: Wed 12 Apr 2017 3:50 pm
by martinu
I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.

Re: Future data formats

Posted: Wed 12 Apr 2017 4:36 pm
by RayProudfoot
martinu wrote:I would definitely like flat file to continue (maybe in addition to SQL) as it is easier to import data into Excel or to an SQL database on a web hosting site.
You're a bit late to the party. ;) Topic was launched over 4 years ago. No further development to Cumulus 1 I'm afraid.

Re: Future data formats

Posted: Fri 14 Apr 2017 3:05 pm
by steve
Just to add to that - MX uses the same flat file format as Cumulus 1, and will possibly add local SQL logging in addition if I ever find the time to finish writing the code.