Welcome to the Cumulus Support forum.
Latest Cumulus MX V4 release 4.0.1 (build 4023) - 16 May 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V4 release 4.0.1 (build 4023) - 16 May 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Dodgy historical data aproach
Moderator: daj
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Dodgy historical data aproach
Hi all,
I looked all over, and I couldn't find a simple enough solution for displaying all of my historical data easily, so I built my own way, piggy backing completely off of Cumulus. I have many years of data in backups from various locations I have used Cumulus, in subfolders throughout my Cumulus installation folder, so this solution may not work for anyone else.
This was built in csharp (dotnet3.5), so it should work everywhere the old and new Cumulus works, although I have never used MX.
Source is included, using VisualStudio2008, but it is really just one Program.cs file, so you could rebuild it under any environment.
It is also plain simple jquery and Flot for graphing, so its all free.
The exe looks for ALL *log.txt files of the MONTHYEARlog.txt format, within the exe's folder and all its subfolders.
It only grabs min/max indoor and outdoor temperatures, and rainfall totals, and if duplicate files are found it simply merges them together.
It outputs these found datasets into a history.js javascript file in the same folder as the exe.
This history.js contains two variables, months and years, both just arrays of sets of data.
The history.htm file included simply uses the included jquery and flot dependencies, and the history.js as well and renders them.
To install, its a bit clunky,
1) Copy HistoryToJavascript.exe to you Cumulus folder.
2) Copy web files, history.html and js folder to your destination web site.
3) Configure Cumulus to execute external program HistoryToJavascript.exe, I did it first as Program to generate history.js more quickly, and afterwards set it to execute Daily.
4) Configure Cumulus to copy file history.js to destination website
5) Update all template webfiles; ie web\indexT.htm to include history link, ie:
<td colspan="4" class="td_navigation_bar">:<a href="index.htm">now</a>:...... to include :<a href="history.htm">history</a>:
I use ftp as my destination website, and the first update or daily rollover the history.js file does not exist, so none is copied, but one is created after this first update. During the second update the history.js file exists and is sent to the remote web server.
After that I set the exe to execute only Daily, so the whole of my 50Mb backup *log.txt data is only processed once a day.
Yes its plain and buggy and chunky and dodgy, but it works enough for me.
Zoom with mouse wheel (can't on my laptop!) and pan with click.
If anyone has any suggestions or could point me in a better direction to achieve this then tell me so
Cheers,
cehbab
I looked all over, and I couldn't find a simple enough solution for displaying all of my historical data easily, so I built my own way, piggy backing completely off of Cumulus. I have many years of data in backups from various locations I have used Cumulus, in subfolders throughout my Cumulus installation folder, so this solution may not work for anyone else.
This was built in csharp (dotnet3.5), so it should work everywhere the old and new Cumulus works, although I have never used MX.
Source is included, using VisualStudio2008, but it is really just one Program.cs file, so you could rebuild it under any environment.
It is also plain simple jquery and Flot for graphing, so its all free.
The exe looks for ALL *log.txt files of the MONTHYEARlog.txt format, within the exe's folder and all its subfolders.
It only grabs min/max indoor and outdoor temperatures, and rainfall totals, and if duplicate files are found it simply merges them together.
It outputs these found datasets into a history.js javascript file in the same folder as the exe.
This history.js contains two variables, months and years, both just arrays of sets of data.
The history.htm file included simply uses the included jquery and flot dependencies, and the history.js as well and renders them.
To install, its a bit clunky,
1) Copy HistoryToJavascript.exe to you Cumulus folder.
2) Copy web files, history.html and js folder to your destination web site.
3) Configure Cumulus to execute external program HistoryToJavascript.exe, I did it first as Program to generate history.js more quickly, and afterwards set it to execute Daily.
4) Configure Cumulus to copy file history.js to destination website
5) Update all template webfiles; ie web\indexT.htm to include history link, ie:
<td colspan="4" class="td_navigation_bar">:<a href="index.htm">now</a>:...... to include :<a href="history.htm">history</a>:
I use ftp as my destination website, and the first update or daily rollover the history.js file does not exist, so none is copied, but one is created after this first update. During the second update the history.js file exists and is sent to the remote web server.
After that I set the exe to execute only Daily, so the whole of my 50Mb backup *log.txt data is only processed once a day.
Yes its plain and buggy and chunky and dodgy, but it works enough for me.
Zoom with mouse wheel (can't on my laptop!) and pan with click.
If anyone has any suggestions or could point me in a better direction to achieve this then tell me so
Cheers,
cehbab
You do not have the required permissions to view the files attached to this post.
-
- Posts: 11
- Joined: Thu 27 Jun 2013 10:39 am
- Weather Station: WS 1081
- Operating System: Raspberry Pi
- Location: Shotton NE Wales
- Contact:
Re: Dodgy historical data aproach
Just for information..........I run MX on raspberry pi. HistoryToJavascript.exe runs (sudo mono History.....) and looks at all the log files. It does NOT however make history.js Have run the program several times. As I say, just for info. I don't know enough programming etc to be able to see were any problem may be.
Cheers............Geoff
Cheers............Geoff
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
If you place it in the same folder as your Cumulus, and run it from the shell, what info does it spit out ?
Do you have any MMMYYlog.txt files in that folder or below it in subfolders ?
For me, it prints out CWD and then all the matching log files that match MMMYYlog.txt
It might be that the path separator is not working under linux, or some other pathing issue.
Does it print anything out at all, and if so could you post that output here ?
Cheers.
Do you have any MMMYYlog.txt files in that folder or below it in subfolders ?
For me, it prints out CWD and then all the matching log files that match MMMYYlog.txt
It might be that the path separator is not working under linux, or some other pathing issue.
Does it print anything out at all, and if so could you post that output here ?
Cheers.
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
There was one bad path thing in the code, I looked up mono porting guidelines, and it told me to never use a \ in a path, so I removed it, using their recommended way of doing things.
Re-uploaded for you to try,
Only 2 small changes,
1) change path concatenation to remove hard-coded \, use Path.Combine(...) instead
2) print out that its writing history.js
Cheers.
New file attached.
Re-uploaded for you to try,
Only 2 small changes,
1) change path concatenation to remove hard-coded \, use Path.Combine(...) instead
2) print out that its writing history.js
Cheers.
New file attached.
You do not have the required permissions to view the files attached to this post.
-
- Posts: 11
- Joined: Thu 27 Jun 2013 10:39 am
- Weather Station: WS 1081
- Operating System: Raspberry Pi
- Location: Shotton NE Wales
- Contact:
Re: Dodgy historical data aproach
Thanks for the update. It now works and makes the file history.js I now need to make sure that MX runs the .exe file. Only other thing I want to look at is the width of the page. My cumulus site has images down each side which get in the way. The graphs look fine. Not put history.htm in my main menu yet, but you can still look at it. Its the /wx/ site.
Thanks for your time with this nice add on.
Cheers............Geoff
Thanks for your time with this nice add on.
Cheers............Geoff
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
Awesome, thanks for testing
The history.htm is very plain, hence I called it dodgy .
If you make something a bit more stylish, then give a shout out and upload it back here for me to use
Also, if you want more/different data in that history.js like humidity or whatever other column is available in the logs then tell me so and its easy to add. I only needed those indoor/outdoor temps and rainfall for now, and it keeps the .js file small.
My graph only had a hard-coded 80% width and a left of 10% set in the style in the .htm, quite dodgy
Also if anyone else wants to use this and the dates can't be read because of locale or something, tell me and I'll update it.
I was lazy and parsed the logs for my default date syntax only...
Cheers.
The history.htm is very plain, hence I called it dodgy .
If you make something a bit more stylish, then give a shout out and upload it back here for me to use
Also, if you want more/different data in that history.js like humidity or whatever other column is available in the logs then tell me so and its easy to add. I only needed those indoor/outdoor temps and rainfall for now, and it keeps the .js file small.
My graph only had a hard-coded 80% width and a left of 10% set in the style in the .htm, quite dodgy
Also if anyone else wants to use this and the dates can't be read because of locale or something, tell me and I'll update it.
I was lazy and parsed the logs for my default date syntax only...
Cheers.
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
Geoff,
Because your dataset is so small, only 2 months, I could spot errors in it very easily.
My dataset is over a 5 year span and I didn't notice that it was showing the data incorrectly.
Oh well..
I updated 2 lines in the htm file, to use the correct month offsets for displaying and bounding the x axis scrolling.
I also updated the csharp source to actually use maximum's instead of minimums for the months view. silly bug
The days data is now being displayed in the correct month, and you can scroll back to the beginning, instead of not showing or scrolling to that first month
Looking at your website, using MX which I had not seen before, it is using HighCharts.
If that is already being included by default in Cumulus MX then it should be quite easy to use that graph methodology to show all history quite easily.
My interface is crap, its the history.js that I wanted like yourself.
Because your dataset is so small, only 2 months, I could spot errors in it very easily.
My dataset is over a 5 year span and I didn't notice that it was showing the data incorrectly.
Oh well..
I updated 2 lines in the htm file, to use the correct month offsets for displaying and bounding the x axis scrolling.
I also updated the csharp source to actually use maximum's instead of minimums for the months view. silly bug
The days data is now being displayed in the correct month, and you can scroll back to the beginning, instead of not showing or scrolling to that first month
Looking at your website, using MX which I had not seen before, it is using HighCharts.
If that is already being included by default in Cumulus MX then it should be quite easy to use that graph methodology to show all history quite easily.
My interface is crap, its the history.js that I wanted like yourself.
You do not have the required permissions to view the files attached to this post.
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
Apologies again for another upload,
I restricted the axis to not be able to scroll past unavailable data to the left and right.
I would have updated the last entry over version _c, but it has been downloaded already...
Only history.htm was changed which is quite irrelevant.
Have a happy new years
I restricted the axis to not be able to scroll past unavailable data to the left and right.
I would have updated the last entry over version _c, but it has been downloaded already...
Only history.htm was changed which is quite irrelevant.
Have a happy new years
You do not have the required permissions to view the files attached to this post.
-
- Posts: 11
- Joined: Thu 27 Jun 2013 10:39 am
- Weather Station: WS 1081
- Operating System: Raspberry Pi
- Location: Shotton NE Wales
- Contact:
Re: Dodgy historical data aproach
Hi cehbab...................Thanks for the updates. All working ok as far as I can tell. Now need to sort MX to handle the .exe file each day.
Best wishes for new year..
Geoff
Best wishes for new year..
Geoff
-
- Posts: 225
- Joined: Sat 27 Aug 2016 12:11 am
- Weather Station: Davis vantage pro 2
- Operating System: Debian
- Location: Wickliffe, Ohio
- Contact:
Re: Dodgy historical data aproach
Seems to work well with my linux system very nice job.
I only have a few months of data so process went quickly, I dont know if you can change to just grab files from data dir and ignore the others, might help with speed for people with years of log files.
Thanks
I only have a few months of data so process went quickly, I dont know if you can change to just grab files from data dir and ignore the others, might help with speed for people with years of log files.
Thanks
Jeff
My Site http://wickliffeweather.com/
WeatherUnderground https://www.wunderground.com/personal-w ... KOHYOUNG21
My Site http://wickliffeweather.com/
WeatherUnderground https://www.wunderground.com/personal-w ... KOHYOUNG21
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
Thanks for the feedback,
I've updated it to support args from the command line
Any argument you add it will search in that folder, multiple args, multiple folders.
It still looks for all files MMMYYlog.txt in all subfolders within those given folders.
If you pass no args it does default and searches from the folder context Cumulus is running in.
Also, speed is not that big of an issue if you only run it daily.
Old Cumulus has that option in its configuration screen, I don't know about the new Cumulus MX though.
Cheers.
I've updated it to support args from the command line
Any argument you add it will search in that folder, multiple args, multiple folders.
It still looks for all files MMMYYlog.txt in all subfolders within those given folders.
If you pass no args it does default and searches from the folder context Cumulus is running in.
Also, speed is not that big of an issue if you only run it daily.
Old Cumulus has that option in its configuration screen, I don't know about the new Cumulus MX though.
Cheers.
You do not have the required permissions to view the files attached to this post.
- kocher
- Posts: 233
- Joined: Sat 19 Apr 2014 7:57 pm
- Weather Station: Davis Vantage Pro2+
- Operating System: Windows 10
- Location: San Sebastian - Spain
- Contact:
Re: Dodgy historical data aproach
Cheers Cehbab
The HistoryToJavascript.exe program stops with an error message when reading the first data file: "abr12log.txt"
I suspect that the cause is because the file has:
Decimal Sign: ,
Separator: ;
The HistoryToJavascript.exe program stops with an error message when reading the first data file: "abr12log.txt"
I suspect that the cause is because the file has:
Decimal Sign: ,
Separator: ;
-
- Posts: 8
- Joined: Thu 29 Dec 2016 5:42 am
- Weather Station: XC0348
- Operating System: Windows XP
Re: Dodgy historical data aproach
Hi Kocher,
Sorry for the delay, could you attach your data file here so I can get it to parse for you ?
Thanks..
cehbab
Sorry for the delay, could you attach your data file here so I can get it to parse for you ?
Thanks..
cehbab
- kocher
- Posts: 233
- Joined: Sat 19 Apr 2014 7:57 pm
- Weather Station: Davis Vantage Pro2+
- Operating System: Windows 10
- Location: San Sebastian - Spain
- Contact:
Re: Dodgy historical data aproach
Thank you very much for your reply.
This is the address of the file "abr12log.txt":
Http://kocher.es/cumulusMX/data/abr12log.txt
This is the address of the file "abr12log.txt":
Http://kocher.es/cumulusMX/data/abr12log.txt