Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

a PHP import script for DayFile and Monthly log files

Discussion and support for 3rd-party (non-Sandaysoft) tools for Cumulus
User avatar
nitrx
Posts: 1300
Joined: Sun Dec 13, 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby nitrx » Sat Jul 24, 2010 9:35 am

Arnie wrote:Ive now changed back to "localhost" but still get the failing to connect to server msg as above

Ron pls check my inputs on the ImportCumulusFile.php

$key = "xxxxxxx"
$dbhost = "localhost"
$dbuser = "xxxx@wentworthvilleweather.com" my server username
$dbpassword = "xxxxxx" my server password
$database = "wentydb"

Is this the correct way?


This should be the correct way well if you have made at least the database and the database-user , note that the user must have all acces to the database BTW. I don't know the config of your database server

Arnie
Posts: 85
Joined: Tue Jan 05, 2010 9:45 am
Weather Station: La Crosse Ws 2355
Operating System: Windows XP SP3
Location: Wentworthville, Sydney Australia
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby Arnie » Sat Jul 24, 2010 10:49 am

nitrx wrote: note that the user must have all acces to the database BTW. I don't know the config of your database server


I'm not too sure what you mean by config of database server!!
Mysql database is listed as wentworthvilleweather_com_wentydb

I can't access the myadminphp page ???

Just a little confused
Cheers
Brian
Sydney Australia

http://www.wentworthvilleweather.com

Image

User avatar
nitrx
Posts: 1300
Joined: Sun Dec 13, 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby nitrx » Sat Jul 24, 2010 10:59 am

Arnie wrote:
nitrx wrote: note that the user must have all acces to the database BTW. I don't know the config of your database server


I'm not too sure what you mean by config of database server!!
Mysql database is listed as wentworthvilleweather_com_wentydb

I can't access the myadminphp page ???

Just a little confused


In that case you should change the phpfile settings to
$database ="wentworthvilleweather_com_wentydb";

You're hosting by David I think you should mail him ?

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Tue Aug 03, 2010 10:37 pm

Not sure if this has been asked or answered before, but having done a qujick search I can't locate it.

Has anyone used, or modified this script to import data from other weather software packages? For example, I have data from Jan 2003 until a few days ago that was collected with FreeWx, earlier data for about 18 months from WD and a fwe months before that from VWS. Having recently upgraded the server the weather station runs on, FreeWx would no longer run (also no longer supported!); WD is one of the MOST convoluted software programs I have ever used to try and set up (plus it has a serious problem importing FreeWx data) and VWS can't run for more than 5 minutes without crashing!) (An aside - I like the cumulus approach - simple, easy to use and doesn't try to cater for 528 weather stations and have 8039 different configuration settings!)

Anyway - whilst I have been able to update the all time records by hand, I still want to have my old data available - so the idea of this script seems ideal to import the FreeWx, WD and VWS files.

Ha anyone done this? With this, or any other script? If not - guess I have a job for the next few days :)
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Wed Aug 04, 2010 6:21 am

Started to implement this script this morning and had a few problems - some from the script itself, others possibly from the combination I use..

First - the script contains an example that obviously threw nitrix. The example given to call the script is shown thus:

// htp://www.myserver.com/ImportCumulusFil ... ayfile.txt

Note the htp instead of http - not a big problem but can obviously catch some people out.

Second - my setup is a Windows 2008 R2 server running IIS 7. I had all sorts of problems even getting the script to run but after a bit of messing around, finally got it to run - sort of.

First - I had to create the tables manually. I am running MySQL 5.0.51 and it doesn't like the COMMENT at the end of the CREATE statements! Took me a while to figure this out (I naturally assumed the statements were all OK). Once I removed this, it created them BUT, once the tables were created, everytime the script was run I ended up with a server 500 error - only by commenting out the CREATE lines could I get the script to successfully run onm this combination (W2008R2, MySQL 5.0.51, PHP 5.2.13).

Now the other quesiton - and possibly a really silly one - using the monthly file is straight forward (although the script doesn't really tell you which one to use, neither does the Wiki article).

But....

What is the name of the day file you process with it? I know this sounds stupid but given the script and examples all use the name 'data/dayfile.txt' and that this file 'doesn't exist (or isn't generated),I had to ask - maybe I am missing something obvious, but there doesn't appear to be a file being generated with the layout required for the daily input (or have I missed a configuration setup somewhere?)
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby steve » Wed Aug 04, 2010 7:40 am

serowe wrote:Now the other quesiton - and possibly a really silly one - using the monthly file is straight forward (although the script doesn't really tell you which one to use, neither does the Wiki article).
The 'monthly log files' are these: http://wiki.sandaysoft.com/a/Monthly_log_files. The instructions can't tell you which one to use, as that depends on which month you're trying to import. The file name depends on the particular month and on the the language setting on your PC.
What is the name of the day file you process with it? I know this sounds stupid but given the script and examples all use the name 'data/dayfile.txt' and that this file 'doesn't exist (or isn't generated),I had to ask - maybe I am missing something obvious, but there doesn't appear to be a file being generated with the layout required for the daily input (or have I missed a configuration setup somewhere?)
The instructions are correct, it's called dayfile.txt and it's in the data folder.

As you're effectively running Windows 7, where have you installed Cumulus? Are you looking in the right place for the data logs?
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Wed Aug 04, 2010 7:55 am

Hmm - the monthly files aren't a problem. The daily files though...

All I see in the \cumulus\data directory are:

Aug10log.txt
today.ini
alltime.ini
altimelog.txt
alltimelog.txt
yesterday.ini
log.xml
and
dayfile.txt BUT this is time stamped at 09:00 am this morning - so I am guessin ghtne that this is retro written ie it isn't kept up to date during the day?

And the prog is installed in c:\cumulus on the server (which is another issue because I would like to reference this daily file but the script doesn't seem to like being pointed to another drive (I have the web site on drive e:)
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby steve » Wed Aug 04, 2010 8:00 am

serowe wrote:dayfile.txt BUT this is time stamped at 09:00 am this morning - so I am guessin ghtne that this is retro written ie it isn't kept up to date during the day?
Yes, it's a summary of the entire day, created at the end of the day. The today.ini file is effectively the equivalent that's updated during the day.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Wed Aug 04, 2010 8:45 am

Actually - just got to thinking about this.

The dayfile.txt I have is one I copied to ensure I had one by that name - ie it wasn't written this morning.

The data is from my monthly file - so what I have actually been testing is my monthly file because, from what I can see, there is no file available conforming to the structure for dayfile.txt in this directory.

Is there something I should check to ensure that this file is written tomorrow morning at changeover time? Given that I only started cumulus yesterday afternoon my time, should I have expected one to be written this morning? And, another stupid question, it is written at changeover time (9am) and not midnight?
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby steve » Wed Aug 04, 2010 9:15 am

serowe wrote:Is there something I should check to ensure that this file is written tomorrow morning at changeover time? Given that I only started cumulus yesterday afternoon my time, should I have expected one to be written this morning?
Yes, at the end of the meteorological day, it should have created the file if it didn't exist and written the first entry (which wouldn't have been for a full day - and I often advise people to delete the very first entry in dayfile.txt so it doesn't skew their data). If it didn't do that, zip up the diags folder and attach it - it's probably best to start a new thread in the Oregon section - and I'll see what happened. Cumulus was running at the time it should have written the entry?
And, another stupid question, it is written at changeover time (9am) and not midnight?
I don't understand the question. Have you chosen 0900 as your start/end of meteorogical day? If so, it writes the summary at 0900. Why would you expect it to write it at midnight?
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Wed Aug 04, 2010 9:58 am

steve wrote:
serowe wrote:
And, another stupid question, it is written at changeover time (9am) and not midnight?
I don't understand the question. Have you chosen 0900 as your start/end of meteorogical day? If so, it writes the summary at 0900. Why would you expect it to write it at midnight?

You answered it (0900) - but in the station configuration there is the option to roll over logs at 9am or midnight - I just wanted to confirm that this particular file followed the config setting.

Re the dayfile - I'll wait until tomorrow morning and see what is created before posting in the Oregon section.
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

User avatar
steve
Cumulus Author
Posts: 26314
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: On tour in Portugal
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby steve » Wed Aug 04, 2010 10:13 am

serowe wrote:I just wanted to confirm that this particular file followed the config setting.
Right, OK. Maybe you haven't seen the relevant bit of the help file yet, where it says (in the data logs section): "The first is a single file called dayfile.txt ; an entry is made to this file every day at midnight or 0900 (1000 in summer) depending on the chosen setting on the Station Configuration screen, summarising the day's data."
Re the dayfile - I'll wait until tomorrow morning and see what is created before posting in the Oregon section.
OK.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Thu Aug 05, 2010 10:54 am

Forgot to mention - file was created on time this orning. Don't know what happened yesterday but - no great loss.
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

serowe
Posts: 484
Joined: Tue Aug 03, 2010 6:23 am
Weather Station: WM918
Operating System: Win Server 2008 R2
Location: Ferntree Gully, VIC, Oz
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby serowe » Fri Aug 06, 2010 9:26 am

David - just struck a problem importing some historical data that needs a change to the structure of one of the fields in the table.

Raincounter is current 6,2 field - in 2008 Melbourne had over 1000 mm of rain - it got to that figure on Dec 2 at 1813 - which is when the import starts to fail :)

You might want to consider upping this a notch to 7,2 - we might have all 4 seasons in one day, but we also get some heavy consistent rain at times :lol:
Punctuation is the difference between 'Let's eat, grandma' and 'Let's eat grandma'

User avatar
nitrx
Posts: 1300
Joined: Sun Dec 13, 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Postby nitrx » Sat Oct 16, 2010 10:38 am

daj wrote:I have written a PHP script to read the contents of the dayfile.txt or a Monthly log file and import these into a MySQL database on your webserver.

Handy if you want to get the Cumulus data onto the web to query it and write your own reports/graphs directly from SQL

Full details on the wiki http://wiki.sandaysoft.com/a/ImportCumulusFile

As an example, I upload my dayfile.txt just after midnight each day and then process it using this script. The result is my Historic Data page ... http://www.grantownweather.co.uk/historic/historic.php... which reader the data live from the SQL database and presents it in a table for viewing.

All my monthly log files are also in SQL so I am working on analysis tools for those too

David can you change the wiki for the import rule it says 'htp://www.myserver.com/ImportCumulusFile.php?type=dayfile&key=letmein&table=Dayfile&file=data/Jan10log.txt ' I had lost it but I saw a trailing point was missing the example should be I think

Code: Select all

htp://www.myserver.com/ImportCumulusFile.php?type=dayfile&key=letmein&table=Dayfile&file=./data/Jan10log.txt
?


Return to “3rd Party Tools”

Who is online

Users browsing this forum: No registered users and 2 guests