Importing IIS logs

Hi, Everyone. I’ve been spending the last couple of days perusing the forums and installation and FAQ pages regarding importing IIS logs. I’ve successfully installed PiWik along with the respective MySQL, PHP and Python needs and am able to successfully log on to the application. I’ve successfully tested the JavaScript with a live site and created another group for importing IIS logs. I’ve successfully ran the Python script and confirmed successful data importing to MySQL via phpMyAdmin. However, when I attempt to load the dashboard of the group that the logs are attached to, I’m not seeing any data. I’ve confirmed that PiWik seems to be recognizing the data exists as I’m able to select the dates of the respective log files, however, the dashboard widgets show up empty.

If anyone would like specific information, please let me know.

Thanks much, in advance, for any insight anyone might offer.

Regards,
Kory

Bump.

Did you setup: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo ?

Also check your error logs maybe? it should work for sure

Hi, Matt. Thanks much for replying. I had not set up auto archiving previously. I wasn’t sure if that was needed for importing old log files. I’ve since done that and will see if that’ll help over the next day. I don’t see anything out of the ordinary in my event viewer logs, specifically nothing related to Piwik.

Regards,
Kory

Hi,
I’m also trying to import IIS logs but I’m stuck at running the Python script. I’ve installed Python 2.7 and have downloaded the latest committed version of import_logs.py. But when I run the command below, I get the following console output:

C:\Python27>python “d:\websites\piwik\misc\log-analytics\import_logs.py” --url=h
ttp://piwikpre.unaids.org/ “d:\tmp\logfiles\ex120831.log” --idsite=2
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log d:\tmp\logfiles\ex120831.log…
4023 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
8180 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current
)
12669 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (curren
t)
17105 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (curren
t)
20995 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (curren
t)
25486 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (curren
t)
29775 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (curren
t)
Fatal error: None
You can restart the import of “d:\tmp\logfiles\ex120831.log” from the point it f
ailed by specifying --skip=0 on the command line.

Am I missing something here? If it would help, here’s the header of the IIS log file I’m using:
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2012-08-31 00:00:01
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes time-taken

Appreciate your help on this.

Best regards,
-Josephine

Josephine, please post a sample log file to reproduce the issue, with the log format definition, and post your bug report in: Log analytics list of improvements · Issue #3163 · matomo-org/matomo · GitHub

Thanks!

Did you ever get a solution to this problem?

I’ve been fighting this error for a week now. Exactly the same response. I’m about to start debugging the python line by line.

Thanks.

Hi,

I don’t know if it’s the same for you guys, but we are getting the error shown above too. No lines recorded and Fatal Error:None.

We ‘think’ we have worked out that if you import a log file of a certain date, Jan 2012 for example it will work. Now if you try and import a log file previous to that, Dec 2011 for example it fails with the above message. Any log file that has a later date than your last successful import seems to work.

Typically we tried out the script with a date much later than the first log file so now we are stuck :slight_smile:

Would be interesting to know if that is the same for you and if maybe it is by design and there is a workaround.

Thanks
Simon

This shouldn’t be right, Piwik should work if importing past dates as well.

Can you please prepare a very small log file (eg. with 3-4 lines) and another one that make the problem reproducible? if you can reproduce, we can create a tickte and fix it

Same problem here, I can’t import log files older than the newest log file imported.

C:\Python27>python C:\Inetpub\piwik\misc\log-analytics\import_logs.py --url=http://piwik/ D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log --i
dsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --skip=11
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log…
3323 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3323 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3323 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3323 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Fatal error: None
You can restart the import of “D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log” from the point it failed by specifying --skip=11 on the comma
nd line.

More proof the issue is with the older date, if I use --dry-run it works:

C:\Python27>python C:\Inetpub\piwik\misc\log-analytics\import_logs.py --url=http://piwik/ D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log --i
dsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --dry-run
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log D:\IISLOGS\GEMINI\W3SVC1908953133\ex121101.log…
4275 lines parsed, 4000 lines recorded, 4000 records/sec (avg), 4000 records/sec (current)
8734 lines parsed, 8000 lines recorded, 4000 records/sec (avg), 4000 records/sec (current)
13116 lines parsed, 12000 lines recorded, 4000 records/sec (avg), 4000 records/sec (current)
17372 lines parsed, 16000 lines recorded, 4000 records/sec (avg), 4000 records/sec (current)
21444 lines parsed, 20117 lines recorded, 4023 records/sec (avg), 4117 records/sec (current)
25578 lines parsed, 24800 lines recorded, 4133 records/sec (avg), 4683 records/sec (current)
29716 lines parsed, 28800 lines recorded, 4114 records/sec (avg), 4000 records/sec (current)
33958 lines parsed, 32800 lines recorded, 4100 records/sec (avg), 4000 records/sec (current)
38288 lines parsed, 36800 lines recorded, 4088 records/sec (avg), 4000 records/sec (current)
42432 lines parsed, 40800 lines recorded, 4080 records/sec (avg), 4000 records/sec (current)
46507 lines parsed, 44966 lines recorded, 4087 records/sec (avg), 4166 records/sec (current)

236146 lines parsed, 232800 lines recorded, 4105 records/sec (avg), 4000 records/sec (current)
240329 lines parsed, 237600 lines recorded, 4117 records/sec (avg), 4800 records/sec (current)
244506 lines parsed, 241600 lines recorded, 4115 records/sec (avg), 4000 records/sec (current)

Logs import summary

243373 requests imported successfully
999 requests were downloads
2712 requests ignored:
    17 invalid log lines
    2695 requests done by bots, search engines, ...
    0 HTTP errors
    0 HTTP redirects
    0 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

243373 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 59 seconds
Requests imported per second: 4119.52 requests per second

C:\Python27>

I need some help here too. I am following the examples, have everything setup as per the instructions but I am unable to run the import script at all. Is there a different windows format we are supposed to use?

C:\Python33>python “C:\Inetpub\wwwroot\piwik\misc\log-analytics\import_logs.py”
–url=subdomain.abc.co.uk E:\LOGFILES\CTN-VMWCWEB01\W3SVC2\u_ex120501.log --enable-
http-errors --enable-http-redirects --enable-static --enable-bots
File “C:\Inetpub\wwwroot\piwik\misc\log-analytics\import_logs.py”, line 430
except Piwik.Error, e:
^
SyntaxError: invalid syntax

Thanks

upgrade your python

Hi,

I can agree with this, I’m getting the same problem. :S

If I move forward in time it runs, however I’ve not managed to get it to run a full log yet. It seems to stop with a Fatal Error: None problem. The debug shows the message 500 Internal Server Error a few lines before hitting a Piwik urlopen error. Then the log wont run again. Nor will logs that are older.

Anyone managed to get round this yet ?

Thanks

Ed

I don’t know much about python so can’t troubleshoot the code too well.

import_logs.py dont support Pythom 3.3.

Try with Python 2.7

I’m already using 2.7.3

Not v3 or higher :frowning:

Have upgraded to latest version 1.11 and still the same issue. :X

We are still waiting for log files to reproduce this problem. We are testing against IIS logs already but the import works. So anyone who experiences this problem: Please provide us a log file (with like 10 entries) that produced this error.

You can send the log to fabian [at] piwik.org or post it at dev.piwik.org.

Hi,

Log sent. This is a NCSA extended format log. Haven’t managed to get import routine to even try a IIS one yet at all (see other thread).

Thanks

Ed