Log analystic - apache pipe problem

Support & Bugs
1

Hi all

i tried to setup log analystics on my server.

my httpd.conf look like this :


LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" common
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" piwik

CustomLog "logs/access_log" combined
# Log to piwik as well
CustomLog "|/home/shomga/public_html/misc/log-analytics/import_logs.py --url=http://shomga.tk --add-sites-new-host --output=/home/shomga/public_html/piwik.log --enable-http-errors --enable-http-redirects --enable-static --enable-bots --force-lowercase-path --enable-http-redirects --enable-reverse-dns --config=/home/shomga/public_html/config/config.ini.php -" piwik

but nothing was imported to site.
i checked the output log and here is the error :


Traceback (most recent call last):
  File "/home/shomga/public_html/misc/log-analytics/import_logs.py", line 1758, in <module>
    config = Configuration()
  File "/home/shomga/public_html/misc/log-analytics/import_logs.py", line 571, in __init__
    self._parse_args(self._create_parser())
  File "/home/shomga/public_html/misc/log-analytics/import_logs.py", line 561, in _parse_args
    self.options.piwik_token_auth = self._get_token_auth()
  File "/home/shomga/public_html/misc/log-analytics/import_logs.py", line 652, in _get_token_auth
    credentials = open(filename, 'r').readline()
IOError: [Errno 2] No such file or directory: 'Content-type: text/html\r\n\r\n/home/shomga/public_html/tmp/cache/token.php'

this error repeated several times.
but the file exist in that directory. what is wrong with my setup.

regards

2

Can you try to run the import_logs as root, maybe it is a permission problem?

Of course don’t run it as root all the time, just for purpose of testing once :wink:

3

yes i tried it and it is fine.


python /home/shomga/public_html/misc/log-analytics/import_logs.py --url=http://shomga.tk --add-sites-new-host --enable-http-errors --enable-http-redirects --enable-static --enable-bots --force-lowercase-path --enable-http-redirects --enable-reverse-dns --config=/home/shomga/public_html/config/config.ini.php /usr/local/apache/logs/access_log



Logs import summary
-------------------

    658 requests imported successfully
    0 requests were downloads
    0 requests ignored:
        0 invalid log lines
        0 requests done by bots, search engines, ...
        0 HTTP errors
        0 HTTP redirects
        0 requests to static resources (css, js, ...)
        0 requests did not match any known site
        0 requests did not match any requested hostname

Website import summary
----------------------

    658 requests imported to 14 sites
        1 sites already existed
        13 sites were created:
            tr (ID: 2)
            s (ID: 3)
            e (ID: 4)
            g (ID: 5)
            d (ID: 6)
            1 (ID: 7)
            s (ID: 8)
            m (ID: 9)
            ra (ID: 10)
            r (ID: 11)
            h (ID: 12)
            k (ID: 13)
            s (ID: 14)
    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 31 seconds
    Requests imported per second: 20.72 requests per second
4

So it works when running as root and not working when running as normal user? that’s interesting… we would need to think of detecting this and printing a useful error message.
maybe you could create a ticket in the issue tracker?

5

Hi, first thank you for all the work and sorry if I reopen a subject, but I am running into the same issue that has been reported by BaharNaranj and I didn’t found a open ticket about it.
Should I open one ?

My configuration is

Piwik 2.3
CentOS 6.5 (SELinux on)
Python 2.6.6

The error message I run into is :


Traceback (most recent call last):
  File "/var/www/html/piwik/misc/log-analytics/import_logs.py", line 1761, in <module>
    config = Configuration()
  File "/var/www/html/piwik/misc/log-analytics/import_logs.py", line 571, in __init__
    self._parse_args(self._create_parser())
  File "/var/www/html/piwik/misc/log-analytics/import_logs.py", line 561, in _parse_args
    self.options.piwik_token_auth = self._get_token_auth()
  File "/var/www/html/piwik/misc/log-analytics/import_logs.py", line 652, in _get_token_auth
    credentials = open(filename, 'r').readline()
IOError: [Errno 13] Permission denied: '/var/www/html/piwik/tmp/cache/token.php'
piped log program '/var/www/html/piwik/misc/log-analytics/import_logs.py --url=http://xxxx.xxxxx.fr/piwik --idsite=1 --enable-bots --enable-http-redirects --enable-static -' failed unexpectedly

The file ‘/var/www/html/piwik/tmp/cache/token.php’ exist and has all permissions set for apache user to run it.

Thank you for any help.

6

I resolved the problem for my installation.
I can recommand to SeLinux user trying to configure piwik this way to specify the login, password and log format used in the import_logs params. (see script)

7

A solution is to use the


--token-auth=

parameter with the import-logs.py script. Tested with a Strato hosted website.