Error in import_logs.py after upgrade to version 2.13.0

After upgrading piwik from 2.12.0 to 2.13.0 we get the following error when import apache logs using import_logs.py:

Traceback (most recent call last):
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 2159, in
config = Configuration()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 822, in init
self._parse_args(self._create_parser())
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 805, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 841, in _get_token_auth
_url=self.options.piwik_url,
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1252, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1211, in _call_wrapper
response = func(*args, **kwargs)
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1196, in _call_api
res = Piwik._call(’/’, final_args, url=url)
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1157, in _call
response = urllib2.urlopen(request, timeout = config.options.request_timeout)
NameError: global name ‘config’ is not defined

We call the script using the following parameter:

        [i]/var/www/html/piwik/misc/log-analytics/import_logs.py -d --login=XXXX --password=XXXXXX --url=http://localhost/piwik $logfile --idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots &>>$PIWIK_LOG[/i]

With version 2.12.0 we used the same call without having any problems.
Do you have any idea?

Cheers
Hartmut

After further analyzing the issue we found the reason for the NameError:

When we start piwik, the global variable ‘config’ is used while defining it.
In import_logs.py line 2158 the definition is called:
config = Configuration()

But while creating this object in import_logs.py line 1157 the
global variable ‘config’ is already used:

response = urllib2.urlopen(request, timeout = config.options.request_timeout)

Our quick Workaround in import_logs.py:
headers[‘User-Agent’] = 'Piwik/LogImport’
request = urllib2.Request(url + path, data, headers)
try:
response = urllib2.urlopen(request, timeout = config.options.request_timeout)
except NameError:
response = urllib2.urlopen(request)

Now import_logs.py runs for us but I suggest to treat this as a bug which should be solved.

Cheers
Hartmut

Hi Hartmut

thanks for the report, i’ve created an issue here: NameError: global name ‘config’ is not defined · Issue #74 · matomo-org/piwik-log-analytics · GitHub feel free to comment there

Got the same problem but i did not understand the solution. I did not understand what i have to replace.

If i do what i understand i get an:
[quote=“File “/home/www/piwik/misc/log-analytics/import_logs.py”, line 1170
except NameError:”]
[/quote]

Bug in Github looks closed or whatever.

Hi Hula,
I am not sure if you really have the same issue.
Please verify if it is the variable config in the urlopen statement which causes the Problem.
As a Workaround we catch the Name error exception.
There we added the 3 lines
Try:
and
except Name error
and
Response = urllib2.urlopen(request)
like this:
try:
response = urllib2.urlopen(request, timeout = config.options.request_timeout)
except NameError:
response = urllib2.urlopen(request)

In the exception block we call the urlopen Statement without the config Parameter. In our case this Approach solves the issue.

In sorry, not sure where.


1166         headers['User-Agent'] = 'Piwik/LogImport'
1167
1168         request = urllib2.Request(url + path, data, headers)
1169         opener = urllib2.build_opener(Piwik.RedirectHandlerWithLogging())
1170         response = opener.open(request, timeout = config.options.request_timeout)
1171         result = response.read()
1172         response = urllib2.urlopen(request, timeout = config.options.request_timeout)
1173         except NameError:
1174         response = urllib2.urlopen(request)
1175         response.close()
1176         return result


Give me an error


 File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1173
    except NameError:
         ^
SyntaxError: invalid syntax
xb2105@xb2:~$


Im on the latest version and that what i get since some days (havent check that due to work)


Traceback (most recent call last):
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 2159, in <module>
    config = Configuration()
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 822, in __init__
    self._parse_args(self._create_parser())
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 805, in _parse_args
    self.options.piwik_token_auth = self._get_token_auth()
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 841, in _get_token_auth
    _url=self.options.piwik_url,
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1252, in call_api
    return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1211, in _call_wrapper
    response = func(*args, **kwargs)
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1196, in _call_api
    res = Piwik._call('/', final_args, url=url)
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1157, in _call
    response = urllib2.urlopen(request, timeout = config.options.request_timeout)
NameError: global name 'config' is not defined

Hi Hula,
you did not add all suggested lines:
try:
response = urllib2.urlopen(request, timeout = config.options.request_timeout)
except NameError:
response = urllib2.urlopen(request)

You forgot the line Try:. Also keep in mind that line indention is important in python.
Cheers
Hartmut

Hi,
ok, make sense. Guess python or whatever of my shared Space is not up to date. It works since years…but now its gone.


xbyyyyyy@xb2:~$ python /home/www/piwik/misc/log-analytics/import_logs.py --url=XXXXXXXXXX --login=XXXXXXXX--password=XXXXXXXX--idsite=1 --enable-reverse-dns /home/log/access_log_10.gz --recorder-max-payload-size=60
  File "/home/www/piwik/misc/log-analytics/import_logs.py", line 1170
    response = urllib2.urlopen(request, timeout = config.options.request_timeout)
           ^
IndentationError: expected an indented block
xbyyyyy@xb2:~$

Will be better to wait for the next version. Can i use an older version of that importer?

Update: Answer to myself. Version 2.12 of that package works.

Got the same issue with "NameError: global name ‘config’ is not defined "

Bug still in the current nightly build :confused:

Still seeing this as well. The fix above doesn’t seem to work.

Just take an older version. Extract it and upload ist. Works for me.

http://builds.piwik.org/

Hi there,
it should be fixed in latest Piwik beta version 2.14.0 - we will next week release the final version but the beta already works well

I updated to the beta because I was having the same problem - now getting the following message when importing:

“Error when connecting to PIWK: HTTP Error 500: Internal Server Error”

FROM SERVER LOGS:
2015-07-05 17:20:38 10.208.128.187 POST / - 80 - 10.208.128.187 Piwik/LogImport 200 0 0 312
2015-07-05 17:20:38 10.208.128.187 POST / - 80 - 10.208.128.187 Piwik/LogImport 200 0 0 296
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 218
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 234
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 249
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 249
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 249
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 265
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 327
2015-07-05 17:20:41 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 296
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 265
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 249
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 265
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 265
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 280
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 265
2015-07-05 17:20:51 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 280
2015-07-05 17:20:52 10.208.128.187 POST /piwik.php - 80 - 10.208.128.187 Piwik/LogImport 500 0 0 312

The website is working just fine. Can log into PIWIK both externally and on the box that is doing the imports.

Any help or direction would be appreciated.

edited

Hi there,

“Error when connecting to PIWK: HTTP Error 500: Internal Server Error”

please check your piwik server error logs, do you find any information there?

Still having this problem with the 2.16.2 version of piwik.
c:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py --url=https://myurl/Analytics/piwik/ c:\inetpub\wwwroot\Analytics\log1.log --login=xxxxxxxxxxx --password=xxxxxxxx --idsite=2 --enable-http-errors
2016-08-29 11:49:19,707: [INFO] Error when connecting to Piwik: HTTP Error 405: Method Not Allowed
Traceback (most recent call last):
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 2336, in
config = Configuration()
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 885, in init
self._parse_args(self._create_parser())
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 865, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 904, in _get_token_auth
_url=self.options.piwik_url,
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 1366, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “C:\inetpub\wwwroot\Analytics\piwik\misc\log-analytics\import_logs.py”, line 1350, in _call_wrapper
if errors == config.options.max_attempts:
NameError: global name ‘config’ is not defined

This is a brand new install. I don’t even know where to start on the fix for this any thoughts?

Patrick, I am currently mit Wirkung with piwik. So I’m afraid but I cannot
help.
Cheers

Finally got it working! My solution was to make sure to set index.php as your default document in the IIS site you are running Piwik from.