We are using Piwik 1.8.2 on a CentOS-Box with PHP 5.1.6 and are using the PiwikAPI to create custumized reports for our clients.
After updating from 1.7.x there is an error in the API. The following call
/index.php?module=API&segment=&method=Actions.getPageUrls&format=xml&idSite=8&expanded=1&date=2011-04-01,2012-06-30&period=range&token_auth=mytoken
produces the following error:
*There is an error. Please report the message (Piwik 1.8.2) and full
backtrace in the Piwik forums
Dear developers,
first of all thanks for all your energy and work you put into this tool!
Unfortunately we face the same issue, and it might be special characters like Chinese, Russian, or Japanese signs we saw as entry in Piwik’s Action Section. Before the new version we escaped special characters ourselves, and these characters were the vertical tab (vt) and “/>” as entry in the Action Section:
private void ReplaceSpecialChars(string filepath, long linenumber)
{
System.IO.StreamReader strm;
string strline;
string strreplace = " ";
//locally
string tempfile = “D:\Temp.xml”;
try
{
System.IO.File.Copy(filepath, tempfile, true);
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
System.IO.StreamWriter strmwriter = new System.IO.StreamWriter(filepath);
strmwriter.AutoFlush = true;
strm = new System.IO.StreamReader(tempfile);
long i = 0;
while (i < linenumber - 1)
{
strline = strm.ReadLine();
strmwriter.WriteLine(strline);
i = i + 1;
}
strline = strm.ReadLine();
Int32 lineposition;
lineposition = strline.IndexOf((char)11);//replace character (vt), Oct=0013 Hex=0x0b
if (lineposition > 0)
{
strreplace = “”;
}
//not sure why we need this either…:
else
{
lineposition = strline.IndexOf("<", 1);
if (lineposition > 0)
{
strreplace = “<”;
}
}
strline = strline.Substring(0, lineposition - 1) + strreplace + strline.Substring(lineposition + 1);
//hack:
//not sure why we need it here, but it works 
//1_2011_8_getVisitDurationAndBounceRate.xml caused problems. In Piwik, the entry that caused problems was:
// /> 1 1 0% 25s 0%
// right behind
// motorrad 2 1 0% 15s 100%
// pdf 2 2 0% 12s 0%
if (strline.Contains("“winking smiley == false)
strline = strline.Replace(”<label", ""winking smiley;
//end hack
strmwriter.WriteLine(strline);
strline = strm.ReadToEnd();
strmwriter.WriteLine(strline);
strm.Close();
strm = null;
strmwriter.Flush();
strmwriter.Close();
strmwriter = null;
}
Help would be highly appreciated !
Greetings from Munich!
Thanks for the report. There is a ticket at: Warning: htmlspecialchars() Invalid multibyte sequence in argument in core/DataTable/Renderer.php on line 223 · Issue #3259 · matomo-org/matomo · GitHub
I put a quick fix in the comment. But i’m interested if you manage to reproduce the issue on a new day with a specific request - then we can try to fix it properly hopefully
Thank you, Matt, for your fast response, your fix and opening a ticket! We appreciate it!
One thing at your URL-example in the ticket: the attribute expanded=1 didn’t show up, not sure if this is a problem in reproducing the error…
@ tvc: Do you have capacity to test matt’s quick fix? Are you admins on your machine? I’m asking as we have to ask a third-party service company which host our applications. They are fast but still it would take some more time.
Thanks!
Hi,
well this quick fix seems to work for me now. Although suppressing the error seems indeed a “quick fix” and is just a way to ignore the error. Hopefully the resulting XML is still valid so the program can interpret it… but that I can handle on the client side.
Thanks so far for the quick fix!
Cheers!
Yes, thanks, Matt!
And tvc - thank you for testing!
Have a good week.
To all users,
Can you please try the following patch:
Index: core/DataTable/Renderer.php
===================================================================
--- core/DataTable/Renderer.php (revision 6483)
+++ core/DataTable/Renderer.php (working copy)
@@ -220,7 +220,7 @@
&& !is_numeric($value))
{
$value = html_entity_decode($value, ENT_QUOTES, 'UTF-8');
- $value = htmlspecialchars($value, ENT_COMPAT, 'UTF-8');
+ $value = htmlspecialchars($value, ENT_QUOTES, 'UTF-8');
We are hoping this could solve the problem and the error message.
Please test and confirm as it is really important (we still haven’t found the solution!)
Thanks!
Dear Matt,
thanks for your patch - but it doesn’t seem to work. When running my report, the following error message is produced - which seems to be the same as with the orginal code.
<div style='word-wrap: break-word; border: 3px solid red; padding:4px; width:70%; background-color:#FFFF96;'>
<strong>There is an error. Please report the message (Piwik 1.8.2)
and full backtrace in the <a href='?module=Proxy&action=redirect&url=http://forum.piwik.org' target='_blank'>Piwik forums</a> (please do a Search first as it might have been reported already!).<br /><br/>
Warning:</strong> <i>htmlspecialchars() [<a href='function.htmlspecialchars'>function.htmlspecialchars</a>]: Invalid multibyte sequence in argument</i> in <b>/var/www/html/piwik-1.8.2/core/DataTable/Renderer.php</b> on line <b>224</b>
<br /><br />Backtrace --><div style="font-family:Courier;font-size:10pt">#0 Piwik_ErrorHandler(...) called at [:]#1 htmlspecialchars(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer.php:224]#2 Piwik_DataTable_Renderer::formatValueXml(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:342]#3 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#4 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#5 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#6 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#7 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#8 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:337]#9 Piwik_DataTable_Renderer_Xml->renderDataTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:142]#10 Piwik_DataTable_Renderer_Xml->renderTable(...) called at [/var/www/html/piwik-1.8.2/core/DataTable/Renderer/Xml.php:33]#11 Piwik_DataTable_Renderer_Xml->render(...) called at [/var/www/html/piwik-1.8.2/core/API/ResponseBuilder.php:221]#12 Piwik_API_ResponseBuilder->getRenderedDataTable(...) called at [/var/www/html/piwik-1.8.2/core/API/ResponseBuilder.php:325]#13 Piwik_API_ResponseBuilder->handleDataTable(...) called at [/var/www/html/piwik-1.8.2/core/API/ResponseBuilder.php:79]#14 Piwik_API_ResponseBuilder->getResponse(...) called at [/var/www/html/piwik-1.8.2/core/API/Request.php:130]#15 Piwik_API_Request->process(...) called at [/var/www/html/piwik-1.8.2/plugins/API/Controller.php:27]#16 Piwik_API_Controller->index(...) called at [:]#17 call_user_func_array(...) called at [/var/www/html/piwik-1.8.2/core/FrontController.php:138]#18 Piwik_FrontController->dispatch(...) called at [/var/www/html/piwik-1.8.2/index.php:53]</div><br />
</pre></div><br />
Do you have any idea how I can extract the corresponding data so that you can reproduce the error? For the interim time I will switch back to your “quick-fix” solution so that my report is not completely broken.
Cheers,
Tino
Thanks for the feedback, unfortunate that it does not work!
Would you be able to reproduce the issue on a new piwik server by any chance? this would really help us reproduce and fix the issue. Thankws!
Dear Matt,
we also tried your second quick fix and get the same error message as without fix. (see attachment).
It works fine with your first quick fix !
We unfortunately can’t promise to try it out on a new installation yet. We need to get the Go from Management for the costs of our third-party service partner having to implement it.
Best regards,
Webstat_Guys
Thanks for trying.
Now could you please try the following patch please:
Index: Renderer.php
===================================================================
--- Renderer.php (revision 6584)
+++ Renderer.php (working copy)
@@ -219,8 +219,8 @@
if(is_string($value)
&& !is_numeric($value))
{
- $value = html_entity_decode($value, ENT_COMPAT, 'UTF-8');
- $value = htmlspecialchars($value, ENT_COMPAT, 'UTF-8');
+ $value = html_entity_decode($value, Piwik_Common::HTML_ENCODING_QUOTE_STYLE, 'UTF-8');
+ $value = htmlspecialchars($value, Piwik_Common::HTML_ENCODING_QUOTE_STYLE, 'UTF-8');
Does the error still show?
Dear Matt,
I am sorry to report, that the new patch doesn’t work either. If you could provide a patch which could show the input on which the code is failing I could apply that, so that you can eventually reproduce this error.
So for the “quick fix” works - but it surely doesn’t report the correct results.
Cheers,
Tino
Can you try:
--- Renderer.php (revision 6621)
+++ Renderer.php (working copy)
@@ -219,11 +219,12 @@
if(is_string($value)
&& !is_numeric($value))
{
+ $value = @iconv('UTF-8', 'UTF-8//IGNORE', $value);
$value = html_entity_decode($value, ENT_COMPAT, 'UTF-8');
$value = htmlspecialchars($value, ENT_COMPAT, 'UTF-8');
Dear Matt,
that seems to be fine! At least we are getting good results an no more error messages. Did you finally have an idea what exactly went wrong?
Cheers,
Tino
Thanks Anthon for finding it, I would never have thought about that - is the problem a malformed UTF8 string?
Is it safe to commit do you think?
Invalid multibyte sequence in argument in /html/piwik/vendor/twig/twig/lib/Twig/Extension/Core.php on line 927
This bug has occurred again in Piwik 2.2.2. I created ticket at http://dev.piwik.org/trac/ticket/5157#ticket