Piwik 1.7 Multi Sites with 1100 sites being tracked

Hi,

I’m starting this thread because I’m not 100% sure where to start troubleshooting this. I am running Piwik 1.7.1 and I am tracking 1100 sites right now. It is set to archive every hour and that appears to go just fine. If I look up individual sites all the stats appear to be accounted for.

When I try and use the multi sites plugin all that appears is a blank page. There are no errors in the web server log. I looked into the multi sites performance issue and found this "All Websites dashboard" should load fast even when tracking hundreds of websites! MultiSites · Issue #1077 · matomo-org/matomo · GitHub, but this should not be an issue on 1.7.

It appears to crop up right when I hit 1000 sites, before then I can view all sites using the plugin.

So my question is, where do I start checking for issues? Does anyone have an idea about what might be causing this?

Thanks

Our tests showed the page loading for more than 1000 websites. But, maybe there is a bug. Most likely though, there is a timeout on your server. PLease check error logs?
you say there is no error but there should be one if you get a blank page with no output. Or, test with a different browser? also test ot generate an error ie <?php echo bla();
and then check it appears in error logs

Thanks for the reply,

I looked at all the error logs I could find and found nothing. They are all empty.

I then did as you suggested and purposely through an error. I created a file called error.php in the web root. This threw an error and it was logged. I then purposely caused an error in index.php which also threw an error and was logged accordingly.

When I call the multi sites plugin, it loads for 2-3 seconds then stops. I do not believe I am receiving any output from the server at all.

Right now we are on Arvixe shared server which I was told by the nice people there would be sufficient. I can now see in the server logs that my CPU use has been limited at some point in the past 24 hours but the time of the CPU throttling does not coincide with when I tried to run the multi sites plugin. I see a spike in usage to near 100% when the archive script runs but that is about it, otherwise it hovers at about 40%.

Thanks for the help, it is much appreciated.

Also, this is just our test install of Piwik, we like it for the API and the great plugin interface. We are actually wanting to use Piwik to track about 4600 sites.

Arvixe will not be sufficient to track so many websites, unfortunately it is very good and cheap, but not for such heavy use :slight_smile: Please try to host Piwik on a more powerful server or VPS in your case. Probably they are throttling to save resources for other users on the same server.

Hey,

Upgraded to a new VPS with Arvixe and still the same issue. Overall responsiveness is better but the same issue still persists. When I click all sites it spends about 1-2 seconds then loads nothing.

Any ideas?

Maybe paginate the results somehow? Do you think that would work?

Did you setup: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo
it is required when tracking so many websites.
After setting this up and making it run at least once until completion, it should work better. If not, check web server error logs. Then report here?

Yep, it is set to auto archive. This appears to run successfully every 30 minutes. Only takes a few minutes to run.

Found an error though, the error log was not getting written due to permissions errors. Fixed that here is what I got

[28-Feb-2012 19:21:30] PHP Fatal error: Class ‘Zend_Registry’ not found in /home/username/public_html/tracker/core/FrontController.php on line 214

That’ surprising error. What if you run archive.php ? what is the output?

Here is the last from the cron task:

X-Powered-By: PHP/5.2.15
Content-type: text/html

[2012-02-29 00:30:05] [ac671448] ---------------------------
[2012-02-29 00:30:05] [ac671448] INIT
[2012-02-29 00:30:05] [ac671448] Querying Piwik API at: PATHTOPIWIK
[2012-02-29 00:30:05] [ac671448] Running as Super User: admin
[2012-02-29 00:30:05] [ac671448] Notes
[2012-02-29 00:30:05] [ac671448] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-02-29 00:30:05] [ac671448] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-02-29 00:30:05] [ac671448] - Archiving was last executed without error 29 min 59s ago
[2012-02-29 00:30:05] [ac671448] Will process 1 websites with new visits since 29 min 59s , IDs: 223
[2012-02-29 00:30:05] [ac671448] ---------------------------
[2012-02-29 00:30:05] [ac671448] START
[2012-02-29 00:30:05] [ac671448] Starting Piwik reports archiving…
[2012-02-29 00:30:06] [ac671448] Archived website id = 223, period = day, Time elapsed: 1.216s
[2012-02-29 00:30:08] [ac671448] Archived website id = 223, period = week, 1 visits, Time elapsed: 1.921s
[2012-02-29 00:30:09] [ac671448] Archived website id = 223, period = month, 1 visits, Time elapsed: 0.619s
[2012-02-29 00:30:10] [ac671448] Archived website id = 223, period = year, 1 visits, Time elapsed: 1.531s
[2012-02-29 00:30:10] [ac671448] Archived website id = 223, today = 1 visits, 4 API requests, Time elapsed: 5.289s [1/1 done]
[2012-02-29 00:30:10] [ac671448] Done archiving!
[2012-02-29 00:30:10] [ac671448] ---------------------------
[2012-02-29 00:30:10] [ac671448] SUMMARY
[2012-02-29 00:30:10] [ac671448] Total daily visits archived: 1
[2012-02-29 00:30:10] [ac671448] Archived today’s reports for 1 websites
[2012-02-29 00:30:10] [ac671448] Archived week/month/year for 1 websites.
[2012-02-29 00:30:10] [ac671448] Skipped 1000 websites: no new visit since the last script execution
[2012-02-29 00:30:10] [ac671448] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
[2012-02-29 00:30:10] [ac671448] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-02-29 00:30:10] [ac671448] Total API requests: 4
[2012-02-29 00:30:10] [ac671448] done: 1/1 100%, 1 v, 1 wtoday, 1 wperiods, 4 req, 5293 ms, no error
[2012-02-29 00:30:10] [ac671448] Time elapsed: 5.294s
[2012-02-29 00:30:10] [ac671448] ---------------------------
[2012-02-29 00:30:10] [ac671448] SCHEDULED TASKS
[2012-02-29 00:30:10] [ac671448] Starting Scheduled tasks…
[2012-02-29 00:30:11] [ac671448] task,output Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 0.584s Piwik_PrivacyManager.deleteLogTables,Time elapsed: 0.000s
[2012-02-29 00:30:11] [ac671448] done

This would be the same as manually calling it right?

  1. I recommend you change the time from 10 seconds to 1800 in the General Settings
  2. When you’re trying to load the “All websites dashboard” make sure you don’t select “Today” in the calendar. When you select yesterday does it load?
  1. try to call the API such as: http://your site.org/index.php?module=API&method=MultiSites.getAll&period=day&date=yesterday&format=xml&token_auth=
    does it load?

Matt,

Thanks for your help.

So far, still the same result

Changed the interval to 1800

called it by clicking on the link
http://PIWIK/index.php?module=MultiSites&action=index&idSite=1&period=day&date=yesterday

and then by trying to call the API
http://PIWIK/index.php?module=API&method=MultiSites.getAll&period=day&date=yesterday&format=xml&token_auth=MYAUTHTOKEN

Still the same result, page attempts to load then nothing. No reports in the error log, firebug doesn’t seem to find anything either.

OK, let’s try something else:
can you execute archive.sh (not.php) the old script that forces to archive everything.
Then, try the API query again?

what’ s strange is there is no error in error log…

Running now, could take a little while.

I am looking at the mysql error logs and the error_log in the Piwik root. Any other place I should be checking?

OK, so ran archive.sh, took awhile, ran and completed with no errors.

Still the same result using the API and just the regular call.

What about a different API call:


http://demo.piwik.org/?module=API&method=VisitsSummary.getVisits&idSite=all&period=day&date=yesterday&format=xml

This would try and fetch all visits from all websites using a different API.

If the page loads and then stops loading, how much time is between the start of loading and end result showing blank page?

Is there no error in the error log at all after this?

Same thing, blank page, about a 1-2 second load time.

Nothing in the error_logs. I have noticed that there has been 220,000 SQL queries in the past 24 hours while digging through and trying to find any error logs at all.

I don’t get it, 2 seconds load time is strange, if it was to fail it should take longer.

Otherwise, apart from all websites dashboard, does everything else work fine?

I think you already confirmed it, but if you do <?php echo bla();
in a new file, it is logged in the error log?

Can you finally ask the web host if they see anything suspicious?

Everything else is working just fine. It seems to be recording all visits and all the other plugins, including the standard APi appear to be working perfectly. No other errors anywhere. I think the Zend Registry error was during install when I tried to load index.php before I was finished uploading Piwik to the server.

Yes, all other errors are reported, both in the browser and in the Apache log file. AKA

Fatal error: Call to undefined function bla() in /home/cpaneluser/public_html/tracker/error.php on line 1

This is driving me crazy.

I opened a ticket with Arvixe to see if they can find any error logs for the server.

OK, I tend to think it’s a server issue as this point, hopefully Arvixe will find out why! please report here. It worked for us in our tests on 1000+ websites, but there is always the possibility of a bug