Log Analytics: Monitor traffic load in bytes

chAlx · February 11, 2014, 8:57am

Some missing features:

view bytes sent for time period
view bytes received for time period
find site resources generating most traffic
detect partial file downloads more precisely

techadmin · March 26, 2014, 1:24pm

I miss exactly the same things!

When is it that easy to setup a log analyzing job like it was in Urchin? I’m probably conservative, but I believe much more in a log file as in any scripts.

cheers

matthieu · March 29, 2014, 4:16am

These are great feature request. Maybe someone will work on them or maybe someone can sponsor us to work on those features!

al_piwik · May 23, 2014, 12:35pm

Hi.

How much donation should we expect for sponsoring this feature?
What’s the best startpoint to implement this feautre?

I think when the log-import methode is used this feature is much easier to add the with the tracker method.

Maybe in the GitHub - SOASTA/boomerang: End user oriented web performance testing and beaconing are some ideas which piwik can reuse.

cheers a

matthieu · May 23, 2014, 1:20pm

The best next step would be to create a ticket in our issue tracker for this feature request. Then we can help with estimating how long it would take to implement and release the feature. I think that’s very important that Piwik offers tracking of bytes and traffic!

al_piwik · May 27, 2014, 10:56am

Hi matt.

ticket is created

github.com/matomo-org/matomo

Log Analytics: Monitor Bandwidth for each page, download, and measure overall traffic in bytes

opened 12:54PM - 27 May 14 UTC

closed 12:35AM - 22 Jan 15 UTC

anonymous-matomo-user

Major Enhancement

As a user, when [importing my server logs in analytics](http://piwik.org/log-ana…lytics/), I want to measure the Bandwidth that was used by each page view. How would it work? - The bandwidth information is commonly available in the server access log files. It is measured in Bytes. - Log analytics script would detect the numeric bandwidth value - Log analytics forwards this value to the [Tracking API](http://developer.piwik.org/api-reference/tracking-api) and this value will be stored in the action. - This value can be attached to downloads and pageviews - The Reporting UI: - for each page: display the total bandwidth for each page view - for each directory (groups of pages): display the aggregated bandwidth value (the sum) of all pages within the directory Proposed implementation: - New column in log_link_visit_action: `bandwidth` - Tracking API add new parameter: file size in bytes`bw_bytes` - Log Analytics parses filesize, and sets &bw_bytes to tracking api requests - first log format we need to support is Apache common log format. - Tracker stores bandwidth in the new column `log_link_visit_action.bandwidth` - Actions/Archiver will aggregate the filesize in the Action report blobs. - This is similar to Average generation time. - The metric is processed for Pages, Page titles, and Download files. - User interface: the new metric "Bytes" will be displayed in the Actions tables - It is displayed in the existing Actions reports (Pages, Page titles, Downloads): this is not a new report. - If a user is not using Log Analytics, or if he is using Log Analytics but the logs don't have the file size bandwidth, then Actions reports will not have the "Bytes" column. Other steps: - Add new overall metric (and sparkline) in Visitors > Overview report:` Total Bandwidth` - eg. Total Bandwidth for piwik.org website = 155 Gb - Update [Tracking API reference](http://developer.piwik.org/api-reference/tracking-api) to document new parameter - Add new FAQ` How do I measure traffic bandwidth used by a page, and/or overall bandwidth?` - Update [feature page](http://piwik.org/features/) and [Log Analytics](http://piwik.org/log-analytics/) page to mention this new feature To be confirmed / optional: - Besides Apache common log format, maybe other log formats contain the bandwidth information - New custom segment "Bandwidth" to let users segment traffic based on the file size in bytes - for example "Show me reports only for file requests of files over 500,000" - We could even measure in Javascript the current page size, which would be useful, but unfortunately it can only be done with approximate value. This counts the number of bytes in DOM tree:` document.documentElement.innerHTML.length`. - This is an approximation only. if there's lots of page content created on the fly from script, that content will count in innerHTML despite not being present in the original source, which could throw your calculation out [source](http://stackoverflow.com/a/4402107/120988)

cheers a

al_piwik · May 27, 2014, 10:59am

Hi Matt.

ticket created

github.com/matomo-org/matomo

Log Analytics: Monitor Bandwidth for each page, download, and measure overall traffic in bytes

opened 12:54PM - 27 May 14 UTC

closed 12:35AM - 22 Jan 15 UTC

anonymous-matomo-user

Major Enhancement

As a user, when [importing my server logs in analytics](http://piwik.org/log-ana…lytics/), I want to measure the Bandwidth that was used by each page view. How would it work? - The bandwidth information is commonly available in the server access log files. It is measured in Bytes. - Log analytics script would detect the numeric bandwidth value - Log analytics forwards this value to the [Tracking API](http://developer.piwik.org/api-reference/tracking-api) and this value will be stored in the action. - This value can be attached to downloads and pageviews - The Reporting UI: - for each page: display the total bandwidth for each page view - for each directory (groups of pages): display the aggregated bandwidth value (the sum) of all pages within the directory Proposed implementation: - New column in log_link_visit_action: `bandwidth` - Tracking API add new parameter: file size in bytes`bw_bytes` - Log Analytics parses filesize, and sets &bw_bytes to tracking api requests - first log format we need to support is Apache common log format. - Tracker stores bandwidth in the new column `log_link_visit_action.bandwidth` - Actions/Archiver will aggregate the filesize in the Action report blobs. - This is similar to Average generation time. - The metric is processed for Pages, Page titles, and Download files. - User interface: the new metric "Bytes" will be displayed in the Actions tables - It is displayed in the existing Actions reports (Pages, Page titles, Downloads): this is not a new report. - If a user is not using Log Analytics, or if he is using Log Analytics but the logs don't have the file size bandwidth, then Actions reports will not have the "Bytes" column. Other steps: - Add new overall metric (and sparkline) in Visitors > Overview report:` Total Bandwidth` - eg. Total Bandwidth for piwik.org website = 155 Gb - Update [Tracking API reference](http://developer.piwik.org/api-reference/tracking-api) to document new parameter - Add new FAQ` How do I measure traffic bandwidth used by a page, and/or overall bandwidth?` - Update [feature page](http://piwik.org/features/) and [Log Analytics](http://piwik.org/log-analytics/) page to mention this new feature To be confirmed / optional: - Besides Apache common log format, maybe other log formats contain the bandwidth information - New custom segment "Bandwidth" to let users segment traffic based on the file size in bytes - for example "Show me reports only for file requests of files over 500,000" - We could even measure in Javascript the current page size, which would be useful, but unfortunately it can only be done with approximate value. This counts the number of bytes in DOM tree:` document.documentElement.innerHTML.length`. - This is an approximation only. if there's lots of page content created on the fly from script, that content will count in innerHTML despite not being present in the original source, which could throw your calculation out [source](http://stackoverflow.com/a/4402107/120988)

cheers a

al_piwik · May 27, 2014, 11:00am

Hi Matt.

#5248 Log Analytics: Monitor Bandwidth for each page, download, and measure overall traffic in bytes · Issue #5248 · matomo-org/matomo · GitHub

cheers a

Techwolf · June 8, 2014, 4:25pm

When piping logs via the import script, there is a request_time that is passed along. Would love that piwik give some stats on how long it took to download some files. I have a download server set up that only serves up files. No html.

andrearivetti · August 6, 2014, 11:05am

Hi!
I am very interested in this feature.
Please keep me informed about the possibility to sponsor it!

Cheers,
Andrea