CDN log processing technologies

Reliable data for reliable reports
CDNs and content publishers need reliable usage traffic reporting and reliable analytics, based upon reliable data. In order to calculate traffic usage and to enable CDN operators and account owners to generate viewing reports, modern CDNs have a number of challenges. The first challenge is to obtain reliable traffic usage and access information: reliable data. The most reliable source of information is in the delivery services access logs. 

Afterburner: industrial CDN log processing engine
Jet-Stream invented Afterburner, powerful CDN log processing technology. Afterburner is not a proprietary analytics suite, but an industrial, scalable log processing engine. 

Supported access log formats
In general, all access log files that are compliant with common, combined or W3C formatting are supported for input. Afterburner has been verified to work with these services: Apache, Lighttpd, IIS, Wowza, NGINX, QuickTime/Darwin Streaming Server, Windows Media Server and Icecast. Afterburner understands and can process these log files and produces similar output log files. 

Compliant log outputs
Afterburner maintains access logs compliancy. Processing is transparent. Afterburner does not alter the log formatting, it resorts logs from per service per server to per service per account and makes the access logs available for further processing per account. Optionally Afterburner can insert additional server and service fields for deeper CDN analysis.

Agents architecture
Afterburner is a set of collector agents, account agents, dechunk agents, interval agents, merge agents and statistics agents. All agents are high performance software agents. All agents can run on a single server or can be run on distributed servers, physical or cloud based. There is no need to install, manage and run agents on the delivery nodes, there is no need for specific overhead equipment. Scalability is key when you need to process access logs, which can grow into TeraBytes.

Intelligent log processing
Jet-Stream has reached out to delivery technology vendors to tackle the HTTP adaptive logging challenges. Thanks to our research, support and specifications, these vendors now log true sessions instead of logging each individual segment, exclusively for Jet-Stream CDNs. This feature reduces the number of log entries and it also guarantees exact session reporting, opposed to an educated guess. Afterburner also has the ability to insert so called 'late coming' log entries which occur when end users consume streams for a longer period (hours, days). Afterburner inserts these log entries, refreshes the output data and instructs analytics suites to refresh their data.

Traffic usage reporting
Afterburner provides a powerful traffic reporting API. CDN account owners and CDN administrators can integrate their billing workflows with the Afterburner API, so they can read out used traffic per year, per month, per service type and per account. Since the data is extracted from the CDN's delivery logs, the reported data is very exact. The CDN operator can decide whether to include or exclude internal CDN traffic from this data.

Free basic analytics
Afterburner bundles a basic and free usage analyzer. This service is provided 'as is'. It is up to the CDN operator to use it or not. The reporting engine is enhanced to support log files for various streaming services. Reported information includes popularity of content per year, month, day and hour, most popular content, traffic, technical reports and other details. Account owners can access this data via their self service portal.

3rd party analytics support
In our long-time experience with offering premium CDN services to hundreds demanding content publishers and CDN operators, we have learned that content publishers have very specific requirements when it comes to analytical features. One size does not fit all. Therefore, Jet-Stream does not offer a locked-in, proprietary analytics environment but lets CDN operators -and even more importantly lets account owners- individually choose to integrate with a wide variety of third party analytics tools and services.

Federated CDN log processing
Afterburner is designed from the ground up for federated CDN log processing. For instance, a single Afterburner setup can simultaneously process logs from multiple CDNs in parallel mode. In federated mode, a terminating CDN Afterburner collects, splits, merges, re-sorts and produces logs per account. Nothing different than it would do for a regular account. A higher layer CDN Afterburner can then log into this lower layer CDN Afterburner account and into all other Afterburner accounts on other CDNs, collect these logs, collect logs from their own delivery nodes, split, merge, resort and produce overall logs for their own accounts. This allows CDNs to fully integrate traffic reporting and usage reporting, in any arrangement, such as bi-lateral, horizontal, vertical, multiple, hierarchical, whether on-net or global. Each CDN keeps 100% control over which access logs are shared with other CDNs so they can protect their customers’ privacy. Each CDN is guaranteed that they can obtain the access logs that they need to process for their accounts. 

Additional benefits
You don't have to grant 3rd party analytics tools full access to your delivery nodes anymore. You don't have to grant 3rd party analytics tools to all your logs anymore, just for those accounts that need to be processed. This saves in traffic consumption, improves the privacy of your customers and can save in 3rd party analytics processing time and licensing costs as well. 

Log processing and analytics can be expensive components in a CDN. To help our customers drive down their costs, Afterburner is licensed unlimited and for free to anyone who licenses the Jet-Stream CDN technology suite.

Read more
The Jet-Stream CDN Technologies White Paper describes Jet-Stream technical vision and technologies. The 150+ pages document can be obtained here.