ISP Backbone Traffic in Japan

updated: August 2017


Overview

We are collecting month-long aggregated traffic logs for different traffic groups from six major ISPs in Japan twice per year, in May and November, in order to analyze the macro-level impact of residential broadband traffic. These traffic groups are carefully selected to be summable, and not to count the same traffic multiple times.

The participating ISPs are IIJ, K-Opticom, KDDI, NTT Communications, and SoftBank (former Softbank Telecom and fomer Softbank BB).

Brief History of the Study

Background

Fig. 1 shows the number of boradband subscriners in Japan. (The gap at 2010/03 is due to a change in data collection methods of some CATV service providers.)

broadband subscribers in Japan
Fig. 1: Broadband Subscribers in Japan

Fig. 2 shows aggregated traffic and its growth rate at major IXes in Japan, namely JPIX, JPNAP, NSPIXP, BBIX, and Equinix.

traffic at major IXes
Fig. 2: Traffic at Major IXes

Methodology

We have been collecting traffic volumes crossing ISP boundaries which can be roughly divided into customer traffic, and external provider traffic such as peering and transit. For practical purposes, we selected the five traffic groups in for data collection:

(A1) RBB customers
represent residential broadband customer lines. This group also includes small business customers using residential broadband access.
(A2) non-RBB customers
represent customer lines other than RBB customers, including leased lines, data centers, and dialup lines. This group includes RBB customers behind leased lines, e.g., second or third level ISPs, since ISPs do not distinguish them from other leased lines. Customer providers, back offices, and CDN caches are also included.
(B1) external 6IXes
represent links for 6 major IXes, namely JPIX, JPNAP and NSPIXP in both Tokyo and Osaka in order to compare measurements at these IXes as well as to know the traffic share of our measurement.
(B2) external domestic
represents domestic external provider links other than the 6IXes, including regional IXes, private peering and transit. We used the term ``domestic'' to indicate that both ends of a link are located in Japan. This group also includes domestic peering with global ASes.
(B3) external international
represents international external provider links with one end point outside of Japan.
figure of traffic groups
Fig. 3: Five traffic groups for data collection at customer and external boundaries of an ISP

These groups are chosen by the existing operational practices of the participating ISPs because it is impossible to draw a strict line for grouping, e.g., residential/business and domestic/international, on the global Internet. We re-aggregate each ISP's aggregated logs, and only the resulting aggregated traffic is used in our study so as to not reveal the share of each ISP.

Each ISP provided month-long traffic logs aggregated for each traffic group by a log aggregation tool we developed. This allows ISPs not to disclose the internal structure of their network or unneeded details of their traffic. The final results were obtained by aggregating all traffic logs provided by the six ISPs.

The time resolution of the logs is 2 hours since it was the highest common factor for month-long data in MRTG and RRDtool; 2-hour boundaries in UTC fall on odd hours in Japanese Standard Time (UTC+9). Note that, IN and OUT are presented from the ISPs' point of view.

Traffic measured by the 6 ISPs

For weekly data analysis, we took the averages of the same weekdays in a month but excluded holidays since holiday traffic patterns are closer to weekends.

Customer Traffic

week plot of customer-bb
Fig. 4: Week View of (A1) Customer Broadband
week plot of customer-other
Fig. 5: Week View of (A2) Customer Other

External Traffic

week plot of external-6ix
Fig. 6: Week View of (B1) External 6IX
week plot of external-domestic
Fig. 7: Week View of (B2) External Domestic
week plot of external-international
Fig. 8: Week View of (B1) External International

Traffic Growth

The monthly average rates in bits/second of the traffic groups are shown in Table 2 and their growth is illustrated in Fig. 9 and 10.

growth of customer traffic
Fig. 9: Growth of Customer Traffic
growth of external traffic
Fig. 10: Growth of External Traffic

The decrease at 2010/05 seems to be an impact of the amended Copyright Act making the download of copyright infringing content illegal. The increase of A2 IN at 2016/11 is a result of re-examination for classifying CDN caches into A2.

Table. 1: Monthly average rates of aggregated customer and external traffic (bps)
yearmonth(A1)customer-bb(A2)customer-other (B1)external-6ix(B2)external-domestic(B3)external-international
in(Gbps)out(Gbps)in(Gbps)out(Gbps) in(Gbps)out(Gbps)in(Gbps)out(Gbps)in(Gbps)out(Gbps)
20040998.1111.814.013.635.930.948.237.825.314.1
10108.3124.915.014.936.331.853.141.627.715.4
11116.0133.016.215.638.033.055.143.328.516.7
20055134.5178.323.723.947.941.673.358.440.124.1
11146.7194.236.129.754.048.180.968.157.139.8
200605173.0226.242.938.366.260.194.977.668.547.8
11194.5264.250.746.768.462.3107.690.594.557.8
200705217.3306.073.857.877.470.8124.5108.4116.471.2
11237.2339.885.463.293.583.4129.0113.3133.781.8
200805269.0374.7107.085.095.788.3141.2119.4152.694.4
11302.0432.9122.488.7107.5102.5155.6132.3176.1110.8
200905349.5501.0154.4121.4111.7104.9185.0155.4213.1126.4
11373.6539.7169.4127.6114.3109.8209.5154.3248.2148.3
201005321.9536.4178.8131.294.191.0194.8121.4286.9155.5
11311.1593.0190.1147.590.191.6198.7117.2330.1144.9
201105302.5662.0193.9174.498.490.0242.9131.5420.9160.5
11293.6744.5221.9207.5102.989.4265.1139.1498.5169.6
201205287.8756.6251.5243.0118.498.6317.4145.1528.7178.8
11294.0840.3268.3257.2103.283.2316.6135.7571.3201.6
201305347.81027.8300.3286.4114.585.5423.3161.3633.9231.6
11370.01146.3336.5326.2138.994.9520.8186.2714.5259.7
201405398.91274.5359.2317.2163.6101.5614.9214.3808.3282.3
11407.61557.0496.1426.1192.3104.6765.1246.5924.6340.6
201505457.01928.9525.6440.2198.9117.5955.6287.5941.5308.1
11452.92336.1581.1503.0251.9137.11306.4366.61059.7307.9
201605551.52863.3652.7570.5277.0112.61765.1453.81080.1292.4
11602.53396.61246.0653.6311.0113.61989.2518.21221.9535.8
201705743.73950.61304.0690.9427.4146.52809.3625.91248.5308.7

Annual growth rates

The annual growth rates over the years are summarized in Table 2. The growth rate is the ratio against the measurement one year before.

Table. 2: Annual Growth Rates
yearmonth(A1)customer-bb(A2)customer-other (B1)external-6ix(B2)external-domestic(B3)external-international
inoutinout inoutinoutinout
2005111.2651.4602.2281.9041.4211.4581.4681.5732.0042.383
2006051.2861.2691.8101.6031.3821.4451.2951.3291.7081.983
111.3261.3601.4041.5721.2671.2951.3301.3291.6551.452
2007051.2561.3531.7201.5091.1691.1781.3121.3971.6991.490
111.2201.2861.6841.3531.3671.3391.1991.2521.4151.415
2008051.2381.2251.4501.4711.2361.2471.1341.1011.3111.326
111.2731.2741.4331.4031.1501.2291.2061.1681.3171.355
2009051.2991.3371.4431.4281.1671.1881.3101.1751.3961.339
111.2371.2471.3841.4391.0631.0711.3461.1661.4091.338
2010050.9211.0711.1581.0810.8420.8681.0530.7811.3461.230
110.8331.0991.1221.1560.7880.8340.9480.7601.3300.977
2011050.9401.2341.0841.3290.9250.9551.3051.1091.4671.032
110.9441.2551.1671.4071.1420.9761.3341.1871.5101.170
2012050.9511.1431.2971.3931.2031.0961.3071.1031.2561.114
111.0011.1291.2091.2401.0030.9311.1940.9761.1461.189
2013051.2081.3581.1941.1790.9670.8671.3341.1121.1991.295
111.2591.3641.2541.2681.3461.1411.6451.3721.2511.288
2014051.1471.2401.1961.1081.4291.1871.4531.3291.2751.219
111.1021.3581.4741.3061.3841.1021.4691.3241.2941.312
2015051.1461.5131.4631.3881.2161.1581.5541.3421.1651.091
111.1111.5001.1711.1801.3101.3111.7071.4871.1460.904
2016051.2071.4841.2421.2961.3930.9581.8471.5781.1470.949
111.3301.4542.1441.2991.2350.8291.5231.4141.1531.149
2017051.3491.3801.9981.2111.5431.3011.5921.3791.1561.056

Estimating Total Customer Traffic in Japan

We can estimate the total customer traffic in Japan using the measurements from the 6 ISPs. We use the participating ISPs' share to compute the total traffic in Japan. For A1, the contract share of broadband services of the participating ISPs are used (assuming that customers traffic usage is similar among ISPs). For A2, the traffic share of the major IXes is used. However, the A2 data is provided only from 3 ISPs, and the A2 ratio differs from ISP to ISP. So, the number for A2 is just a very rough estimate.

estimated A1 total
Fig. 11: Estimated A1 Total in Japan. Mobile shows the total 3G/LTE data traffic in Japan.
estimated A2 total
Fig. 11: Estimated A2 Total in Japan
Table. 3: Estimated Total Customer Traffic in Japan
yearmonth6 ISPsestimated A1 total3 ISPsestimated A2 total
contract share(%)in(Gbps)out(Gbps)IX traffic share(%)in(Gbps)out(Gbps)
20040951.818921614.99491
1051.820923915.29998
1151.722425714.0116111
20050551.925934414.9159160
1149.729539115.9227187
20060549.335145916.7257229
1148.939854016.1315290
20070548.644763017.5422330
1148.049470816.6515381
20080546.957379917.9598475
1146.165593918.7655474
20090545.5768110017.4887698
1144.7836121017.6963725
20100543.4742124016.91060776
1143.5715136017.01120868
20110543.7692152013.814101260
1143.9668170012.817301620
20120543.7659173012.420301960
1144.1667191011.224002300
20130545.276922709.5631403000
1144.483325808.6738803760
20140544.190528908.7641003620
1143.992935507.1369605980
20150542.1109045807.3671405980
1143.1105054206.7985607410
20160541.6132068804.871340017700
1141.2146082504.532750014400
20170541.3180095605.122550013500

References

  1. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
    The Impact and Implications of the Growth in Residential User-to-User Traffic.
    SIGCOMM2006, pp207-218. Pisa, Italy. September 2006. (pdf).
  2. Kenjiro Cho, Kensuke Fukuda, Hiroshi Esaki and Akira Kato.
    Observing Slow Crustal Movement in Residential User Traffic.
    ACM CoNEXT2008. Madrid, Spain. December 2008. (pdf).