This dcument is a copy of a document on http://www.iepg.org/settlements.html
G. Huston IEPG December 1994This document examines the area of network peering within the Internet environment, and identifies a set of parameters within which peering arrangements may be negotiated between Internet Service Providers (ISPs).
The first (and most common international peering model) is that of bilateral settlements, where each Telcom provider invoices the originating end user, and then financial settlements are used in accordance with originating call imbalance (where the Telcom generating the greater number of calls pays the other party according to a bilaterally negotiated structure, using a financial transaction to reconcile a position of perceived imbalance between the providers within the underlying call exchange volumes).
The second model is that of Sender Keep All (SKA) where each Telcom invoices their originating clients user for the end to end services, but no financial settlement is made across the bilateral peering structure.
The third model is that of transit fees, where the one party invoices the other party for services provided. For example this arrangement is commonly used as the basis of the long distance provider - local access provider interconnection arrangements.
Within the Internet to date only the second and third models can be observed within the deployed infrastructure.
Where there is a perceived peering requirement the SKA model is generally used, where no financial settlements are used. The SKA arrangement forms the basis of the CIX, MAE-East, FIX and SWAB interconnection structures (and others).
Generally an SKA arrangement is only stable where the parties involved perceive equal benefit from the interconnection, and in general the SKA arrangements are only prevalent within the US and Europe where the parties involved all operate national transit structures, and in the area of peering of international services.
The third model (transit fees) is commonly seen as a component of the relationship between a transit provider and a local access provider, where the local provider is charged for transit services within a larger service agreement. The US model of regional networks, transit providers and trunk network access points (NAPs) is now introducing elements of the bilateral settlement service model into the Internet, where a number of regional providers are using a financial settlement model with the trunk domestic transit network providers.
Within the Internet the common basis of the settlement appears to be one of flat fees, without reference to exchanged data volumes.
This is possibly a reflection of the attributes of the Internet technology, where traditional accounting elements of "calls" are not readily apparent.
One of the significant issues here is determination of whether an ISP is a peer to, or a client of another ISP. This is perhaps most significant in the case where the exchange environment is one of SKA, where in effect the successful assertion by one party of a change to peer status, from a previous relationship of a client to the second party results in effect in the dropping of client service revenue.
The Telcom resolution of such matters has typically been through an administrative process of "licensed" service providers, who become peer entities through a process of administrative fiat. This is not normally an option for the ISP domain, which typically operates within the deregulated valued added network service provider regulatory environment, where individual "licenses" are replaced with generic class licenses or similar deregulated structures.
The early attempts of the CIX (Commercial Internet eXchange) exchange arrangements were based on a description of the infrastructure of each party, where acknowledgment of peer capability were based on the operation of a national transit infrastructure of a minimum specified capability. This specification of peering within the CIX has been modified so that currently CIX peer status for an ISP is based on payment of the CIX Association membership fee. It is also noted that the CIX model is not one which intrinsically admits bilateral peer relationships - the relationship is an multilateral one, where each ISP executes a single agreement with the CIX Association, and then effectively is a component of the Association. The consequence of this multilateral arrangements is that the peering settlements can be regarded as an instance of SKA peering.
Other models use a functional peer specification, where, if the ISP attaches to a nominated physical exchange structure, then the ISP is then in a position to open bilateral negotiations with any other ISP also directly attached to the structure. This model is inherently a more flexible model, as the bilateral exchange structure allows each represented ISP to make their own determination of whether to agree to a peer relationship or not, and also allows each bilateral peer arrangement to be executed individually, admitting a mix of SKA and financial settlement arrangements.
There are three generic peering structure options: that of direct point-to-point interconnection, multilateral exchange points and multi-party bilateral exchange points.
Direct point-to-point peering is where each bilateral peer arrangement is implemented through a physical interconnection. While this was a feature of the emerging Internet infrastructure of the 1980's it is no longer a widely deployed mechanism. Such arrangements become highly unstable because of the problems with scaling the peering to accommodate a number of peer entities. The scaling results in a N2 interconnection environment, which becomes highly unstable in terms of the underlying routing and transit arrangements.
Multilateral exchange points are seen within the CIX exchange architecture, and within the European EBONE structure. The issue of such multilateral exchange architectures is the enforcement of a single multilateral peering structure, which is only stable when all parties are in a position of approximate equal positioning with respect to the mutual traffic exchange. No parties are effectively in a position to offer charged transit services within such exchange structures, and while the SKA exchange is a model which is highly regarded by many ISPs, the inability to match transit ISPs with local ISPs across such multilateral exchanges can be a significant prohibitive factor. This observation is most relevant in the international domain, where international transit costs can be reduced through traffic aggregation across large transit links, which in turn can only be provided on a stable basis if settlement structures allow a pricing component for an international transit provider ISP to offer aggregated transit services to local ISPs.
Mutual bilateral exchange points offer the greatest degree of flexibility, and have been used for some years within the United States at the Federal Internet eXchange structures (the FIXes), the MAE-East international exchange structure (a metropolitan distributed ethernet operated across the greater Washington DC area), the Japanese Internet eXchange (JIX) and the NSFnet NAPs. Here a single physical ISP connection to an exchange environment can be used to execute individual bilateral peer agreements with some, or all, of the similarly connected ISPs, and the peer agreements are capable of admitting selective transit provision between arbitrary parties.
Peering issues can be view from a perspective of "domestic" and "intercontinental" peering. This is admittedly a somewhat arbitrary classification, and a refinement of this can take the form of description using a terminology of "service domains", or, more accurately "market domains". In this sense peering requirements within an environment of overlapping market domains admit a set of issues which are somewhat distinct from those seen within an environment of non-overlapping market domains. This distinction can be mapped into national and intercontinental market spaces, given the high cost differential between operation in a single national domain and operation across multiple national domains across continents.
The following sections of this document examine the aspects of peering within the domain of overlapping markets.
One scenario is that a mutually acceptable peering relationship cannot be negotiated, and all ISPs operate disconnected network domains with dedicated international connections. The most likely outcome of such a situation is that third party peering would take place, with transit traffic flowing between the two entities being exchanged within the domain of a mutually connected third party ISP (or set of ISPs). For example, in the case for Australia this would result in a situation where domestic traffic would be passed across the Pacific, routed across a number of network domains within the US, and then passed back across the Pacific.
It must be noted that this is not an entirely novel situation, and the Internet has seen such arrangements appear in the past. Such arrangements have arisen, in general, as the outcome of an inability to negotiate a stable domestic peering structure. However such arrangements have proved to be, in general, relatively short-lived due to the high cost of operating such international transit environments, the instability of the intercontinental Internet infrastructure in such a configuration, and the unwillingness of foreign third party ISPs to act (often unwittingly) as agents for domestic interconnection in the longer term. As a result of these factors such off-shore connectivity structures have generally been augmented with domestic peering structures.
The resultant general operating environment of the Internet is that each national (or regional) collection of ISPs should act to ensure full domestic interconnectivity between such ISPs as a consequence of the common desire for rational cost containment, and there is a consequent priority to reach acceptable ISP peering arrangements as a matter of some priority.
Bilateral connection arrangements are used where each bilateral party uses a private exchange facility, and negotiates a distinct agreement.
While this is relatively straightforward within a two party agreement (at least in engineering terms), there are a number of additional issues when additional ISPs enter into the domain, and for each existing ISP there are increasingly higher costs for bilaterals (in terms of engineering support) for each new ISP.
These bilateral private connection structures have also been observed as short-lived structures within the Internet to date, as the issues involving engineering support, integrity of transit provision and the increased overhead of an N2 interconnection environment, are, as noted previously, relatively unstable under pressures of scaling to a larger number of ISPs.
The exchange structure uses the concept of a single point as a mutual exchange structure, allowing each party to participate in a fully connected exchange environment with a single external connection (the word "point" is perhaps misleading in a physical sense, as the exchange "point" is in effect a single coherent layer 2 or layer 3 point, or a single coherently presented common transit domain, and as such can exist within a single physical location, or can be implemented with distributed access points). The essential attribute of this structure is the ability to execute a comprehensive set of agreements with all other ISPs using a single external connection.
As noted above a multilaterally structured exchange point (where a single peering model is used for all peering ISPs) is an essential precondition for a layer 3 (routed) exchange environment, while a layer 2 exchange environment can support either a fully bilaterally defined environment, or, with the use of Route Servers, can support any form of hybrid peering models which use both multilateral and bilateral exchange models.
While the SKA peering arrangement does have strong proponents with the Internet ISP community, the model of SKA has stability problems where either one ISP acts in a transit role for the other, or where one ISP is providing a set of services used by the other ISP. In such cases there is little incentive for the service providing ISP to enter into an SKA agreement with the second ISP, and a financial settlement may be the only viable means of reaching a peering agreement.
Indeed such financial settlement environments are most effective in those environments where there is a perceived imbalance in their relationship, where there is a perceived imbalance in the provision of services from one party to the other.
SKA peering is one where traffic is exchanged between ISPs without mutual charge. Within a national structure it is typically the case that the marginal cost of international traffic transfer to and from the rest of the Internet is significantly higher than domestic traffic transfer it would be likely that any SKA peering would only relate to domestic traffic, and international traffic would either be provided by a separate agreement, or provided independently by each party.
Even then it is unlikely that SKA peering would be stable unless both networks provided infrastructure functions either independently, or shared the operational load of provision of infrastructure functions. These functions include USENET news flows, NTP reference signals, DNS forwarders and caching services, and information caches, multicast services and similar infrastructural services.
The essential criteria for a stable SKA peering structure is perceived equality in the peering relationship. There are a number of ways in which this can be achieved, including the use entry pricing into the peering environment, or the use of peering criteria (such as specification of ISP network infrastructure, or network level of service and coverage areas as eligibility for peering).
One method to implement an approximate parity in such a SKA peering environment is to define a SKA peering in terms of traffic peering at the client level only, forcing each peering ISP to be self sufficient in the provision of ISP infrastructure services (such as the requirement to provide services including DNS, NTP, USENET, Archie, HTTPD and similar) which would not be provided across a peering point. This may not result in the most efficient or effective Internet infrastructure.
An alternative structure is to use bilateral financial settlements as the basis of peering. Here the configuration where one party provides common infrastructure services to the other party can be configured in a relatively stable fashion, using a financial settlement to offer the transfer costs of single service provision.
There are a number of ways in which financial settlement can be determined, including direct negotiation, traffic exchange levels, network routing exchanges, and similar.
Traffic volume is perhaps the most obvious means of determining a metric to be used as a basis of financial settlement, and such a metric can accommodate a number of variations in peering models. However it must be noted that traffic volumes on the Internet are not as rigidly structured as those within the Telcom environment, and within this environment it is not possible to readily determine which party's client generated a bidirectional traffic flow across the peering structure.
Observation of traffic statistics would indicate that traffic received is perhaps the only viable means of determining relative traffic volumes, and in this case the settlement traffic unit price would have to match the average marginal cost of domestic transit traffic for the settlement to be attractive to both parties. The figure here indicates a theoretical position where one provider acts in a common service position to the other two providers, and accordingly operates in a net traffic imbalance position to the other parties, providing more data than is being received. Within these two bilateral peering structures financial settlements may be a means of reconciling such a position while preserving the underlying efficiencies of single point provision of infrastructural services.
Within the Internet the concept of an originating party holds only for TCP sessions, and this concept does not readily translate into measurement technologies, or to actual end client billing models. Accordingly ISP peering settlement models based on settlement of number, volume, and duration of originating TCP sessions are not readily reconcilable within the Internet domain. The use of received traffic volumes is a coarse approximation to volume of originating sessions, based on the observation that the most predominant interaction at the user level is the importation of data.
The alternative of SKA effectively imposes a model of self-sufficiency on each ISP which effectively implies that each peering entity uses the peering structure solely for the exchange of user data. This is not generally the case, as peering structures typically involve both peer interaction and the provision of transit or infrastructure services (within a client / service provider model). The major challenge with an SKA peering structure is whether it is possible to appropriately define what is encompassed within common peer interactions and what falls within a client / service provider domain (where SKA agreements would not apply).
The decision point here is that of determining whether the exchange operates at the network layer, and becomes an active layer 3 network entity, or whether it operates at layer 2 as a common physical medium.
A choice of layer three structure does impose an immediate consequence of a multilateral peering environment, where each connecting entity must peer with the common structure, rather than elect to undertake individual bilateral agreements. The implementation of a layer three peering environment is the easiest to engineer and support, as the engineering model is that of a single router owned and operated by the common peering entity. Each participating party must connect to this router and then is able to conduct an inclusive routing exchange. There is no requirement for co-location of foreign equipment, and there is a single active peering element.
The disadvantage of a layer three structure is typically administrative, where there is the need to carefully construct a completely independent peering host entity which is capable of representing the "common peering position" to each new party, and is also capable of owning and operating equipment which implements the peering structure.
The alternative is a layer 2 structure, and this has typically entailed the provision of the peering point as an Ethernet (or) cable segment, and each party provides a router to attach to the cable. This does minimise the administrative requirements, as the common property is a single cable segment, and there is no single common administrative entity required to operate such a peering structure. However it does replace the common equipment requirement with a co-location requirement. This can be addressed with the use of publicly available distributed layer 2 transport structures (such as FastPac2 within Australia) which effectively creates a distributed layer 2 environment using a Closed User Group approach.
This then allows the construction of a scalable and flexible peering structure where the entry to peering is based on participation in a distributed layer 2 structure. Each party can then operate with layer 3 route peering tools that allows any arbitrary set of peering structures to be supported, and the structure dictates no pre-conditions concerning the nature of any particular bilateral peering arrangement, or even that a peering agreement must be executed. Any connected site may offer co-location service to third parties if they so wish, although there is no obligation for any party to do so.