Internet Service Guarantees
Or at least past attempts at service guarantees
Peter Dordal, Loyola University CS Department
Home & Small Office VoIP
Make no mistake: if you try to fit 20 VoIP calls on a line with bandwidth
sufficient only for 10, it will fail. Actually, if you try to fit 20 VoIP
calls on a line with bandwidth sufficient only for 18, it will fail, and if
the bandwidth is sufficient only for 19, then users will complain.
But suppose you have "more than enough" Internet capacity. Here's the
problem: what if you have a "few" VoIP lines and you're also downloading
bulk data, all on the same internet connection. How
do you make sure the VoIP packets get priority? If you've got a
1Mbps link, one DS0 (µ-law) line will use (not counting headers) 3% of the
bandwidth. Counting headers but using reasonable compression, you might have
five voice lines using well under 10% of the bandwidth. But if the router
queue is long, you don't want VoIP traffic waiting at the back end.
For outbound traffic, the solution is simple. You get a little box that
connects to (a) your ISP, (b) your internal data LAN, and (c) your phone
lines. The box simply implements outbound priority queuing, sending VoIP
packets at a higher priority. Such boxes are standard hardware items.
The harder direction is inbound. What if, at the router, a 1.0-second queue
has built up, because someone is downloading a large file? Your inbound VoIP
packets will likely sit at the back of this queue. If your ISP has a lot of
sense, they will allow at their end of your link only a small queue, so this
does not happen. Failing that, though, you can:
(a) negotiate with your ISP for them to do
priority routing of VoIP packets at their
end too. Their router can almost certainly easily be configured to act
just like the little box at your end. Note that this option, though, means
telling your ISP you want to use your Internet for voice; they may want to
charge you extra.
(b) Try to throttle your inbound
data by shaping (ie slowing) the stream of outboundACKs.
little box, above, can estimate when the VoIP packets are slowing down,
and, at that point, it might slow down outbound data ACKs to just a
sufficient extent that the inbound VoIP stream returns to normal, or until
the inbound data rate falls below (no-VoIP-data-rate −
bandwidth-allocated-for-voice). It's dicey; I'm not sure how often this is
done. Perhaps the biggest problem with slowing ACKs is that initially it
does not work at all (because the upstream queue is still draining), and
then all of a sudden it has a dramatic and probably-too-large effect.
Integrated Services Architecture (ISA) and RSVP
ISA is the overall architecture, and RSVP (ReSerVation Protocol) is a
protocol for reserving bandwidth for voice calls or any other designated
The first point is that even without real-time traffic, we would still like
to do some forms of traffic management. For example, we would probably like
to prioritize telnet/ssh traffic, which are time-sensitive to at least some
degree, to avoid queuing problems. Example of telnet + file download through
a slow link. This is the same situation as the voice example above.
Simple case with all non-realtime traffic: we're only interested in
Realtime traffic: may need throughput, delay, jitter, loss bounds
Notion of "flows": common user, endpoints, QoS needs
RSVP: Each flow can make a reservation
with the routers. Note that this is a major
retreat from the datagram-routing stateless-router model. Were virtual
circuits the better routing model all along?
Problem with the Internet
1. IP routers are stateless; that is, they use datagram routing. THE biggest
argument in favor of ATM, by ATM proponents, was that virtual circuits
allowed routers to know about connections and thus reserve capacity. How can
a datagram router reserve anything?
IP routers are subject to being rebooted, and links are subject to failure.
If routers have HARD reservation state, they MUST not lose it when
rebooting. If a link fails, hosts may then not be able to tear down a
connection (or if an intermediate router fails!). But routers with "hard"
reservation state MUST receive teardown notification.
The RSVP solution: "soft state" that times out. Soft state can be refreshed
if lost (though with some small probability of failure)
2. Not all routers may start providing RSVP services at once! (Actually, not
any routers may start providing RSVP services ever.)
3. We need to support "all" known major needs for QoS. Voice is one, but the
other biggie is multicast video/audio for teleconferencing. Unicast video is
also supported, but multicasting is technically more complicated to
Overview of multicasting
Typical model: many more receivers
than senders (though teleconferencing may involve lots of people sending, in
Construction of multicast tree
within the routers; this tree also is maintained through soft state.
Multicast, like RSVP, has major "rollout" problems, although one way that
has been addressed is through tunneling (= manual multicast-tree
We need to avoid IPv4 fragmentation, so that receivers can identify packets
by flow info
- "Soft state" is used; the state times out if not refreshed
- Reservations are requested by receiver (though reservation is for data
sent by sender)
- Reservations are in terms of delay and rate/burst
Basic Ethernet version of multicast: using multicast Ethernet addresses
IP: goal is to construct a "spanning tree" (draw picture). Compare to
A receiver R wishing to join the tree send JOIN messages to the sender S.
However, the routers through which the JOIN message passes on the way from R
to S may have nothing to do with the routers on the path from S to R. S then
sends R's identity out towards R; routers forward it along the S-to-R path
but take no action unless there is a need to create a new branch.
In some cases, sender doesn't even necessarily know who the receivers are! Receivers just contact routers.
sender S sends, routers retransmit on all links if
packet arrived directly from S; that is, arrived via shortest path to S.
Shortest-Path multicast: routers with no members of multicast group G
send this info to neighbors.
Kind of expen$ive.
IP addrs for multicast group: 224.x.y.z ... 239.x.y.z
RSVP: reservations compatible with multicast: they are receiver-initiated.
(Multicast senders may not even know who all the receivers are!)
Different receivers may have different needs; these can sometimes be
accommodated by routers, without bothering the sender.
- transparent to senders
- reservations are for a single direction (a "flow")
- suitable for multicast receivers!!!
- mechanisms for CHANGES IN ROUTE
how RSVP routers recognize flows; IP6 flowID
RSVP Architecture service categories:
Within each category, a sender negotiates a "TSpec", somewhat like GCRA
- Guaranteed, (hard delay bound)
- Controlled load (bandwidth guarantee, loss rates, soft delay bound)
- Best effort
Note that the MaxDelay, while bounded, may be relatively large.
- guaranteed throughput
- bounded queuing delay
- no queuing losses
Controlled-load: approximates best-effort service on lightly loaded
networks. That is, queue bursts are "rare".
Controlled-load is good for adaptive
applications. Voice & video can be made adaptive, though for voice the
main adaptation is to increase the total delay. Voice bandwidth is only
adaptive if you are using an adaptive codec (eg speex).
- specified throughput
- no uniform bound on queuing delay, but there is a good mean
delay bound; ie a bound for "most" packets
- queuing losses are very low
Routers involved in RSVP typically do weighted fair queuing on whole service
categories, not per-flow!
typical reservation timeout: 30 sec
RSVP does include teardown messages (both directions), but RSVP won't fall
apart if they are not sent. The reservation will go away after ~30 sec of no
RESV messages (below).
RSVP is transparent to nonparticipating ISPs (though the service agreements
may be impossible to guarantee).
Receivers send periodic (30 sec) RESV messages to refresh routers; routers
forward these "up the path". Note that the receiver sends these to the
router from which it is receiving the reserved flow. Each router figures out
reverse path, and RESV packets are sent along this reverse path by the receiver.
- works well if receiver crashes or becomes disconnected
- works well if router crashes
- how does it figure the path? Because routers send PATH mesg.
Each RSVP sender host transmits RSVP "Path" messages downstream along the
uni-/multicast routes provided by the routing protocol(s), following the
paths of the data. These Path messages store "path state" in each node
along the way. This path state includes at least the unicast IP
address of the previous hop node, which is used to route the Resv messages
hop-by-hop in the reverse direction.
Sometimes RESV messages can be merged. For example, if receiving nodes A and
B each connect to R, which in turn connects to S, and A and B each send RESV
messages towards S, then R may be able to consolidate these and send only a
single RESV message to S.
Sender sends its TSpec in PATH mesg; note that some receivers may ask for a
Unidirectional reservations and different paths each direction
Suppose A⟶B traffic travels A⟶R0⟶R1⟶R2⟶R5⟶B, and B⟶A traffic travels
B⟶R5⟶R4⟶R3⟶R0⟶A, and A is the RSVP sender.
B sends RESV message R5⟶R2⟶R1⟶R0⟶A by virtue of saved state at these
routers, which recognize RESV packets specially. R5 sees a RESV packet
destined for A; it must know to send it to R2 (which it determines from its
reservation state) rather than to R4.
Reservations are initiated by requests sent by receiver to sender (these
follow the standard receiver-to-sender route); sender sends PATH packets
back down to receiver; these lay groundwork for return-path computation. The
PATH message also contains sender's traffic specification, or Tspec.
Problem: too many reservations. Routers have too much to do. And how do we
decide who gets to reserve what?
1. Charge $ for each reservation made
2. Charge a flat additional rate for the right to make reservations
3. Anyone can ask for a reservation, but the answer may be "no". Maybe there
would be a cap on size
Traffic Specifications, or Tspecs
RFC2210: standard Tspec has
- token bucket rate
- token bucket size
- peak data rate
- minimum policed unit m
- max packet size M
NO MaxDelay!!! But see notes in rfc 2212:
applications can use the Tspec values above
to estimate their queuing delay, which when added to propagation delay
gives an accurate total.
How we might do RSVP over the backbone: see notes in rfc2208. Basic idea: do
something else there!! Maybe just trust to best-effort?
How VOIP may facilitate adoption of RSVP (well, probably not outside of
merging of multicast reservations
Reasons for lower QoS on some reserved paths:
MaxDelay QoS only: some receivers may accept larger buffer!!!
- lower QoS toleration/affordability
- stepped-down bandwidth
- subscription to only selective "channels"
link-level cooperation with ATM links
One potential advantage over ATM: QoS can change dynamically (this is
relevant for video). Also RSVP supports multicast.
Admission control: how exercised???
RSVP did not scale well; every router on the path had to maintain connection
state for every connection through it. Thus began a search for something
Problem: Integrated Services may not scale well. Few ISPs have adopted it.
Can we come up with a simpler mechanism?
Differentiated Services: basically just two service classes: high
and low (now three
A simple priority-queue mechanism to handle the service classes is enough.
Rules on which packets can be "premium": we set a maximum rate of such
packets arriving from border router?
Goal: set some rules on admitting premium packets, and hope that their total
numbers to any given destination is small enough that we can meet service
targets (not exactly guarantees)
Packets are marked at ingress only.
This simplifies things. Interior routers just have to grant priority status
to the marked packets. Marking patterns are few in number ("hi" and "low"
might be enough, though there are 12 mark patterns identified below); we are
not marking on a per-flow basis
like RSVP! Interior routers, in other words, just see large flows, and route
DS may start with a Service Level Agreement (SLA) with your ISP, that
defines just what traffic (eg VOIP) will get what service. No need to change
applications. DS bits will probably be set by the ISP. The DS field
repurposes the old IP4 TOS field, widely ignored in the past.
The ISP (not the user!) would mark
VOIP packets as they enter, subject to
some ceiling. These are routed internally (within the ISP) with
premium service. The ISP negotiates with its
ISP for a total bulk delivery of premium packets.
One possible arrangement is that the leaf ISPs would use RSVP, but the
Internet core would run DS. Packets would be DS-marked as they enter the
core, based on their RSVP status.
DS bulk-traffic management generally works best with a "statistical" model:
if you as an ISP are allocating 10% of your bandwidth to VoIP, you limit
each ingress point to 10%. And you hope
that the traffic is uniformly distributed to each egress point, so that at
each egress your outbound VoIP share is also 10%. But note that it doesn't
have to work out this way: you might have five equal-sized ingress points
each accepting 10%, and all of it going to a single egress point that then
has 50% VoIP traffic. Or a 100Mbps ingress point might have 10Mbps (10%) of
VoIP traffic, but all of it goes to a 20Mbps egress link, amounting to 50%.
DS field: this is the first six bits of the second byte of the IP header,
after 0x45 and before the 16-bit length field. The remaining two bits are
the ECN bits.
The 6 bits are divided into 3 bits for class and 3 bits for drop_precedence.
However, some bit patterns are reserved for legacy uses.
Two basic strategies: Expedited Forwarding
(EF) and Assured Forwarding
101 110 "EF", or "Expedited Forwarding": best service.
This amounts to marking packets as EF packets, and then giving them the top
Assured Forwarding: 3 bits of Class, 3 bits of Drop Precedence
100 class 4: best
011 class 3
010 class 2
001 class 1
010: don't drop
(See Stallings fig 8e:19.13 / 9e:20.8)
Main thing: The classes each get priority
service, over best-effort.
What happens when your traffic must be carried by another
ISP? Do they respect the first ISP's DS bits?
What happens if you send more than you've contracted for?
Routers SHOULD implement priority queues for service categories
Basic idea: get your traffic marked for the appropriate class. Then what?
000 000: current best-effort status
xxx 000: traditional IPv4 precedence
PHBs (Per-Hop Behaviors): implemented by all routers
Only "boundary" routers do traffic policing/shaping/classifying/re-marking
to manage categories (re-marking is really part of shaping/policing)
EF: Expedited Forwarding
This is the best service, implemented as basically the highest-priority
service. Packets should experience lowest queuing delay. The lower-priority
traffic should not affect the overall EF performance, but too much EF
traffic can degrade EF
performance. As usual, EF packets will go into the topmost band of the
priority queue at each router.
(We may give bulk traffic some
guaranteed share, but if we have to be worrying about that then we've
probably oversold EF.)
Functionality depends on ensuring that there is not too much EF traffic.
Basically, we control at the boundary the total volume of EF traffic (eg to
a level that cannot saturate the slowest link), so that we have plenty of
capacity for EF traffic.
EF provides a minimum-rate guarantee.
This can be tricky: if we accept input traffic from many sources, and have
four traffic outlets R1, R2, R3, R4, then we should
only accept enough EF traffic that any one
Ri can handle it. But we might go for a more statistical model, if in
practice 1/4 of the traffic goes to each Ri.
AF: Assured Forwarding
Simpler than EF, but no guarantee. Traffic totals can be higher. There is an
easy way to send more traffic: it is just marked as having a lower drop
precedence. This marking would initially be done by the sender, but the
ingress router (at whichever level) is allowed to lower the precedence if it
detects more AF traffic than it has agreed to allow. Such a router would use
a token-bucket filter to determine which packets were conforming and which
were not; the non-conforming packets would have their precedence level
In-out marking: we will refer to high and low precedence as "in" or "out";
packets will be marked by the ingress policer. (Actually, we have three
precedence levels to use for marking.)
The drop precedence level of a packet could
be assigned, for example, by using a leaky bucket traffic policer, which
has as its parameters a rate and a size, which is the sum of two burst
values: a committed burst size and an excess burst size. A packet is
assigned low drop precedence if the number of tokens in the bucket is
greater than the excess burst size [ie bucket is full],
medium drop precedence if the number of tokens in the bucket is greater
than zero, but at most the excess burst size, and high drop precedence if
the bucket is empty.
RIO: RED with In/Out: RED = Random Early Detection
Packet "mangling" is used to mark DS bits; the routers have a sufficient
number of priority bands for the drop precedences.
(It is not quite clear how to handle the different classes; they might get
classful TBF service)
AF works nicely with RIO routers: RED with In and Out (or In, Middle, and
Out): each traffic "level" is subject to a different drop threshold. This
strategy does not involve reordering!