1. Policy Proposal Name: Multihomed Microallocations
2. Proposal Originator:
  a. name: William Herrin
  b. email:  bill@herrin.us
  c. telephone:
  d. organization: self
3. Proposal Version: 1.0
4. Date: 3 August 2009
5. Proposal type: new
6. Policy term: permanent
7. Policy statement:

4.4 IPv4 Allocations and Assignments to Small Multihomed Organizations

4.4.1 Section 4.4 specifies criteria for allocating /23 and /24 IPv4 address blocks to end users and ISPs where the requesting organization is multihomed with multiple Internet vendors but does not meet the minimum usage criteria for address allocation or assignment under Sections 4.2 and 4.3.

4.4.2 Except as specified in section 4.4, the requesting organization must also meet all criteria for receiving addresses specified in section 4.2 if an ISP or section 4.3 if an end user.

4.4.3 Criteria for allocation or assignment

4.4.3.1 The requesting organization must hold exactly one AS number and must already announce IPv4 addresses to the Internet via BGP using its AS number.

4.4.3.2. The requesting organization must announce IPv4 addresses to the Internet via at least two distinct Internet vendors.

4.4.3.3. Notwithstanding 4.4.3.2, the requesting organization must spend at least $8000/year on Internet transit services.

4.4.3.4. Upon annual renewal of the allocation or assignment received under section 4.4, if the requesting organization fails to demonstrate that it continues to announce IPv4 addresses to the Internet via at least two distinct Internet vendors the allocation or assignment is revoked and returned to ARIN.

4.4.3.5. The requesting organization must agree to withdraw any other BGP routes it announces from the BGP table within 6 months of receiving an allocation or assignment under section 4.4. If the organization continues to receive IP addresses from its ISPs, those IP addresses will be single-homed within the ISP's larger aggregate announcement.

4.4.3.6. If the requesting organization fails to announce the allocation or assignment received under section 4.4 to the Internet using its AS number for at least 4 months total within a service year, the allocation or assignment is revoked and returned to ARIN.

4.4.3.7. If the requesting organization already holds IPv4 addresses directly from ARIN, from any other RIR or legacy addresses, the organization must agree to renumber out of those addresses and surrender them to the appopriate RIR within 6 months of receiving an allocation or assignment under section 4.4.

4.4.3.8. The requesting organization agrees to return the allocation or assignment received under section 4.4 to ARIN within 6 months of receiving another allocation or assignment from any RIR.

4.4.3.9. For allocations of /23 and larger, the requesting organization shall meet the utilization rate criteria described in section 4.2 for ISPs and section 4.3 for end users. As /24 is the smallest address block known to be generally routable on the Internet, no utilization criteria will be applied to requests for a /24.

8. Rationale

The reason behind a /20 minimum assignment for single-homed orgs is fairly straightforward: an ARIN allocation adds a route to the BGP table which wouldn't otherwise be needed. Routes are expensive and the cost falls into overhead since it isn't recoverable directly from the org announcing the route. And we're not really certain how many routes we can handle before the network falls over. So, we restrict the availability of non-aggregable IP addresses to just very large organizations. For smaller orgs, renumbering sucks but at least it only costs the renumbering org, not everyone else.

The reason behind nothing smaller than a /24 is also straightforward: many if not most ISPs filter out BGP announcements smaller than /24. There is tremendous inertia behind /24 as the minimum backbone-routable quantity going back to the pre-CIDR days of class-C addresses. So, an ARIN allocation smaller than /24 would generally be wasted addresses, unusable on the Internet.

But why peg multihomed orgs at /22 instead of /24? Multivendor multihomed orgs have to announce a route anyway, regardless of whether the addresses are from an ISP or directly from ARIN. Their routes are not aggregable, even if assigned from ISP space. That's the way the technology works and no new tech in the pipeline is likely to change it.

With load balanced server clusters and NAT you can pack a heck of a lot of punch into a multihomed /24 if you want to. And as a community it's to our benefit to want registrants to pack the maximum punch into their address space: IPv4 addresses are becoming scarce. So why restrict ARIN assignments to folks who can write papers which justify a /22?

FAQ

Q. Why not just use ISP addresses if you're too small?

A1. ISPs have conflicting requirements placed on their use of IP addresses. They want, for example, to prevent address spoofing at their borders by not allowing traffic from their addresses to enter their network from another ISP. These goals conflict with multivendor multihoming and generally make a mess when a customer wants to multihome.

A2. Renumbering is expensive and painful. We should require it only when it serves a reasonable public policy goal such as reducing the consumption of BGP routing slots.

A3. We've seen common counterproductive situations where multihomed end users make many discontiguous /24 announcements until they eventually seek ARIN address space.

Q. But aren't your routes aggregable with your ISP's routes if you use your ISP's address space?

A. No. For routes to be aggregable, two things must be true:

1. The routes must be contiguous, sharing a common network/netmask.
2. The routes must share an identical network topology.

In the mutlivendor multihomed case addressed by this proposal, the routes almost never share an identical network topology. As a result the routes can not be aggregated even if cut from the ISP's address space. Single-homed networks are excluded from this proposal precisely because they always share a network topology with the ISP.

Q. Can't other organizations filter routes at the RIR minimums and user the ISP's covering route to reach you?

A. Maybe. With a bunch of ifs. You might not be connected to the ISP who has the covering route, and what if he doesn't have the more specific route to you?

The answer is more decisive on a practical level: The author was unable to identify any ISPs who presently both filters on RIR minimums and chooses not to carry a default route to an upstream ISP that doesn't filter. Hence there is no apparent route filtering value to the RIR minimums.

Q. What's so messy about multihoming with a cutout from an ISP's allocation?

A. Many things. Here are some of them:

* As an ISP I want to drop packets from the Internet that purport to be from my addresses (spoofed packets). I can't do that with packets from a multihomed network: in the normal course of failure recovery or traffic engineering, the multihomed user may originate packets to hosts on my network using the IP addresses I assigned him, but via his other ISPs. These packets will arrive at my Internet interfaces and if I drop them as spoofed packets, I've broken my customer's connectivity.

* As an ISP I want to reject false route announcements entering my system which purport to serve my addresses. And I want my pager to go off and let me know someone is trying to hijack my address space. My multihomed customer will, in some circumstances, want me to route all packets to him via his other ISP. That means I have to accept his route announcement from other ISPs. To do this, I have to write some tricky filtering rules that accept his routes right smack in the middle of the address space where I generally want to reject routes.

* As an ISP, I want to aggregate my contiguous address space into a single route announcement. It's part of being a good citizen: don't waste the TCAM slots in everybody else's router. But I have to carefully exclude my multihomed customer's routes from that aggregation. Packets follow the more specific route. If he announces his more specific route to his other ISP but I roll his route up into my less specific route when I announce it then all his packets will go to his other ISP instead of to me and I won't get paid.

* Lots of folks disaggregate their route announcements for the purpose of traffic engineering. Two or three hops away from their system, those TE routes are irrelevant. In theory I should be able to filter a lot of that out by discarding the routes smaller than the RIR minimum allocation for that /8. That would save me money and make my routing updates work faster. But if I try it, I end up filtering mutlihomed customers so that I can only reach them via the ISP that assigned their addresses. At best that damages the effectiveness of my routing. At worst it cuts me off from sites my customers want to access when my competitors who just accept /24 everywhere don't have a problem. Oops.

Q. Where do the announced addresses in 4.4.3.1 come from?

A. Most likely from one of the ISPs as described in NRPM section 4.2.3.6. You go through the process of getting them assigned and announced to demonstrate that you're not a poser. Then you get addresses from ARIN under this proposal and return the 4.2.3.6 addresses to your ISP.

Q. What does "distinct vendors" mean?

A. It means two different ISPs like Verizon and Sprint. Two T1s with a single ISP can be accomodated without announcing a route into the Internet backbone, so such a connection does not qualify for addresses under this proposal.

Q. $8000? What's that all about?

A. The best available estimate of the systemic cost of carrying a route in the Internet backbone table is around $8000/year. See http://bill.herrin.us/network/bgpcost.html for the cost estimate. If you're going to play in the backbone, you should really be putting more money into the system than you're taking out. If you have two $600 T1's then you're spending nearly twice that anyway. This  limits the folks who want to multihome their $50 DSL and cable modems.

Q. How reliable is that estimate? Does it change? Shouldn't ARIN routinely update that estimate rather than codifying the specific number in the policy?

A. In theory ARIN staff should set the number. In practice, professional cost analysts are expensive and hard data on things like router count is almost impossible to get anyway. Even if a more reliable cost analysis could be produced, we still wouldn't know what multiple of that cost was "fair" for the pay-in. 1x? 2x? 5x? Let's just pick a number that's our best guess at fair, and move forward with it.

Besides, the $8k rule will probably turn out to be a non-operative part of the policy. Companies providing $50 DSL service are disinclined to set up BGP sessions with their customers anyway. I include it in the name of caution so that we're proof against a deluge of multihomed cable-modem users but I expect that with some experience under our belts we'll find that we can safely submit a follow-on policy proposal that removes the $8k requirement.

Q. I have to renumber when exactly?

A. If you have IP addresses under section 4.4, you get to announce that one allocation or assignment to the Internet via BGP. In fact, we'd really prefer that you only announce one single route, even if you get a /23 or larger. You don't get to collect two assignments and then ten discontiguous assignments and burn up the BGP table. Until you reach the minimums in sections 4.2 or 4.3, you renumber each time you grow large enough to justify the next bigger allocation or assignment. Yes, that's unfortunate and painful. But burning up the BGP table would be even more unfortunate.

Practically speaking, you'll renumber zero or one times. Either you'll never renumber because the /24 was enough to do the job, or when you run out of space in your original assignment, you'll be big enough to find a way to meet the minimum size criteria in section 4.2 or 4.3 so that you don't have to renumber again.

Q. But renumbering is expensive! What's the difference between having to renumber under this proposal and having to renumber when I change ISPs?

A. You'll have to renumber less often if at all. The big deal is that you don't have to renumber merely because you changed vendors and you don't run into a sticky mess of   filtering rules as ISPs try to keep control of their address space.

Q. Doesn't this discriminate against some kinds of multihoming?

A. In addition to multihoming with your own AS number, its possible to have two ISPs seperately announce your addresses or announce with a private AS number that they strip from their peer announcements. This proposal is more than complex enough. For the sake of making verification simple, let's just say that tiny registrants will announce with their own AS number, period.

Q. Does this proposal affect IPv6 allocations and assignments?

A. It does not appear to impact ISP allocations whose criteria is spelled out in NRPM section 6.5.1.1. It does impact end user assignments under NRPM section 6.5.8.1. End users who qualify for addresses under this policy will also be qualified for an IPv6 /48.

Q. Have there been previous policy proposals to extend allocations and assignments from ARIN to /24?

A. Yes. See the discussions in March and April of 2007 for proposal 2007-6. http://lists.arin.net/pipermail/arin-ppml/

In proposal 2007-6, the /22 size for multihomers in section 4.3 was simply changed to /24. It met the following criticisms:

* Could make it easier for spammers. This seems to reflect some concern at the time over whether ARIN's policies made things easy for spammers to hop IP addresses and was probably a red herring. Spammers aren't interested in registering with anybody. They want address space as anonymously as possible for as long as possible as cheaply as possible exerting as little effort as possible. All of these things make address space under this proposal unattractive to spammers.

* Could create a land rush. This seems like an unreasonable concern for the instant proposal: anyone who can justify addresses under this proposal can justify the same addresses from their ISP already. So why hassle them with ISP addresses?

* Could create a new swamp. The renumbering requirements in this proposal prevent that problem.

* Unfair to ISPs since it only applies to end users. This proposal applies to both.

* Staff worried that it could increase request workload. If it does, the fees could presumably be set accordingly.

Proposal 2007-6 also tried to make the following point: Don't penalize the honest. An org that has ponied up the cash to be multihomed with multiple vendors can often restructure their network to require a /22 long enough to get one. Refusing ARIN assignments smaller than /22 encourages behavior which is contrary to ARIN's general public policy goal of conservation. We'll be better off as a community if the folks who are completely honest about their need get the same or better treatment than the ones who lie.

Implementation notes:

This proposal asks for verification of multihoming somewhat beyond what the rest of the NRPM requires. It does this in order to prevent a land rush of cheaters. Not that there's likely to be a land rush of cheaters (or if there is that they're likely to ask for less than a /22), but better safe than sorry.

Verifying that there's a BGP announcement is trivial: go to any of the hundreds of looking glasses. For the four-month rule, staff may want to let it be practiced in the breach for now. That is, don't go out and look unless someone complains. Writing software that actively checks for it can be part of the address recovery strategy after depletion.

Same deal with the route withdrawals: if slot consumption bugs the ISPs, let them write a script which trolls for cheaters and then complain.

To verify multihoming, I would suggest asking the registrant to provide a letter of service from two ISPs in which they indicate that they're under contract to announce routes for AS XXX.

To verify $8k, you might ask for a month's bills and check that 12 months worth adds up to $8k.

There's a respectable chance that folks requesting a /23 or larger will continue to grow. It would be nice if ARIN staff made reasonable efforts to reserve adjacent space for a year or so they may be able to get the next larger size without having to renumber. With the scarcity of IPv4 addresses this won't always be possible, but it would be good to do as a "best efforts" kind of thing. .

9. Timetable for implementation: immediate

100 messages later, here's a summary of the discussion on lowering the ARIN minimum allocation for Multihomed organizations:

1. Singlehomed and single-vendor multihomed uses a-priori excluded from the discussion. Single-homed users' routes are aggregable with the ISP's if assigned by the ISP. If assigned by ARIN they're not aggregable. Multi-vendor multihomed users' routes are not aggregable regardless of who assigns them. Hence it is appropriate to consider the two cases (multihomed, not multihomed) seperately.

2. Current ARIN minimum allocations are /22 for multihomed organizations.

3. RIR minimums are about routing. They help suppress the growth of the backbone BGP table by encouraging aggregability. They appear to serve no other purpose and are actually counterproductive to ARIN's companion goal of allocating the minimum number of addresses an organization needs.

4. If a multihomed organization announcing a /24 cut from an ISPs's space trades that /24 for a CIDR block from ARIN, there is zero net impact on the routing table.

5. Should make cautious, conservative changes to the minimum allocation and look for unintended consequences before lowering the minimums futher.

6. Doubtful value to lowering the minimums beyond /24 since routes smaller than /24 are not generally routable on the backbone right now. /24 is the defacto minimum enforced by the ISPs who comprise the core of the Internet. Would probably be counterproductive since we'd  allocate scarce IP addresses in quantities that aren't actually usable for any valid purpose.

7. Unable to identify any ISPs who  presently both filter on RIR minimums and choose not to carry a default route to an upstream ISP that doesn't filter. Hence no apparent route filtering value to the RIR minimums, at least not in practice.

8. Common counterproductive situations with the current policy observed where an organization seperately announced as many as 6 discontiguous /24's from ISPs' space into the BGP table before finally requesting and receiving a single ARIN allocation.

9. The current minimum can result in allocating more addresses than the requesting organization actually wants, as long as it finds a way to qualify.

10. Worries about the effect of multihomed DSL users on the routing table, regardless of whether addresses are assigned from ISP space or from ARIN. Some observe that ISPs generally won't provide BGP service to DSL or cable modem users.

11. Worries about whether the availability of  provider-indepentent (PI) space if you but multihome would push folks who don't already multihome to start.

With all of this in mind, it seems to me that there are two ways we can reasonably go with the policy:

Alternative #1: Lower the minimum for multihomed organizations to /23. Give it a year to see if we get unintended consequences. Then lower it to /24.

Alternative #2: Craft a fresh policy for small multihomed organizations to fill the gap between the current /22 ARIN minimum and the /24 defacto minimum on the backbone. Limit folks to 1 assignment longer than /24 and none once they have at least a /22 so that starting allocations smaller doesn't induce routing table growth. Exclude folks for whom their ISP bills are less than the systemic cost of carrying a route. Make sure folks getting these allocations are and stay multihomed with multiple vendors who are really doing BGP. Make sure they're not just using it as an excuse to get PI space. Maybe even raise the regular minimum from /22 back to /20 in order to encourage aggregation if this /24 policy works out.

Thoughts?