Origin Incomplete - BGP multihoming and load balancing issues

Intro


I have been recently working on the BGP deployment in one of our office branches. After long weeks of paper work spent on requesting public IP address range and BGP AS number assignment from IANA, negotiating and signing the contracts with ISPs, ordering required network devices (I bought Cisco 2811 routers) I finally got what I needed and started implementing BGP in my site. After connecting all the devices together and providing initial IP configuration my BGP infrastructure looks as shown on the following picture:

BGP multihoming and load balancing issues

R1(config)# router bgp IJKL
R1(config-router)# network I.J.K.0 mask 255.255.255.0
R1(config-router)# neighbor A.B.C.1 remote-as ABCD
R1(config-router)# neighbor I.J.K.2 remote-as IJKL
R1(config-router)# neighbor I.J.K.2 next-hop-self

R2(config)# router bgp IJKL
R2(config-router)# network I.J.K.0 mask 255.255.255.0
R2(config-router)# neighbor E.F.G.1 remote-as EFGH
R2(config-router)# neighbor I.J.K.1 remote-as IJKL
R1(config-router)# neighbor I.J.K.1 next-hop-self

First of all, after setting up initial BGP peering between my routers and ISP endpoints, my routers got propagated with the whole global BGP routing table which in fact killed their memory and made them completely inoperational. So I had to call both ISPs and ask them to propagate only the default route in BGP updates. After completing that, the BGP started working stable, however as I wanted to make sure that such a situation doesn't happen again in future, I implemented two route-maps:
  • allowing BGP updates from ISPs containing only the paths originating from the neighboring ISP AS
  • allowing BGP updates to ISPs containing only the path to my company network

R1(config)# ip as-path access-list 10 permit ^ABCD$
R1(config)# access-list 10 permit I.J.K.0 0.0.0.255
R1(config)# route-map ISP1IN permit 10
R1(config-route-map)# match as-path 10
R1(config-route-map)# exit
R1(config)# route-map ISP1OUT permit 10
R1(config-route-map)# match ip address 10
R1(config-route-map)# exit
R1(config)# router bgp IJKL
R1(config-router)# neighbor A.B.C.1 route-map ISP1IN in
R1(config-router)# neighbor A.B.C.1 route-map ISP1OUT out

R2(config)# ip as-path access-list 10 permit ^EFGH$
R2(config)# access-list 10 permit I.J.K.0 0.0.0.255
R2(config)# route-map ISP2IN permit 10
R2(config-route-map)# match as-path 10
R2(config-route-map)# exit
R2(config)# route-map ISP2OUT permit 10
R2(config-route-map)# match ip address 10
R2(config-route-map)# exit
R2(config)# router bgp IJKL
R2(config-router)# neighbor E.F.G.1 route-map ISP2IN
R2(config-router)# neighbor E.F.G.1 route-map ISP2OUT out

As everything has been working fine so far I proceed to load balancing deployment.

Outbound traffic load balancing


In multihomed BGP environment with 2 ISPs the outbound traffic should be automatically load balanced in such a way that:
  • all traffic that reaches R1 and matches the default route should go via ISP1 and use R2 as a backup route
  • all traffic that reaches R2 and matches the default route should go via ISP2 and use R1 as a backup route
That's what I expected to see when looking into the routing tables on my routers, but surprisingly the output was completely different:

R1#show ip route

...

Gateway of last resort is I.J.K.2 to network 0.0.0.0

B*    0.0.0.0/0 [200/100] via I.J.K.2, 1w1d
      A.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        A.B.C.0/30 is directly connected, FastEthernet0/1
L        A.B.C.2/32 is directly connected, FastEthernet0/1
      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        I.J.K.0/24 is directly connected, FastEthernet0/0
L        I.J.K.1/32 is directly connected, FastEthernet0/0

R2#show ip route

...

Gateway of last resort is E.F.G.1 to network 0.0.0.0

B*    0.0.0.0/0 [20/0] via E.F.G.1, 1w1d
      E.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        E.F.G.0/30 is directly connected, FastEthernet0/1
L        E.F.G.2/32 is directly connected, FastEthernet0/1
      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        I.J.K.0/24 is directly connected, FastEthernet0/0
L        I.J.K.2/32 is directly connected, FastEthernet0/0

So all my outbound traffic was flowing via ISP2. Afer deeper research I found the following detailed information:

R1#show ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 55
Paths: (2 available, best #1, table default)
  Not advertised to any peer
  EFGH
    I.J.K.2 from I.J.K.2 (I.J.K.2)
      Origin IGP, metric 100, localpref 100, valid, internal, best
  ABCD
    A.B.C.D from A.B.C.D (A.B.C.D)
      Origin incomplete, localpref 100, valid, external

R2#show ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 8
Paths: (1 available, best #1, table default)
  Advertised to update-groups:
     1         
  EFGH
    E.F.G.1 from E.F.G.1 (E.F.G.1)
      Origin IGP, localpref 100, valid, external, best

So it revealed that R1 had been receiving BGP updates from ISP1 with some strange incomplete value of Origin attribute. I've made some googling then and found that according to Cisco the incomplete value of BGP origin attribute means that the route is inserted into the BGP updates from some other routing protocol. So there was something on the ISP1 side. I raised a case with ISP1, but in meantime I found the information on the following blog that it's possible to change the Origin attribute values using route maps. I tested that and finally got the expected results:


R1#show ip route

...

Gateway of last resort is I.J.K.2 to network 0.0.0.0

B*    0.0.0.0/0 [20/100] via A.B.C.1, 1w1d
      A.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        A.B.C.0/30 is directly connected, FastEthernet0/1
L        A.B.C.2/32 is directly connected, FastEthernet0/1
      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        I.J.K.0/24 is directly connected, FastEthernet0/0
L        I.J.K.1/32 is directly connected, FastEthernet0/0

R1#show ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 55
Paths: (2 available, best #1, table default)
  Not advertised to any peer
  EFGH
    I.J.K.2 from I.J.K.2 (I.J.K.2)
      Origin IGP, metric 100, localpref 100, valid, internal, best
  ABCD
    A.B.C.D from A.B.C.D (A.B.C.D)
      Origin IGP, localpref 100, valid, external

R2#show ip route

...

Gateway of last resort is E.F.G.1 to network 0.0.0.0

B*    0.0.0.0/0 [20/0] via E.F.G.1, 1w1d
      E.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        E.F.G.0/30 is directly connected, FastEthernet0/1
L        E.F.G.2/32 is directly connected, FastEthernet0/1
      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        I.J.K.0/24 is directly connected, FastEthernet0/0
L        I.J.K.2/32 is directly connected, FastEthernet0/0

R2#sh ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 21
Paths: (2 available, best #1, table default)
  Advertised to update-groups:
     1         
  ABCD
    E.F.G.1 from E.F.G.1 (E.F.G.1)
      Origin IGP, localpref 100, valid, external, best
  EFGH
    I.J.K.1 from I.J.K.1 (I.J.K.1)
      Origin IGP, metric 100, localpref 100, valid, internal

When I finally get a response from ISP1 I was informed that what I am requesting for is not possible to implement and that I am wrong, so I sent them all the above information and after a week of silence I got an information that I am right indeed and that they were wrong. I was said that their router placed on my endpoint was messing the BGP updates however they managed to fix that. I tested that and indeed after removing my route-map I had back the outbound BGP failover on my site.

Inbound traffic load sharing


Lets go the inbound traffic policing. Basically, in multihomed BGP environment with 2 ISPs it's not possible to implement inbound BGP load balancing however it's possible to implement load sharing instead. How do we achieve that is by splitting the entire network into smaller, logical subnets that are advertised on the edge routers in parallel with the summarized network. As the subnet mask length in routing decision process has always precedence over the routing protocol metric we can guarantee that the traffic destined to a particular logical subnet will always pass via only one of the ISPs. In case of one of the ISPs or links failure, the whole subnet is still advertised on the failover peer. So how do we configure it? Please, have a look on the attached listing:

R1(config)# router bgp IJKL
R1(config-router)# network I.J.K.0 mask 255.255.255.128

R2(config)# router bgp IJKL
R2(config-router)# network I.J.K.128 mask 255.255.255.128

So I split my network into 2 separate subnets:

  • I.J.K.0/25 advertised on R1 only
  • I.J.K.128/25 advertised on R2 only
while the whole I.J.K.0/24 network is still being advertised on both routers. I've tested any possible scenario and it looks like it's working fine. Fortunately, no ISP related issues this time ;).


6 comments:

  1. How do we achieve that is by splitting the entire network into smaller, logical subnets that are advertised on the edge routers in parallel with the summarized network.

    ReplyDelete
  2. the whole subnet is still advertised on the failover peer. So how do we configure it? Please, have a look on the attached listing.......

    ReplyDelete
  3. if ISP don't accept the /25 network then will it work ?

    ReplyDelete
    Replies
    1. @vishwas: For best practice, please consult with your ISP. It worked like a charm in my case.

      Delete
  4. I have a situation in a customer site implementation. I'm unable to see the figure attached. It will be easy to understand this and resolve the issue at customer implementation. Can you please send me that figure. mailtobashitha@gmail.com

    ReplyDelete