Origin Incomplete - BGP multihoming and load balancing issues

Intro

I have been recently working on the BGP deployment in one of our office branches. After long weeks of paper work spent on requesting public IP address range and BGP AS number assignment from IANA, negotiating and signing the contracts with ISPs, ordering required network devices (I bought Cisco 2811 routers) I finally got what I needed and started implementing BGP in my site. After connecting all the devices together and providing initial IP configuration my BGP infrastructure looks as shown on the following picture:

R1(config)# router bgp IJKL

R1(config-router)# network I.J.K.0 mask 255.255.255.0

R1(config-router)# neighbor A.B.C.1 remote-as ABCD

R1(config-router)# neighbor I.J.K.2 remote-as IJKL

R1(config-router)# neighbor I.J.K.2 next-hop-self

R2(config)# router bgp IJKL

R2(config-router)# network I.J.K.0 mask 255.255.255.0

R2(config-router)# neighbor E.F.G.1 remote-as EFGH

R2(config-router)# neighbor I.J.K.1 remote-as IJKL

R1(config-router)# neighbor I.J.K.1 next-hop-self

First of all, after setting up initial BGP peering between my routers and ISP endpoints, my routers got propagated with the whole global BGP routing table which in fact killed their memory and made them completely inoperational. So I had to call both ISPs and ask them to propagate only the default route in BGP updates. After completing that, the BGP started working stable, however as I wanted to make sure that such a situation doesn't happen again in future, I implemented two route-maps:

• allowing BGP updates from ISPs containing only the paths originating from the neighboring ISP AS
• allowing BGP updates to ISPs containing only the path to my company network

R1(config)# ip as-path access-list 10 permit ^ABCD$

R1(config)# access-list 10 permit I.J.K.0 0.0.0.255

R1(config)# route-map ISP1IN permit 10

R1(config-route-map)# match as-path 10

R1(config-route-map)# exit

R1(config)# route-map ISP1OUT permit 10

R1(config-route-map)# match ip address 10

R1(config-route-map)# exit

R1(config)# router bgp IJKL

R1(config-router)# neighbor A.B.C.1 route-map ISP1IN in

R1(config-router)# neighbor A.B.C.1 route-map ISP1OUT out

R2(config)# ip as-path access-list 10 permit ^EFGH$

R2(config)# access-list 10 permit I.J.K.0 0.0.0.255

R2(config)# route-map ISP2IN permit 10

R2(config-route-map)# match as-path 10

R2(config-route-map)# exit

R2(config)# route-map ISP2OUT permit 10

R2(config-route-map)# match ip address 10

R2(config-route-map)# exit

R2(config)# router bgp IJKL

R2(config-router)# neighbor E.F.G.1 route-map ISP2IN

R2(config-router)# neighbor E.F.G.1 route-map ISP2OUT out

As everything has been working fine so far I proceed to load balancing deployment.

Outbound traffic load balancing

In multihomed BGP environment with 2 ISPs the outbound traffic should be automatically load balanced in such a way that:

• all traffic that reaches R1 and matches the default route should go via ISP1 and use R2 as a backup route
• all traffic that reaches R2 and matches the default route should go via ISP2 and use R1 as a backup route

That's what I expected to see when looking into the routing tables on my routers, but surprisingly the output was completely different:

R1#show ip route

...

Gateway of last resort is I.J.K.2 to network 0.0.0.0

B*    0.0.0.0/0 [200/100] via I.J.K.2, 1w1d
      A.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        A.B.C.0/30 is directly connected, FastEthernet0/1
L        A.B.C.2/32 is directly connected, FastEthernet0/1
      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        I.J.K.0/24 is directly connected, FastEthernet0/0
L        I.J.K.1/32 is directly connected, FastEthernet0/0

R2#show ip route

...

Gateway of last resort is E.F.G.1 to network 0.0.0.0

B*    0.0.0.0/0 [20/0] via E.F.G.1, 1w1d

      E.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        E.F.G.0/30 is directly connected, FastEthernet0/1

L        E.F.G.2/32 is directly connected, FastEthernet0/1

      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        I.J.K.0/24 is directly connected, FastEthernet0/0

L        I.J.K.2/32 is directly connected, FastEthernet0/0

So all my outbound traffic was flowing via ISP2. Afer deeper research I found the following detailed information:

R1#show ip bgp 0.0.0.0

BGP routing table entry for 0.0.0.0/0, version 55

Paths: (2 available, best #1, table default)

  Not advertised to any peer

  EFGH

    I.J.K.2 from I.J.K.2 (I.J.K.2)

      Origin IGP, metric 100, localpref 100, valid, internal, best

  ABCD

    A.B.C.D from A.B.C.D (A.B.C.D)

      Origin incomplete, localpref 100, valid, external

R2#show ip bgp 0.0.0.0

BGP routing table entry for 0.0.0.0/0, version 8

Paths: (1 available, best #1, table default)

  Advertised to update-groups:

     1         

  EFGH
    E.F.G.1 from E.F.G.1 (E.F.G.1)

      Origin IGP, localpref 100, valid, external, best

So it revealed that R1 had been receiving BGP updates from ISP1 with some strange incomplete value of Origin attribute. I've made some googling then and found that according to Cisco the incomplete value of BGP origin attribute means that the route is inserted into the BGP updates from some other routing protocol. So there was something on the ISP1 side. I raised a case with ISP1, but in meantime I found the information on the following blog that it's possible to change the Origin attribute values using route maps. I tested that and finally got the expected results:

R1#show ip route

...

Gateway of last resort is I.J.K.2 to network 0.0.0.0

B*    0.0.0.0/0 [20/100] via A.B.C.1, 1w1d

      A.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        A.B.C.0/30 is directly connected, FastEthernet0/1

L        A.B.C.2/32 is directly connected, FastEthernet0/1

      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        I.J.K.0/24 is directly connected, FastEthernet0/0

L        I.J.K.1/32 is directly connected, FastEthernet0/0

R1#show ip bgp 0.0.0.0

BGP routing table entry for 0.0.0.0/0, version 55

Paths: (2 available, best #1, table default)

  Not advertised to any peer

  EFGH

    I.J.K.2 from I.J.K.2 (I.J.K.2)

      Origin IGP, metric 100, localpref 100, valid, internal, best

  ABCD

    A.B.C.D from A.B.C.D (A.B.C.D)

      Origin IGP, localpref 100, valid, external

R2#show ip route

...

Gateway of last resort is E.F.G.1 to network 0.0.0.0

B*    0.0.0.0/0 [20/0] via E.F.G.1, 1w1d

      E.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        E.F.G.0/30 is directly connected, FastEthernet0/1

L        E.F.G.2/32 is directly connected, FastEthernet0/1

      I.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C        I.J.K.0/24 is directly connected, FastEthernet0/0

L        I.J.K.2/32 is directly connected, FastEthernet0/0

R2#sh ip bgp 0.0.0.0

BGP routing table entry for 0.0.0.0/0, version 21

Paths: (2 available, best #1, table default)

  Advertised to update-groups:

     1         

  ABCD

    E.F.G.1 from E.F.G.1 (E.F.G.1)

      Origin IGP, localpref 100, valid, external, best

  EFGH

    I.J.K.1 from I.J.K.1 (I.J.K.1)

      Origin IGP, metric 100, localpref 100, valid, internal

When I finally get a response from ISP1 I was informed that what I am requesting for is not possible to implement and that I am wrong, so I sent them all the above information and after a week of silence I got an information that I am right indeed and that they were wrong. I was said that their router placed on my endpoint was messing the BGP updates however they managed to fix that. I tested that and indeed after removing my route-map I had back the outbound BGP failover on my site.

Inbound traffic load sharing

Lets go the inbound traffic policing. Basically, in multihomed BGP environment with 2 ISPs it's not possible to implement inbound BGP load balancing however it's possible to implement load sharing instead. How do we achieve that is by splitting the entire network into smaller, logical subnets that are advertised on the edge routers in parallel with the summarized network. As the subnet mask length in routing decision process has always precedence over the routing protocol metric we can guarantee that the traffic destined to a particular logical subnet will always pass via only one of the ISPs. In case of one of the ISPs or links failure, the whole subnet is still advertised on the failover peer. So how do we configure it? Please, have a look on the attached listing:

R1(config)# router bgp IJKL
R1(config-router)# network I.J.K.0 mask 255.255.255.128

R2(config)# router bgp IJKL
R2(config-router)# network I.J.K.128 mask 255.255.255.128

So I split my network into 2 separate subnets:

• I.J.K.0/25 advertised on R1 only
• I.J.K.128/25 advertised on R2 only

while the whole I.J.K.0/24 network is still being advertised on both routers. I've tested any possible scenario and it looks like it's working fine. Fortunately, no ISP related issues this time ;).