Möhrenfeld

I faced an interesting problem today. In a customer L3VPN/VRF a route was hidden and it was not apparent why. In JunOS when you do a show route ... command you will get a summary at the top that tells you some statistics. In this case, four routes are hidden (three were expected, the one displayed here should not have been hidden). You can display hidden routes by adding the hidden option to the show route command:

user@core1> show route table customer-vrf.inet.0 10.1.2.0/22 hidden

customer-vrf.inet.0: 31 destinations, 34 routes (30 active, 0 holddown, 4 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.2.0/22      [BGP ] 1d 17:20:31, MED 0, localpref 100, from 198.51.100.27
                      AS path: 65003 ?, validation-state: unverified
                      to 198.51.100.174 via ae1.0, Push 16, Push 300928(top)
                    > to 198.51.100.66 via ae17.0, Push 16, Push 302608(top)

So why is it hidden? To get more details you can use the show route detail command. I cut out some lines not relevant here and if you need even more information use the show route extensive option.

user@core1> show route table customer-vrf.inet.0 10.1.2.0/22 hidden detail

customer-vrf.inet.0: 31 destinations, 34 routes (30 active, 0 holddown, 4 hidden)
10.1.2.0/22 (1 entry, 0 announced)
        BGP                 /-101
                Route Distinguisher: 198.51.100.27:5
                Next hop type: Indirect, Next hop index: 0
                Address: 0x271f862c
                Next-hop reference count: 5
                Source: 198.51.100.27
                Next hop type: Router, Next hop index: 0
                Next hop: 198.51.100.174 via ae1.0 weight 0x1
                Label operation: Push 16, Push 300928(top)
                Next hop: 198.51.100.66 via ae17.0 weight 0x1, selected
                Label operation: Push 16, Push 302608(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Protocol next hop: 198.51.100.27
                Label operation: Push 16
                State: <Secondary Hidden Int Ext ProtectionCand>
                Local AS: 65555 Peer AS: 65555
                Age: 1d 17:20:29 	Metric: 0 	Metric2: 2003 
                Validation State: unverified 
                Task: BGP_65555.198.51.100.27
---------->     AS path: 65003 ? (Looped: 65003)     <--------------------------
                AS path: Recorded
                Communities: target:65555:55
                Import
                VPN Label: 16
                Localpref: 100
                Router ID: 198.51.100.27
---------->     Hidden reason: AS path loop     <-------------------------------
                Primary Routing Table bgp.l3vpn.0

As we can see the router is hiding this route because it detects an AS loop. The looped AS is displayed as AS65003 which is interesting because that AS is not the local AS for this router. AS65555 is used as the local AS. AS65003 is only used on the customer (CPE) router that is announcing this route into the VRF.

Why is the router deciding that it must prevent an AS loop with AS65003? To answer this question we must first introduce the local-as command. local-as allows you to “fake” your AS so that it appears different to your eBGP peer. This also affects AS path loop detection, as the AS specified in local-as is also used in detecting BGP routing loops. This option is system-wide! So if you specify local-as anywhere in your router configuration it will be used for loop detection everywhere, even in VRFs. The documentation states this somewhat lengthy in the last info box at the bottom:

Note: If you configure the local AS values for any BGP group, the detection of routing loops is performed using both the AS and the local AS values for all BGP groups.

[..]

When you configure the local AS within a VRF, this impacts the AS path loop-detection mechanism. All of the local-as statements configured on the device are part of a single AS domain. The AS path loop-detection mechanism is based on looking for a matching AS present in the domain.

This is the answer to our problem. Even though the VRF for the customer is not using AS65003 locally, somewhere else in a (global) BGP group local-as 65003 was configured and this was influencing the AS path loop detection on the router.

We can confirm this with show as-path domain:

user@core1> show as-path domain
Domain: 2        Primary: 65555
    References:         10 Paths:     138718
    Flags: Master
    Local AS: 65555  Loops: 1
    Local AS: 65003  Loops: 1

After removing the local-as command from the configuration (it was for a test and no longer needed) the route became active as expected.

What can we learn from this:

  • Be aware that local-as has side-effects
  • local-as influences BGP AS path loop detection regardless of where it is configured
  • When the customer announces you BGP routes check the AS they’re using for conflicts (and if in doubt get them an official AS)