Möhrenfeld

Debian Secure APT Keys for Third-Party Repositories

blog@karotte.org (Sebastian Wiesinger) — Thu, 13 Feb 2020 00:00:00 +0000

In most (if not all) installation instructions for Debian Third-Party repositories you will find a sentence like this:

wget -O - https://packages.example.com/example.key | apt-key add -

This allows the package maintainers key to sign ALL Debian packages, not only the packages from their repository. I’m not sure why this is still so common when a more secure alternative exists.

Instead of adding the key to the global apt key repository, you can just put the (dearmored) version in the shared key directory:

wget -O - https://packages.example.com/example.key | gpg --dearmor > /usr/share/keyrings/example.gpg

After that, just specify the key in a signed-by option for your repository in /etc/apt/sources.list.d/example.list:

deb [arch=amd64 signed-by=/usr/share/keyrings/example.gpg] https://packages.example.com/latest/ buster main
deb-src [signed-by=/usr/share/keyrings/example.gpg] https://packages.example.com/latest/ buster main

Now the key can only sign packages for these repositories. This is effective since stretch according to the Debian Wiki.

Juniper krt queue problems

blog@karotte.org (Sebastian Wiesinger) — Thu, 22 Mar 2018 00:00:00 +0000

Juniper is somewhat (in-)famous for unhelpful and/or superfluous error messages. We faced a new one on our Juniper MX boxes recently. A normal-looking static route was committed, the next-hop was reachable via an irb interface bound to a VPLS instance:

[edit routing-options static]
    +    route 10.1.2.168/29 {
    +        next-hop 10.1.2.167;
    +        tag 666;
    +    }

Directly after committing the following messages occurred in the log:

kernel: iflist_ifindex_lookup(651) error: 2, errmsg: ifle for ifindex 5106 is not found
kernel: iflist_ifindex_lookup(651) error: 2, errmsg: ifle for ifindex 5106 is not found
rpd[14024]: RPD_SYSTEM: Get index for rt table CUSTOMER-TRANSFER failed of gf 11 kqp_op 2: Resource temporarily unavailable
rpd[14024]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
kernel: Timeout for req id=41835 ifd_index=285

After that the following errors occurred again and again over time:

rpd[14024]: RPD_KRT_Q_RETRIES: nexthop add: No error: 0
rpd[14024]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
rpd[14024]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable
rpd[14024]: RPD_KRT_Q_RETRIES: nexthop add: Resource temporarily unavailable

This in itself is not very helpful. The only hint was the “KRT_Q” part. JunOS has a kernel routing table (krt). Routes acquired from routing protocols and static configuration are placed in this table. This table is then syncronized to the linecards (FPC/MPC) via the krt queue.

To see what’s going on there are hidden commands to see the krt status and queue contents. First we look at the show krt state command:

root@core1> show krt state
General state:
        Install job is not running
        Number of operations queued: 1
                Routing table adds: 0
                Interface routes: 0
                High pri multicast   Adds/Changes: 0
                Indirect Next Hop    Adds/Changes: 0       Deletes: 0
                MPLS        Adds: 0       Changes: 0
                High pri    Adds: 1       Changes: 0       Deletes: 0
                Normal pri Indirects: 0
                Normal pri  Adds: 0       Changes: 0       Deletes: 0
                GMP GENCFG Objects: 0
                Routing Table deletes: 0
        Number of operations deferred: 1 <-------------------
        Number of operations canceled: 0
        Number of async queue entries: 1 <-------------------
        Number of async non queue entries: 0
        Time until next queue run: 0.847768
        Routes learned from kernel: 34

You can see there is one “async entry” queued and also one “deferred operation”. Normally these should all be at zero after routes are installed in the linecards. In our case however this one operation was stuck there and did not disappear.

To see what exactly is queued, you can look at the queue with show krt queue:

root@core1> show krt queue
[..]
High-priority add queue: 1 queued
                ADD nhtype Router index 0 (46562)
                    error 'ENOENT -- Item not found'
                    kqp '0x10959000'
[..]

We can determine that the operation was stuck in the “High-priority add queue”. Also there is an error (that is not very helpful) and from the “ADD nhtype” we can guess that the operation tried to add some kind of next-hop entry.

This caused us to look at the static route next-hop again. The next-hop was pointing to 10.1.2.167. This in return was located on an irb:

root@core1> show route 10.1.2.167
10.1.2.160/29  *[Direct/0] 2d 01:49:44
                    > via irb.510

And there we have the problem. The network on the IRB is 10.1.2.160/29. 10.1.2.167 is the last address in this network, so it is the broadcast address. Setting the next-hop of a static route to the broadcast address is a bad idea (and was a typo in our case). After changing the next-hop to point to another address the krt queue cleared up and the error messages vanished.

In conclusion, the error messages were cryptic and without some detective work it was not apparent what went wrong. In my opinion this problem should’ve been catched before the route got stuck in the krt queue and it should’ve produced a more helpful error message.

This blog entry is also for people who google this problem. Sadly this is often the most helpful thing to do when faced with cryptic Juniper error messages.

Juniper local-as and BGP loops

blog@karotte.org (Sebastian Wiesinger) — Mon, 27 Nov 2017 00:00:00 +0000

I faced an interesting problem today. In a customer L3VPN/VRF a route was hidden and it was not apparent why. In JunOS when you do a show route ... command you will get a summary at the top that tells you some statistics. In this case, four routes are hidden (three were expected, the one displayed here should not have been hidden). You can display hidden routes by adding the hidden option to the show route command:

user@core1> show route table customer-vrf.inet.0 10.1.2.0/22 hidden

customer-vrf.inet.0: 31 destinations, 34 routes (30 active, 0 holddown, 4 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.2.0/22      [BGP ] 1d 17:20:31, MED 0, localpref 100, from 198.51.100.27
                    AS path: 65003 ?, validation-state: unverified
                    to 198.51.100.174 via ae1.0, Push 16, Push 300928(top)
                    > to 198.51.100.66 via ae17.0, Push 16, Push 302608(top)

So why is it hidden? To get more details you can use the show route detail command. I cut out some lines not relevant here and if you need even more information use the show route extensive option.

user@core1> show route table customer-vrf.inet.0 10.1.2.0/22 hidden detail

customer-vrf.inet.0: 31 destinations, 34 routes (30 active, 0 holddown, 4 hidden)
10.1.2.0/22 (1 entry, 0 announced)
        BGP                 /-101
                Route Distinguisher: 198.51.100.27:5
                Next hop type: Indirect, Next hop index: 0
                Address: 0x271f862c
                Next-hop reference count: 5
                Source: 198.51.100.27
                Next hop type: Router, Next hop index: 0
                Next hop: 198.51.100.174 via ae1.0 weight 0x1
                Label operation: Push 16, Push 300928(top)
                Next hop: 198.51.100.66 via ae17.0 weight 0x1, selected
                Label operation: Push 16, Push 302608(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Protocol next hop: 198.51.100.27
                Label operation: Push 16
                State: <Secondary Hidden Int Ext ProtectionCand>
                Local AS: 65555 Peer AS: 65555
                Age: 1d 17:20:29 	Metric: 0 	Metric2: 2003 
                Validation State: unverified 
                Task: BGP_65555.198.51.100.27
    ---------->     AS path: 65003 ? (Looped: 65003)     <--------------------------
                    AS path: Recorded
                    Communities: target:65555:55
                    Import
                    VPN Label: 16
                    Localpref: 100
                    Router ID: 198.51.100.27
    ---------->     Hidden reason: AS path loop     <-------------------------------
                    Primary Routing Table bgp.l3vpn.0

As we can see the router is hiding this route because it detects an AS loop. The looped AS is displayed as AS65003 which is interesting because that AS is not the local AS for this router. AS65555 is used as the local AS. AS65003 is only used on the customer (CPE) router that is announcing this route into the VRF.

Why is the router deciding that it must prevent an AS loop with AS65003? To answer this question we must first introduce the local-as command. local-as allows you to “fake” your AS so that it appears different to your eBGP peer. This also affects AS path loop detection, as the AS specified in local-as is also used in detecting BGP routing loops. This option is system-wide! So if you specify local-as anywhere in your router configuration it will be used for loop detection everywhere, even in VRFs. The documentation states this somewhat lengthy in the last info box at the bottom:

Note: If you configure the local AS values for any BGP group, the detection of routing loops is performed using both the AS and the local AS values for all BGP groups.

[..]

When you configure the local AS within a VRF, this impacts the AS path loop-detection mechanism. All of the local-as statements configured on the device are part of a single AS domain. The AS path loop-detection mechanism is based on looking for a matching AS present in the domain.

This is the answer to our problem. Even though the VRF for the customer is not using AS65003 locally, somewhere else in a (global) BGP group local-as 65003 was configured and this was influencing the AS path loop detection on the router.

We can confirm this with show as-path domain:

user@core1> show as-path domain
Domain: 2        Primary: 65555
    References:         10 Paths:     138718
    Flags: Master
    Local AS: 65555  Loops: 1
    Local AS: 65003  Loops: 1

After removing the local-as command from the configuration (it was for a test and no longer needed) the route became active as expected.

What can we learn from this:

Be aware that local-as has side-effects
local-as influences BGP AS path loop detection regardless of where it is configured
When the customer announces you BGP routes check the AS they’re using for conflicts (and if in doubt get them an official AS)

Configure JunOS via SSH and NETCONF

blog@karotte.org (Sebastian Wiesinger) — Fri, 12 May 2017 00:00:00 +0000

I was looking for an easy and fast way to push configuration to our Juniper devices. Preferably one that doesn’t need anything special except a ssh connection.

I started with a standard Juniper configuration snippet. Something like this:

policy-options {
replace:
    policy-statement deny-everything {
        then reject;
    }
}

How do we get this on the device? Luckily Juniper (as well as other vendors) supports a feature called NETCONF ({% include rfc.html rfc=“6241” %}) which uses a XML RPC API to talk to the device. You need to enable it together with SSH:

system {
    services {
        ssh;
        netconf {
            ssh;
        }
    }
}

In the Juniper NETCONF documentation we find the useful <load-configuration> command. This can be used to load configuration data into the candidate configuration of the JunOS device. It supports XML syntax (which is used internally by JunOS), configuration text or set commands. For our snippet we need text mode, which means the following syntax:

<rpc>
    <load-configuration action="merge" format="text">
        <configuration-text>
        <!-- configuration data -->
        </configuration-text>
    </load-configuration>
</rpc>

Combining this with our snippet we end up with:

<rpc>
    <load-configuration action="merge" format="text">
        <configuration-text>
          policy-options {
          replace:
              policy-statement deny-everything {
                  then reject;
              }
          }
        </configuration-text>
    </load-configuration>
</rpc>

This will instruct the device to merge the configuration snippet with the current configuration. The replace: tag means that the deny-everything statement will be replaced with the new version rather than merged.

Now we can send this to the device. Fortunately there is a NETCONF command for SSH. This means you can invoke a NETCONF session by simply opening a SSH session to the device and specifying the NETCONF subsystem:

ssh admin@router netconf

That’s all it takes. You now have an open NETCONF session to the system. You should get a NETCONF “hello” response, something like this:

<!-- No zombies were killed during the creation of this user interface -->
<!-- user admin, class j-super-user -->
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <capabilities>
    <capability>urn:ietf:params:netconf:base:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:validate:1.0</capability>
    <capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</capability>
    <capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>
    <capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</capability>
    <capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</capability>
    <capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</capability>
    <capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file</capability>
    <capability>http://xml.juniper.net/netconf/junos/1.0</capability>
    <capability>http://xml.juniper.net/dmi/system/1.0</capability>
  </capabilities>
  <session-id>48653</session-id>
</hello>
]]>]]>
<!-- session end at 2017-05-11 11:25:45 CEST -->

Now we just need to push the configuration to the system. We can do this by simply using echo to pipe it into the SSH session:

$ echo '    <rpc>
>         <load-configuration action="merge" format="text">
>             <configuration-text>
>               policy-options {
>               replace:
>                   policy-statement deny-everything {
>                       then reject;
>                   }
>               }
>             </configuration-text>
>         </load-configuration>
>     </rpc>
> ' | ssh admin@router netconf

If everything worked as expected you get back a response that contains this:

<load-configuration-results>
<ok/>
</load-configuration-results>

Great! The snipped is now loaded in the candidate configuration (uncommitted):

admin@lab-mx960> configure
Entering configuration mode
The configuration has been changed but not committed

[edit]
admin@lab-mx960# show | diff
[edit policy-options]
+   policy-statement deny-everything {
+       then reject;
+   }

[edit]
admin@lab-mx960#

Do a commit and you are done.

{% include danger.html content=“There is no error checking done. If the code snippet is invalid or has other problem you must check this yourself, for example by doing a show | diff on the device.” %}

If you want you can also do the commit automatically by appending it to the RPC call:

<rpc>
    <load-configuration action="merge" format="text">
        <configuration-text>
          policy-options {
          replace:
              policy-statement deny-everything {
                  then reject;
              }
          }
        </configuration-text>
    </load-configuration>
</rpc>
<rpc>
    <commit/>
</rpc>

This will send the snippet to the device and do a commit immediately after.

{% include danger.html content=“Again, there is no error checking! This will commit whatever you send, as long as it passes the syntax check. Also it will commit anything else that was not committed before and is still in the candidate configuration! You must make sure that no one else is editing the configuration at the same time!” %}

As the XML “wrapper” around the configuration is always the same you could also put it out of the way into a separate script. I wrote one and called it netconf-merge-wrapper:

#! /bin/bash
# Use like: bgpq3 ... stuff | netconf-merge-wrapper | ssh admin@router netconf
cat <<_END
<rpc>
    <load-configuration action="merge" format="text">
        <configuration-text>
_END
cat -
cat <<_END
        </configuration-text>
    </load-configuration>
</rpc>
_END
if [ "$1" == "commit" ]; then
cat <<_END
<rpc>
    <commit/>
</rpc>
_END
fi

As the comment states you could use that script with bgpq3. bgpq3 builds automatic prefix-lists, route-filters or as-path-groups for Cisco and Juniper devices by supplying an AS number or AS-SET.

So for example if you want to generate a route-filter for all AS-APPLE prefixes, you can use it like that:

bgpq3 -J3EAl as-apple AS-APPLE | netconf-merge-wrapper | ssh admin@router netconf

If you are brave you can commit it right away as well:

bgpq3 -J3EAl as-apple AS-APPLE | netconf-merge-wrapper commit | ssh admin@router netconf

In production I would invest in a lot more error checking before rolling stuff out all over the place. For that I would use python and ncclient. Still, I hope this illustrates that it is relatively easy to push stuff to the NETCONF interface, even without skills in programming.