Considering IPSec VPN different topologies there is often a need for reliability in communication between peers. Physical link redundancy is the minimal requirement which should be met in order to fulfill service level agreements. Taking into account IPSec network configuration complexity it is relatively hard to configure failover between data channels while using multiple outbound connections. This article will explain a design approach using two outbound channels, multiple routing instances per router and dynamic routing.
While initiating IPSec site-to-site communication between peers those peers should be defined either by static IP address, dynamic IP address, hostname etc [1]. In most cases it would be wise to build four different IPSec tunnels between two peers (in case each peer has two outbound connections) and choose the path through the best channel (either measured by latency, speed or other factors). However in practice it is quite hard to achieve and typical configuration uses only two tunnels between peers – usually primary ISP – primary ISP connection and secondary ISP – secondary ISP connection.
Both peers have to be able to transmit information through both internet service providers simultaneously. Only in this case it is possible to form successful IPSec negotiations between those. In typical scenario it is possible to build multiple virtual routers in one physical device and separate outbound traffic between equally treated outbound virtual routers. In case there is a master routing instance e.g. master virtual router it is wise to bind internal networking interfaces to this. It will automatically solve several problems with direct export of interface routes between routing instances and so forth.
Let`s take into account two IPSec peers: peer A and peer B. Peer A has two outbound connections with ISP red and ISP blue and also local network defined by VPN domain x.x.x.x/24. Peer B also has two outbound connections with ISP green and ISP yellow and local network defined by VPN domain y.y.y.y/24. The main goal is to build ISP failure resilient IPSec VPN secured communication between VPN domains x.x.x.x/24 and y.y.y.y/24.
First of all it is necessary to build up a basic routing system inside each peer using multiple virtual routers and assign physical interfaces accordingly. After interfaces assignment and configuration it is necessary to test connectivity between peers and assume that all ISP routes are configured properly. There is no need to configure outgoing default route in the master routing instance as it is intended to be imported automatically from both ISP routing instances belonging to the same peer.
Some vendors’ equipment allows creation of virtual tunnel interfaces in order to terminate IPSec traffic. In this case it is necessary to create separate tunnel interface in each specific ISP virtual routing instance – e.g. tunnel red, tunnel blue, tunnel green and tunnel yellow.
After tunnel establishment using specific IPSec negotiation parameters we will have two tunnel between four ISP interfaces – ISP red <-> ISP green and ISP blue <-> ISP yellow. However, routing in the master instance is not built yet therefore it is not possible to test failover scenario.
We have to create two OSPF areas per each tunnel pair. For instance it is possible to create OSPF area 1 and OSPF area 2 and assign tunnel interfaces red and green to area 1 and interfaces blue and yellow to area 2. At this point it is necessary to determine whether the OSPF adjacencies between routers are formed or not.
There might be a problem at this step with OSPF interface types – for the successful formation of adjacency the multicast address is used and usually the multicast interface types cannot work through IPSec VPN tunnel directly. However, it is possible to specify tunnel interface type in relevant OSPF area – in this case it should be marked as point – to – point (p2p) type [2]. In this case the members of relevant OSPF area will change their state from init to full.
According to the fact that direct interface routes are shared between instances ISP virtual routers should have relevant routes to the VPN domains through master instance interface. Those routes should also be marked as directly connected in each instance. Therefore there is no need for additional import of routing table entries from master instances.
If adjacencies are correctly formed the next step which should be taken is the determination of OSPF export policy – e.g. specifying the rules for routing entries export from one OSPF peer to another.
It is possible to determine different OSPF export and import policies according to route prefix, metrics, type etc. In this example we have to export route leading to the relevant VPN domain and the policy could use either exact prefix or the route type. For instance we might configure a policy which exports all directly connected routes through OSPF, in our example it will export interface routes and VPN domain route.
If we are dealing with the so called hub and spoke topology where all traffic should flow through the central gateway it is possible to simplify export policy and export only default route leading to 0.0.0.0/0 supernet.
After specifying export policies and applying those to relevant OSPF area settings after LSA update we should see that VPN domain routes are correctly imported to non-default routing instances. Yet again at this point it is not possible to test routing and failover because master routing instance is not aware of OSPF routing yet.
The next step which should be taken is importing OSPF routes from non-default routing instances to master routing instance. At this point it is necessary to determine whether we want to have load balancing and asymmetric traffic flow between ISP interfaces or not. In case we do there is a need to specify two import policies from ISP virtual routers to master routing instance and configure them to import OSPF routes with specified preference. For instance it is possible to specify that all OSPF imported routes will be treated equally and will have local preference of 150 or relevant equal metric.
When the routing decision will be taken usually it will use per packet or per flow load balancing depending on the type of peer. If the device is flow-based it will be per flow, if the device is discrete or packet-based (ordinary router) it will be per-packet.
Figure 1 shows the schematic view of networking devices configuration.
Figure 1. Schematic configuration
If the first virtual routers are attached to better ISPs (according to jitter, latency, speed etc) then it is wise not to use asymmetric routing at all and to have a second tunnel only for the failover purposes.
In this case the import policy should distinguish the route imported from OSPF area 1 with a better preference or metric. Depending on the vendor of the device this behavior might differ – for instance routing decision might prefer smaller preference over the bigger number. This way, the policy might assign local preference of 140 for the routes imported through ISPF area 1 and local preference of 150 for the routes which are imported through OSPF area 2.
When the configuration is made it is possible to test the traffic flow through the tunnel. Basically the routing tables should look like this (interface routes are truncated):
ISP red:
0.0.0.0/0 through default gateway, static, default preference
y.y.y.y/24 through tunnel red, imported through OSPF, default preference
x.x.x.x/24 through master instance interface, directly connected, default preference
ISP blue:
0.0.0.0/0 through default gateway, static, default preference
y.y.y.y/24 through tunnel blue, imported through OSPF, default preference
x.x.x.x/24 through master instance interface, directly connected, default preference
ISP green:
0.0.0.0/0 through default gateway, static, default preference
x.x.x.x/24 through tunnel green, imported through OSPF, default preference
y.y.y.y/24 through master instance interface, directly connected, default preference
ISP yellow:
0.0.0.0/0 through default gateway, static, default preference
x.x.x.x/24 through tunnel yellow, imported through OSPF, default preference
y.y.y.y/24 through master instance interface, directly connected, default preference
Peer A master routing instance:
x.x.x.x/24 through interface, default preference
y.y.y.y/24 through tunnel red, preference 140, directly connected
y.y.y.y/24 through tunnel blue, preference 150, directly connected
Peer B master routing instance:
y.y.y.y/24 through interface, default preference
x.x.x.x/24 through tunnel green, preference 140, directly connected
x.x.x.x/24 through tunnel yellow, preference 150, directly connected
In this case the routing is symmetric and the IPSec VPN tunnel between ISP red and ISP green is preferred always when available.
The failover scenario might be different – let’s simulate what happens in case the core routing of ISP red fails to forward packets in the network. In this case the IPSec VPN tunnel between ISP red and ISP green will fail. The next thing which will happen is that tunnel red interface is not able to communicate and send OSPF updates to tunnel green interface and vice versa. Thus the OSPF protocol communication between peers will also fail. ISP red and ISP green routing tables will lose remote peer VPN domain routes automatically and will have only local VPN domain route and also default route which is static.
Master routing instance periodically updates itself importing OSPF routes from non-default routing instances. The update interval might be determined automatically and also might not, this is yet again device specific. However, during the next update it will import only OSPF routes from secondary ISPs and will look like this:
Peer A master routing instance:
x.x.x.x/24 through interface, default preference
y.y.y.y/24 through tunnel blue, preference 150, directly connected
Peer B master routing instance:
y.y.y.y/24 through interface, default preference
x.x.x.x/24 through tunnel yellow, preference 150, directly connected
The routing process will forward packets using the only route left – e.g. through secondary OSPF route.
When the problem at ISP side is solved the VPN tunnel and OSPF adjacency between ISP red and ISP green is established again. The OSPF will import relevant routes to the master routing instance and it will again have two different routes to the same destination with different preference, this forwarding traffic through the first tunnel. As it is seen, the configuration is resilient and failover does not require any manual intervention, all operations are done automatically.
This application can be used in different networking environments with different tasks, also critical environments – e.g. banking, e-commerce, military. However it is quite simple to configure and deploy and therefore it can be also used in different not so critical areas – e.g. agile software development, scada systems, remote access for different branches etc.
In this example only two ISP connections from both sides are used. However it is possible to use more and thus create more robust networking – the configuration will simply involve additional virtual routers and OSPF areas.
References:
S. Kent, R. Atkinson. Security Architecture for the Internet Protocol. // [WWW-document] URL http://www.ietf.org/rfc/rfc2401.txt
Juniper Networks. [SRX] Device running OSPF over IPSec VPN in full-mesh network is stuck in 'init' state. // [WWW-document] URL http://kb.juniper.net/InfoCenter/index?page=content&id=KB19472&actp=RSS&smlogin=true