======================================
Using DevStack with neutron Networking
======================================

This guide will walk you through using OpenStack neutron with the ML2
plugin and the Open vSwitch mechanism driver.


Using Neutron with a Single Interface
=====================================

In some instances, like on a developer laptop, there is only one
network interface that is available. In this scenario, the physical
interface is added to the Open vSwitch bridge, and the IP address of
the laptop is migrated onto the bridge interface. That way, the
physical interface can be used to transmit tenant network traffic,
the OpenStack API traffic, and management traffic.


Physical Network Setup
----------------------

In most cases where DevStack is being deployed with a single
interface, there is a hardware router that is being used for external
connectivity and DHCP. The developer machine is connected to this
network and is on a shared subnet with other machines.

.. nwdiag::

        nwdiag {
                inet [ shape = cloud ];
                router;
                inet -- router;

                network hardware_network {
                        address = "172.18.161.0/24"
                        router [ address = "172.18.161.1" ];
                        devstack-1 [ address = "172.18.161.6" ];
                }
        }


DevStack Configuration
----------------------

The following is a complete `local.conf` for the host named
`devstack-1`. It will run all the API and services, as well as
serving as a hypervisor for guest instances.

::

        [[local|localrc]]
        HOST_IP=172.18.161.6
        SERVICE_HOST=172.18.161.6
        MYSQL_HOST=172.18.161.6
        RABBIT_HOST=172.18.161.6
        GLANCE_HOSTPORT=172.18.161.6:9292
        ADMIN_PASSWORD=secrete
        DATABASE_PASSWORD=secrete
        RABBIT_PASSWORD=secrete
        SERVICE_PASSWORD=secrete
        SERVICE_TOKEN=secrete

        # Do not use Nova-Network
        disable_service n-net
        # Enable Neutron
        ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt,q-l3


        ## Neutron options
        Q_USE_SECGROUP=True
        FLOATING_RANGE="172.18.161.0/24"
        FIXED_RANGE="10.0.0.0/24"
        Q_FLOATING_ALLOCATION_POOL=start=172.18.161.250,end=172.18.161.254
        PUBLIC_NETWORK_GATEWAY="172.18.161.1"
        Q_L3_ENABLED=True
        PUBLIC_INTERFACE=eth0
        Q_USE_PROVIDERNET_FOR_PUBLIC=True
        OVS_PHYSICAL_BRIDGE=br-ex
        PUBLIC_BRIDGE=br-ex
        OVS_BRIDGE_MAPPINGS=public:br-ex


Adding Additional Compute Nodes
-------------------------------

Let's suppose that after installing DevStack on the first host, you
also want to do multinode testing and networking.

Physical Network Setup
~~~~~~~~~~~~~~~~~~~~~~

.. nwdiag::

        nwdiag {
                inet [ shape = cloud ];
                router;
                inet -- router;

                network hardware_network {
                        address = "172.18.161.0/24"
                        router [ address = "172.18.161.1" ];
                        devstack-1 [ address = "172.18.161.6" ];
                        devstack-2 [ address = "172.18.161.7" ];
                }
        }


After DevStack installs and configures Neutron, traffic from guest VMs
flows out of `devstack-2` (the compute node) and is encapsulated in a
VXLAN tunnel back to `devstack-1` (the control node) where the L3
agent is running.

::

    stack@devstack-2:~/devstack$ sudo ovs-vsctl show
    8992d965-0ba0-42fd-90e9-20ecc528bc29
        Bridge br-int
            fail_mode: secure
            Port br-int
                Interface br-int
                    type: internal
            Port patch-tun
                Interface patch-tun
                    type: patch
                    options: {peer=patch-int}
        Bridge br-tun
            fail_mode: secure
            Port "vxlan-c0a801f6"
                Interface "vxlan-c0a801f6"
                    type: vxlan
                    options: {df_default="true", in_key=flow, local_ip="172.18.161.7", out_key=flow, remote_ip="172.18.161.6"}
            Port patch-int
                Interface patch-int
                    type: patch
                    options: {peer=patch-tun}
            Port br-tun
                Interface br-tun
                    type: internal
        ovs_version: "2.0.2"

Open vSwitch on the control node, where the L3 agent runs, is
configured to de-encapsulate traffic from compute nodes, then forward
it over the `br-ex` bridge, where `eth0` is attached.

::

    stack@devstack-1:~/devstack$ sudo ovs-vsctl show
    422adeea-48d1-4a1f-98b1-8e7239077964
        Bridge br-tun
            fail_mode: secure
            Port br-tun
                Interface br-tun
                    type: internal
            Port patch-int
                Interface patch-int
                    type: patch
                    options: {peer=patch-tun}
            Port "vxlan-c0a801d8"
                Interface "vxlan-c0a801d8"
                    type: vxlan
                    options: {df_default="true", in_key=flow, local_ip="172.18.161.6", out_key=flow, remote_ip="172.18.161.7"}
        Bridge br-ex
            Port phy-br-ex
                Interface phy-br-ex
                    type: patch
                    options: {peer=int-br-ex}
            Port "eth0"
                Interface "eth0"
            Port br-ex
                Interface br-ex
                    type: internal
        Bridge br-int
            fail_mode: secure
            Port "tapce66332d-ea"
                tag: 1
                Interface "tapce66332d-ea"
                    type: internal
            Port "qg-65e5a4b9-15"
                tag: 2
                Interface "qg-65e5a4b9-15"
                    type: internal
            Port "qr-33e5e471-88"
                tag: 1
                Interface "qr-33e5e471-88"
                    type: internal
            Port "qr-acbe9951-70"
                tag: 1
                Interface "qr-acbe9951-70"
                    type: internal
            Port br-int
                Interface br-int
                    type: internal
            Port patch-tun
                Interface patch-tun
                    type: patch
                    options: {peer=patch-int}
            Port int-br-ex
                Interface int-br-ex
                    type: patch
                    options: {peer=phy-br-ex}
        ovs_version: "2.0.2"

`br-int` is a bridge that the Open vSwitch mechanism driver creates,
which is used as the "integration bridge" where ports are created, and
plugged into the virtual switching fabric. `br-ex` is an OVS bridge
that is used to connect physical ports (like `eth0`), so that floating
IP traffic for tenants can be received from the physical network
infrastructure (and the internet), and routed to tenant network ports.
`br-tun` is a tunnel bridge that is used to connect OpenStack nodes
(like `devstack-2`) together. This bridge is used so that tenant
network traffic, using the VXLAN tunneling protocol, flows between
each compute node where tenant instances run.


DevStack Compute Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The host `devstack-2` has a very minimal `local.conf`.

::

    [[local|localrc]]
    HOST_IP=172.18.161.7
    SERVICE_HOST=172.18.161.6
    MYSQL_HOST=172.18.161.6
    RABBIT_HOST=172.18.161.6
    GLANCE_HOSTPORT=172.18.161.6:9292
    ADMIN_PASSWORD=secrete
    MYSQL_PASSWORD=secrete
    RABBIT_PASSWORD=secrete
    SERVICE_PASSWORD=secrete
    SERVICE_TOKEN=secrete

    ## Neutron options
    PUBLIC_INTERFACE=eth0
    ENABLED_SERVICES=n-cpu,rabbit,q-agt

Network traffic from `eth0` on the compute nodes is then NAT'd by the
controller node that runs Neutron's `neutron-l3-agent` and provides L3
connectivity.


Neutron Networking with Open vSwitch and Provider Networks
==========================================================

In some instances, it is desirable to use neutron's provider
networking extension, so that networks that are configured on an
external router can be utilized by neutron, and instances created via
Nova can attach to the network managed by the external router.

For example, in some lab environments, a hardware router has been
pre-configured by another party, and an OpenStack developer has been
given a VLAN tag and IP address range, so that instances created via
DevStack will use the external router for L3 connectivity, as opposed
to the neutron L3 service.

Physical Network Setup
----------------------

.. nwdiag::

        nwdiag {
                inet [ shape = cloud ];
                router;
                inet -- router;

                network provider_net {
                        address = "203.0.113.0/24"
                        router [ address = "203.0.113.1" ];
                        controller;
                        compute1;
                        compute2;
                }

                network control_plane {
                        router [ address = "10.0.0.1" ]
                        address = "10.0.0.0/24"
                        controller [ address = "10.0.0.2" ]
                        compute1 [ address = "10.0.0.3" ]
                        compute2 [ address = "10.0.0.4" ]
                }
        }


On a compute node, the first interface, eth0 is used for the OpenStack
management (API, message bus, etc) as well as for ssh for an
administrator to access the machine.

::

        stack@compute:~$ ifconfig eth0
        eth0      Link encap:Ethernet  HWaddr bc:16:65:20:af:fc
                  inet addr:10.0.0.3

eth1 is manually configured at boot to not have an IP address.
Consult your operating system documentation for the appropriate
technique. For Ubuntu, the contents of `/etc/network/interfaces`
contains:

::

        auto eth1
        iface eth1 inet manual
                up ifconfig $IFACE 0.0.0.0 up
                down ifconfig $IFACE 0.0.0.0 down

The second physical interface, eth1 is added to a bridge (in this case
named br-ex), which is used to forward network traffic from guest VMs.

::

        stack@compute:~$ sudo ovs-vsctl add-br br-ex
        stack@compute:~$ sudo ovs-vsctl add-port br-ex eth1
        stack@compute:~$ sudo ovs-vsctl show
        9a25c837-32ab-45f6-b9f2-1dd888abcf0f
            Bridge br-ex
                Port br-ex
                    Interface br-ex
                        type: internal
                Port phy-br-ex
                    Interface phy-br-ex
                        type: patch
                        options: {peer=int-br-ex}
                Port "eth1"
                    Interface "eth1"


Service Configuration
---------------------

**Control Node**

In this example, the control node will run the majority of the
OpenStack API and management services (keystone, glance,
nova, neutron)


**Compute Nodes**

In this example, the nodes that will host guest instances will run
the ``neutron-openvswitch-agent`` for network connectivity, as well as
the compute service ``nova-compute``.

DevStack Configuration
----------------------

The following is a snippet of the DevStack configuration on the
controller node.

::

        HOST_IP=10.0.0.2
        SERVICE_HOST=10.0.0.2
        MYSQL_HOST=10.0.0.2
        RABBIT_HOST=10.0.0.2
        GLANCE_HOSTPORT=10.0.0.2:9292
        PUBLIC_INTERFACE=eth1

        ADMIN_PASSWORD=secrete
        MYSQL_PASSWORD=secrete
        RABBIT_PASSWORD=secrete
        SERVICE_PASSWORD=secrete
        SERVICE_TOKEN=secrete

        ## Neutron options
        Q_USE_SECGROUP=True
        ENABLE_TENANT_VLANS=True
        TENANT_VLAN_RANGE=3001:4000
        PHYSICAL_NETWORK=default
        OVS_PHYSICAL_BRIDGE=br-ex

        Q_USE_PROVIDER_NETWORKING=True
        Q_L3_ENABLED=False

        # Do not use Nova-Network
        disable_service n-net

        # Neutron
        ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt

        ## Neutron Networking options used to create Neutron Subnets

        FIXED_RANGE="203.0.113.0/24"
        NETWORK_GATEWAY=203.0.113.1
        PROVIDER_SUBNET_NAME="provider_net"
        PROVIDER_NETWORK_TYPE="vlan"
        SEGMENTATION_ID=2010

In this configuration we are defining FIXED_RANGE to be a
publicly routed IPv4 subnet. In this specific instance we are using
the special TEST-NET-3 subnet defined in `RFC 5737 <http://tools.ietf.org/html/rfc5737>`_,
which is used for documentation.  In your DevStack setup, FIXED_RANGE
would be a public IP address range that you or your organization has
allocated to you, so that you could access your instances from the
public internet.

The following is the DevStack configuration on
compute node 1.

::

        HOST_IP=10.0.0.3
        SERVICE_HOST=10.0.0.2
        MYSQL_HOST=10.0.0.2
        RABBIT_HOST=10.0.0.2
        GLANCE_HOSTPORT=10.0.0.2:9292
        ADMIN_PASSWORD=secrete
        MYSQL_PASSWORD=secrete
        RABBIT_PASSWORD=secrete
        SERVICE_PASSWORD=secrete
        SERVICE_TOKEN=secrete

        # Services that a compute node runs
        ENABLED_SERVICES=n-cpu,rabbit,q-agt

        ## Neutron options
        PHYSICAL_NETWORK=default
        OVS_PHYSICAL_BRIDGE=br-ex
        PUBLIC_INTERFACE=eth1
        Q_USE_PROVIDER_NETWORKING=True
        Q_L3_ENABLED=False

Compute node 2's configuration will be exactly the same, except
``HOST_IP`` will be ``10.0.0.4``

When DevStack is configured to use provider networking (via
``Q_USE_PROVIDER_NETWORKING`` is True and ``Q_L3_ENABLED`` is False) -
DevStack will automatically add the network interface defined in
``PUBLIC_INTERFACE`` to the ``OVS_PHYSICAL_BRIDGE``

For example, with the above  configuration, a bridge is
created, named ``br-ex`` which is managed by Open vSwitch, and the
second interface on the compute node, ``eth1`` is attached to the
bridge, to forward traffic sent by guest VMs.

Miscellaneous Tips
==================


Disabling Next Generation Firewall Tools
----------------------------------------

DevStack does not properly operate with modern firewall tools.  Specifically
it will appear as if the guest VM can access the external network via ICMP,
but UDP and TCP packets will not be delivered to the guest VM.  The root cause
of the issue is that both ufw (Uncomplicated Firewall) and firewalld (Fedora's
firewall manager) apply firewall rules to all interfaces in the system, rather
then per-device.  One solution to this problem is to revert to iptables
functionality.

To get a functional firewall configuration for Fedora do the following:

::

         sudo service iptables save
         sudo systemctl disable firewalld
         sudo systemctl enable iptables
         sudo systemctl stop firewalld
         sudo systemctl start iptables


To get a functional firewall configuration for distributions containing ufw,
disable ufw.  Note ufw is generally not enabled by default in Ubuntu.  To
disable ufw if it was enabled, do the following:

::

        sudo service iptables save
        sudo ufw disable

Configuring Extension Drivers for the ML2 Plugin
------------------------------------------------

Extension drivers for the ML2 plugin are set with the variable
``Q_ML2_PLUGIN_EXT_DRIVERS``, and includes the 'port_security' extension
by default. If you want to remove all the extension drivers (even
'port_security'), set ``Q_ML2_PLUGIN_EXT_DRIVERS`` to blank.